Modified the async producer so it re-queues failed batches. #7
Closed
hiloboy0119 wants to merge 1 commit intoapache:trunkfrom
Closed
Modified the async producer so it re-queues failed batches. #7hiloboy0119 wants to merge 1 commit intoapache:trunkfrom
hiloboy0119 wants to merge 1 commit intoapache:trunkfrom
Conversation
… cause the queue to fill up under backpressure.
Member
|
This pull request doesn't merge cleanly anymore, can you please close it? Sorry for not replying to this earlier, but pull requests are not currently monitored as the Kafka project uses JIRA and Review Board for contributions. There is a plan to change this and we would like to close stale PRs before we start. Unfortunately we can't do it ourselves (a JIRA needs to be filed with Apache Infra) so your help is appreciated. If this change is still relevant, please see http://kafka.apache.org/contributing.html. Alternatively wait until the new approach based on GitHub pull requests is in place (hopefully soon). |
ymatsuda
added a commit
to ymatsuda/kafka
that referenced
this pull request
Aug 26, 2015
Parth-Brahmbhatt
pushed a commit
to Parth-Brahmbhatt/kafka
that referenced
this pull request
Oct 15, 2015
BUG-42616: Support for more than one remote JMX ports for clients and…
asfgit
pushed a commit
that referenced
this pull request
May 23, 2016
… as Apache Kafka 0.10.0.0 Merge tag '0.10.0.0-rc6' into 0.10.0 Release candidate #7 for Apache Kafka 0.10.0.0
wuqingjun
pushed a commit
to wuqingjun/kafka
that referenced
this pull request
Feb 27, 2019
Fork 1.1 branch from apache/kafka
guozhangwang
referenced
this pull request
in guozhangwang/kafka
Jun 13, 2019
Fixing scala test case bugs
lianetm
added a commit
to lianetm/kafka
that referenced
this pull request
Jun 12, 2023
…he#7) TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>
lianetm
added a commit
to lianetm/kafka
that referenced
this pull request
Jun 12, 2023
…he#7) TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>
lianetm
pushed a commit
to lianetm/kafka
that referenced
this pull request
Jun 22, 2023
…ch into new consumer threading refactor This change introduces the FetchRequestManager that will be responsible for: - Formatting fetch requests to the background thread - Configuring the callback on fetch responses for the background thread The response handler will collect the fetch responses from the broker and create CompletedFetch instances as is done in Fetcher. The foreground logic will decompress the data into a Record, which will then be deserialized into a ConsumerRecord for returning to the user.
fvaleri
pushed a commit
to fvaleri/kafka
that referenced
this pull request
Sep 17, 2025
…-for-remote-leader_fix fix: Some fixes for consumer fetch
Raikion201
added a commit
to Raikion201/kafka
that referenced
this pull request
Apr 17, 2026
Issue apache#4 — Quota entity serialization: Sort entity keys alphabetically before joining with "|" separator, so (user+client-id) always serializes identically regardless of HashMap iteration order. Previously could produce different strings for the same entity, causing silent duplicates. Issue apache#5 — Hardcoded credentials: SecurityStoreConfig.resolveValue() now supports ${VAR} and ${VAR:-default} syntax for environment variable substitution. Updated server.properties to use ${KAFKA_SECURITY_STORE_USER:-kafka} and ${KAFKA_SECURITY_STORE_PASSWORD:-kafka}. Issue apache#6 — Byte cast: Use rs.getShort() to match SMALLINT column type, then cast to byte. Previous getInt() + (byte) cast had overflow risk in theory. Issue apache#7 — MetadataLoader silent partial load: injectSecurityData() now throws on failure instead of silently returning stale image. Prevents users from retaining revoked access. Issue apache#10 — apiKey filter comments: Extracted SECURITY_STORE_API_KEYS constant with inline comments identifying each record type. Replaced boolean chain with Set.contains(). Issue apache#11 — persistUpsertion fragility: Now searches records by (name, mechanism) instead of assuming the target is always the last record. Throws IllegalStateException if no matching record is found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Raikion201
added a commit
to Raikion201/kafka
that referenced
this pull request
Apr 17, 2026
Issue apache#4 — Quota entity serialization: Sort entity keys alphabetically before joining with "|" separator, so (user+client-id) always serializes identically regardless of HashMap iteration order. Issue apache#5 — Hardcoded credentials: SecurityStoreConfig.resolveValue() now supports ${VAR} and ${VAR:-default} syntax for environment variable substitution. server.properties updated to use env vars by default. Issue apache#6 — Byte cast: Use rs.getShort() to match SMALLINT column type, then cast to byte. Issue apache#7 — MetadataLoader silent partial load: injectSecurityData() now throws on failure instead of silently returning stale image. Issue apache#10 — apiKey filter comments: Extracted SECURITY_STORE_API_KEYS constant with inline comments. Issue apache#11 — persistUpsertion fragility: Searches records by (name, mechanism) instead of assuming the target is last. Also add .gitignore entries for .metals/, .bloop/, and runtime logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
blitzy Bot
pushed a commit
to blitzy-public-samples/blitzy-kafka
that referenced
this pull request
Apr 18, 2026
Resolve all 9 Minor and 10 Info findings from the Checkpoint 1 code review, correcting factual inaccuracies, citation line-range imprecisions, and cross- artifact consistency drift. No modifications to pre-existing Kafka source, tests, build files, or comments — Audit Only rule preserved. Findings by file: accepted-mitigations.md #1 [MINOR] AclCache imports corrected: org.apache.kafka.server.immutable (PCollections-backed Kafka-internal) instead of Guava's com.google.common.collect. apache#2 [MINOR] API surface rewritten to reflect PCollections-style structural- sharing methods .updated()/.added()/.removed() instead of Guava builder pattern. apache#3 [MINOR] ZstdCompression BufferPool path split: wrap-for-output uses zstd-jni RecyclingBufferPool.INSTANCE (L55-L63), wrap-for- input uses ChunkedBytesStream (L65-L75), wrap-for-zstd-input uses anonymous Kafka-owned BufferPool delegating to BufferSupplier (L77-L98). apache#4 [INFO] MAX_RECORDS_PER_USER_OP citation corrected: declaration at QuorumController.java:L185; AclControlManager.java:L52 is the static import only. apache#5 [INFO] AclCache.removeAcl(Uuid) line corrected to L91-L103 (was L89+). references.md apache#6 [MINOR] SafeObjectInputStream citation range tightened from L17-L25 (class header + imports only) to L25-L62 covering the class declaration, DEFAULT_NO_DESERIALIZE_CLASS_NAMES blocklist (L27-L37), resolveClass (L43-L52), and isBlocked helper (L54-L62). apache#7 [INFO] PropertyFileLoginModule citation corrected to L42-L50, pointing at the Javadoc PLAINTEXT warning (L47-L48) plus the class declaration (L50). remediation-roadmap.md apache#8 [INFO] Gantt markers sanitised: all :done/:active markers replaced with :crit (illustrative critical emphasis) or plain markers to avoid any visual suggestion of work already performed. Explanatory blockquote added clarifying the marker change. severity-matrix.md apache#9 [MINOR] 7 occurrences of parenthesised '(Accepted Mitigation)' replaced with bracketed '[Accepted Mitigation]' per Global Conventions for plain-text markers. Cross-validated 9 bracketed instances, 0 parenthesised remaining. README.md apache#11 [MINOR] HEAD commit reference corrected to the pre-audit baseline 6d16f68 (was 8a99096, a mid-audit snapshot); baseline attestation now refers to the commit immediately before the audit began. apache#12 [MINOR] Snapshot date unified to 2026-04-17 across all artifacts. apache#14 [INFO] '25 files' claim qualified as 'planned at project completion' vs 'delivered at this checkpoint (15 files)'. attack-surface-map.md apache#16 [MINOR] Clients module category count corrected from 'six' to 'nine' (actual Mermaid edges: C1, C2, C3, C4, C5, C7, C8, C9, C10). apache#17 [MINOR] Connect module category count corrected from 'five' to 'seven' (actual Mermaid edges: C1, C4, C6, C7, C8, C9, C10). oauth-jwt-validation-paths.md apache#18 [INFO] Outer citation ranges tightened: BrokerJwtValidator.configure at L107-L138 (not L102-L134); OAuthBearerUnsecuredValidatorCallbackHandler.handleCallback at L154-L177 (not L161-L204, which spanned unrelated helpers); allowableClockSkewMs helper cited separately at L194-L207. executive-summary.html Cross-ref A [MINOR] HEAD commit aligned to 6d16f68 at three sites (L621, L668, L1544); methodology Mermaid node re-labelled 'Baseline 6d16f68'. Cross-ref B [MINOR] Snapshot date aligned to 2026-04-17 at two sites (L619, L1542). Out-of-scope (Info-level forward-refs): apache#10, apache#13, apache#15 — Links to docs/security-audit/findings/*.md deliverables not yet present at Checkpoint 1; expected per scope boundary; will resolve at Checkpoint 2 when the 10 per-category findings files land. Validation results (Phase 3): - Mermaid fences: all balanced (20 blocks total, all typed) - HTML tag balance: 22 sections + all 20+ tag types balanced - CDNs intact: reveal.js 5.1.0, Mermaid 11.4.0, Font Awesome 6.6.0 - Emojis: zero across all 15 artifacts - TODOs/placeholders introduced: zero - Gantt markers: :crit + plain only (no :done/:active) - Cross-artifact consistency: zero wrong SHA/date values remaining - Citation ranges: 12 verified against AclCache, QuorumController, AclControlManager, ZstdCompression, SafeObjectInputStream, PropertyFileLoginModule, BrokerJwtValidator, and OAuthBearerUnsecuredValidatorCallbackHandler. Audit Only rule verification: git diff --name-status 6d16f68..HEAD returns only 'A' entries, all under docs/security-audit/. Zero modifications, deletions, or renames of any pre-existing Kafka path.
blitzy Bot
pushed a commit
to blitzy-public-samples/blitzy-kafka
that referenced
this pull request
Apr 18, 2026
QA Checkpoint #1 identified 9 MINOR documentation-quality findings in the Apache Kafka 4.2 security audit deliverables. All 9 findings are documentation corrections confined to the docs/security-audit/ tree; no source code, tests, or build configuration touched — fully compliant with the Audit Only rule. FIXES APPLIED (by QA finding number): Issue #1 [MINOR] — findings/07-external-function-callback-misuse.md L247 Validation Checklist cited legacy path 'internals/secured/BrokerJwtValidator.java'. Updated to current Kafka 4.2 canonical path 'clients/src/main/java/org/apache/kafka/common/security/oauthbearer/BrokerJwtValidator.java' with an explanatory note that the class was reorganized out of the internals/secured sub-package in a prior Kafka refactor. Issue apache#2 [MINOR] — findings/08-deserialization-attacks.md L305 Same pattern as #1 — Validation Checklist updated from 'internals/secured/{Broker,Client}JwtValidator.java' to 'clients/.../oauthbearer/{Broker,Client}JwtValidator.java' with explanatory note. Issue apache#3 [MINOR] — findings/09-information-leakage.md L245 Validation Checklist cited legacy path 'connect/runtime/src/main/java/org/apache/kafka/connect/runtime/RecordRedactor.java'. Updated to current canonical path 'metadata/src/main/java/org/apache/kafka/metadata/util/RecordRedactor.java' with explanatory note. Issue apache#4 [MINOR] — findings/09-information-leakage.md L248 Validation Checklist BrokerJwtValidator and ClientJwtValidator paths updated to current 'oauthbearer/' canonical paths with explanatory note. Issue apache#5 [MINOR] — findings/10-public-api-developer-misuse.md L298 Validation Checklist BrokerJwtValidator path updated to current 'oauthbearer/BrokerJwtValidator.java:L131' canonical path with explanatory note. Issue apache#6 [MINOR] — findings/10-public-api-developer-misuse.md L302 Validation Checklist cited legacy path 'server-common/src/main/java/org/apache/kafka/server/config/ReplicationConfigs.java'. Updated to current canonical path 'server/src/main/java/org/apache/kafka/server/config/ReplicationConfigs.java' with explanatory note that the file moved from the server-common module to the server module in a prior Kafka refactor. Issue apache#7 [MINOR] — references.md Section 3.1 Configuration Added missing entry for 'AllowedPaths.java' ('clients/src/main/java/org/apache/kafka/common/config/internals/AllowedPaths.java'), inserted between the DirectoryConfigProvider and EnvVarConfigProvider entries. Finding 01 cites AllowedPaths 14 times; this bibliography gap is now closed. Issue apache#8 [MINOR] — references.md Section 7 Server Module Added missing entry for 'SocketServerConfigs.java' ('server/src/main/java/org/apache/kafka/network/SocketServerConfigs.java'), inserted after the ReplicationConfigs entry with an inline note about the 'org.apache.kafka.network' vs 'org.apache.kafka.server.config' package mismatch. Findings 03 (11 cites) and 10 (5 cites) reference SocketServerConfigs; this bibliography gap is now closed. Issue apache#9 [MINOR] — findings/01 and findings/10 section header numbering Harmonized H2 section headers to match the numbered 1-10 pattern used by findings 02-09. Applied 20 header replacements total: 10 in finding 01 ('## Category' -> '## 1. Category', etc.), 10 in finding 10 (same pattern). Validation Checklist and Key Insights remain unnumbered per the existing majority convention. Content substance is unchanged; only section prefixes updated. VALIDATION RESULTS: - All 6 canonical file paths verified via 'test -f' to exist in the Kafka source tree at HEAD. - Zero stale 'internals/secured/', 'connect/runtime/.../RecordRedactor', or 'server-common/.../ReplicationConfigs' references remain across the audit corpus. - All 10 findings now have exactly 10 numbered H2 section headers (verified via 'grep -cE "^## [0-9]+\. "'). - Markdown fence balance intact (all diagram files: 4 fences each; findings: all balanced). - Cross-referenced anchors (DISALLOW_NONE, ALLOW_LEADING_ZEROS, AllowedPaths, MAX_RECORDS_PER_USER_OP) preserved. - references.md entries verified present (AllowedPaths=1 match, SocketServerConfigs=1 match). AUDIT ONLY RULE COMPLIANCE: Modifications confined exclusively to documentation artifacts under docs/security-audit/. Zero source code, test, build-configuration, or inline-comment modifications. The untracked 'blitzy/' directory (pre-existing baseline) is NOT part of this commit. Files changed: 6 (+46 / -26 lines) M docs/security-audit/findings/01-filesystem-access-path-traversal.md M docs/security-audit/findings/07-external-function-callback-misuse.md M docs/security-audit/findings/08-deserialization-attacks.md M docs/security-audit/findings/09-information-leakage.md M docs/security-audit/findings/10-public-api-developer-misuse.md M docs/security-audit/references.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I'm working on an application that needs the throughput offered by an async producer but also needs to handle send failures gracefully like a sync producer. I modified the ProducerSendThread so it will re-queue failed batches. This allowed me to determine the behavior I wanted from my producer with the queue config parameters.
A "queue.enqueue.timeout.ms=0" allowed me to get runtime exceptions when sends failed often enough to fill the queue. This also allowed me to use "queue.buffering.max.messages" to control how tolerant the application is to network blips.