Unmap before resizing by lizziew · Pull Request #6 · apache/kafka

lizziew · 2013-07-25T07:07:49Z

While I was studying how MappedByteBuffer works, I saw a sharing runtime exception on Windows. I applied what I learned to generate a patch which uses an internal open JDK API to solve this problem.

Caused by: java.io.IOException: The requested operation cannot be performed
on a
file with a user-mapped section open
at java.io.RandomAccessFile.setLength(Native Method)
at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262)
at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247)
at kafka.log.Log.rollToOffset(Log.scala:518)
at kafka.log.Log.roll(Log.scala:502)
at kafka.log.Log.maybeRoll(Log.scala:484)
at kafka.log.Log.append(Log.scala:297)

lizziew · 2013-07-25T08:23:02Z

Here's some background info from my blog:

http://datasieve.blogspot.com/2013/07/directbytebuffer-in-java.html

ijuma · 2015-07-20T11:06:42Z

This pull request doesn't merge cleanly anymore, can you please close it?

Sorry for not replying to this earlier, but pull requests are not currently monitored as the Kafka project uses JIRA and Review Board for contributions. There is a plan to change this and we would like to close stale PRs before we start. Unfortunately we can't do it ourselves (a JIRA needs to be filed with Apache Infra) so your help is appreciated.

If this change is still relevant, please see http://kafka.apache.org/contributing.html. Alternatively wait until the new approach based on GitHub pull requests is in place (hopefully soon).

parallel execution

[WIP] New API

KAFKA-2979: Refactored ACLs to their own class.

incr ver

guozhangwang · 2016-12-21T01:22:07Z

@lizziew Is this PR still valid? If yes could you rebase so we can review and merge?

asfbot · 2016-12-21T01:22:49Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/293/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2016-12-21T01:22:56Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/294/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2016-12-21T01:22:58Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/292/
Test FAILed (JDK 7 and Scala 2.10).

ijuma · 2017-12-22T01:24:43Z

Closing as this is inactive.

MAPR-25297 Pass hadoop.login=hybrid to JVM in case of secure cluster

…server implementation (apache#6) Reviewers: Radai Rosenblatt

…pObserver implementation (apache#6) TICKET = LI_DESCRIPTION = Reviewers: Radai Rosenblatt EXIT_CRITERIA = MANUAL ["describe exit criteria"]

Fixing scala test case issues

…pObserver implementation (apache#6) TICKET = LI_DESCRIPTION = Reviewers: Radai Rosenblatt EXIT_CRITERIA = MANUAL ["describe exit criteria"]

…pObserver implementation (apache#6) TICKET = LI_DESCRIPTION = The observer interface lets us provide implementation which provides the usage accounting data unit for the C2S V3 service. Reviewers: Radai Rosenblatt EXIT_CRITERIA = MANUAL [""]

This is the commit message #6: non zero jitter and new unit test

…pObserver implementation (apache#6) TICKET = LI_DESCRIPTION = The observer interface lets us provide implementation which provides the usage accounting data unit for the C2S V3 service. Reviewers: Radai Rosenblatt EXIT_CRITERIA = MANUAL [""]

Changes: 1. Objects owned by the background thread are not instantiated until the background thread runs (via Suppliers) 2. Using a class called ConsumerTestBuilder to reduce a lot of inconsistency in the way the objects were built up for tests 3. Ensuring resources are properly using Closeable and using IdempotentCloser to ensure they're only closed once

…eqeust-for-remote-leader feat: build consumer fetch request for remote leader

Issue apache#4 — Quota entity serialization: Sort entity keys alphabetically before joining with "|" separator, so (user+client-id) always serializes identically regardless of HashMap iteration order. Previously could produce different strings for the same entity, causing silent duplicates. Issue apache#5 — Hardcoded credentials: SecurityStoreConfig.resolveValue() now supports ${VAR} and ${VAR:-default} syntax for environment variable substitution. Updated server.properties to use ${KAFKA_SECURITY_STORE_USER:-kafka} and ${KAFKA_SECURITY_STORE_PASSWORD:-kafka}. Issue apache#6 — Byte cast: Use rs.getShort() to match SMALLINT column type, then cast to byte. Previous getInt() + (byte) cast had overflow risk in theory. Issue apache#7 — MetadataLoader silent partial load: injectSecurityData() now throws on failure instead of silently returning stale image. Prevents users from retaining revoked access. Issue apache#10 — apiKey filter comments: Extracted SECURITY_STORE_API_KEYS constant with inline comments identifying each record type. Replaced boolean chain with Set.contains(). Issue apache#11 — persistUpsertion fragility: Now searches records by (name, mechanism) instead of assuming the target is always the last record. Throws IllegalStateException if no matching record is found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Issue apache#4 — Quota entity serialization: Sort entity keys alphabetically before joining with "|" separator, so (user+client-id) always serializes identically regardless of HashMap iteration order. Issue apache#5 — Hardcoded credentials: SecurityStoreConfig.resolveValue() now supports ${VAR} and ${VAR:-default} syntax for environment variable substitution. server.properties updated to use env vars by default. Issue apache#6 — Byte cast: Use rs.getShort() to match SMALLINT column type, then cast to byte. Issue apache#7 — MetadataLoader silent partial load: injectSecurityData() now throws on failure instead of silently returning stale image. Issue apache#10 — apiKey filter comments: Extracted SECURITY_STORE_API_KEYS constant with inline comments. Issue apache#11 — persistUpsertion fragility: Searches records by (name, mechanism) instead of assuming the target is last. Also add .gitignore entries for .metals/, .bloop/, and runtime logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Resolve all 9 Minor and 10 Info findings from the Checkpoint 1 code review, correcting factual inaccuracies, citation line-range imprecisions, and cross- artifact consistency drift. No modifications to pre-existing Kafka source, tests, build files, or comments — Audit Only rule preserved. Findings by file: accepted-mitigations.md #1 [MINOR] AclCache imports corrected: org.apache.kafka.server.immutable (PCollections-backed Kafka-internal) instead of Guava's com.google.common.collect. apache#2 [MINOR] API surface rewritten to reflect PCollections-style structural- sharing methods .updated()/.added()/.removed() instead of Guava builder pattern. apache#3 [MINOR] ZstdCompression BufferPool path split: wrap-for-output uses zstd-jni RecyclingBufferPool.INSTANCE (L55-L63), wrap-for- input uses ChunkedBytesStream (L65-L75), wrap-for-zstd-input uses anonymous Kafka-owned BufferPool delegating to BufferSupplier (L77-L98). apache#4 [INFO] MAX_RECORDS_PER_USER_OP citation corrected: declaration at QuorumController.java:L185; AclControlManager.java:L52 is the static import only. apache#5 [INFO] AclCache.removeAcl(Uuid) line corrected to L91-L103 (was L89+). references.md apache#6 [MINOR] SafeObjectInputStream citation range tightened from L17-L25 (class header + imports only) to L25-L62 covering the class declaration, DEFAULT_NO_DESERIALIZE_CLASS_NAMES blocklist (L27-L37), resolveClass (L43-L52), and isBlocked helper (L54-L62). apache#7 [INFO] PropertyFileLoginModule citation corrected to L42-L50, pointing at the Javadoc PLAINTEXT warning (L47-L48) plus the class declaration (L50). remediation-roadmap.md apache#8 [INFO] Gantt markers sanitised: all :done/:active markers replaced with :crit (illustrative critical emphasis) or plain markers to avoid any visual suggestion of work already performed. Explanatory blockquote added clarifying the marker change. severity-matrix.md apache#9 [MINOR] 7 occurrences of parenthesised '(Accepted Mitigation)' replaced with bracketed '[Accepted Mitigation]' per Global Conventions for plain-text markers. Cross-validated 9 bracketed instances, 0 parenthesised remaining. README.md apache#11 [MINOR] HEAD commit reference corrected to the pre-audit baseline 6d16f68 (was 8a99096, a mid-audit snapshot); baseline attestation now refers to the commit immediately before the audit began. apache#12 [MINOR] Snapshot date unified to 2026-04-17 across all artifacts. apache#14 [INFO] '25 files' claim qualified as 'planned at project completion' vs 'delivered at this checkpoint (15 files)'. attack-surface-map.md apache#16 [MINOR] Clients module category count corrected from 'six' to 'nine' (actual Mermaid edges: C1, C2, C3, C4, C5, C7, C8, C9, C10). apache#17 [MINOR] Connect module category count corrected from 'five' to 'seven' (actual Mermaid edges: C1, C4, C6, C7, C8, C9, C10). oauth-jwt-validation-paths.md apache#18 [INFO] Outer citation ranges tightened: BrokerJwtValidator.configure at L107-L138 (not L102-L134); OAuthBearerUnsecuredValidatorCallbackHandler.handleCallback at L154-L177 (not L161-L204, which spanned unrelated helpers); allowableClockSkewMs helper cited separately at L194-L207. executive-summary.html Cross-ref A [MINOR] HEAD commit aligned to 6d16f68 at three sites (L621, L668, L1544); methodology Mermaid node re-labelled 'Baseline 6d16f68'. Cross-ref B [MINOR] Snapshot date aligned to 2026-04-17 at two sites (L619, L1542). Out-of-scope (Info-level forward-refs): apache#10, apache#13, apache#15 — Links to docs/security-audit/findings/*.md deliverables not yet present at Checkpoint 1; expected per scope boundary; will resolve at Checkpoint 2 when the 10 per-category findings files land. Validation results (Phase 3): - Mermaid fences: all balanced (20 blocks total, all typed) - HTML tag balance: 22 sections + all 20+ tag types balanced - CDNs intact: reveal.js 5.1.0, Mermaid 11.4.0, Font Awesome 6.6.0 - Emojis: zero across all 15 artifacts - TODOs/placeholders introduced: zero - Gantt markers: :crit + plain only (no :done/:active) - Cross-artifact consistency: zero wrong SHA/date values remaining - Citation ranges: 12 verified against AclCache, QuorumController, AclControlManager, ZstdCompression, SafeObjectInputStream, PropertyFileLoginModule, BrokerJwtValidator, and OAuthBearerUnsecuredValidatorCallbackHandler. Audit Only rule verification: git diff --name-status 6d16f68..HEAD returns only 'A' entries, all under docs/security-audit/. Zero modifications, deletions, or renames of any pre-existing Kafka path.

Adds docs/security-audit/findings/02-low-level-code-safety.md covering every JVM-to-native boundary Kafka crosses (zstd-jni 1.5.6-10, snappy-java 1.1.10.7, lz4-java 1.8.0, rocksdbjni 10.1.3) plus JVM-side memory-pool management (SimpleMemoryPool non-strict mode). Document follows the audit-wide template with 10 numbered H2 sections: Category, Definition, Kafka Surface Inventory, Evidence, Attack Vector, Severity, Business Impact, Accepted Mitigations Already Present, Recommended Future Remediation, Cross-References. All five sub-findings (02.1 ZstdCompression, 02.2 SnappyCompression, 02.3 Lz4Compression, 02.4 RocksDBStore, 02.5 SimpleMemoryPool) are rated [Low] per severity-matrix.md L160-L164. Citations use the canonical Source: <path>:L<start>-L<end> format with line ranges matching Accepted Mitigation apache#6 in accepted-mitigations.md. Audit-Only rule honored: no modifications to any existing Kafka source, tests, build configuration, inline comments, or runtime behavior. Closing sentence in Section 9 explicitly restates the rule.

QA Checkpoint #1 identified 9 MINOR documentation-quality findings in the Apache Kafka 4.2 security audit deliverables. All 9 findings are documentation corrections confined to the docs/security-audit/ tree; no source code, tests, or build configuration touched — fully compliant with the Audit Only rule. FIXES APPLIED (by QA finding number): Issue #1 [MINOR] — findings/07-external-function-callback-misuse.md L247 Validation Checklist cited legacy path 'internals/secured/BrokerJwtValidator.java'. Updated to current Kafka 4.2 canonical path 'clients/src/main/java/org/apache/kafka/common/security/oauthbearer/BrokerJwtValidator.java' with an explanatory note that the class was reorganized out of the internals/secured sub-package in a prior Kafka refactor. Issue apache#2 [MINOR] — findings/08-deserialization-attacks.md L305 Same pattern as #1 — Validation Checklist updated from 'internals/secured/{Broker,Client}JwtValidator.java' to 'clients/.../oauthbearer/{Broker,Client}JwtValidator.java' with explanatory note. Issue apache#3 [MINOR] — findings/09-information-leakage.md L245 Validation Checklist cited legacy path 'connect/runtime/src/main/java/org/apache/kafka/connect/runtime/RecordRedactor.java'. Updated to current canonical path 'metadata/src/main/java/org/apache/kafka/metadata/util/RecordRedactor.java' with explanatory note. Issue apache#4 [MINOR] — findings/09-information-leakage.md L248 Validation Checklist BrokerJwtValidator and ClientJwtValidator paths updated to current 'oauthbearer/' canonical paths with explanatory note. Issue apache#5 [MINOR] — findings/10-public-api-developer-misuse.md L298 Validation Checklist BrokerJwtValidator path updated to current 'oauthbearer/BrokerJwtValidator.java:L131' canonical path with explanatory note. Issue apache#6 [MINOR] — findings/10-public-api-developer-misuse.md L302 Validation Checklist cited legacy path 'server-common/src/main/java/org/apache/kafka/server/config/ReplicationConfigs.java'. Updated to current canonical path 'server/src/main/java/org/apache/kafka/server/config/ReplicationConfigs.java' with explanatory note that the file moved from the server-common module to the server module in a prior Kafka refactor. Issue apache#7 [MINOR] — references.md Section 3.1 Configuration Added missing entry for 'AllowedPaths.java' ('clients/src/main/java/org/apache/kafka/common/config/internals/AllowedPaths.java'), inserted between the DirectoryConfigProvider and EnvVarConfigProvider entries. Finding 01 cites AllowedPaths 14 times; this bibliography gap is now closed. Issue apache#8 [MINOR] — references.md Section 7 Server Module Added missing entry for 'SocketServerConfigs.java' ('server/src/main/java/org/apache/kafka/network/SocketServerConfigs.java'), inserted after the ReplicationConfigs entry with an inline note about the 'org.apache.kafka.network' vs 'org.apache.kafka.server.config' package mismatch. Findings 03 (11 cites) and 10 (5 cites) reference SocketServerConfigs; this bibliography gap is now closed. Issue apache#9 [MINOR] — findings/01 and findings/10 section header numbering Harmonized H2 section headers to match the numbered 1-10 pattern used by findings 02-09. Applied 20 header replacements total: 10 in finding 01 ('## Category' -> '## 1. Category', etc.), 10 in finding 10 (same pattern). Validation Checklist and Key Insights remain unnumbered per the existing majority convention. Content substance is unchanged; only section prefixes updated. VALIDATION RESULTS: - All 6 canonical file paths verified via 'test -f' to exist in the Kafka source tree at HEAD. - Zero stale 'internals/secured/', 'connect/runtime/.../RecordRedactor', or 'server-common/.../ReplicationConfigs' references remain across the audit corpus. - All 10 findings now have exactly 10 numbered H2 section headers (verified via 'grep -cE "^## [0-9]+\. "'). - Markdown fence balance intact (all diagram files: 4 fences each; findings: all balanced). - Cross-referenced anchors (DISALLOW_NONE, ALLOW_LEADING_ZEROS, AllowedPaths, MAX_RECORDS_PER_USER_OP) preserved. - references.md entries verified present (AllowedPaths=1 match, SocketServerConfigs=1 match). AUDIT ONLY RULE COMPLIANCE: Modifications confined exclusively to documentation artifacts under docs/security-audit/. Zero source code, test, build-configuration, or inline-comment modifications. The untracked 'blitzy/' directory (pre-existing baseline) is NOT part of this commit. Files changed: 6 (+46 / -26 lines) M docs/security-audit/findings/01-filesystem-access-path-traversal.md M docs/security-audit/findings/07-external-function-callback-misuse.md M docs/security-audit/findings/08-deserialization-attacks.md M docs/security-audit/findings/09-information-leakage.md M docs/security-audit/findings/10-public-api-developer-misuse.md M docs/security-audit/references.md

…ndings Address QA Final Checkpoint apache#4 findings for unmitigated Critical/High CVEs in pinned runtime dependencies. Per the user-specified Audit Only rule, no source code or gradle/dependencies.gradle modifications are performed; this commit only enhances the docs/security-audit/ deliverable to surface CVE findings maximally (markdown files explicitly related to the analysis are permitted by the rule). New consolidated CVE advisory hub: - docs/security-audit/cve-snapshot.md (new, 477 lines) Aggregates the 3 gating findings (lz4-java CVE-2025-12183 CVSS 8.8 and CVE-2025-66566 CVSS 8.2 [Critical x 2]; Jetty CVE-2026-1605 CVSS 7.5 [High] GzipHandler native-memory DoS) plus 5 informational findings (CVE-2026-2332 Jetty HTTP/1.1 chunk-ext Medium; CVE-2026-5795 Jetty JASPI Medium but not exploitable in Kafka; CVE-2025-68161 Log4j2 operator-configuration-dependent Medium; CVE-2026-0636 and CVE-2026-5588 Bouncy Castle Low and non-exploitable in Kafka). Includes Mermaid reachability flowchart, per-CVE mechanism, business impact, and future-state operator compensating controls. Enhanced existing audit artifacts with CVE cross-references: - docs/security-audit/README.md: dependency table now has CVE Snapshot column; cve-snapshot.md added to Core Documents navigation; audit file count updated 25 -> 26 and 6 -> 7 core markdowns; perofrmace typo preserved verbatim. - docs/security-audit/dependency-inventory.md: 12 distinct edits add a CVE column, enrich Mermaid per-dependency colouring with CVE overlay, and document per-dependency CVE posture with fix versions. - docs/security-audit/severity-matrix.md: 9 edits add new supply-chain rows (02.3 lz4-java Critical x 2; 06.6 Jetty GzipHandler High) and update pie chart totals to reflect new roll-up counts. - docs/security-audit/remediation-roadmap.md: 8 edits add Section 3.2.7 (lz4-java KIP-track migration), Section 3.4.4 (Jetty 12.0.22 supply chain upgrade), new Gantt bar (st3), and Section 2/7/8 updates; all future-state language (consider/evaluate/may/could) - no imperatives. - docs/security-audit/findings/02-low-level-code-safety.md: 12 coordinated edits add lz4-java CVE evidence (02.3), severity entry, and business impact narrative. - docs/security-audit/findings/06-network-subprocess-access.md: 14 coordinated edits add Jetty GzipHandler CVE evidence (06.6), new Section 4.6 and Section 5.6, severity matrix entry, and consequence apache#6 in business impact. - docs/security-audit/executive-summary.html: 3 edits flag CVE findings on Slide 18 dependency table via orange/red triangle-exclamation icons with title attributes citing the CVE identifiers and badges. 22-slide discipline preserved; 0 emojis; reveal.js 5.1.0, Font Awesome 6.6.0, and Mermaid 11.4.0 CDN references intact. No changes to any source tree, build file, or gradle manifest. Only docs/security-audit/ paths touched. Audit Only rule fully honored. QA findings addressed: - Issue 1 (lz4-java Critical x 2): documented with severity, CVSS, CWE, business impact, and future-state remediation path (KIP-track) - Issue 2 (Jetty High): documented with severity, CVSS, GzipHandler DoS attack narrative, pre-authentication reachability, and Jetty 12.0.32+ / 12.1.6+ (or 12.0.34 / 12.1.8) upstream fix versions - Issue 3/4 (Medium/Low informational): all 5 CVEs documented with appropriate severity notes and scope qualifiers (not exploitable in Kafka / operator-configuration-dependent / not reachable)

Unmap before resizing

804116d

Finetune patch

da110ce

ymatsuda referenced this pull request in confluentinc/kafka Aug 5, 2015

Merge pull request #6 from ymatsuda/parallel

055f6d7

parallel execution

ymatsuda pushed a commit to ymatsuda/kafka that referenced this pull request Aug 21, 2015

Merge pull request apache#6 from guozhangwang/NewAPI

c20e8aa

[WIP] New API

junrao mentioned this pull request Oct 19, 2015

KAFKA-1686: Implement SASL/Kerberos. #191

Closed

benstopford pushed a commit to benstopford/kafka that referenced this pull request Dec 22, 2015

Merge pull request apache#6 from benstopford/ben-KAFKA-2979

568b9e4

KAFKA-2979: Refactored ACLs to their own class.

resetius added a commit to resetius/kafka that referenced this pull request Jun 7, 2016

Merge pull request apache#6 from aozeritsky/0.8.2-stbx

41ad4a1

incr ver

ijuma closed this Dec 22, 2017

jamesrgrinter pushed a commit to jamesrgrinter/kafka that referenced this pull request Feb 2, 2018

Merge pull request apache#6 from mapr/mapr-25297

3d7ce68

MAPR-25297 Pass hadoop.login=hybrid to JVM in case of secure cluster

stanislavkozlovski mentioned this pull request Oct 28, 2018

KAFKA-3932 - Consumer fails to consume in a round robin fashion #5838

Closed

kehuum pushed a commit to kehuum/kafka that referenced this pull request Mar 27, 2019

HOTFIX - LIKAFKA-21968: Add broker-side observer interface and NoOpOb…

fb2a630

…server implementation (apache#6) Reviewers: Radai Rosenblatt

guozhangwang referenced this pull request in guozhangwang/kafka Jun 13, 2019

Merge pull request #6 from brary/newinter

0826ceb

Fixing scala test case issues

abbccdda referenced this pull request in abbccdda/kafka May 22, 2020

add individual end quorum request

9739913

This is the commit message #6: non zero jitter and new unit test

ijuma mentioned this pull request Sep 5, 2021

KAFKA-13273: Add support for Java 17 #11296

Merged

3 tasks

fvaleri pushed a commit to fvaleri/kafka that referenced this pull request Sep 17, 2025

Merge pull request apache#6 from FrankYang0529/build-consumer-fetch-r…

dd7fa5b

…eqeust-for-remote-leader feat: build consumer fetch request for remote leader

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unmap before resizing#6

Unmap before resizing#6
lizziew wants to merge 2 commits intoapache:0.8from
lizziew:0.8

lizziew commented Jul 25, 2013

Uh oh!

lizziew commented Jul 25, 2013

Uh oh!

ijuma commented Jul 20, 2015

Uh oh!

guozhangwang commented Dec 21, 2016

Uh oh!

asfbot commented Dec 21, 2016

Uh oh!

asfbot commented Dec 21, 2016

Uh oh!

asfbot commented Dec 21, 2016

Uh oh!

ijuma commented Dec 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lizziew commented Jul 25, 2013

Uh oh!

lizziew commented Jul 25, 2013

Uh oh!

ijuma commented Jul 20, 2015

Uh oh!

guozhangwang commented Dec 21, 2016

Uh oh!

asfbot commented Dec 21, 2016

Uh oh!

asfbot commented Dec 21, 2016

Uh oh!

asfbot commented Dec 21, 2016

Uh oh!

ijuma commented Dec 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants