Skip to content

Rack aware replication#15

Closed
jmlvanre wants to merge 171 commits intoapache:0.8from
jmlvanre:rack-aware-replication
Closed

Rack aware replication#15
jmlvanre wants to merge 171 commits intoapache:0.8from
jmlvanre:rack-aware-replication

Conversation

@jmlvanre
Copy link
Copy Markdown
Contributor

@jmlvanre jmlvanre commented Feb 1, 2014

Adding rack-aware replication option. rack-id defaults to -1. use the max-rack-replication option when creating a topic to distribute replicas such that no more than max-rack-replication replicas are hosted on the same rack-id. This option is also enforced when adding new partitions. The option does not enforce manual (re)assignment.

Neha Narkhede and others added 30 commits January 13, 2012 19:53
…hed by vtkstef; reviewed by nehanarkhede

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1231276 13f79535-47bb-0310-9956-ffa450edef68
…d by Pierre-Yves Ritschard; reviewed by Jun Rao; KAFKA-247

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1232500 13f79535-47bb-0310-9956-ffa450edef68
…ut occurs; patched by Jun Rao; reviewed by Joel Koshy; KAFKA-241

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1233501 13f79535-47bb-0310-9956-ffa450edef68
… by Jay Kreps and Neha Narkhede; KAFKA-261

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1239740 13f79535-47bb-0310-9956-ffa450edef68
… not pulling data from some partitions; patched by nehanarkhede; reviewed by joelkoshy

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1239766 13f79535-47bb-0310-9956-ffa450edef68
…by Sam Shah; reviewed by nehanarkhede

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1241249 13f79535-47bb-0310-9956-ffa450edef68
…th uneven distribution on the source brokers; patched by John Fung; reviewed by Neha Narkhede

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1241528 13f79535-47bb-0310-9956-ffa450edef68
…o release partitions that it does not own; patched by Neha Narkhede; reviewed by Jun Rao

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1242552 13f79535-47bb-0310-9956-ffa450edef68
…educe the number of rebalancing attempts; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-265

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1243721 13f79535-47bb-0310-9956-ffa450edef68
…ao; reviewed by Neha Narkhede; KAFKA-272

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1244755 13f79535-47bb-0310-9956-ffa450edef68
…Ye; reviewed by Jun Rao; KAFKA-268

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1291490 13f79535-47bb-0310-9956-ffa450edef68
… by John Fung; reviewed by Jun Rao; KAFKA-254

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1291535 13f79535-47bb-0310-9956-ffa450edef68
…atched by John Fung; reviewed by Jun Rao; KAFKA-255

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1291536 13f79535-47bb-0310-9956-ffa450edef68
…e; reviewed by Jun Rao; KAFKA-277

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1293010 13f79535-47bb-0310-9956-ffa450edef68
… Rao; reviewed by Neha Narkhede; KAFKA-283

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1293720 13f79535-47bb-0310-9956-ffa450edef68
…uring rebalance; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-286

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1294302 13f79535-47bb-0310-9956-ffa450edef68
…sent; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-290

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1294959 13f79535-47bb-0310-9956-ffa450edef68
…t; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-292

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1297324 13f79535-47bb-0310-9956-ffa450edef68
Swapnil Ghike and others added 21 commits December 3, 2013 16:50
…srRequest should gracefully handle leader == -1; patched by Swapnil Ghike; reviewed by Jun Rao
…rolled shutdown; patched by Jun Rao; reviewed by Swapnil Ghike
…Apis; patched by Jun Rao; reviewed by Neha Narkhede
…low throughput partitions; reviewed by Jun Rao and Guozhang Wang
…Rebalance directly; reviewed by Guozhang Wang and Jun Rao
…e change requests; reviewed by Jun Rao and Guozhang Wang
…isk; patched by Jun Rao; reviewed by Jay Kreps, Neha Narkhede and Guozhang Wang
… fixes in kafka-1020; patched by Jun Rao and Guozhang Wang; reviewed by Neha Narkhede and Joel Koshy
…afka-create-topics shell; patched by Guozhang Wang; reviewed by Jun Rao
… max-rack-replication option when creating a topic to distribute replicas such that no more than max-rack-replication replicas are hosted on the same rack-id. This option is also enforced when adding new partitions. The option does not enforce manual (re)assignment.
@jmlvanre jmlvanre closed this Feb 1, 2014
guozhangwang referenced this pull request in confluentinc/kafka Aug 5, 2015
Add examples of Processor and KStreamJob; add collector and coordinator to Processor.apply.
ymatsuda pushed a commit to ymatsuda/kafka that referenced this pull request Aug 28, 2015
resetius added a commit to resetius/kafka that referenced this pull request Jun 7, 2016
KAFKA-2169: Moving to zkClient 0.5 release.
lianetm added a commit to lianetm/kafka that referenced this pull request Jun 5, 2023
…e#15)

KAFKA-14966 - Extract reusable logic for offsetsForTimes
mjsax pushed a commit to mjsax/kafka that referenced this pull request Jul 21, 2024
…pc_handler

KSTREAMS-5764: Implement InitStreamsApp RPC in the group coordinator
haianh1233 added a commit to haianh1233/kafka that referenced this pull request Apr 17, 2026
New tests:
- apache#8  Port conflict (HTTP == PLAINTEXT port) → rejected
- apache#9  HTTP port=0 (random) works
- apache#10 HTTP + HTTPS coexist on same broker
- apache#11 advertised.listeners with HTTP parsed correctly
- apache#12 HTTP without httpAcceptorFactory → IllegalStateException
- apache#13 inter.broker.listener=HTTPS also rejected (not just HTTP)
- apache#14 Custom listener name mapped to HTTP protocol (MY_REST_API:HTTP)
- apache#15 HTTPS with valid SSL config succeeds

All 15 SocketServerHttpTest pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
blitzy Bot pushed a commit to blitzy-public-samples/blitzy-kafka that referenced this pull request Apr 18, 2026
Resolve all 9 Minor and 10 Info findings from the Checkpoint 1 code review,
correcting factual inaccuracies, citation line-range imprecisions, and cross-
artifact consistency drift. No modifications to pre-existing Kafka source,
tests, build files, or comments — Audit Only rule preserved.

Findings by file:

accepted-mitigations.md
  #1 [MINOR] AclCache imports corrected: org.apache.kafka.server.immutable
              (PCollections-backed Kafka-internal) instead of Guava's
              com.google.common.collect.
  apache#2 [MINOR] API surface rewritten to reflect PCollections-style structural-
              sharing methods .updated()/.added()/.removed() instead of
              Guava builder pattern.
  apache#3 [MINOR] ZstdCompression BufferPool path split: wrap-for-output uses
              zstd-jni RecyclingBufferPool.INSTANCE (L55-L63), wrap-for-
              input uses ChunkedBytesStream (L65-L75), wrap-for-zstd-input
              uses anonymous Kafka-owned BufferPool delegating to
              BufferSupplier (L77-L98).
  apache#4 [INFO]  MAX_RECORDS_PER_USER_OP citation corrected: declaration at
              QuorumController.java:L185; AclControlManager.java:L52 is
              the static import only.
  apache#5 [INFO]  AclCache.removeAcl(Uuid) line corrected to L91-L103 (was L89+).

references.md
  apache#6 [MINOR] SafeObjectInputStream citation range tightened from L17-L25
              (class header + imports only) to L25-L62 covering the class
              declaration, DEFAULT_NO_DESERIALIZE_CLASS_NAMES blocklist
              (L27-L37), resolveClass (L43-L52), and isBlocked helper
              (L54-L62).
  apache#7 [INFO]  PropertyFileLoginModule citation corrected to L42-L50,
              pointing at the Javadoc PLAINTEXT warning (L47-L48) plus
              the class declaration (L50).

remediation-roadmap.md
  apache#8 [INFO]  Gantt markers sanitised: all :done/:active markers replaced
              with :crit (illustrative critical emphasis) or plain markers
              to avoid any visual suggestion of work already performed.
              Explanatory blockquote added clarifying the marker change.

severity-matrix.md
  apache#9 [MINOR] 7 occurrences of parenthesised '(Accepted Mitigation)'
              replaced with bracketed '[Accepted Mitigation]' per Global
              Conventions for plain-text markers. Cross-validated 9
              bracketed instances, 0 parenthesised remaining.

README.md
  apache#11 [MINOR] HEAD commit reference corrected to the pre-audit baseline
               6d16f68 (was 8a99096, a
               mid-audit snapshot); baseline attestation now refers to the
               commit immediately before the audit began.
  apache#12 [MINOR] Snapshot date unified to 2026-04-17 across all artifacts.
  apache#14 [INFO]  '25 files' claim qualified as 'planned at project completion'
               vs 'delivered at this checkpoint (15 files)'.

attack-surface-map.md
  apache#16 [MINOR] Clients module category count corrected from 'six' to 'nine'
               (actual Mermaid edges: C1, C2, C3, C4, C5, C7, C8, C9, C10).
  apache#17 [MINOR] Connect module category count corrected from 'five' to
               'seven' (actual Mermaid edges: C1, C4, C6, C7, C8, C9, C10).

oauth-jwt-validation-paths.md
  apache#18 [INFO]  Outer citation ranges tightened:
               BrokerJwtValidator.configure at L107-L138 (not L102-L134);
               OAuthBearerUnsecuredValidatorCallbackHandler.handleCallback
               at L154-L177 (not L161-L204, which spanned unrelated
               helpers); allowableClockSkewMs helper cited separately at
               L194-L207.

executive-summary.html
  Cross-ref A [MINOR] HEAD commit aligned to 6d16f68 at three sites
                       (L621, L668, L1544); methodology Mermaid node
                       re-labelled 'Baseline 6d16f68'.
  Cross-ref B [MINOR] Snapshot date aligned to 2026-04-17 at two sites
                       (L619, L1542).

Out-of-scope (Info-level forward-refs):
  apache#10, apache#13, apache#15 — Links to docs/security-audit/findings/*.md deliverables
                   not yet present at Checkpoint 1; expected per scope
                   boundary; will resolve at Checkpoint 2 when the 10
                   per-category findings files land.

Validation results (Phase 3):
  - Mermaid fences: all balanced (20 blocks total, all typed)
  - HTML tag balance: 22 sections + all 20+ tag types balanced
  - CDNs intact: reveal.js 5.1.0, Mermaid 11.4.0, Font Awesome 6.6.0
  - Emojis: zero across all 15 artifacts
  - TODOs/placeholders introduced: zero
  - Gantt markers: :crit + plain only (no :done/:active)
  - Cross-artifact consistency: zero wrong SHA/date values remaining
  - Citation ranges: 12 verified against AclCache, QuorumController,
                     AclControlManager, ZstdCompression,
                     SafeObjectInputStream, PropertyFileLoginModule,
                     BrokerJwtValidator, and
                     OAuthBearerUnsecuredValidatorCallbackHandler.

Audit Only rule verification:
  git diff --name-status 6d16f68..HEAD returns only 'A' entries,
  all under docs/security-audit/. Zero modifications, deletions, or
  renames of any pre-existing Kafka path.
blitzy Bot pushed a commit to blitzy-public-samples/blitzy-kafka that referenced this pull request Apr 18, 2026
Adds docs/security-audit/findings/04-module-system-builtin-abuse.md,
the code-grounded audit finding covering enumeration position 4 of 10
in the security audit: module-system and built-in abuse.

Enumerates six pluggable SPI sub-findings and one built-in mitigation:
  04.1 Connect REST extensions (ServiceLoader)            [Medium]
  04.2 Connect plugin.path + DelegatingClassLoader        [Medium]
  04.3 MirrorMaker 2 FORWARDING_ADMIN_CLASS               [Medium]
  04.4 Tiered Storage RemoteStorageManager / RLMM         [Medium]
  04.5 OAuth JwtValidator / JwtRetriever                  [Low]
  04.6 Metrics reporters (kafka.metrics.reporters, metric.reporters) [Low]
  04.7 Built-in StandardAuthorizer + MAX_RECORDS_PER_USER_OP (accepted mitigation)

Every citation is a verified file-path + line-range reference into the
tracked source (no code changes made). MirrorClientConfig
FORWARDING_ADMIN_CLASS is correctly cited at Importance.LOW (L164).

Cross-references accepted-mitigation apache#15 (AclControlManager bounded list),
remediation-roadmap sections 3.3.4, 3.3.5, 3.4.2, and related findings
01.4, 07.4, 09, 10.

This change is part of the audit-only deliverable tree under
docs/security-audit/. Compliant with the Audit Only rule: no existing
source code, tests, configuration, comments, or runtime behaviour are
modified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.