input: add thread.ring_buffer.retry_limit option#11394
input: add thread.ring_buffer.retry_limit option#11394jinyongchoi wants to merge 2 commits intofluent:masterfrom
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a per-input configurable retry limit for threaded input ring buffer writes: new Changes
Sequence Diagram(s)sequenceDiagram
participant InputThread as Input Thread
participant RingBuffer as Ring Buffer
participant Metrics as Metrics/Logger
rect rgba(200,200,255,0.5)
InputThread->>RingBuffer: try enqueue(record)
alt enqueue succeeds
RingBuffer-->>InputThread: ack
else ring full
RingBuffer-->>InputThread: reject
InputThread->>InputThread: sleep(100ms) and retry up to ins.ring_buffer_retry_limit
opt retries exhausted
InputThread->>Metrics: increment ring_buffer_retry_failures
InputThread->>Metrics: log error "could not enqueue records"
InputThread-->>RingBuffer: drop record
end
end
end
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR adds a configurable retry limit for threaded input plugins' ring buffer writes. Previously, the retry limit was hardcoded to 10 attempts (1 second total with 100ms sleep between retries). This caused data loss during temporary backpressure situations that lasted longer than 1 second.
Changes:
- Added
thread.ring_buffer.retry_limitconfiguration option with default value of 10 to maintain backward compatibility - Modified ring buffer write logic to use the configurable retry limit instead of hardcoded value
- Updated documentation to explain the new configuration option
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| include/fluent-bit/flb_input.h | Added ring_buffer_retry_limit field to flb_input_instance struct |
| src/flb_input.c | Added configuration map entry, initialization, and property parsing for the new retry limit option |
| src/flb_input_chunk.c | Removed hardcoded retry limit and updated to use instance-level configurable value |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jinyongchoi
force-pushed
the
feature/input-ring-buffer-retry-limit
branch
from
0e15cc1 to
dba41e4
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
jinyongchoi
force-pushed
the
feature/input-ring-buffer-retry-limit
branch
from
dba41e4 to
4a748ca
Compare
|
@cosmo0920 @edsiper |
jinyongchoi
force-pushed
the
feature/input-ring-buffer-retry-limit
branch
from
4a748ca to
20b9f36
Compare
Add a new configuration option 'thread.ring_buffer.retry_limit' for threaded input plugins. This option controls the maximum number of retry attempts when the ring buffer is full before dropping data. The default value is 10 (maintaining backward compatibility). Fixes fluent#11393 Signed-off-by: jinyong.choi <inimax801@gmail.com>
Replace the hardcoded retry limit (10) with the configurable 'ring_buffer_retry_limit' value from the input instance. This allows users to increase retry attempts for handling temporary backpressure situations without dropping data. Fixes fluent#11393 Signed-off-by: jinyong.choi <inimax801@gmail.com>
jinyongchoi
force-pushed
the
feature/input-ring-buffer-retry-limit
branch
from
20b9f36 to
311d857
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
include/fluent-bit/flb_input.h (1)
386-386: Considerunsigned int(oruint32_t) to reflect the non-negative constraint at the type level.The field is validated to be
> 0inflb_input.c, but its type isint, allowing negative values to compile silently if the validation path is ever bypassed or the field is set directly. The neighbouring ring buffer fields already use distinct types (size_t,uint8_t) to encode their semantics.♻️ Optional type adjustment
- int ring_buffer_retry_limit; /* ring buffer write retry limit */ + unsigned int ring_buffer_retry_limit; /* ring buffer write retry limit */This also means the validation in
flb_input.cand the usage inflb_input_chunk.cshould check> 0against an unsigned type (which is naturally ≥ 0), or keep the> 0guard as-is for clarity.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@include/fluent-bit/flb_input.h` at line 386, Change the type of the struct field ring_buffer_retry_limit from int to an unsigned type (preferably uint32_t or unsigned int) in flb_input.h, then update all validation/uses in flb_input.c and flb_input_chunk.c to treat it as unsigned (keep the existing > 0 guard if you want to preserve explicit non-zero checks, or replace checks with != 0 where clearer) and adjust any signed-to-unsigned comparisons/casts to avoid warnings; ensure all references (e.g., the validation logic in flb_input.c and any loops or counters in flb_input_chunk.c) compile cleanly with the new unsigned type.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@include/fluent-bit/flb_input.h`:
- Line 386: Change the type of the struct field ring_buffer_retry_limit from int
to an unsigned type (preferably uint32_t or unsigned int) in flb_input.h, then
update all validation/uses in flb_input.c and flb_input_chunk.c to treat it as
unsigned (keep the existing > 0 guard if you want to preserve explicit non-zero
checks, or replace checks with != 0 where clearer) and adjust any
signed-to-unsigned comparisons/casts to avoid warnings; ensure all references
(e.g., the validation logic in flb_input.c and any loops or counters in
flb_input_chunk.c) compile cleanly with the new unsigned type.
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
include/fluent-bit/flb_input.hsrc/flb_input.csrc/flb_input_chunk.c
🚧 Files skipped from review as they are similar to previous changes (2)
- src/flb_input.c
- src/flb_input_chunk.c
When a threaded input plugin's ring buffer is full, the input thread retries writing to the buffer before dropping data. Previously, this retry limit was hardcoded to 10 (1 second total with 100ms sleep between retries).
This patch makes the retry limit configurable via the new 'thread.ring_buffer.retry_limit' option. The default value remains 10 for backward compatibility. Increasing this value allows the system to handle temporary backpressure situations without dropping data.
Fixes #11393
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes / Observability