feat!(internal_logs source): turn ON rate limiting by default #24024

pront · 2025-10-17T18:34:20Z

Summary

This branch makes Vector's internal logging significantly more production-ready by:

Preventing log flooding by default (rate limiting ON)
- todo: remove redundant tags - will followup in another PR
Maintaining existing rate limiting behavior
Improving developer experience (comprehensive docs + easy opt-out mechanisms)
Setting foundation for future improvements

Technically a breaking change but very minor impact.

Vector configuration

How did you test this PR?

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Related: #22410

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

…-flag

thomasqueirozb

This should make life much easier! :)

lib/tracing-limit/src/lib.rs

…ed duplication

The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in #24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes #24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in #24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes #24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com>

* chore(ci): reorganize integration test files (#24108) * chore(ci): reorganize integration test files * revert nats data move * fix paths to data * fix more paths * amqp fix * Fix logstash int tests shared files paths * Rename scripts/integration -> tests/integration * scripts/integration -> tests/integration in source files * Fix scripts->tests * Fix gcp paths * Fix http-client pem path * move nats data --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(vdev): move all utils in a new utils folder (#24143) * chore(vdev): move all utils in a new utils folder * move remaining files to utils dir * move remaining files to utils dir * fmt * chore(vdev): make modules visible to rustfmt (#24162) * Remove mod inside cli_subcommands macro * cargo fmt * chore(vdev): apply vdev rust check fixes * Link to rustfmt issue * fix(vdev): release prepare vrl version pinning (#24158) * Add --dry-run to release prepare * Add error handling and checks to pin_vrl_version * Add wrapper to toml * Remove wrapper, parse as Table instead * Fix vrl pinning logic * Enable preserve_order feature in toml crate * Use dependency instead of whole toml * Fix dry run docs * Fix dry run wording * refactor to use toml_edit * Add update_vrl_to_version to add unit test * Use indoc in prepare.rs * Remove preserve_order feature * chore(deps): update VRL to add missing stdlib fns from 0.28 (#24178) * chore(ci): temporarily remove homebrew publish step from publish workflow (#24185) This temporarily removes the publish-homebrew job from the publish workflow to address issue #24139. This is step 1 of the plan to fix the homebrew publishing process. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * fix(blackhole sink): disable rate limiting for periodic stats messages (#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in #24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes #24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com> * chore(internal logs): Disable rate limiting for critical internal error logs (#24192) * Improve debug log in aws_ecs_metrics * Add internal_log_rate_limit = false to config reload errors * Disable rate limiting for lua build failure * Disable rate limiting for amqp build failure * Add internal_log_rate_limit = false to important failures during reload * Disable log rate limit for repeated events in process.rs * Update src/config/watcher.rs * Fix message format * fix(tracing): prevent panic for traces without standard fields (#24191) * fix(tracing): prevent panic for traces without std fields * forbid unwrap in this lib * clippy fix * improve events_with_custom_fields_no_message_dont_panic, advance to next window * chore(tracing): do not rate limit utlization report (#24202) * chore(internal metrics): move config_reload_* metrics to VectorReload* (#24203) * chore(internal metrics): move config_reload_* metrics to VectorReloaded and VectorReloadedError * update topology_doesnt_reload_new_data_dir test * chore(dev): cargo fmt * strengthen tests, since now we have a better interface * chore(releasing): Add 0.51.0 known issues (#24211) * Format fixes * Add 0.51.0 known issues * Add dots * Reword log panic known issue * Reword VRL missing fns * Backtick versions * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Fix cue docs fmt --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Bump vector version to 0.51.1 * Bump VRL to 0.28.1 * Generate 0.51.1 cue * Add vrl_changelog * Remove whitespace * chore(releasing): Updated distribution/install.sh vector version to 0.51.1 * chore(releasing): Add 0.51.1 to versions.cue * chore(releasing): Created release md file * Add description * Reorder items and fix format * Finally fix styling * Bump date --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

* chore(ci): reorganize integration test files (#24108) * chore(ci): reorganize integration test files * revert nats data move * fix paths to data * fix more paths * amqp fix * Fix logstash int tests shared files paths * Rename scripts/integration -> tests/integration * scripts/integration -> tests/integration in source files * Fix scripts->tests * Fix gcp paths * Fix http-client pem path * move nats data --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(vdev): move all utils in a new utils folder (#24143) * chore(vdev): move all utils in a new utils folder * move remaining files to utils dir * move remaining files to utils dir * fmt * chore(vdev): make modules visible to rustfmt (#24162) * Remove mod inside cli_subcommands macro * cargo fmt * chore(vdev): apply vdev rust check fixes * Link to rustfmt issue * fix(vdev): release prepare vrl version pinning (#24158) * Add --dry-run to release prepare * Add error handling and checks to pin_vrl_version * Add wrapper to toml * Remove wrapper, parse as Table instead * Fix vrl pinning logic * Enable preserve_order feature in toml crate * Use dependency instead of whole toml * Fix dry run docs * Fix dry run wording * refactor to use toml_edit * Add update_vrl_to_version to add unit test * Use indoc in prepare.rs * Remove preserve_order feature * chore(deps): update VRL to add missing stdlib fns from 0.28 (#24178) * chore(ci): temporarily remove homebrew publish step from publish workflow (#24185) This temporarily removes the publish-homebrew job from the publish workflow to address issue #24139. This is step 1 of the plan to fix the homebrew publishing process. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * fix(blackhole sink): disable rate limiting for periodic stats messages (#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in #24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes #24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com> * chore(internal logs): Disable rate limiting for critical internal error logs (#24192) * Improve debug log in aws_ecs_metrics * Add internal_log_rate_limit = false to config reload errors * Disable rate limiting for lua build failure * Disable rate limiting for amqp build failure * Add internal_log_rate_limit = false to important failures during reload * Disable log rate limit for repeated events in process.rs * Update src/config/watcher.rs * Fix message format * fix(tracing): prevent panic for traces without standard fields (#24191) * fix(tracing): prevent panic for traces without std fields * forbid unwrap in this lib * clippy fix * improve events_with_custom_fields_no_message_dont_panic, advance to next window * chore(tracing): do not rate limit utlization report (#24202) * chore(internal metrics): move config_reload_* metrics to VectorReload* (#24203) * chore(internal metrics): move config_reload_* metrics to VectorReloaded and VectorReloadedError * update topology_doesnt_reload_new_data_dir test * chore(dev): cargo fmt * strengthen tests, since now we have a better interface * chore(releasing): Add 0.51.0 known issues (#24211) * Format fixes * Add 0.51.0 known issues * Add dots * Reword log panic known issue * Reword VRL missing fns * Backtick versions * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Fix cue docs fmt --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Bump vector version to 0.51.1 * Bump VRL to 0.28.1 * Generate 0.51.1 cue * Add vrl_changelog * Remove whitespace * chore(releasing): Updated distribution/install.sh vector version to 0.51.1 * chore(releasing): Add 0.51.1 to versions.cue * chore(releasing): Created release md file * Add description * Reorder items and fix format * Finally fix styling * Bump date --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> chore(releasing): pull in `internal_logs` fix into 0.51.1 (#24225) * fix(internal_logs source): remove rate limit (#24218) * fix(internal_logs source): remove rate limit * add changelog * Debug commit * Fix validated - Revert "Debug commit" This reverts commit c7b9ec9. * add unit test * fix check-events * Pull in internal_logs source fix * Add internal_logs known issue * Bump date * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

* chore(releasing): prepare v0.51.1 release (#24214) * chore(ci): reorganize integration test files (#24108) * chore(ci): reorganize integration test files * revert nats data move * fix paths to data * fix more paths * amqp fix * Fix logstash int tests shared files paths * Rename scripts/integration -> tests/integration * scripts/integration -> tests/integration in source files * Fix scripts->tests * Fix gcp paths * Fix http-client pem path * move nats data --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(vdev): move all utils in a new utils folder (#24143) * chore(vdev): move all utils in a new utils folder * move remaining files to utils dir * move remaining files to utils dir * fmt * chore(vdev): make modules visible to rustfmt (#24162) * Remove mod inside cli_subcommands macro * cargo fmt * chore(vdev): apply vdev rust check fixes * Link to rustfmt issue * fix(vdev): release prepare vrl version pinning (#24158) * Add --dry-run to release prepare * Add error handling and checks to pin_vrl_version * Add wrapper to toml * Remove wrapper, parse as Table instead * Fix vrl pinning logic * Enable preserve_order feature in toml crate * Use dependency instead of whole toml * Fix dry run docs * Fix dry run wording * refactor to use toml_edit * Add update_vrl_to_version to add unit test * Use indoc in prepare.rs * Remove preserve_order feature * chore(deps): update VRL to add missing stdlib fns from 0.28 (#24178) * chore(ci): temporarily remove homebrew publish step from publish workflow (#24185) This temporarily removes the publish-homebrew job from the publish workflow to address issue #24139. This is step 1 of the plan to fix the homebrew publishing process. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * fix(blackhole sink): disable rate limiting for periodic stats messages (#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in #24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes #24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com> * chore(internal logs): Disable rate limiting for critical internal error logs (#24192) * Improve debug log in aws_ecs_metrics * Add internal_log_rate_limit = false to config reload errors * Disable rate limiting for lua build failure * Disable rate limiting for amqp build failure * Add internal_log_rate_limit = false to important failures during reload * Disable log rate limit for repeated events in process.rs * Update src/config/watcher.rs * Fix message format * fix(tracing): prevent panic for traces without standard fields (#24191) * fix(tracing): prevent panic for traces without std fields * forbid unwrap in this lib * clippy fix * improve events_with_custom_fields_no_message_dont_panic, advance to next window * chore(tracing): do not rate limit utlization report (#24202) * chore(internal metrics): move config_reload_* metrics to VectorReload* (#24203) * chore(internal metrics): move config_reload_* metrics to VectorReloaded and VectorReloadedError * update topology_doesnt_reload_new_data_dir test * chore(dev): cargo fmt * strengthen tests, since now we have a better interface * chore(releasing): Add 0.51.0 known issues (#24211) * Format fixes * Add 0.51.0 known issues * Add dots * Reword log panic known issue * Reword VRL missing fns * Backtick versions * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Fix cue docs fmt --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Bump vector version to 0.51.1 * Bump VRL to 0.28.1 * Generate 0.51.1 cue * Add vrl_changelog * Remove whitespace * chore(releasing): Updated distribution/install.sh vector version to 0.51.1 * chore(releasing): Add 0.51.1 to versions.cue * chore(releasing): Created release md file * Add description * Reorder items and fix format * Finally fix styling * Bump date --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> chore(releasing): pull in `internal_logs` fix into 0.51.1 (#24225) * fix(internal_logs source): remove rate limit (#24218) * fix(internal_logs source): remove rate limit * add changelog * Debug commit * Fix validated - Revert "Debug commit" This reverts commit c7b9ec9. * add unit test * fix check-events * Pull in internal_logs source fix * Add internal_logs known issue * Bump date * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Delete changelog entries --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

vectordotdev#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in vectordotdev#24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes vectordotdev#24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com>

* chore(releasing): prepare v0.51.1 release (vectordotdev#24214) * chore(ci): reorganize integration test files (vectordotdev#24108) * chore(ci): reorganize integration test files * revert nats data move * fix paths to data * fix more paths * amqp fix * Fix logstash int tests shared files paths * Rename scripts/integration -> tests/integration * scripts/integration -> tests/integration in source files * Fix scripts->tests * Fix gcp paths * Fix http-client pem path * move nats data --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(vdev): move all utils in a new utils folder (vectordotdev#24143) * chore(vdev): move all utils in a new utils folder * move remaining files to utils dir * move remaining files to utils dir * fmt * chore(vdev): make modules visible to rustfmt (vectordotdev#24162) * Remove mod inside cli_subcommands macro * cargo fmt * chore(vdev): apply vdev rust check fixes * Link to rustfmt issue * fix(vdev): release prepare vrl version pinning (vectordotdev#24158) * Add --dry-run to release prepare * Add error handling and checks to pin_vrl_version * Add wrapper to toml * Remove wrapper, parse as Table instead * Fix vrl pinning logic * Enable preserve_order feature in toml crate * Use dependency instead of whole toml * Fix dry run docs * Fix dry run wording * refactor to use toml_edit * Add update_vrl_to_version to add unit test * Use indoc in prepare.rs * Remove preserve_order feature * chore(deps): update VRL to add missing stdlib fns from 0.28 (vectordotdev#24178) * chore(ci): temporarily remove homebrew publish step from publish workflow (vectordotdev#24185) This temporarily removes the publish-homebrew job from the publish workflow to address issue vectordotdev#24139. This is step 1 of the plan to fix the homebrew publishing process. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * fix(blackhole sink): disable rate limiting for periodic stats messages (vectordotdev#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in vectordotdev#24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes vectordotdev#24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com> * chore(internal logs): Disable rate limiting for critical internal error logs (vectordotdev#24192) * Improve debug log in aws_ecs_metrics * Add internal_log_rate_limit = false to config reload errors * Disable rate limiting for lua build failure * Disable rate limiting for amqp build failure * Add internal_log_rate_limit = false to important failures during reload * Disable log rate limit for repeated events in process.rs * Update src/config/watcher.rs * Fix message format * fix(tracing): prevent panic for traces without standard fields (vectordotdev#24191) * fix(tracing): prevent panic for traces without std fields * forbid unwrap in this lib * clippy fix * improve events_with_custom_fields_no_message_dont_panic, advance to next window * chore(tracing): do not rate limit utlization report (vectordotdev#24202) * chore(internal metrics): move config_reload_* metrics to VectorReload* (vectordotdev#24203) * chore(internal metrics): move config_reload_* metrics to VectorReloaded and VectorReloadedError * update topology_doesnt_reload_new_data_dir test * chore(dev): cargo fmt * strengthen tests, since now we have a better interface * chore(releasing): Add 0.51.0 known issues (vectordotdev#24211) * Format fixes * Add 0.51.0 known issues * Add dots * Reword log panic known issue * Reword VRL missing fns * Backtick versions * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Fix cue docs fmt --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Bump vector version to 0.51.1 * Bump VRL to 0.28.1 * Generate 0.51.1 cue * Add vrl_changelog * Remove whitespace * chore(releasing): Updated distribution/install.sh vector version to 0.51.1 * chore(releasing): Add 0.51.1 to versions.cue * chore(releasing): Created release md file * Add description * Reorder items and fix format * Finally fix styling * Bump date --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> chore(releasing): pull in `internal_logs` fix into 0.51.1 (vectordotdev#24225) * fix(internal_logs source): remove rate limit (vectordotdev#24218) * fix(internal_logs source): remove rate limit * add changelog * Debug commit * Fix validated - Revert "Debug commit" This reverts commit c7b9ec9. * add unit test * fix check-events * Pull in internal_logs source fix * Add internal_logs known issue * Bump date * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Delete changelog entries --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

vectordotdev#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in vectordotdev#24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes vectordotdev#24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com>

* chore(releasing): prepare v0.51.1 release (vectordotdev#24214) * chore(ci): reorganize integration test files (vectordotdev#24108) * chore(ci): reorganize integration test files * revert nats data move * fix paths to data * fix more paths * amqp fix * Fix logstash int tests shared files paths * Rename scripts/integration -> tests/integration * scripts/integration -> tests/integration in source files * Fix scripts->tests * Fix gcp paths * Fix http-client pem path * move nats data --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com> * chore(vdev): move all utils in a new utils folder (vectordotdev#24143) * chore(vdev): move all utils in a new utils folder * move remaining files to utils dir * move remaining files to utils dir * fmt * chore(vdev): make modules visible to rustfmt (vectordotdev#24162) * Remove mod inside cli_subcommands macro * cargo fmt * chore(vdev): apply vdev rust check fixes * Link to rustfmt issue * fix(vdev): release prepare vrl version pinning (vectordotdev#24158) * Add --dry-run to release prepare * Add error handling and checks to pin_vrl_version * Add wrapper to toml * Remove wrapper, parse as Table instead * Fix vrl pinning logic * Enable preserve_order feature in toml crate * Use dependency instead of whole toml * Fix dry run docs * Fix dry run wording * refactor to use toml_edit * Add update_vrl_to_version to add unit test * Use indoc in prepare.rs * Remove preserve_order feature * chore(deps): update VRL to add missing stdlib fns from 0.28 (vectordotdev#24178) * chore(ci): temporarily remove homebrew publish step from publish workflow (vectordotdev#24185) This temporarily removes the publish-homebrew job from the publish workflow to address issue vectordotdev#24139. This is step 1 of the plan to fix the homebrew publishing process. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * fix(blackhole sink): disable rate limiting for periodic stats messages (vectordotdev#24190) * fix(blackhole sink): disable rate limiting for periodic stats messages The blackhole sink's informational "Collected events" messages were being rate-limited since rate limiting was enabled by default in vectordotdev#24024. This is undesirable because: 1. These are deliberately scheduled periodic messages (controlled by the `print_interval_secs` config option), not error conditions that could flood the logs 2. Users explicitly configure the frequency - rate limiting defeats that explicit configuration and breaks user expectations 3. The interval timer already prevents log flooding, making additional rate limiting redundant 4. The blackhole sink is used for debugging/testing, where predictable output is essential This fix adds `internal_log_rate_limit = false` to both info! calls, similar to how the console sink disables rate limiting for its critical operational messages. Fixes vectordotdev#24188 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changelog fragment for blackhole sink rate limiting fix --------- Co-authored-by: Claude <noreply@anthropic.com> * chore(internal logs): Disable rate limiting for critical internal error logs (vectordotdev#24192) * Improve debug log in aws_ecs_metrics * Add internal_log_rate_limit = false to config reload errors * Disable rate limiting for lua build failure * Disable rate limiting for amqp build failure * Add internal_log_rate_limit = false to important failures during reload * Disable log rate limit for repeated events in process.rs * Update src/config/watcher.rs * Fix message format * fix(tracing): prevent panic for traces without standard fields (vectordotdev#24191) * fix(tracing): prevent panic for traces without std fields * forbid unwrap in this lib * clippy fix * improve events_with_custom_fields_no_message_dont_panic, advance to next window * chore(tracing): do not rate limit utlization report (vectordotdev#24202) * chore(internal metrics): move config_reload_* metrics to VectorReload* (vectordotdev#24203) * chore(internal metrics): move config_reload_* metrics to VectorReloaded and VectorReloadedError * update topology_doesnt_reload_new_data_dir test * chore(dev): cargo fmt * strengthen tests, since now we have a better interface * chore(releasing): Add 0.51.0 known issues (vectordotdev#24211) * Format fixes * Add 0.51.0 known issues * Add dots * Reword log panic known issue * Reword VRL missing fns * Backtick versions * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Fix cue docs fmt --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Bump vector version to 0.51.1 * Bump VRL to 0.28.1 * Generate 0.51.1 cue * Add vrl_changelog * Remove whitespace * chore(releasing): Updated distribution/install.sh vector version to 0.51.1 * chore(releasing): Add 0.51.1 to versions.cue * chore(releasing): Created release md file * Add description * Reorder items and fix format * Finally fix styling * Bump date --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> chore(releasing): pull in `internal_logs` fix into 0.51.1 (vectordotdev#24225) * fix(internal_logs source): remove rate limit (vectordotdev#24218) * fix(internal_logs source): remove rate limit * add changelog * Debug commit * Fix validated - Revert "Debug commit" This reverts commit c7b9ec9. * add unit test * fix check-events * Pull in internal_logs source fix * Add internal_logs known issue * Bump date * Update website/cue/reference/releases/0.51.0.cue Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> * Delete changelog entries --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

pront added 3 commits October 17, 2025 12:50

feat(internal_logs source): turn ON rate limiting by default

e72fcaf

make fmt

c2f2757

wip

7c08da2

pront force-pushed the pront-tracing-limit-flag branch from 3882b2c to 762b078 Compare October 17, 2025 21:54

pront added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Oct 17, 2025

preserve master groupping behavior

1ad0df9

pront force-pushed the pront-tracing-limit-flag branch from 762b078 to 1ad0df9 Compare October 17, 2025 22:14

use 'component_id' and add changelog

1871245

github-actions bot added the domain: topology Anything related to Vector's topology code label Oct 17, 2025

pront added 4 commits October 20, 2025 10:07

remove unused 'vrl_position' special field

b6e52fa

make fmt

5cfb9c0

comments

5188f8d

changelog

383d557

pront changed the title ~~feat(internal_logs source): turn ON rate limiting by default~~ feat!(internal_logs source): turn ON rate limiting by default Oct 20, 2025

pront added 4 commits October 20, 2025 10:20

make fmt

7fc81d4

refactor tests

03ae4de

Merge remote-tracking branch 'origin/master' into pront-tracing-limit…

d894534

…-flag

improve cli doc

5e771d2

pront marked this pull request as ready for review October 20, 2025 15:24

pront requested review from a team as code owners October 20, 2025 15:24

thomasqueirozb approved these changes Oct 20, 2025

View reviewed changes

lib/tracing-limit/src/lib.rs Show resolved Hide resolved

lib/tracing-limit/src/lib.rs Show resolved Hide resolved

added rate_limit_same_message_different_component test case and remov…

4d04661

…ed duplication

buraizu self-assigned this Oct 20, 2025

pront enabled auto-merge October 20, 2025 19:05

buraizu approved these changes Oct 20, 2025

View reviewed changes

buraizu removed their assignment Oct 20, 2025

pront added this pull request to the merge queue Oct 20, 2025

Merged via the queue into master with commit 3146eca Oct 20, 2025
78 checks passed

pront deleted the pront-tracing-limit-flag branch October 20, 2025 19:52

pront mentioned this pull request Nov 7, 2025

fix(blackhole sink): disable rate limiting for periodic stats messages #24190

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat!(internal_logs source): turn ON rate limiting by default #24024

feat!(internal_logs source): turn ON rate limiting by default #24024

Uh oh!

pront commented Oct 17, 2025 •

edited

Loading

Uh oh!

thomasqueirozb left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat!(internal_logs source): turn ON rate limiting by default #24024

feat!(internal_logs source): turn ON rate limiting by default #24024

Uh oh!

Conversation

pront commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pront commented Oct 17, 2025 •

edited

Loading