queue: add new BackpressureQueue[T] variant by Roasbeef · Pull Request #9838 · lightningnetwork/lnd

Roasbeef · 2025-05-20T00:03:46Z

In this commit, we add a new type of queue: the back pressure queue. This is a bounded queue based on a simple channel, that will consult a predicate to decide if we should preemptively drop a message or not.

We then provide a sample predicate for this use case, based on random early dropping. Given a min and max threshold, we'll start to drop message randomly once we get past the min threshold, ramping up to the max threshold where we'll start to always drop the message.

coderabbitai · 2025-05-20T00:03:52Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

github-actions · 2025-05-20T00:03:58Z

Pull reviewers stats

Stats of the last 30 days for lnd:

User	Total reviews	Time to review	Total comments
guggero 🥇	25 ▀▀▀	23h 19m	49 ▀▀▀
ziggie1984 🥈	12 ▀	16h 37m	35 ▀▀
bhandras 🥉	11 ▀	4h 35m	13 ▀
yyforyongyu	11 ▀	1d 5h 12m	20 ▀
Roasbeef	6 ▀	9h 9m	4
bitromortac	5 ▀	1h 41m	6
ellemouton	4	1d 6h 20m	5
morehouse	2	7h 12m	1
mohamedawnallah	2	6d 14h 50m ▀▀	11 ▀
NishantBansal2003	2	5d 15h 32m ▀▀	0
ffranr	1	19m	0
sputn1ck	1	23h 39m	2
GeorgeTsagk	1	3d 36m ▀	0
saubyk	1	20h 37m	0
MPins	1	8d 20h 40m ▀▀▀	4

gijswijs

I did a pass of this and made a bunch of comments. There seems to be a lot of LLM fluff (although I could be mistaken) that I don't like.

queue/back_pressure.go

queue/back_pressure_test.go

Roasbeef · 2025-06-11T20:16:42Z

Pushed up a new version. PTAL.

gijswijs

Did a second pass. Mainly nits (line length! 😄) and some clarifications from my side about the use of generics in the tests.

Suffice to say that I really like this. I already used it in #10219 so I would love to see this merged.

@yyforyongyu Could you have a go at this as well?

queue/back_pressure_test.go

gijswijs · 2025-09-18T12:43:19Z

queue/back_pressure_test.go

+}
+
+// intQueueMachine is a concrete wrapper for queueMachine[int] for rapid.
+type intQueueMachine struct {


No, I meant

type intQueueMachine struct { tb rapid.TB capacity int queue *BackpressureQueue[int] modelQueue []int dropPredicate DropPredicate[int] itemGenerator *rapid.Generator[int] }

and take it from there.

gijswijs · 2025-09-18T12:47:14Z

queue/back_pressure_test.go

+}
+
+// intQueueMachine is a concrete wrapper for queueMachine[int] for rapid.
+type intQueueMachine struct {


This is what copilot came up with. Github doesn't allow me to upload .diff files, hence the .txt extension. no-generics.txt

queue/back_pressure.go

queue/back_pressure_test.go

queue/back_pressure.go

gijswijs

Some minor issues still, mostly validation and better comments. I don't think release notes are in place here, so maybe add the no-changelog label to the PR to skip that CI check.

queue/back_pressure.go

gijswijs · 2026-03-03T14:34:36Z

As I'm building on top of this in my onion messaging branch I made a few changes to the commit from this PR. Wanted to flag them in case you'd like to incorporate any of these upstream.

Split DropPredicate[T] into two types:

Added DropCheckFunc (func(queueLen int) bool) for length-only drop decisions.
RandomEarlyDrop now returns DropCheckFunc instead of DropPredicate[T], since RED only considers queue depth and doesn't need the item. This removes the type parameter from RandomEarlyDrop.
Added AsDropPredicate[T] adapter to bridge the two when BackpressureQueue (which still takes DropPredicate[T]) needs a length-only check.

New methods on BackpressureQueue:

TryEnqueue(item T) bool — non-blocking send with drop check.
Len() int, ReceiveChan() <-chan T, Close() — needed by the actor package's BackpressureMailbox, which uses BackpressureQueue as its core buffer while implementing the Mailbox interface's select-based iteration and lifecycle methods.

Minor behavioral change:

RandomEarlyDrop: a nil randSrc now falls back to rand.Float64 instead of panicking.

Better RED comment:

Added a note in the linear-scaling branch that minThreshold < maxThreshold is guaranteed at that point, so the denominator can't be zero.

Tests:

Updated existing tests to use the new AsDropPredicate[int](RandomEarlyDrop(...)) calling convention and explicit NewBackpressureQueue[int](...) type params.
Added tests for TryEnqueue, Len/ReceiveChan, and Close.

go.mod:

Bumped Go version to 1.25.5, added local replace for fn/v2.

This is the amended commit

Roasbeef · 2026-03-06T18:09:33Z

Thanks for putting that together! Incorporated most of these changes in the latest push:

The DropCheckFunc / AsDropPredicate[T] split is in, RandomEarlyDrop now returns (DropCheckFunc, error) with the type parameter removed.
TryEnqueue, Len(), ReceiveChan(), and Close() are all added.
Dequeue now handles the closed channel case properly via item, ok := <-q.ch, returning a new ErrQueueClosed sentinel.
Constructor and RED threshold validation return concrete sentinel errors (ErrInvalidCapacity, ErrNilPredicate, ErrNegativeMinThreshold, ErrInvalidThresholdOrder) instead of panicking.
The denominator safety comment is in the linear-scaling branch.
Tests condensed w/ subtests and table-driven validation cases.

In this commit, we add a new type of queue: the back pressure queue. This is a bounded queue based on a simple channel, that will consult a predicate to decide if we should preemptively drop a message or not. We then provide a sample predicate for this use case, based on random early dropping. Given a min and max threshold, we'll start to drop message randomly once we get past the min threshold, ramping up to the max threshold where we'll start to always drop the message.

lightninglabs-deploy · 2026-03-20T18:37:24Z

@gijswijs: review reminder

Roasbeef · 2026-03-26T23:52:13Z

Folded into 6b806f2

Roasbeef added enhancement Improvements to existing features / behaviour p2p Code related to the peer-to-peer behaviour labels May 20, 2025

gijswijs mentioned this pull request May 26, 2025

Basic structures for onion messages into LND #9868

Merged

gijswijs requested changes Jun 11, 2025

View reviewed changes

saubyk assigned Roasbeef Jun 11, 2025

saubyk added this to the v0.20.0 milestone Jun 11, 2025

saubyk added this to lnd v0.20 Jun 11, 2025

saubyk moved this to In progress in lnd v0.20 Jun 11, 2025

Roasbeef force-pushed the backpressure-queue branch from 2e346d0 to 3511604 Compare June 11, 2025 20:16

Roasbeef requested a review from gijswijs June 11, 2025 20:16

saubyk requested a review from yyforyongyu August 12, 2025 16:38

saubyk modified the milestones: v0.20.0, v0.21.0 Sep 5, 2025

saubyk removed this from lnd v0.20 Sep 5, 2025

This was referenced Sep 15, 2025

Async handling of Onion Messages with BackpressureQueue #10219

Merged

[epic]: Implement Onion Messaging (BOLT4) and Support Downstream BOLT12 Features #10220

Closed

gijswijs requested changes Sep 18, 2025

View reviewed changes

Roasbeef force-pushed the backpressure-queue branch from 3511604 to 3998b98 Compare October 9, 2025 20:49

Roasbeef requested a review from gijswijs October 9, 2025 20:49

Roasbeef removed the request for review from yyforyongyu October 31, 2025 00:50

saubyk added this to v0.21 Jan 13, 2026

saubyk moved this to In review in v0.21 Jan 13, 2026

gijswijs requested changes Feb 5, 2026

View reviewed changes

saubyk removed this from v0.21 Mar 5, 2026

saubyk modified the milestones: v0.21.0, v0.22.0 Mar 5, 2026

Roasbeef removed this from the v0.22.0 milestone Mar 6, 2026

Roasbeef added this to lnd v0.20 Mar 6, 2026

Roasbeef added the no-changelog label Mar 6, 2026

Roasbeef force-pushed the backpressure-queue branch from 3998b98 to 5759675 Compare March 6, 2026 18:02

Roasbeef force-pushed the backpressure-queue branch from 0379b2d to 8cef46e Compare March 7, 2026 01:50

Roasbeef requested a review from gijswijs March 10, 2026 02:34

Roasbeef closed this Mar 26, 2026

github-project-automation bot moved this to Done in lnd v0.20 Mar 26, 2026

Conversation

Roasbeef commented May 20, 2025

Uh oh!

coderabbitai bot commented May 20, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

github-actions bot commented May 20, 2025

Pull reviewers stats

Uh oh!

gijswijs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roasbeef commented Jun 11, 2025

Uh oh!

gijswijs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gijswijs Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

gijswijs Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gijswijs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gijswijs commented Mar 3, 2026

Uh oh!

Roasbeef commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lightninglabs-deploy commented Mar 20, 2026

Uh oh!

Roasbeef commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CodeRabbit Configuration File (`.coderabbit.yaml`)

Roasbeef commented Mar 6, 2026 •

edited

Loading