Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Jun 19, 2024

Motivation

In Pulsar Functions and Pulsar IO Connectors, there's a Pulsar Producer cache, that
doesn't expire the producers and doesn't have a maximum size.
This is a problem for functions/connectors that produce to a large number of topics.

Modifications

  • Extract common ProducerCache
    • common Producer cache with separate cache areas for
      • producers created by calling Context, SinkContext, SourceContext methods
      • producers created in Pulsar Sources
        • multiple topics are possible by returning destination topics by SinkRecord.getDestinationTopic call
  • Keep backwards compatibility on the cache key
    • useThreadLocalProducers will use the Thread's ID as part of the key so that each thread has it's own isolated producer
    • In Sources, the SinkRecord.getPartitionId() is used as part of the cache key
  • Cache size is limited by the number of partitions in the producer
    • Default limit of the cache is 10000 partitions in total. Older entries will get removed from the cache if the cache overflows
  • Default cache entry expiration in 300 seconds after last access

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 3.4.0 milestone Jun 19, 2024
@lhotari lhotari self-assigned this Jun 19, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 19, 2024
@lhotari lhotari requested a review from nlu90 June 20, 2024 05:34
@lhotari
Copy link
Member Author

lhotari commented Jun 20, 2024

There's also #22950 improving code and fixing bugs in this area.

@lhotari lhotari force-pushed the lh-producer-cache-expiration branch 3 times, most recently from 2992fb6 to 27d42b7 Compare June 20, 2024 13:33
@lhotari lhotari force-pushed the lh-producer-cache-expiration branch from 27d42b7 to 6b36081 Compare June 20, 2024 13:47
lhotari added a commit that referenced this pull request Jun 25, 2024
lhotari added a commit that referenced this pull request Jun 25, 2024
…onnectors (#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
lhotari added a commit that referenced this pull request Jun 25, 2024
lhotari added a commit that referenced this pull request Jun 25, 2024
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants