Skip to content

Conversation

@klwilson227
Copy link
Contributor

Add some basic support for testing deduplication feature.

  1. Ability to set the producer name in the test client.
  2. Ability to set the sendTimeout in the test client.
  3. Ability to set the separator used between producername and threadnumber. As well as the topic and topic number. Defaults to '-'
  4. Ability to customize format each payload record if enabled.
    • Ability to customize/replace the formatting class which is used.
    • DefaultFormatter provided:
    • MetaData replacements: %p replaced with producer Name, %t nanoseconds when replacement occured, %i external messageId
    • Other replacements follow java sprintf formatting with %d %s %f %l with randomly generated strings or numbers.

@klwilson227 klwilson227 force-pushed the SupportTestDeduplication branch from 18831ac to f6d9802 Compare April 15, 2021 22:01
Copy link
Contributor

@merlimat merlimat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the goal is testing the deduping mechanism, we should probably also add something in the performance consumer to validate that no messages with duplicated sequences are received.

@klwilson227 klwilson227 force-pushed the SupportTestDeduplication branch 2 times, most recently from d07e66f to bc293c8 Compare April 16, 2021 14:13
@klwilson227
Copy link
Contributor Author

Merlimat, RE the consumer side, I'm sure the consumer side would also benefit as well for a possible demo/software acceptance level validation. However, in my particular use case the validation and scanning for duplicates and lost records was easier implemented in the Data Sink which received the data hence I did not invest extra to build a data validation layer within pulsar test client, only the ability to publish data using both the dedup enabled and dedup disabled.

Since, I did not have the requirement, it was not coded. And I did not want to assume that this was a requirement, as a poor implementation may show both strengths and weaknesses in the platform. I can certainly take it in that direction. But would think that may be a separate PR.

@klwilson227 klwilson227 force-pushed the SupportTestDeduplication branch from bc293c8 to 636306a Compare April 16, 2021 14:31
@klwilson227 klwilson227 requested a review from merlimat April 20, 2021 15:21
@klwilson227
Copy link
Contributor Author

klwilson227 commented Apr 21, 2021

/pulsabot rerun CI-CPP,Python Tests

@merlimat merlimat added this to the 2.8.0 milestone Apr 21, 2021
@merlimat merlimat added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Apr 21, 2021
@merlimat merlimat merged commit e4630a9 into apache:master Apr 21, 2021
@klwilson227 klwilson227 deleted the SupportTestDeduplication branch April 22, 2021 19:09
@lhotari
Copy link
Member

lhotari commented Apr 26, 2021

This PR introduced a flaky test, reported as #10378. I have created a PR to fix the flakiness in #10379 . Please review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tool type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants