Skip to content

Conversation

@BewareMyPower
Copy link
Contributor

@BewareMyPower BewareMyPower commented Oct 14, 2020

Motivation

When creating schema for a partitioned topic, e.g. my-topic, only one z-node /schemas/public/default/my-topic would be created to record the schema's metadata. However, multiple ledgers would be created because each single partition would try to create the schema: first append a ledger, then write the same z-node.

#2959 and #6683 solved the race condition of accessing the same z-node by calling putSchema() again. However, the unused ledgers of other partitions were created and were not recorded in the z-node. So, there's no way to locate these ledgers from ZK, which means there's no way to delete a topic's all associated schema ledgers.

Modifications

  • Record these unused ledgers' id under the topic's z-node, e.g. /schemas/public/default/my-topic/unusedLedgers/0 for topic my-topic and ledger 0.
  • Add a unit test to check if unusedLedgers z-node has these ledgers' id and these ledgers exist, after a partitioned topic's schema was created.
  • Make BookkeeperSchemaStorage.delete() delete ledgers found from ZK, instead of recording ledgers in memory.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Run test PartitionedTopicsSchemaTest.testUnusedLedgers.

@BewareMyPower
Copy link
Contributor Author

/pulsarbot run-failure-checks

@BewareMyPower
Copy link
Contributor Author

/pulsarbot run-failure-checks

@codelipenghui codelipenghui added this to the 2.7.0 milestone Oct 15, 2020
@jiazhai jiazhai changed the title Record unused schema ledgers to ZK for partitioned topics [WIP]Record unused schema ledgers to ZK for partitioned topics Oct 19, 2020
@BewareMyPower
Copy link
Contributor Author

Close this PR first because it's a temporary solution for current schema design. After discussing with @codelipenghui , the schema storage design should change, then this PR may not be needed.

@BewareMyPower BewareMyPower deleted the bewaremypower/record-unused-schema-ledgers branch September 16, 2022 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants