Fix pallet bags list and doc by gui1117 · Pull Request #10231 · paritytech/substrate

gui1117 · 2021-11-10T22:43:03Z

with current sp-io storage API we can't know how much we deleted when calling sp_io::storage::clear_prefix.

This information was not in the doc of remove_all function in frame-support., this is fixed.
This was misused in pallet-bags-list, this is fixed too.

polkadot companion: paritytech/polkadot#4306
skip check-dependent-cumulus

shawntabrizi · 2021-11-10T23:41:52Z

frame/support/src/storage/types/map.rs

+	/// Remove all values of the storage in the overlay and up to `limit` in the backend.
+	///


Its just a crazy api. This is absolutely not an "expected" behavior, and we shouldn't really expose such weird things in this way...

We need apis like this which are both bounded and behave in expected ways.

yes I agree, IMO we should at least have sp_io::storage::clear_prefix return the exact number of value removed (in both overlay and backend).

shawntabrizi

seems like we need this right away, but hopefully this is not the final state of things

frame/election-provider-support/src/lib.rs

bkchr · 2021-11-12T14:08:38Z

frame/election-provider-support/src/lib.rs

+	///
+	/// This function should never be called in production settings because it can lead to an
+	/// unbounded amount of storage accesses.
+	fn clear();


Maybe call this unsafe_clear then? To make the user aware of the problems?

also in pallet-bags-list when implementing the function SortedListProvider::regenerate it calls clear inside. Should we also prepend unsafe to regenerate ? WDYT @kianenigma

Yes we should

addressed in fede886

frame/support/src/storage/mod.rs

xlc · 2021-11-16T21:44:18Z

Um... Does this means remove_all is unsafe to be used by parachains? Do we need to re-audit all the usage of it?

gui1117 · 2021-11-17T00:54:14Z

Um... Does this means remove_all is unsafe to be used by parachains? Do we need to re-audit all the usage of it?

remove_all for counted storage map doesn't allow to give bounds on the number of value to remove from backend, thus in case you remove too many keys, this can go beyond PoV size.

Also benchmarks should already count the number of storage access in general.

xlc · 2021-11-17T01:09:23Z

Ok. So as long as we correctly benchmark all the calls using remove_all we should be mostly fine.

gui1117 · 2021-11-17T01:29:25Z

bot merge

paritytech-processbot · 2021-11-17T01:29:33Z

Error:
Error: When trying to meet the "Project Owners" approval requirements: this pull request does not belong to a project defined in Process.json.

Approval by "Project Owners" is only attempted if other means defined in the criteria for merge are not satisfied first.

The following errors might have affected the outcome of this attempt:

'Batch: Availability and Validity' does not match any projects in polkadot's Process.json
'Batch: Codebase Restructure' does not match any projects in polkadot's Process.json

Merge failed. Check out the criteria for merge. If you're not meeting the approval count, check if the approvers are members of substrateteamleads or core-devs.

gui1117 · 2021-12-01T05:22:49Z

companion needs a review

shawntabrizi · 2021-12-02T16:08:30Z

reminder on this one

gui1117 · 2021-12-03T04:06:46Z

bot merge

paritytech-processbot · 2021-12-03T04:06:56Z

Waiting for commit status.

paritytech-processbot · 2021-12-03T04:11:14Z

Merge cancelled due to error. Error: Companion paritytech/polkadot#4306 has error: Checks failed for bfccb973507ebc8d3a28a514e6a8f38073216fdb

gui1117 · 2021-12-03T05:58:01Z

bot merge

* fix bags list * improve doc * doc * inner doc * fix test * Update docs in frame/election-provider-support/src/lib.rs * fix staking impl * prepend unsafe to clear and regenerate * fix test Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

gui1117 added 2 commits November 11, 2021 07:37

fix bags list

c1550d2

improve doc

29ce8c0

github-actions bot added the A0-please_review Pull request needs code review. label Nov 10, 2021

gui1117 added B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed. labels Nov 10, 2021

doc

ed1fb34

gui1117 requested review from emostov and kianenigma November 10, 2021 22:46

inner doc

c2f6366

shawntabrizi reviewed Nov 10, 2021

View reviewed changes

shawntabrizi approved these changes Nov 10, 2021

View reviewed changes

fix test

eb90700

kianenigma approved these changes Nov 11, 2021

View reviewed changes

KiChjang reviewed Nov 12, 2021

View reviewed changes

frame/election-provider-support/src/lib.rs Outdated Show resolved Hide resolved

Update docs in frame/election-provider-support/src/lib.rs

2e72037

gui1117 mentioned this pull request Nov 12, 2021

use CountedMap in pallet-bags-list #10179

Merged

fix staking impl

ce7fff7

bkchr reviewed Nov 12, 2021

View reviewed changes

emostov reviewed Nov 13, 2021

View reviewed changes

frame/support/src/storage/mod.rs Show resolved Hide resolved

gui1117 added 2 commits November 16, 2021 10:12

prepend unsafe to clear and regenerate

fede886

Merge remote-tracking branch 'origin/master' into gui-fix-bags-list

61e8612

emostov approved these changes Nov 16, 2021

View reviewed changes

fix test

ee451a5

Merge remote-tracking branch 'origin/master' into gui-fix-bags-list

ae9c70e

This was referenced Nov 17, 2021

Improve sp_io::storage::clear_prefix API to return the number of key deleted #10288

Closed

companion for #10231 paritytech/polkadot#4306

Merged

Merge remote-tracking branch 'origin/master' into gui-fix-bags-list

18a0d8b

gui1117 mentioned this pull request Dec 2, 2021

Used CountedStorageMap in pallet-staking #10233

Merged

shawntabrizi and others added 2 commits December 2, 2021 13:13

Merge branch 'master' into gui-fix-bags-list

790456b

Merge remote-tracking branch 'origin/master' into gui-fix-bags-list

56e3ede

gui1117 mentioned this pull request Dec 3, 2021

Allow to pass a bound on StorageCountedMap::remove_all (similarly to StorageMap::removal_all) paritytech/polkadot-sdk#294

Open

paritytech-processbot bot merged commit b6d489e into master Dec 3, 2021

paritytech-processbot bot deleted the gui-fix-bags-list branch December 3, 2021 05:58

aurexav added a commit to darwinia-network/darwinia-common that referenced this pull request Sep 8, 2022

Companion of paritytech/substrate#10231, paritytech/substrate#10233

ecb22b3

		/// Remove all values of the storage in the overlay and up to `limit` in the backend.
		///

Conversation

gui1117 commented Nov 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shawntabrizi Nov 10, 2021

Choose a reason for hiding this comment

Uh oh!

gui1117 Nov 10, 2021

Choose a reason for hiding this comment

Uh oh!

shawntabrizi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bkchr Nov 12, 2021

Choose a reason for hiding this comment

Uh oh!

gui1117 Nov 15, 2021

Choose a reason for hiding this comment

Uh oh!

emostov Nov 15, 2021

Choose a reason for hiding this comment

Uh oh!

gui1117 Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xlc commented Nov 16, 2021

Uh oh!

gui1117 commented Nov 17, 2021

Uh oh!

xlc commented Nov 17, 2021

Uh oh!

gui1117 commented Nov 17, 2021

Uh oh!

paritytech-processbot bot commented Nov 17, 2021

Uh oh!

gui1117 commented Dec 1, 2021

Uh oh!

shawntabrizi commented Dec 2, 2021

Uh oh!

gui1117 commented Dec 3, 2021

Uh oh!

paritytech-processbot bot commented Dec 3, 2021

Uh oh!

paritytech-processbot bot commented Dec 3, 2021

Uh oh!

gui1117 commented Dec 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gui1117 commented Nov 10, 2021 •

edited

Loading