BREAKING Overlay transaction support.#3263
BREAKING Overlay transaction support.#3263cheme wants to merge 203 commits intoparitytech:masterfrom
Conversation
initial values.
cheme
added
the
A3-in_progress
pepyakin
left a comment
pepyakin
left a comment
There was a problem hiding this comment.
This looks like a half way through to #2980! (With another part being an runtime instance state/linear memory concerns). But even with that, I think it might be super useful!
We even might be able to construct ext_try from primitives introduced in this PR and with a future introduced primitives for cloning a wasm instance!
Remove bench feature, exposing 'set_storage' publicly. Minor fixes.
Maybe start_transactional_layer |
I forgot to update my comment, I change it to 'storage_start_transaction' but maybe 'start_transactional_layer' is better? |
Co-Authored-By: Sergei Pepyakin <s.pepyakin@gmail.com>
|
I am actually not sure about the wording "layer". I.e. transactional layer of what? What if we get another kind of layer? On the other hand, "storage transaction" is unambigious. |
primitives/io/src/lib.rs
Outdated
|
|
||
| /// Interface for managing transaction within the runtime. | ||
| #[runtime_interface] | ||
| pub trait Transaction { |
There was a problem hiding this comment.
I wonder if StorageTransactions will be a better name?
| // You should have received a copy of the GNU General Public License | ||
| // along with Substrate. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| //! Types and method for managing a stack of transactional values. |
There was a problem hiding this comment.
| //! Types and method for managing a stack of transactional values. | |
| //! Types and methods for managing a stack of transactional values. |
primitives/state-machine/src/ext.rs
Outdated
| (_, Some(overlay_key)) => if overlay_key.1.value.is_some() { | ||
| return Some(overlay_key.0.to_vec()) | ||
| } else { | ||
| // TODO make this function non recursive to avoid this clone |
There was a problem hiding this comment.
Does this TODO need to be fixed before landing this? If not, an issue is required.
There was a problem hiding this comment.
No those TODO were just a note to myself, I am removing them.
primitives/state-machine/src/ext.rs
Outdated
| }; | ||
| // TODO no need to query child at each recursive iter here | ||
| // and also find a way to remove the clone (non recursive and global mut handle should do the | ||
| // trick). |
There was a problem hiding this comment.
Does this TODO need to be fixed before landing this? If not, an issue is required.
primitives/state-machine/src/ext.rs
Outdated
| .map(|&(ref k, ref v)| (k.to_vec(), Some(v.to_vec()))) | ||
| .chain(self.overlay.committed.top.clone().into_iter().map(|(k, v)| (k, v.value))) | ||
| .chain(self.overlay.prospective.top.clone().into_iter().map(|(k, v)| (k, v.value))) | ||
| .chain(self.overlay.changes.iter_values(None).map(|(k, v)| (k.to_vec(), v.map(|s| s.to_vec())))) |
There was a problem hiding this comment.
The line exceeds the recommended limit for 100 characters. Can we wrap this line?
| TxPending, | ||
| /// The transaction has been discarded. | ||
| /// Data from a `LayerEntry` pointing to this layer state should | ||
| /// not be returned and can be remove. |
| /// Get latest prospective value, excludes | ||
| /// committed values. | ||
| pub(crate) fn get_prospective(&self, states: &States) -> Option<&V> { | ||
| let self_len = self.len(); |
There was a problem hiding this comment.
I find it a bit confusing that the definition of len is in another module. Can we move these definitions closer together?
| /// triggering. | ||
| /// | ||
| /// With this default values it should be very unlikelly that gc is needed | ||
| /// during a block processing for most use case. |
| // This information is redundant as it could be | ||
| // calculated by iterating backward over `history` | ||
| // field. | ||
| // Managing this cache allow use to have |
| None | ||
| } | ||
|
|
||
| /// Push a value without checking without transactional layer |
pepyakin
dismissed
their stale review
Significant change of the implementation after my look.
cover all case but is a good entry point to read this PR.
|
This is stale, closing it, @athei will be working on the subject, keeping it open will only create confusion. |
11 participants
BREAKING: this pr adds externalities, and change mutability of existing one, the corresponding
This PR is a refactor of overlay_change to allow transactional support, it is similar to paritytech/polkadot-sdk#370 goals, but does not stack
Ext, it only stacks storage.It also switches from a stack of hashmap (previously
prospectiveandtop) to a single hashmap (containing history of values) and a transaction global state. Those values with state are managedwith 'historied-data' crate (simple vec of data to query in front of a reference global state).
Under this design access to data is not badly impacted by the number of open transactional layers.
I did fuzz this code a bit against a partial simple layered hashmap implementation.
Usage from a runtime with a function, in a similar way as
ext_tryfrom paritytech/polkadot-sdk#370, there is the very simplewith_transactionfunction: internally it uses three host functionsext_start_transaction,ext_commit_transactionandext_discard_transaction. This does not look as good as the singleext_trybut is clearer: memory mgmt seems way simplier (as there is none).Note that to call global state action, modification of local values need to be synchronize.
eg discard_transaction on states must be follow by apply_discard_transaction for all values related to this state and then follow by ensure_running.
Technically we only maintain a counter of current number of stacked transaction as a global state (this start at 1 and can be 0 only to update some values: case of discarding content, but will then return to 1 through 'finalize_discard' call).
Local state is either committed or the number of stacked transaction when the value was changed, this state is stored with the value history.
polkadot companion: paritytech/polkadot#999