feat: batch address trees

[sergeytimoshin]

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

• NEW: getBatchAddressUpdateInfo endpoint.
  Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

• 50 addresses appended to the queue
• 5 batch address append operations with batch_size=10 invoked

Asserts:

• getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
• Final root is checked against reference tree after applying batch address append txs.

TODO:

• Add test for the case when we have 2 addresses in 1 ix.

[ananas-block]

Suggested change

	for (new_address, seq) in event
	.new_addresses
	.iter()
	.zip(event.address_sequence_numbers.iter())
	{
	let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
	.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
	.clone();
	state_update_event.addresses.push(AddressQueueUpdate {
	tree: tree_info.tree.into(),
	address: new_address.address,
	queue_index: seq.seq,
	});
	}
	for (new_address, seq) in event
	.new_addresses
	.iter()
	.zip(event.address_sequence_numbers.iter())
	{
	state_update_event.addresses.push(AddressQueueUpdate {
	tree: new_address.mt_pubkey.into(),
	address: new_address.address,
	queue_index: seq.seq,
	});
	}

[ananas-block]

shouldn't this be in persisted_indexed_merkle_tree ?

[sergeytimoshin]

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

[ananas-block]

Thanks!
TLDR on comments:

1. are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
2. found some dead code in parse_public_transaction_event
3. PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
4. Final root assert in test
5. couple smaller improvements

[sergeytimoshin]

Try to remove

[ananas-block]

why was this necessary?
(I didn't want to release light-hasher again.)

[sergeytimoshin]

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph