Stress-testing, performance, and miner optimizations for subnets

[gregorycoppola]

Is your feature request related to a problem? Please describe.
There has been a variety of discussion on the stacks-blockchain side about the miner's performance during the Steady Lads mint.

Since this repo is a fork of stacks-blockchain, we could have analogous issues here.

We should start stress testing the hyper-chain.

If we find any fixes, we can backport them to stacks-blockchain, so that Hiro can contribute to stacks-blockchain organically through pursuing its business interest in hyper-chains.

Describe the solution you'd like

1. stress testing of hyper-chains
2. performance measurements
3. improvements

Additional context
This bug was found recently in the miner code stacks-network/stacks-core#3184

Maybe there are more things like this. In either way, we should presumably be fixing in both places.

[kantai]

I think that the first area for pushing on performance in the hyperchain implementation is in confirmation times. One of the major performance benefits of hyperchains is that because they functionally have a single miner (whether they truly have a single miner, or a federated set), microblocks are guaranteed to be confirmed by the subsequent anchor block. This means that transactions can confirm much faster than in the stacks-node: hyperchain microblocks count as confirmed blocks in hyperchains.

The question is, in the current implementation, how well does microblock production work? If I broadcast a transaction to a hyperchain mempool, how quickly does it get included in a microblock? How much does this depend on stacks block confirmation times? Bitcoin confirmation times? How often does the hyperchain miner's block commit end up "missing" the intended stacks block?

[saralab]

Profiling results so far here: https://www.notion.so/hiropbc/Profile-Results-1-10k-L2-Mints-712e8e92ac674f118662535eb4344d62

[gregorycoppola]

Update: https://www.notion.so/In-Memory-Mempool-to-Eliminate-Time-Between-Processing-Transactions-663677018afc4ea2a62c9a7aa4667e22

[elenananana]

nitpick- is it possible to change the name of this issue to swap out hyper-chain for subnet?