draft: In-Memory Mempool, Nonces in MARF, Sort by fee rate#152
Closed
gregorycoppola wants to merge 38 commits intomasterfrom
Closed
draft: In-Memory Mempool, Nonces in MARF, Sort by fee rate#152gregorycoppola wants to merge 38 commits intomasterfrom
gregorycoppola wants to merge 38 commits intomasterfrom
Conversation
added 30 commits
August 22, 2022 17:57
INFO [1662211650.175497] [src/core/mempool.rs:1208] [relayer] total_inside_time: 131.676642ms total_outside_time: 3.345164806s total_consider_next_time: 0ns total_update_nonce_time: 1.000607786s total_bump_nonce_time: 211.635373ms
INFO [1662211651.242151] [src/core/mempool.rs:1208] [relayer] total_inside_time: 823ns total_outside_time: 1.066637144s total_consider_next_time: 0ns total_update_nonce_time: 226.748852ms total_bump_nonce_time: 0ns
INFO [1662211651.246141] [src/chainstate/stacks/miner.rs:1556] [relayer] Miner: mined anchored block, block_hash: 5164702a422df4d3845a1e0a80a44ed44f18d31b30a7d85d688dd0ff9d33ecf7, height: 6, txs_len: 97, parent_block: 022a224627f4faf7836fee9bc4b71336ba82686b9ef6367818fbc52927184d8b, parent_microblock: 0000000000000000000000000000000000000000000000000000000000000000, parent_microblock_sequence: 0, state_root_hash: 3ba082e05a3cf0acf9a2b1906dc41ab9fc38115a0c1828d598cd2a219ba9afc5, header_serialized: 00000000000000000100000000000000060101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101022a224627f4faf7836fee9bc4b71336ba82686b9ef6367818fbc52927184d8b000000000000000000000000000000000000000000000000000000000000000000000af0be4d34d166315da903bac0c4de0566267642883efa0eb454b024adff48d93ba082e05a3cf0acf9a2b1906dc41ab9fc38115a0c1828d598cd2a219ba9afc5000000000000000000000000000000000000000000000000000000000000000096fb7eab48064e4ae9ab06bfa077b6eb061e0fc300000000
INFO [1662211651.248047] [src/chainstate/stacks/miner.rs:2257] [relayer] Miner: mined anchored block, block_hash: 5164702a422df4d3845a1e0a80a44ed44f18d31b30a7d85d688dd0ff9d33ecf7, height: 6, tx_count: 97, parent_stacks_block_hash: 022a224627f4faf7836fee9bc4b71336ba82686b9ef6367818fbc52927184d8b, parent_stacks_microblock: 0000000000000000000000000000000000000000000000000000000000000000, parent_stacks_microblock_seq: 0, block_size: 19919, execution_consumed: {"runtime": 192000000, "write_len": 96, "write_cnt": 96, "read_len": 158016, "read_cnt": 384}, assembly_time_ms: 6277, tx_fees_microstacks: 0
INFO [1662211669.608239] [src/core/mempool.rs:1208] [relayer] total_inside_time: 130.217509ms total_outside_time: 4.627667364s total_consider_next_time: 0ns total_update_nonce_time: 401.612964ms total_bump_nonce_time: 78.106139ms
INFO [1662211672.159179] [src/core/mempool.rs:1208] [relayer] total_inside_time: 546ns total_outside_time: 2.550927313s total_consider_next_time: 0ns total_update_nonce_time: 312.177831ms total_bump_nonce_time: 0ns
INFO [1662211672.161181] [src/chainstate/stacks/miner.rs:1556] [relayer] Miner: mined anchored block, block_hash: 0673a0cb675e44b7a53375ca28250679dfc78f1ca3ed8bdc1d859c9da13b3101, height: 7, txs_len: 97, parent_block: 5164702a422df4d3845a1e0a80a44ed44f18d31b30a7d85d688dd0ff9d33ecf7, parent_microblock: 0000000000000000000000000000000000000000000000000000000000000000, parent_microblock_sequence: 0, state_root_hash: f926b1a7628e0abdd8eef01dce751865a1ca87554b39e6d1419b318ed0895791, header_serialized: 000000000000000001000000000000000701010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101015164702a422df4d3845a1e0a80a44ed44f18d31b30a7d85d688dd0ff9d33ecf70000000000000000000000000000000000000000000000000000000000000000000050a56dd7aef8f6618ecd5390212c65a889c70df3e08b1564841732b16848c646f926b1a7628e0abdd8eef01dce751865a1ca87554b39e6d1419b318ed08957910000000000000000000000000000000000000000000000000000000000000000f87d64298aa3d3b1ab4e28c566f72aca57ba37c400000000
INFO [1662211672.162465] [src/chainstate/stacks/miner.rs:2257] [relayer] Miner: mined anchored block, block_hash: 0673a0cb675e44b7a53375ca28250679dfc78f1ca3ed8bdc1d859c9da13b3101, height: 7, tx_count: 97, parent_stacks_block_hash: 5164702a422df4d3845a1e0a80a44ed44f18d31b30a7d85d688dd0ff9d33ecf7, parent_stacks_microblock: 0000000000000000000000000000000000000000000000000000000000000000, parent_stacks_microblock_seq: 0, block_size: 19919, execution_consumed: {"runtime": 192000000, "write_len": 96, "write_cnt": 96, "read_len": 158016, "read_cnt": 384}, assembly_time_ms: 7778, tx_fees_microstacks: 0
INFO [1662211689.786656] [src/core/mempool.rs:1208] [relayer] total_inside_time: 37.56342ms total_outside_time: 4.793066709s total_consider_next_time: 0ns total_update_nonce_time: 456.440545ms total_bump_nonce_time: 60.261131ms
INFO [1662211689.801095] [src/core/mempool.rs:1208] [relayer] total_inside_time: 587ns total_outside_time: 14.403023ms total_consider_next_time: 0ns total_update_nonce_time: 0ns total_bump_nonce_time: 0ns
INFO [1662211689.807623] [src/chainstate/stacks/miner.rs:1556] [relayer] Miner: mined anchored block, block_hash: 5d29579091ac39c7c0e73adaddda9bd018999d3fab59b5b173f6cf0e7d4a1f3a, height: 8, txs_len: 30, parent_block: 0673a0cb675e44b7a53375ca28250679dfc78f1ca3ed8bdc1d859c9da13b3101, parent_microblock: 0000000000000000000000000000000000000000000000000000000000000000, parent_microblock_sequence: 0, state_root_hash: 3ff9156233289f07a0c0640d1cb2a91bca2a3933ab2f7c0a149eee5600bd8bd3, header_serialized: 000000000000000001000000000000000801010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010673a0cb675e44b7a53375ca28250679dfc78f1ca3ed8bdc1d859c9da13b310100000000000000000000000000000000000000000000000000000000000000000000713dcea01d289df735f377fbddd6e3455c309153f3f7e223b8437e28292b979c3ff9156233289f07a0c0640d1cb2a91bca2a3933ab2f7c0a149eee5600bd8bd300000000000000000000000000000000000000000000000000000000000000005d05aa064adba33241567f1edff273f3c00ee73e00000000
iINFO [1662239879.963000] 60304 /new_burn_block INFO [1662239880.1000] 60304 /new_block: #txs=2 txids=0x37e672b5bccd663b4554f92075319a45eb1fa28ae97fc2bb6dc61ccad0724aca,0x88871220f5b5100e40f7871104069004357ce97fe182840903272b3a178f57ea INFO [1662239880.27000] 60303 /new_mempool_tx INFO [1662239880.894000] found transaction 0x88871220f5b5100e40f7871104069004357ce97fe182840903272b3a178f57ea user2 INFO [1662239880.894000] wait for transaction 0x22d05846c113eb7b5fa4d6588db7c718ee388eab7542a7b3b2962c5f0872d353 user3 INFO [1662239898.891000] 60303 /new_block: #txs=2 txids=0x616acf76bb6edf76b1dce295c2ee8485a6c65b89d3a603b66a06f3938e8bdd03,0xff87cb72b4ab2ff7acf6cc9d82bb6b2b2b29a740327d23892c12dd2ea4098ade INFO [1662239900.087000] 60304 /new_burn_block INFO [1662239900.232000] 60304 /new_block: #txs=2 txids=0x6c51cd072828d03798dd733bff7d4aa974c638b6ae748848f506f7a8fb7b1d09,0x22d05846c113eb7b5fa4d6588db7c718ee388eab7542a7b3b2962c5f0872d353 INFO [1662239900.382000] 60303 /new_mempool_tx INFO [1662239900.907000] found transaction 0x22d05846c113eb7b5fa4d6588db7c718ee388eab7542a7b3b2962c5f0872d353 user3 INFO [1662239900.907000] start submitting mints INFO [1662239900.907000] 60304 will stop showing mempool alerts INFO [1662239918.962000] 60303 /new_block: #txs=2 txids=0x9fafbd3d4cd8d3a7992d2e52d1df300df4f81c8446d487d961c848ea40263290,0x2eec342a180eb8b3692aa18cf3c305b6f2fedb63934f323c0c5be07a4b564362 INFO [1662239920.205000] 60304 /new_burn_block INFO [1662239920.331000] 60304 /new_block: #txs=1 txids=0x54ad99bb932640fa14562d8d25b7d799e5645feb6c53e907bb253b68b531a751 INFO [1662239925.546000] 60303 /new_mempool_tx INFO [1662239939.022000] 60303 /new_block: #txs=2 txids=0xf2e5bd7ca5a85bccc9161cbe84dcd4569f8c973756809b7e2e21ff4bb504ab3d,0xb38874db434ac5f9db66b88bee82fa631ea5bd2ac9cdc0212a1b47b720f6c0e9 INFO [1662239940.318000] 60304 /new_burn_block INFO [1662239946.566000] 60304 /new_block: #txs=1697 txids= INFO [1662239951.799000] 60303 /new_mempool_tx INFO [1662239959.089000] 60303 /new_block: #txs=2 txids=0x0bf390ba9081e042038ed59bb8b8967187b32226981de720aab1eeee6d251f08,0xd1473b869e4c98f24ee9178ca766ddd3fd9176142bb117d2533f158fbd74fd6c INFO [1662239960.43000] 60304 /new_burn_block INFO [1662239967.468000] 60304 /new_block: #txs=1919 txids= INFO [1662239972.62000] 60303 /new_mempool_tx INFO [1662239979.155000] 60303 /new_block: #txs=2 txids=0x9898195b555dd4aee07fdfeba39e2333bf74292b157fe3dce89f58a53b33d3a5,0x555175ae7aeb932b20dbd5e7c1bb279485652e45362928bb5ad110fb1c2b0643 INFO [1662239980.572000] 60304 /new_burn_block INFO [1662239986.019000] 60304 /new_block: #txs=1498 txids= INFO [1662239991.271000] 60303 /new_mempool_tx INFO [1662239999.22000] 60303 /new_block: #txs=2 txids=0x2bebeb4cef3e6639bc5f98f7552994b03d048325fda1cb09a549403b0fd61470,0x7b0396728556e45156ceb5ecb777478b27c8e8b398ceb9bb27f727407282985d INFO [1662240000.684000] 60304 /new_burn_block INFO [1662240005.5000] 60304 /new_block: #txs=1342 txids= INFO [1662240010.658000] 60303 /new_mempool_tx INFO [1662240019.283000] 60303 /new_block: #txs=2 txids=0x758e823038e92d197528b671aa392edb484d1c7b1e57245d712251cc8b0f58c2,0x2dc738dfcaa1e29f7cd2c9b0eaef0b0f765a488c042153c8c743d26bda4acae2 INFO [1662240020.796000] 60304 /new_burn_block INFO [1662240025.753000] 60304 /new_block: #txs=1351 txids= INFO [1662240030.915000] 60303 /new_mempool_tx INFO [1662240039.354000] 60303 /new_block: #txs=2 txids=0xd51c8e1e5bb4e04e9da453360b267c852a027c16593100c6df1657960d543e44,0x92ba59cc8ccbbca5b61b757fec3b0cd155a9c8199bba373862367c35d6e85e1f INFO [1662240040.915000] 60304 /new_burn_block INFO [1662240041.033000] start collecting mints INFO [1662240041.033000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240045.234000] 60304 /new_block: #txs=1398 txids= INFO [1662240046.353000] 60303 /new_mempool_tx INFO [1662240051.034000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240059.448000] 60303 /new_block: #txs=2 txids=0x0f3dd92e9e4ebfed4d542f551085933d17e19b2504c8d53012a125a04d8f7bec,0xc47b041a7ac493544bbd0db86651d520be39419946982d6aa85898de24782ec6 INFO [1662240061.028000] 60304 /new_burn_block INFO [1662240061.034000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240063.676000] 60304 /new_block: #txs=802 txids= INFO [1662240063.992000] 60303 /new_mempool_tx INFO [1662240071.04000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240079.564000] 60303 /new_block: #txs=2 txids=0x384fb97a620d71a51f64aa8e661314bf23f4df4cbde990366510ac4f731ad3f8,0x759a2d41b1aa641dc5849e2cdbd571642f9c73d88fec7476645b5cb2cad35aef INFO [1662240081.04000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240081.148000] 60304 /new_burn_block INFO [1662240081.271000] 60304 /new_block: #txs=1 txids=0x46cdf9b1ea7023322d75951a61dc3e3874bb39a23ea61fd28ead338d977210fc INFO [1662240081.648000] 60303 /new_mempool_tx INFO [1662240091.042000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240099.658000] 60303 /new_block: #txs=2 txids=0x7b5bfd54e03ddf400cffd8a7fe4ff58fa5679aab3a509d0c513d7f50d4039f42,0xf208b7a5ec2399187ab332196f15db4b4a2e5ca541270ad74c3921db353142e1 INFO [1662240101.043000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240101.274000] 60304 /new_burn_block INFO [1662240101.394000] 60304 /new_block: #txs=1 txids=0xe5cb9ecdbf6eaad5e46e69cb6bd2a33aa7b9ad70f6f2edd669ccd173e0e75dae INFO [1662240101.82000] 60303 /new_mempool_tx INFO [1662240111.051000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240119.761000] 60303 /new_block: #txs=2 txids=0x252a53ef9f32f69c65116f267e88681768aae6d2bd787511cc220c1fa772549d,0x07e9fa227a2d0f6e428dde8eb1a2f9ebb6177c03b6776a9daca6939368df0206 INFO [1662240121.052000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240121.399000] 60304 /new_burn_block INFO [1662240121.52000] 60304 /new_block: #txs=1 txids=0x9f7bee887d805806f1d3636f721cac32048cc3230eeb9ec38ec6fe744ee09373 INFO [1662240121.908000] 60303 /new_mempool_tx INFO [1662240131.058000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240139.864000] 60303 /new_block: #txs=2 txids=0x47d563767416101b09b43c908f8ef1cd2f1157d8f13584928f544e48bb92845e,0x898699b8bedff63a3cdbb23702257d6f5c7a630f09784ebf885c12d37499535a INFO [1662240141.059000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240141.524000] 60304 /new_burn_block INFO [1662240141.649000] 60304 /new_block: #txs=1 txids=0xc13a6a5ddee81807007e2af63393cce798a35a612799dc0191400976659603bc INFO [1662240142.021000] 60303 /new_mempool_tx INFO [1662240151.065000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240159.971000] 60303 /new_block: #txs=2 txids=0xff0382274b1bb687c3f9a7b1b8788ea7a23f984345501d27cd8443439ad321cb,0xbf733dc707195ff52bd47370b3dfff9009cd58e1f8850d46a32516268427c50a INFO [1662240161.066000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240161.643000] 60304 /new_burn_block INFO [1662240161.758000] 60304 /new_block: #txs=1 txids=0x06aac209d32fe5779171da69cbf0a7d6c0662173e6432a1ff1aade925e691dc7 INFO [1662240162.177000] 60303 /new_mempool_tx INFO [1662240171.071000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240180.064000] 60303 /new_block: #txs=2 txids=0xe8a2e0cb2669f9cbd545c3e5f62e6f7dac3491bb8f6e7fbe69abf438fcc70d1c,0xe55efab1ab3921b5aa1aff806e550715a216ef38b990e0766a41dcb5e06bb13b INFO [1662240181.071000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240181.765000] 60304 /new_burn_block INFO [1662240181.903000] 60304 /new_block: #txs=1 txids=0x3048e42f094baec86fc6a793cc831116828e9204a9fbe5dba580e677b1dab5b1 INFO [1662240182.4000] 60303 /new_mempool_tx INFO [1662240191.077000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240200.178000] 60303 /new_block: #txs=2 txids=0x64a4b09de6dc0638c89d74f47179b652cf22f27924e6fd621f7434971d868242,0x5fbe4a9f4a0389355c0ee83f583ccd41c956910aa9ee12fc93608cbab4c4b3b2 INFO [1662240201.077000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240201.896000] 60304 /new_burn_block INFO [1662240202.017000] 60304 /new_block: #txs=1 txids=0xacd63d3692f92293afcc5836cf35c2d28b5013fa0beabc3610b7c93b20dfeb47 INFO [1662240202.393000] 60303 /new_mempool_tx INFO [1662240211.082000] this_num_blocks_seen 0 last_num_blocks_seen 0 INFO [1662240220.286000] 60303 /new_block: #txs=2 txids=0x9fb55e9eda27d12c83dbc5e8a25c3bc2a14d895e3bedb2e5a454db1234126f1b,0xd2e82ed180b9454b80b1020ee04117565479a44e4054fadf8493b96d9570ee9c
added 8 commits
September 4, 2022 02:04
one caveat is i think we are still dropping transactionsf rom the mempool which isn't right
…re if we need this tho
Codecov Report
@@ Coverage Diff @@
## master #152 +/- ##
===========================================
+ Coverage 78.11% 90.47% +12.36%
===========================================
Files 260 6 -254
Lines 173164 294 -172870
===========================================
- Hits 135260 266 -134994
+ Misses 37904 28 -37876 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Contributor
Author
|
Closing in favor of #157. |
Contributor
|
Can this PR be closed @gregorycoppola? |
Contributor
|
I think this one can be closed. #203 brought in the changes from upstream that solve the same problem. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR shows how to cache the mempool "in memory" in order to attain vast speed-ups when the mempool is under stress.
The current mempool walk implied by repeated calls to
get_next_tx_to_considerhas, it seems, complexityN^3in the size of the mempool. (We need to do on order ofNcalls toget_next_tx_to_consider, each one takingN^2time to do the join.Also, deserializing a mempool transactions seems to take around 1ms. Thus, to deserialize a 100k tx mempool, it takes 100s. Thus, we move the transactions to memory to avoid deserializing.
Those with access see: https://www.notion.so/hiropbc/Productionization-of-In-Memory-Mempool-c0599678903341d1bd70a728b0e1ab85
Changes