WIP: State indexer #488

superzordon · 2023-03-25T22:23:46Z

No description provided.

… to be consumed by another job

lazynina · 2023-04-11T15:07:43Z

.ipynb_checkpoints/deso_blocks-checkpoint.ipynb

@@ -0,0 +1,188 @@
+{


Do we need this file or was this just for testing?

lazynina · 2023-04-11T15:08:53Z

lib/block_producer.go

 		// Create a new view object.
 		utxoView, err := NewUtxoView(desoBlockProducer.chain.db, desoBlockProducer.params,
-			desoBlockProducer.postgres, desoBlockProducer.chain.snapshot)
+			desoBlockProducer.postgres, desoBlockProducer.chain.snapshot, nil)


Is there a rule-of-thumb for when we need to include the event manager or not?

I think we always include it except for weird edge-cases. In this case, getBlockTemplate runs outside of the main event loop, and just generates blocks for miners. I would expect mempool may be the one other place where we don't include the EventManager.

lazynina · 2023-04-11T15:13:25Z

lib/block_view.go

+				TxHash:      txHash,
+				IsConnected: true,
+			})
+		}


Do we have some emit logic at the transaction level? If I just do a basic transfer, do we get all the transaction fields passed through somewhere? Wondering if we can get all the outputs in the new state syncer db, so we can see "oh you sent money to x, y, and z"

lazynina · 2023-04-11T15:17:41Z

lib/block_view_creator_coin.go

 	})

+	if bav.EventManager != nil && emitMempoolTxn {
+		bav.EventManager.mempoolTransactionConnected(&MempoolTransactionEvent{


do we want to emit state changes for balance changes when the creator gets a founder reward?

lazynina · 2023-04-11T15:20:47Z

lib/block_view_flush.go

 		return bav.FlushToDbWithTxn(txn, blockHeight)
 	})
+	if bav.EventManager != nil {
+		bav.EventManager.dbFlushed(&DBFlushedEvent{


Need to look into what this does

lazynina · 2023-04-11T17:48:45Z

lib/state_change_syncer.go

+		// Get the cached mempool transaction from the connected mempool map.
+		if connectedMempoolTx, ok := stateChangeSyncer.ConnectedMempoolTxMap[*event.TxHash]; ok {
+			// Check to see if the index in question has a "core_state" annotation in its definition.
+			if !isCoreStateKey(connectedMempoolTx.KeyBytes) {
+				return
+			}
+
+			stateChangeEntry.Encoder = connectedMempoolTx.Encoder
+			stateChangeEntry.KeyBytes = connectedMempoolTx.KeyBytes
+			stateChangeEntry.UtxoOps = connectedMempoolTx.UtxoOps
+		} else {
+			return
+		}


Suggested change

// Get the cached mempool transaction from the connected mempool map.

if connectedMempoolTx, ok := stateChangeSyncer.ConnectedMempoolTxMap[*event.TxHash]; ok {

// Check to see if the index in question has a "core_state" annotation in its definition.

if !isCoreStateKey(connectedMempoolTx.KeyBytes) {

return

}

stateChangeEntry.Encoder = connectedMempoolTx.Encoder

stateChangeEntry.KeyBytes = connectedMempoolTx.KeyBytes

stateChangeEntry.UtxoOps = connectedMempoolTx.UtxoOps

} else {

return

}

// Get the cached mempool transaction from the connected mempool map.

connectedMempoolTx, ok := stateChangeSyncer.ConnectedMempoolTxMap[*event.TxHash]

if !ok {

return

}

// Check to see if the index in question has a "core_state" annotation in its definition.

if !isCoreStateKey(connectedMempoolTx.KeyBytes) {

return

}

stateChangeEntry.Encoder = connectedMempoolTx.Encoder

stateChangeEntry.KeyBytes = connectedMempoolTx.KeyBytes

stateChangeEntry.UtxoOps = connectedMempoolTx.UtxoOps

early return and avoiding else statements where possible.

lazynina · 2023-04-11T17:52:26Z

lib/state_change_syncer.go

+	if isEncoder, encoder := StateKeyToDeSoEncoder(stateChangeEntry.KeyBytes); isEncoder && encoder != nil {
+		encoderType = encoder.GetEncoderType()
+	} else {
+		// If the keyBytes is not an encoder, then we decode the entry from the key value.


maybe add a comment that this is expected for things such as LikeEntry where the entry is decoded from the key. the comment is a little off at the end - the phrase "key value" is a little confusing.

lazynina · 2023-04-11T17:53:21Z

lib/state_change_syncer.go

+
+	// Encode the state change entry. We encode as a byte array, so the consumer can buffer just the bytes needed
+	// to decode this entry when reading from file.
+	entryBytes := EncodeToBytes(0, stateChangeEntry, false)


hmmm we probably need a real block height in this call to EncodeToBytes, right? This gets passed down into RawEncodeWithoutMetadata

lazynina · 2023-04-11T17:55:06Z

lib/state_change_syncer.go

+	writeBytes := EncodeByteArray(entryBytes)
+
+	decodedStateChangeEntry := &StateChangeEntry{}
+	DecodeFromBytes(decodedStateChangeEntry, bytes.NewReader(entryBytes))


do we want to catch the error from DecodeFromBytes?

lazynina · 2023-04-11T17:56:57Z

lib/state_change_syncer.go

+
+	// Update the state change file size.
+	transactionLen := uint32(len(writeBytes))
+	unflushedBytes.StateChangeBytesSize += transactionLen


maybe add overflow check? We have these SafeUint utils lying around

+1 (assuming we keep this field-- see my other comment)

diamondhands0 · 2023-08-03T16:16:01Z

lib/state_change_syncer.go

+	}
+
+	// Get the relevant deso encoder for this keyBytes.
+	var encoderType EncoderType


More comments here. I want to better understand when something hits the first case vs the second. I think it happens whenever we have a field in the db that's a "key only" field with no Entry as a value.

diamondhands0 · 2023-08-03T16:16:34Z

lib/state_change_syncer.go

+	err = mempoolUtxoView.FlushToDbWithTxn(txn, uint64(server.blockchain.bestChain[len(server.blockchain.bestChain)-1].Height))
+
+	if err != nil || originalCommittedFlushId != stateChangeSyncer.BlockSyncFlushId {


The second part of this is tricky. I think it's checking whether a block came in while we were mempooling. Regardless, add comment.

diamondhands0 · 2023-08-03T16:17:19Z

lib/state_change_syncer.go

+	AncestralRecord DeSoEncoder
+	// The ancestral record represented in bytes.
+	AncestralRecordBytes []byte


Add comment on when ancestral record bytes is used. I assume in hypersync it's not pupulated for example. Also I think it's only for upsert?

Yep, it's only for upsert. We only use ancestral records when syncing mempool entries. When the new block syncs, we need to revert any mempool entries we've stored in the database before applying the committed entries. We do this reversion by reverting to the ancestral record value.

diamondhands0 · 2023-08-03T16:17:41Z

lib/state_change_syncer.go

+	StateChangeBytes     []byte
+	StateChangeBytesSize uint64
+	// This is a list of the indexes of the state change bytes that should be written to the state change index file.
+	StateChangeOperationIndexes []uint64


Give comments on these fields that explains how they work.

diamondhands0 · 2023-08-03T16:18:47Z

lib/state_change_syncer.go

+
+	// During blocksync, we flush all entries by index to the state change file once the sync is complete.
+	// This is used to track whether this procedure has been initiated.
+	BlocksyncCompleteEntriesFlushed bool


Explain the deeper reasons for why we do this optimization, rather than just flushing with each block. We want to give a lot of context for optimizations like these so that we can revisit them intelligently in the future. Otherwise, people are scared to change things like this for fear it will break things in unexpected ways.

diamondhands0 · 2023-08-03T23:57:25Z

cmd/node.go

 	// Setup chain database
 	dbDir := lib.GetBadgerDbPath(node.Config.DataDirectory)
-	opts := lib.PerformanceBadgerOptions(dbDir)
+	opts := badger.DefaultOptions(dbDir)


Revert to PerformanceBadgerOptions per our conversation

diamondhands0 · 2023-08-03T23:58:18Z

lib/snapshot.go

 	// Initialize the ancestral records database
 	snapshotDirectory := filepath.Join(GetBadgerDbPath(mainDbDirectory), "snapshot")
-	snapshotOpts := PerformanceBadgerOptions(snapshotDirectory)
+	snapshotOpts := DefaultBadgerOptions(snapshotDirectory)


Revert to PerformanceBadgerOptions per our conversation

diamondhands0 · 2023-08-04T00:25:06Z

scripts/nodes/n0_test

  --add-ips=localhost:19000 \
  --testnet=true  \
  --api-port=18001 \
  --protocol-port=18000 \


Can we delete the whole nodes folder here? People should only use the ones in backend from now on. These are outdated and misleading.

This makes sense to me 👍

…fees

diamondhands0 · 2023-11-08T01:44:26Z

lib/block_producer.go

-			_, _, _, _, err = utxoViewCopy._connectTransaction(
-				mempoolTx.Tx, mempoolTx.Hash, int64(mempoolTx.TxSizeBytes), uint32(blockRet.Header.Height), true,
-				false /*ignoreUtxos*/)
+			_, _, _, _, err = utxoViewCopy._connectTransaction(mempoolTx.Tx, mempoolTx.Hash, int64(mempoolTx.TxSizeBytes), uint32(blockRet.Header.Height), true, false)


It would be nice if you could do a pass to fix all the line lengths that the refactoring broke. With vim you can highlight a line and hit "gq" and it does it. But not sure what the shortcut is in other things.

diamondhands0 · 2023-11-08T01:49:30Z

lib/blockchain.go

+				if innerErr = bc.blockView.FlushToDbWithTxn(txn, blockHeight); innerErr != nil {
 					return errors.Wrapf(innerErr, "ProcessBlock: Problem writing utxo view to db on simple add to tip")
 				}
+				if bc.eventManager != nil && !bc.eventManager.isMempoolManager {


Do me a favor and just add a comment explaining why we put this event trigger here rather than elsewhere. For example, is there a reason why we need to put it in the badger Update() vs outside of it? Just explain it, or explain that the placement doesn't matter, in case someone needs to move things around in the future.

diamondhands0 · 2023-11-08T02:15:05Z

lib/state_change_syncer.go

+					EncoderBytes:  dbEntry.Value,
+				}
+
+				fmt.Printf("Flushing entry %d to file\n", ii)


Do you want to leave all these prints in here? Fine if they're intentional I'm just flagging it.

superzordon added 13 commits January 16, 2023 14:39

Add additional event manager type to emit state change events to file…

644125c

… to be consumed by another job

Remove unused function

1430b71

Add operation type

e796854

Update index size

ec188b7

Update how struct size is calculated

4d78cfd

Update tests

46750d7

Resolve conflicts

17af313

Remove protobuf, rely on deso encoders for v1

6f73cfc

Update state syncer encoding to use deso encoders

87ccba4

Update encoding format

7384681

Add db flush events, update encoding types

f95299a

Update to state syncer

f503b0c

Fix config flags

a9f1d1e

superzordon requested a review from a team as a code owner March 25, 2023 22:23

superzordon added 15 commits March 25, 2023 18:31

Fix tests

5db28fd

Fix tests

45f81d0

Update logging

6cf05f3

Updates to state syncer

6a8affb

Merge conflicts

97dbcd2

Create state change encoder

608d47a

Update state change syncer

9bfbb24

Updates to state syncer encoder

6f40217

Clean up state change file handling

0f4cef6

Simplify state syncer, use utxobundle, add documentation

61f10fa

Fix tests

a6735ef

More test fixes

d5cd870

Update tests, run go fmt

c012f71

Go fmt

057446b

Add additional entry type support to state syncer

72ef3f1

lazynina reviewed Apr 11, 2023

View reviewed changes

diamondhands0 reviewed Aug 3, 2023

View reviewed changes

diamondhands0 reviewed Aug 4, 2023

View reviewed changes

superzordon added 6 commits August 4, 2023 10:27

Update isMempool signatures

3e7d53a

Track mempool transactions

acfb50e

Fix mempool flush

8000070

Address feedback, fix CI

8650802

Formatting

f5ad020

Update mempool sync routine to reduce number of re-flushes of mempool

2526ddd

diamondhands0 approved these changes Aug 9, 2023

View reviewed changes

superzordon added 8 commits August 16, 2023 19:39

Updates to state change metadata

dda2f5c

Add mempool lock to prevent concurrent map reads, update transaction …

60485b5

…fees

Updates to extra data decoder

ba48700

Updates to get pkid routine

7d90843

Updates to sync procedure

697a240

Update ancestral record decoding

048f7d9

Update state change entry struct and encoders

40019ae

Merge main, resolve conflicts

16fb4aa

diamondhands0 reviewed Nov 8, 2023

View reviewed changes

Address formatting and print feedback

3cc00f9

lazynina approved these changes Nov 8, 2023

View reviewed changes

superzordon merged commit c79a756 into deso-protocol:main Nov 9, 2023

tholonious mentioned this pull request Nov 9, 2023

Merge branch 'main' into feature/proof-of-stake #807

Merged

		err = mempoolUtxoView.FlushToDbWithTxn(txn, uint64(server.blockchain.bestChain[len(server.blockchain.bestChain)-1].Height))

		if err != nil \|\| originalCommittedFlushId != stateChangeSyncer.BlockSyncFlushId {

WIP: State indexer #488

WIP: State indexer #488

Uh oh!

Conversation

superzordon commented Mar 25, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diamondhands0 Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

diamondhands0 Aug 3, 2023 •

edited

Loading