Make order matching parallel in EndBlock

[codchen]

EDIT: going to break this down into smaller PRs

This PR includes the following changes:

• Replace all maps used during order matching to sync.Map
• Add a new numIncomingPaths field on ContractInfo that's automatically calculated upon contract registration. This field indicates the number of contracts that depend on the current contract
• Add a ParallelRunner that takes in a runnable and a list of ContractInfo, and invoke runnable for ContractInfo according to their topological order while maximizing parallelism. Note that the runner implicitly performs topological sort so the list being passed in doesn't have to be topologically sorted already. (The old TopologicalSort utility is now only used for validation upon contract registration)
• Use ParallelRunner for endBlockForContract in dex EndBlock

[philipsu522]

nit - could probably move this to a helper function where we pass in GetBlockOrders and AddOrder functions as parameters and reuse it for all types below

[codchen]

yeah good call. I'll make an update

[codchen]

well since golang doesn't support passing type as parameters or generics, this is probably as far as it goes: https://github.com/sei-protocol/sei-chain/pull/159/files#diff-f8d8121f7fdd1b62bd88a2ffb5afeeb0b601ed29a813686e9d327f538720a514R102

[philipsu522]

So i understand correctly - we have a topo sorted graph and here is the logic to grab all frontier nodes (that is - contracts with no dependencies) and execute them together. And the next set of frontier nodes is determined by for len(validContractAddresses) > 0 { in x/dex/module.go

[codchen]

The runner will figure out what the next nodes to process within a single Run() call. I updated comments for this file and will just paste it here:

We define "frontier contract" as a contract which:
1. Has not finished running yet, and
2. either:
   a. has no other contracts depending on it, or
   b. for which all contracts that depend on it have already finished.
Consequently, the set of frontier contracts will mutate throughout the
`Run` method, until all contracts finish their runs.
The key principle here is that at any moment, we can have all frontier
contracts running concurrently, since there must be no ancestral
relationships among them due to the definition above.
The simplest implementation would be:
`
while there is any contract left:
    run all frontier contracts concurrently
    wait for all runs to finish
    update the frontier set
`
We can illustrate why this implementation is not optimal with the following
example:
    Suppose we have four contracts, where A depends on B, and C depends on
    D. The run for A, B, C, D takes 5s, 5s, 8s, 2s, respectively.
    With the implementation above, the first iteration would take 8s since
    it runs A and C, and the second iteration would take 5s since it runs
    B and D. However C doesn't actually need to wait for B to finish, and
    if C runs immediately after A finishes, the whole process would take
    max(5 + 5, 8 + 2) = 10s, which is 3s faster than the implementation
    above.
So we can optimize the implementation to be:
`
while there is any contract left:
    run all frontier contracts concurrently
    wait for any existing run (could be from previous iteration) to finish
    update the frontier set
`
With the example above, the whole process would take 3 iterations:
Iter 1 (A, C run): 5s since it finishes when A finishes
Iter 2 (B run): 3s since it finishes when C finishes
Iter 3 (D run): 2s since it finishes when B, D finish

[philipsu522]

I see, so it's improving among the concurrency in the sense that the simple solution is basically sort of like bfs and we process all nodes at level n before moving to level n+1, while the optimized solution is to remove the restriction of processing all contracts at the same level. And this optimization is done by keeping a list which may contain existing contracts when checking frontier contracts each loop. This is awesome, tyty

[codchen]

yeah exactly

[philipsu522]

nit - can we make this validContractsInfo so we know its multiple contracts rather than one

[codchen]

updated

[LCyson]

do we still need this PR?

[codchen]

@LCyson yes this is to actually add the core parallelization logic, using the data structure utilities merged in other PRs