WIP: Performance optimized indexes

[braydonf]

This is a work in progress to optimize indexes for these use cases:

1. Query for balances of addresses regardless of the transaction count. The balance of an address is now calculated with each new block, so the value is calculated only once, and not for every query. Summary information is also included:
   • Balance for an address
   • Total transaction count for an address
   • Total received satoshis to an address (not excluding change)
   • The total change received to the address
2. Query transaction history for an address and many addresses. A key only index is kept sorted by height for all txids for an address. Requesting the latest 10 transactions performs the same regardless of the number of the number of transactions for the address. Pagination is optimized by skipping over stream items to only read those that apply (see tests for read times)
3. Query UTXOS for an address and many addresses. Similarly to transaction history, a key only index is kept sorted by height that can be paginated. One stream is necessary for getting a general overview of all UTXOS (see tests for read times for many addresses).
4. Query for full information for a transaction. Previous output information for inputs is necessary for determining which address balance is spent from an input and what fees there are for a transaction as is needed when viewing transaction details. Previous output information has information for:
   • The associated address
   • The number of satoshis
   • The spending txid, inputIndex and height
5. Query for the latest block history by time. Basic summary information about blocks is indexed for faster query performance. This is especially important for the landing page block explorer view. The following information is indexed and available:
   • Timestamp of the block
   • Total transaction count in the block
   • Block hash
   • The total bytes for the block
   • The coinbase script (for determining mining information)
   • The height of the block

Note: The initial sync times for these indexes remains comparable to current implementation and within the same amount of time it takes for the initial bitcoind sync. This is around 24 hours for livenet and 1 hour for testnet.

Necessary updates to the Insight API: bitpay/insight-api#461

Closes and resolves: bitpay/bitcore-lib#36, bitpay/insight-api#444, #400, #373, #397, #375, bitpay/insight-api#459, bitpay/insight-api#351, bitpay/insight-api#309

For more verbose details with performance comparisons take a look at: https://github.com/braydonf/bitcore-storage-tests

[braydonf]

As summary of results here thus far:

Appending txid values to key/values dramatically helps improve the query time for Node.js for txids for multiple addresses (https://github.com/braydonf/bitcore-storage-tests#multi-address-transaction-history-queries) however read-modify-write performance with LevelDB isn't great. While syncing with livenet it will mostly freeze at around block height 180,000. Writing txids for addresses with key only values is write optimized; however, the performance while streaming or using an iterator carries overhead. Tests with using the highWatermark setting with a leveldown iterator show that this can be faster if done in C++, with an array returned. This also parallels results from LMDB multi-values iteration that had good performance.

The read performance for txids for several address when queried from MongoDB is promising (https://github.com/braydonf/bitcore-storage-tests#multi-address-transaction-history-queries). However writing blocks across multiple collections atomically isn't supported, making a simple write a more complex multi-phase commit with the additional overhead. Even without all necessary indexes (sans unspent info), the sync times for full 1MB blocks was around 0.5 blocks per second. An additional year would estimate at 30 additional hours.

[braydonf]

Will close this for the time being until there are more concrete results, as I'm not convinced this is the best direction.

[braydonf]

Anyone following this issue can continue on this PR: #422