Duneapi package integration

[josojo]

Unfortunately, this integration comes with a bigger refactor.

The package offers the ability to wait until a query calculation is finished. This is really nice and allows us to cut the amount of python script by a factor of 2: Previously, we had one script for executing the queries and a second one for fetching the results. Now, with the waiting feature, we can do it all in one go.

The duneapi package also allowed me to store the fetched dune data in a cleaner structure, as the parsing of the fetched result is better. Due to that I also needed to touch the rust code.

There is still a lot to be refactored to make the repo clean, but this should be okay as a first go.

testplan:

cd dune_api_scripts
python3 -m venv env
source ./env/bin/activate
pip install -r requirements.txt
source .env
export DUNE_DATA_FOLDER=./data/dune_data/
export APP_DATA_REFERRAL_RELATION_FILE=./data/referral_data/app_data_referral_relationship.json
python -m dune_api_scripts.store_query_result_all_distinct_app_data

Now, the referral data needs to be downloaded from ipfs. This takes quite long, if this should be done for all appdata. Hence, I would recommend deleting all entries, but 1, from the newly created distinct_app_data.json file in the data folder.
Once it is done, one can run

cargo run

And wait until app_data_referral_relationship.json is written.

Now, one can run:

python -m dune_api_scripts.store_query_result_for_todays_trading_data
cargo run

and check that trading volume was downloaded and is served by checking:

http://127.0.0.1:8080/api/v1/profile/0x019d6ad53704fc851f1cc9effc12e89de4a1fc2c

[bh2smith]

Looks pretty good overall.

I see that many of the python scripts have a common block of duplicated setup. Not necessary for this PR, but a lot of this could be moved into a general_init.py to avoid having to do Dune.new_from_environment followed by dune.fetch and then setting data_set so many times.

Another thing is that is don't see the need for having unique query IDs everywhere. You can always keep the original queries at those existing IDs, but it seems like everything here could be done from a single query ID since the results are merely being fetched.

[bh2smith]

One thing I noticed that we could probably move towards with these kinds of python projects is writing a single bash script containing all of this stuff

python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
cp .env_sample .env

[bh2smith]

PS - the reason CI is failing is because duneapi requires python@3.10

[bh2smith]

This change removed the dune.initiate_query and only left execute which means we were refreshing a static query without updating the values. Fix is made in #8