Example project to load minute price data from github CSV files.
Download QuestDB for windows
Unzip the files downloaded
No installation is required.
Note: for simplicity, move the unzipped files to c:\questdb, and create shortcut on your desktop for questdb.exe.
Run the questdb server by launching: questdb.exe (or your shortcut).
Windows will raise an alert,
Click on the "More Info" button,
Then click on "Run anyway".
Keep this terminal window opened so the server can run in the background at "localhost:9000"
(See bottom of this file for print screens of windows alert windows)
Open your browser at
localhost:9000
This will bring up the web console
To store minute price data, we want a structured time series database that will prevent duplicate entries for a datetime+ticker.
In the web console, run the following SQL Script
CREATE TABLE 'minute' (
Ticker SYMBOL capacity 256 CACHE,
Close DOUBLE,
High DOUBLE,
Low DOUBLE,
Open DOUBLE,
Volume LONG,
'Adj Close' DOUBLE,
Datetime TIMESTAMP
) timestamp (Datetime) PARTITION BY DAY WAL
DEDUP UPSERT KEYS(Datetime, Ticker);
This will create a table with Open, High, Low, Close, Volume and 'Adj Close' columns. The Datetime column will be the timestamp required by any timeseries database and the ticker will be a special type of column to manage our duplicates (read the docs on QuestDB for more explanations).
Clone thos project to your local machine.
Open the project in an IDE (such as VS Code), with a terminal window:
python -m venv env
.env\script\activate
pip install -r requirements.txt
python build_db.py
Script build_db.py has two main functions.
main()
Loads a csv file from local data folder and sends it to QuestDB
main2()
raw_url : list of links to online csv files (github for example)
loop : iterates through this list to download, structure data and load to questdb
Update this list with your prefered data files
