Running and developing locally

[aaronS7]

Saw this project a while ago on hackernews and rediscovered it not too long ago.

Do you have any instructions on building the project? Developing locally, etc?

Read through the readme, but can't get the docker image to run after building locally.

Getting this error when trying to build the image locally and running with this command in particular (using the turbine-go-rewrite-1 branch as it seems like that is where things are going):
sudo docker run -v $(pwd)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:9c3d31e dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

 /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:9c3d31e dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json
Traceback (most recent call last):
  File "/app/cmd/sql-flow.py", line 130, in <module>
    cli()
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cmd/sql-flow.py", line 116, in dev_invoke
    invoke(conn, config, fixture)
  File "/app/sqlflow/lifecycle.py", line 29, in invoke
    h = handlers.new_handler_from_conf(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sqlflow/handlers.py", line 197, in new_handler_from_conf
    ).init()
      ^^^^^^
  File "/app/sqlflow/handlers.py", line 52, in init
    f = open(self._batch_file, 'w+')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/sqlflow/resultscache/consumer_batch.json'

but let me know if I should wait til that is completed!

[turbolytics]

Hello @aaronS7! Thank you for revisiting SQLFlow!

We have a series of documentation pages planned for this week. I created an issue specifically to address the dev onboarding:

https://github.com/turbolytics/sql-flow-docs/issues

---

I updated the README and created a new command make setup-dev to initialize the missing directory.

The error you are seeing is due to a missing cache directory. We'll do a better job of documenting this and providing tooling to automatically provision the necessary resources.

Could you please try:

make setup-dev
docker run -v $(pwd)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:latest dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

Let me know if you run into any more issues and I'd be happy to help.

[aaronS7]

I got this so far, but I'll look into it to see if it's on my end, but figure I post here in case you know of a quick thing I'm missing.

Installing tools...
creating resultscache directory... '/tmp/sqlflow/resultscache'
Traceback (most recent call last):
  File "/app/cmd/sql-flow.py", line 130, in <module>
    cli()
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cmd/sql-flow.py", line 116, in dev_invoke
    invoke(conn, config, fixture)
  File "/app/sqlflow/lifecycle.py", line 29, in invoke
    h = handlers.new_handler_from_conf(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sqlflow/handlers.py", line 197, in new_handler_from_conf
    ).init()
      ^^^^^^
  File "/app/sqlflow/handlers.py", line 52, in init
    f = open(self._batch_file, 'w+')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/sqlflow/resultscache/consumer_batch.json'

[turbolytics]

Could you please post the docker run command you’re using just to doublecheck?

[turbolytics]

@aaronS7 could you please try to invoke a different config file to test:

docker run -v $(pwd)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:latest dev invoke /tmp/conf/config/examples/basic.agg.mem.yml /tmp/conf/fixtures/simple.json

This is the in mem pipeline so it shouldn't rely on the consumer_batch.json that is giving errors.

[aaronS7]

@turbolytics The command above does work! Thanks!

[turbolytics]

If you need any help developing a custom pipeline, i'd be happy to assist!