Downgrading dql from version 0.6.2 to 0.5.26.

[Jengro777]

Downgrading dql from version 0.6.2 to 0.5.26.
We can't upgrade dql==0.5.26 version as dql==0.6.2 newer versions require PostgreSQL > 10, but we target older versions at the moment.

What type of PR is this?

• Refactor
• Feature
• Bug Fix
• New Query Runner (Data Source)
• New Alert Destination
• Other

Description

How is this tested?

• Unit tests (pytest, jest)
• E2E Tests (Cypress)
• Manually
• N/A

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

[justinclift]

We can't upgrade dql==0.5.26 version as dql==0.6.2 newer versions require PostgreSQL > 10, but we target older versions at the moment.

When you say we target older versions, are you meaning the version in our Dockerfile?

If that's what you're meaning, then we should just update the version in the Dockerfile to something newer. I know for sure that much newer versions of PostgreSQL work (13, 14, 15).

Updating that version number has been on our ToDo list for ages anyway.

Hmmm... I'll make a PR with the updated PostgreSQL version now. 1 minute... 😄

[Jengro777]

Yes, by downgrading dql to version 0.5.26, the Docker image should return to normal. Let's proceed with this solution for now. I also noticed that there is a new pull request for upgrading this package. Let's get Redash back to a functioning state as a restorable node, so that we can revert back to it if needed.

[justinclift]

Ahhh cool, hopefully this dql downgrade works.

I've submitted PR #70 for a newer PostgreSQL in the meantime, but we can hold off on that for a bit to see how this older dql library goes. 😄

[justinclift]

@Avey777 Coming back to this now that I've had some sleep, I'm kind of wondering about that PostgreSQL version requirement. Where did you see that info?

Asking because looking at its PyPi entry, dql is a library for DynamoDB rather than PostgreSQL and there also doesn't seem to be a mention of it in the sql change log:

https://github.com/stevearc/dql/blob/master/CHANGES.rst

[Jengro777]

@Avey777 Coming back to this now that I've had some sleep, I'm kind of wondering about that PostgreSQL version requirement. Where did you see that info?

Asking because looking at its PyPi entry, dql is a library for DynamoDB rather than PostgreSQL and there also doesn't seem to be a mention of it in the sql change log:

https://github.com/stevearc/dql/blob/master/CHANGES.rst

I saw this issue mentioned in other documentation, and it could possibly be caused by other dependencies associated with dql. Rolling back to sql version 0.5.26 has resolved the issue in our current Docker images. However, I cannot be 100% certain that it is solely a sql problem.

[justinclift]

Yeah, it did seem to help getting the um... basic image (without every data source library) working again, so it definitely improved things. I was just thinking to merge that PostgreSQL upgrade, thus looked at it a bit more.

We can hold off on that a bit longer though, until we have the other libraries taken care of too. 😄

[Jengro777]

influxdb 5.2.3 depends on python-dateutil>=2.6.0
pyhive 0.6.1 depends on python-dateutil
vertica-python 0.9.5 depends on python-dateutil>=1.5
td-client 1.0.0 depends on python-dateutil
dql 0.5.26 depends on python-dateutil<2.7.0
memsql 3.2.0 depends on python-dateutil<3.0
atsd-client 3.0.5 depends on python-dateutil
azure-kusto-data 0.0.35 depends on python-dateutil>=2.8.0
pandas 1.1.0 depends on python-dateutil>=2.7.3

This is the prompt I received while installing requirements_all_ds.txt. From the above, it can be seen that updating the version of dql is very important.
@justinclift

[justinclift]

Heh Heh Heh, now I'm getting confused.

Um... if you want to update the dql library version then you're welcome to. If/when you do it, please merge PR #70 at the same time so it upgrades the PostgreSQL version as well. 😄

[Jengro777]

I think we should first address the issue of upgrading the dql version before considering merging PR #70.

[Jengro777]

@justinclift
I have tried upgrading dql from version 0.5.26 to 0.6.2, and the test suite did not throw any errors. However, the Docker images are still not functioning properly.

[justinclift]

Any obvious error in the Docker images?

[Jengro777]

Any obvious error in the Docker images?

I am not very familiar with Docker, but you can try to resolve the issue. We can only move forward once this problem is resolved.

[justinclift]

I think we should first address the issue of upgrading the dql version before considering merging PR #70.

Oops, sorry. I didn't see that and merged the PR already.

Ok, I'll try out the newest preview Docker image that's been generated and lets see what shows up in the docker logs.

[justinclift]

There might actually be something wrong with the Docker download servers at the moment. Download speed to my house (I mostly work from home) is around 0.5MB/second. That's unusually slow.

[justinclift]

Interestingly, while seeding the database with the initial tables (docker-compose run --rm server create_db), this warning message is displayed in my terminal window:

/usr/local/lib/python3.7/site-packages/snowflake/connector/options.py:108: UserWarning: You have an incompatible version of 'pyarrow' installed (10.0.0), please install a version that adheres to: 'pyarrow<10.1.0,>=10.0.1; extra == "pandas"'
  "pyarrow", installed_pyarrow_version, pandas_pyarrow_extra

Doesn't look like a fatal error, but we should add that to the list of things to look at.

---

Continuing on, the server seems to be operating ok to a basic level. eg the initial setup page shows, and once that's configured the normal Redash user home page is happy:

Where are you hitting a problem?

---

Oh, if you have an existing postgres-data directory with data in it, then you'll probably need to move it out of the way before you try starting the Docker containers.

The newer PostgreSQL version would otherwise see the database files from the older version of PostgreSQL, and will refuse to operate. Maybe that's the problem you're having?

[Jengro777]

You should try upgrading dql from version 0.5.26 to 0.6.2 first before attempting anything else.

[justinclift]

Thanks, good point. 😄