chore: fix pip-compile-multi by pinning a few deps

[mistercrunch]

I noticed dependabot wasn't really doing anything on the python side and also encountered issues while trying to bump a dependency before.

So I ran pip-compile-multi, waited for minutes for it to do its thing, and got this error message:

$ pip-compile-multi
Locking requirements/base.in to requirements/base.txt. References: []
Locking requirements/integration.in to requirements/integration.txt. References: []
Locking requirements/development.in to requirements/development.txt. References: ['requirements/base.in']
Package urllib3[socks] was resolved to different versions in different environments: 1.26.18 and 2.2.0
Traceback (most recent call last):
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/cli_v1.py", line 26, in cli
    recompile()
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/actions.py", line 31, in recompile
    compile_topologically(env_confs, deduplicator)
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/actions.py", line 38, in compile_topologically
    if env.maybe_create_lockfile():
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/environment.py", line 51, in maybe_create_lockfile
    self.create_lockfile()
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/environment.py", line 72, in create_lockfile
    self.fix_lockfile()
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/environment.py", line 134, in fix_lockfile
    sections = [
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/environment.py", line 135, in <listcomp>
    self.fix_pin(section)
  File "/Users/max/.pyenv/versions/3.9.16/envs/superset/lib/python3.9/site-packages/pipcompilemulti/environment.py", line 211, in fix_pin
    raise RuntimeError(
RuntimeError: Please add constraints for the package version listed above

From there I guessed that I could pin those indirect dependencies to a number that satistified all of the things referenced in the "# via {some_lib}"

After fixing one I hit another bump and ended up with the two lines in base.in that unlocked the run and ultimately bumping all those libs here.

🤞 pray for unit tests / CI to go through!

[john-bodley]

@mistercrunch this feels somewhat icky and will likely corner us in the future. Have you tried simply running pip-compile-multi --upgrade to give it a fighting chance of finding an eligible set of frozen dependencies.

[mistercrunch]

oh didn't know that was an option, let me re-try

[mistercrunch]

form pip-compile-multi --help

  --upgrade / --no-upgrade        Upgrade package version (default true)

Trying to run it I get the same error as above

[mistercrunch]

I think there's no way around it short of changing the logic. Maybe there's a way to target different ranges of our top-level deps to avoid this fix, but I'm unclear on how to proceed for that.

Note that currently pip-compile-multi fails so that means we're stuck without something like this. I'm open to other approaches, but we need to fix this.

I also noticed that dependabot seems to fail for python/pip, not sure if it's because of this issue, or something else.

[john-bodley]

I can take a look later today to see if I can wrangle the issue.

[mistercrunch]

Awesome, please do! I'll put this on hold for now.

[mistercrunch]

Bonus point if you can get dependabot back on track :)

[john-bodley]

@mistercrunch I don't have the ability to push to preset-io. Here's the contents of my proposed commit which is a slight variation of your change—though with looser constraints after I determined the differences in the dependencies (listed as via in the *.txt files):

Author: John Bodley <john.bodley@gmail.com>
Date:   Thu Feb 1 12:40:35 2024 +1300

    Reconstrain dependencies

diff --git a/requirements/base.in b/requirements/base.in
index 2a3c58f84b..7a01a030dd 100644
--- a/requirements/base.in
+++ b/requirements/base.in
@@ -18,7 +18,7 @@
 #
 -e file:.
 
-# pinning these two dependencies that pip-compile-multi complained about
-# please oh please python overlords can you add support for diamond deps!
-urllib3[socks]==1.26.18
-referencing==0.31.0
+# The following constraints are currently required as these packages resolved to
+# different versions in different environments due to differing sets of dependencies.
+referencing>=0.28.0,<0.32.0  # jsonschema-path 0.3.2
+urllib3>=1.25.4,<1.27  # botocore 1.34.32
diff --git a/requirements/base.txt b/requirements/base.txt
index 4f876bd537..92ebd7190e 100644
--- a/requirements/base.txt
+++ b/requirements/base.txt
@@ -1,4 +1,4 @@
-# SHA1:4e4c4b98ddadacf7f8054c4afaa91bbc20c9f8e3
+# SHA1:f2d183fdbf060b653ccae2c5ce1ac2cba47e4255
 #
 # This file is autogenerated by pip-compile-multi
 # To update, run:
@@ -13,7 +13,7 @@ amqp==5.2.0
     # via kombu
 apispec[yaml]==6.4.0
     # via flask-appbuilder
-apsw==3.45.0.0
+apsw==3.45.1.0
     # via shillelagh
 async-timeout==4.0.3
     # via redis
@@ -313,7 +313,7 @@ pyyaml==6.0.1
     #   apispec
 redis==4.6.0
     # via apache-superset
-referencing==0.31.0
+referencing==0.31.1
     # via
     #   -r requirements/base.in
     #   jsonschema
diff --git a/requirements/development.txt b/requirements/development.txt
index ce34c9ae10..12d999ba81 100644
--- a/requirements/development.txt
+++ b/requirements/development.txt
@@ -14,9 +14,9 @@ astroid==3.0.2
     # via pylint
 asttokens==2.4.1
     # via stack-data
-boto3==1.34.31
+boto3==1.34.32
     # via tabulator
-botocore==1.34.31
+botocore==1.34.32
     # via
     #   boto3
     #   s3transfer
@@ -92,7 +92,7 @@ rfc3986==2.0.0
     # via tableschema
 s3transfer==0.10.0
     # via boto3
-sqloxide==0.1.39
+sqloxide==0.1.43
     # via -r requirements/development.in
 stack-data==0.6.3
     # via ipython

[mistercrunch]

@john-bodley please re-open on apache/ and re can reference this PR for details