SSL Certificate Verification Fails When Accessing GTDB URL and a workaround using mirror

[Stian-2rz]

Hey GToTree community,
I'm encountering a certificate verification error when trying to accessing anything GTDB related (Same as #87 ), but cleanly installing in a new environment didn't solve it, but changing the links did. gtotree was working just fine until today, but now running

gtt-get-accessions-from-GTDB -t "Archaea" --get-taxon-counts

Results in (same as in #87 ):

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>

Full trace

File "/albedo/home/storset/.conda/envs/gtotree/bin/gtt-get-accessions-from-GTDB", line 614, in <module>
    main()
  File "/albedo/home/storset/.conda/envs/gtotree/bin/gtt-get-accessions-from-GTDB", line 94, in main
    gtdb_tab = get_gtdb_tab(args.do_not_check_GTDB_version, args.store_GTDB_metadata_in_current_working_dir)
  File "/albedo/home/storset/.conda/envs/gtotree/bin/gtt-get-accessions-from-GTDB", line 521, in get_gtdb_tab
    check_stored_gtdb_up_to_date(GTDB_dir)
  File "/albedo/home/storset/.conda/envs/gtotree/bin/gtt-get-accessions-from-GTDB", line 490, in check_stored_gtdb_up_to_date
    gtdb_version_info = urllib.request.urlretrieve("https://data.gtdb.ecogenomic.org/releases/latest/VERSION", location + "GTDB-latest-version-info.txt")
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/albedo/home/storset/.conda/envs/gtotree/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>

I tried updating certifi but the error remains the same.

Unlike in issue #87, a clean install of a new version of gtotree in a new environment did not change the behavior.

I was able to work around it by changing the urls in the scripts to the GTDB mirror.
i.e, changing line 465 of gtt-get-accessions-from-GTDB from:

     gtdb_version_info = urllib.request.urlretrieve("https://data.gtdb.ecogenomic.org/releases/latest/VERSION.txt", location + "GTDB-version-info.txt")

to

gtdb_version_info = urllib.request.urlretrieve("https://data.ace.uq.edu.au/public/gtdb/data/releases/latest/VERSION.txt", location + "GTDB-version-info.txt")

seems to fix the issue.

I don't know if its a local, gtotree, or gtdb issue, but based on how chrome yells at me about security when I try to access the regular GTDB url makes me think its a GTDB issue.

As I'm not the first to encounter this, maybe a --use-mirror-links flag or something similar could provide a solution when the GTDB certificate decides to be weird?
Best,
Stian

[AstrobioMike]

Heya, @Stian-2rz!

Thanks so much for the note about this and the quick fix for anyone who might hit it too 👍

I think that's a great idea to add a flag like that – also because even under normal circumstances, one location works better for some folks and the other for other folks depending on where they are. So it wouldn't hurt to have both at the ready

So even if I put in a way to bypass the certificate check, I'll be adding that flag when I can get to this

Thanks again!

[AstrobioMike]

as of v1.8.15, bit-get-accessions-from-GTDB has a flag (--use-ecogenomics) that will pull from there instead of the data.ace link 👍 Thanks, @Stian-2rz!

(and a temporary setting to ignore the certificate check, because for me, at least, the data.ace link is much faster)