Error in .util_download

[mavino]

library(BiocFileCache)
path <- tempfile()
bfc <- BiocFileCache(path, ask = FALSE)
remoteresource <- "https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData"
proxy <- "http://prox1:3128/"
bfcadd(bfc, rname = "uniquename", fpath = remoteresource, proxy = proxy)
Error in .util_download(x, rid[i], proxy, progress, config, "bfcadd()",  : 
  bfcadd() failed; see warnings()
In addition: Warning messages:
1: download failed
  web resource path: ‘https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData’
  local file path: ‘/tmp/RtmpUyiOdh/fileff3f2260f4cc2/ff3f27e6e41cc_stsMap_0.0.1.RData’
  reason: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_disk()`:
! Couldn't resolve proxy name [bioconductorhubs.blob.core.windows.net]:
Could not resolve proxy: prox1 
2: bfcadd() failed; resource removed
  rid: BFC1
  fpath: ‘https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData’
  reason: download failed 

packageDescription("BiocFileCache")
Package: BiocFileCache
Title: Manage Files Across Sessions
Version: 2.99.5
Authors@R: c(person("Lori", "Shepherd", email =
       "lori.shepherd@roswellpark.org", role = c("aut", "cre")),
       person("Martin", "Morgan", email =
       "martin.morgan@roswellpark.org", role = "aut"))
Description: This package creates a persistent on-disk cache of
       files that the user can add, update, and retrieve. It is
       useful for managing resources (such as custom Txdb
       objects) that are costly or difficult to create, web
       resources, and data files used across sessions.
Depends: R (>= 3.4.0), dbplyr (>= 1.0.0)
Imports: methods, stats, utils, dplyr, RSQLite, DBI, filelock,
       curl, httr2
BugReports: https://github.com/Bioconductor/BiocFileCache/issues
DevelopmentURL: https://github.com/Bioconductor/BiocFileCache
License: Artistic-2.0
Encoding: UTF-8
RoxygenNote: 7.3.2
biocViews: DataImport
VignetteBuilder: knitr
Suggests: testthat, knitr, BiocStyle, rmarkdown, rtracklayer
git_url: https://git.bioconductor.org/packages/BiocFileCache
git_branch: devel
git_last_commit: e88fb8e
git_last_commit_date: 2025-05-15
Repository: Bioconductor 3.22
Date/Publication: 2025-05-18
NeedsCompilation: no
Packaged: 2025-05-18 20:18:32 UTC; biocbuild
Author: Lori Shepherd [aut, cre], Martin Morgan [aut]
Maintainer: Lori Shepherd <lori.shepherd@roswellpark.org>
Built: R 4.5.0; ; 2025-07-15 14:00:26 UTC; unix

[lshep]

can you please share the full sessionInfo()

[mavino]

sessionInfo()
R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 20.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Toronto
tzcode source: system (glibc)

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] rtracklayer_1.69.1    GenomicRanges_1.61.1  Seqinfo_0.99.1       
 [4] IRanges_2.43.0        S4Vectors_0.47.0      HiContactsData_1.11.0
 [7] ExperimentHub_2.99.5  AnnotationHub_3.99.6  BiocFileCache_2.99.5 
[10] dbplyr_2.5.0          BiocGenerics_0.55.0   generics_0.1.4       
[13] HiCExperiment_1.9.0  

loaded via a namespace (and not attached):
 [1] tidyselect_1.2.1            dplyr_1.1.4                
 [3] blob_1.2.4                  filelock_1.0.3             
 [5] Biostrings_2.77.2           bitops_1.0-9               
 [7] fastmap_1.2.0               RCurl_1.98-1.17            
 [9] GenomicAlignments_1.45.1    XML_3.99-0.18              
[11] lifecycle_1.0.4             KEGGREST_1.49.1            
[13] RSQLite_2.4.1               magrittr_2.0.3             
[15] compiler_4.5.0              rlang_1.1.6                
[17] tools_4.5.0                 yaml_2.3.10                
[19] knitr_1.50                  S4Arrays_1.9.1             
[21] bit_4.6.0                   curl_6.4.0                 
[23] DelayedArray_0.35.2         abind_1.4-8                
[25] BiocParallel_1.43.4         withr_3.0.2                
[27] purrr_1.1.0                 grid_4.5.0                 
[29] Rhdf5lib_1.31.0             SummarizedExperiment_1.39.1
[31] cli_3.6.5                   crayon_1.5.3               
[33] rstudioapi_0.17.1           httr_1.4.7                 
[35] tzdb_0.5.0                  rjson_0.2.23               
[37] DBI_1.2.3                   cachem_1.1.0               
[39] rhdf5_2.53.1                parallel_4.5.0             
[41] AnnotationDbi_1.71.0        BiocManager_1.30.26        
[43] XVector_0.49.0              restfulr_0.0.16            
[45] matrixStats_1.5.0           vctrs_0.6.5                
[47] Matrix_1.7-3                jsonlite_2.0.0             
[49] bit64_4.6.0-1               strawr_0.0.92              
[51] glue_1.8.0                  codetools_0.2-20           
[53] BiocVersion_3.22.0          GenomeInfoDb_1.45.7        
[55] BiocIO_1.19.0               UCSC.utils_1.5.0           
[57] tibble_3.3.0                pillar_1.11.0              
[59] rappdirs_0.3.3              rhdf5filters_1.21.0        
[61] R6_2.6.1                    httr2_1.2.0                
[63] vroom_1.6.5                 evaluate_1.0.4             
[65] lattice_0.22-7              Biobase_2.69.0             
[67] png_0.1-8                   Rsamtools_2.25.1           
[69] memoise_2.0.1               Rcpp_1.1.0                 
[71] InteractionSet_1.37.0       SparseArray_1.9.0          
[73] xfun_0.52                   MatrixGenerics_1.21.0      
[75] pkgconfig_2.0.3         

[mavino]

my original problem came from doing this

library(HiCExperiment)
library(HiContactsData)
coolf <- HiContactsData('yeast_wt', 'mcool')
Error in .util_download(x, rid[i], proxy, progress, config, "bfcadd()",  : 
  bfcadd() failed; see warnings()
In addition: Warning messages:
1: download failed
  web resource path: ‘https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3’
  local file path: ‘/home/mariano/.cache/R/ExperimentHub/ff3f27250a95_experimenthub.sqlite3’
  reason: `method` must be a single string, not `NULL`. 
2: bfcadd() failed; resource removed
  rid: BFC3
  fpath: ‘https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3’
  reason: download failed 

[lshep]

We switched from using httr to httr2 and have been trying to accurate test the proxy setting. Can you try setting the https_proxy environment variable instead to see if that help? See #54

If not can you also provide the traceback() after the error occurs

[mavino]

I am not sure if I am doing this right...not an expert
following the beginning of #54 but not running the command remotes::install_github("#57")

library(BiocFileCache)
path <- tempfile()
bfc <- BiocFileCache(path, ask = FALSE)
remoteresource <- "https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData"

## Get HTTPS proxy settings from established environment variable
proxy <- Sys.getenv("https_proxy")

## Alternatively, set it manually
proxy <- Sys.getenv("https://proxyserver:8080")

stopifnot(nzchar(proxy))
Error: nzchar(proxy) is not TRUE

library(BiocFileCache)
path <- tempfile()
bfc <- BiocFileCache(path, ask = T)
remoteresource <- "https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData"

## Get HTTPS proxy settings from established environment variable
proxy <- Sys.getenv("https_proxy")

## Alternatively, set it manually
proxy <- Sys.getenv("https://proxyserver:8080")

stopifnot(nzchar(proxy))
bfcadd(bfc, rname = "uniquename", fpath = remoteresource, proxy = proxy)
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): remoteresource <- "https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData"
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): 
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): ## Get HTTPS proxy settings from established environment variable
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): proxy <- Sys.getenv("https_proxy")
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): 
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): ## Alternatively, set it manually
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): proxy <- Sys.getenv("https://proxyserver:8080")
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): 
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): stopifnot(nzchar(proxy))
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): bfcadd(bfc, rname = "uniquename", fpath = remoteresource, proxy = proxy)
/tmp/RtmpUyiOdh/fileff3f2f66579f
  does not exist, create directory? (yes/no): yes

when ask = F, I get the error I am showing you above (nzchar(proxy) is not TRUE)
so when I set up ask=T it asks me to create a directory, I say yes and that's not error.
Now I am not really sure if I am doing this test right.
When I re-run the first command here on top with traceback

library(BiocFileCache)
path <- tempfile()
bfc <- BiocFileCache(path, ask = FALSE)
remoteresource <- "https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData"
proxy <- "http://prox1:3128/"
bfcadd(bfc, rname = "uniquename", fpath = remoteresource, proxy = proxy)
Error in .util_download(x, rid[i], proxy, progress, config, "bfcadd()",  : 
  bfcadd() failed; see warnings()
In addition: Warning messages:
1: download failed
  web resource path: ‘https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData’
  local file path: ‘/tmp/RtmpUyiOdh/fileff3f24eb192e2/ff3f24c14d510_stsMap_0.0.1.RData’
  reason: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_disk()`:
! Couldn't resolve proxy name [bioconductorhubs.blob.core.windows.net]:
Could not resolve proxy: prox1 
2: bfcadd() failed; resource removed
  rid: BFC1
  fpath: ‘https://bioconductorhubs.blob.core.windows.net/annotationhub/goldenpath/hg19/database/stsMap_0.0.1.RData’
  reason: download failed
traceback()
4: stop(call, " failed; see warnings()")
3: .util_download(x, rid[i], proxy, progress, config, "bfcadd()", 
       ...)
2: bfcadd(bfc, rname = "uniquename", fpath = remoteresource, proxy = proxy)
1: bfcadd(bfc, rname = "uniquename", fpath = remoteresource, proxy = proxy)

[lshep]

How did you determine the remoteresource url? I think it might be an outdated path?

Note sure if that will entirely fix the proxy error but it is also worth noting.

[mavino]

I am sorry I do not understand your question,
I copied your remoteresource url from your #54.
Yesterday I had bioconductor 3.21 and everything was working...the I updated to 3.22, re-updating all the packages and BiocFileCache is not working anymore.
I tried to re-download earlier version 2.16.0 which came with bioconductor 3.21 and still got the problem.
No I returned back to 2.99.5 and I am stuck...

[lshep]

3.21 still uses httr instead of htt2.
3.22 reflects the updates to httr2.
httr2 handles proxy and global variables very differently unfortunately so it is considered a major update. We were working with members of the community to test being behind a proxy.
We appreciate your patience as we help debug.

So to understand more fully too -- You want to ideally run the following without ERROR:

library(HiCExperiment)
library(HiContactsData)
coolf <- HiContactsData('yeast_wt', 'mcool')

Are you actually behind a proxy where you are running this command to have to set it? I assumed you were based on the code provided but if you just copied and pasted the previous issues code that might not be the case (since the proxy provided is the same used in the other issue) -- if you are behind a proxy you need to provide your proxy information

Can you repeat the above with a traceback and sessionInfo?

[mavino]

coolf <- HiContactsData('yeast_wt', 'mcool')
Error in .util_download(x, rid[i], proxy, progress, config, "bfcadd()",  : 
  bfcadd() failed; see warnings()
In addition: Warning messages:
1: download failed
  web resource path: ‘https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3’
  local file path: ‘/home/mariano/.cache/R/ExperimentHub/109c714239bf92_experimenthub.sqlite3’
  reason: `method` must be a single string, not `NULL`. 
2: bfcadd() failed; resource removed
  rid: BFC6
  fpath: ‘https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3’
  reason: download failed 
> sessionInfo()
R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 20.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Toronto
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] HiContactsData_1.11.0 ExperimentHub_2.99.5  AnnotationHub_3.99.6 
[4] BiocFileCache_2.99.5  dbplyr_2.5.0          BiocGenerics_0.55.0  
[7] generics_0.1.4        HiCExperiment_1.9.0  

loaded via a namespace (and not attached):
 [1] KEGGREST_1.49.1             SummarizedExperiment_1.39.1
 [3] httr2_1.2.0                 rhdf5_2.53.1               
 [5] Biobase_2.69.0              lattice_0.22-7             
 [7] tzdb_0.5.0                  rhdf5filters_1.21.0        
 [9] vctrs_0.6.5                 tools_4.5.0                
[11] stats4_4.5.0                curl_6.4.0                 
[13] parallel_4.5.0              AnnotationDbi_1.71.0       
[15] tibble_3.3.0                RSQLite_2.4.1              
[17] blob_1.2.4                  pkgconfig_2.0.3            
[19] Matrix_1.7-3                S4Vectors_0.47.0           
[21] lifecycle_1.0.4             compiler_4.5.0             
[23] Biostrings_2.77.2           Seqinfo_0.99.1             
[25] codetools_0.2-20            InteractionSet_1.37.0      
[27] GenomeInfoDb_1.45.7         yaml_2.3.10                
[29] pillar_1.11.0               crayon_1.5.3               
[31] BiocParallel_1.43.4         DelayedArray_0.35.2        
[33] cachem_1.1.0                abind_1.4-8                
[35] tidyselect_1.2.1            purrr_1.1.0                
[37] dplyr_1.1.4                 BiocVersion_3.22.0         
[39] fastmap_1.2.0               grid_4.5.0                 
[41] cli_3.6.5                   SparseArray_1.9.0          
[43] magrittr_2.0.3              S4Arrays_1.9.1             
[45] withr_3.0.2                 filelock_1.0.3             
[47] UCSC.utils_1.5.0            rappdirs_0.3.3             
[49] bit64_4.6.0-1               XVector_0.49.0             
[51] httr_1.4.7                  matrixStats_1.5.0          
[53] bit_4.6.0                   png_0.1-8                  
[55] memoise_2.0.1               GenomicRanges_1.61.1       
[57] IRanges_2.43.0              BiocIO_1.19.0              
[59] rlang_1.1.6                 Rcpp_1.1.0                 
[61] glue_1.8.0                  DBI_1.2.3                  
[63] BiocManager_1.30.26         rstudioapi_0.17.1          
[65] vroom_1.6.5                 jsonlite_2.0.0             
[67] strawr_0.0.92               R6_2.6.1                   
[69] Rhdf5lib_1.31.0             MatrixGenerics_1.21.0      
> traceback()
9: stop(call, " failed; see warnings()")
8: .util_download(x, rid[i], proxy, progress, config, "bfcadd()", 
       ...)
7: bfcadd(bfc, rname = paste0(tolower(.class), ".sqlite3"), fpath = remote_db, 
       proxy = proxy)
6: bfcadd(bfc, rname = paste0(tolower(.class), ".sqlite3"), fpath = remote_db, 
       proxy = proxy)
5: .updateHubDB(hub_bfc, .class, url, proxy, localHub)
4: .create_cache(.class, url, cache, proxy, localHub, ask)
3: .Hub("ExperimentHub", hub, cache, proxy, localHub, ask, ...)
2: ExperimentHub::ExperimentHub()
1: HiContactsData("yeast_wt", "mcool")

I am not behind a proxy
thank you

[lshep]

Thank you and thank you for baring with the debugging process.
Do you get the same error calling ExperimentHub directly

library(ExperimentHub)
eh = ExperimentHub()

[mavino]

> library(ExperimentHub)
> eh = ExperimentHub()
Error in .util_download(x, rid[i], proxy, progress, config, "bfcadd()",  : 
  bfcadd() failed; see warnings()
In addition: Warning messages:
1: download failed
  web resource path: ‘https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3’
  local file path: ‘/home/mariano/.cache/R/ExperimentHub/109c714da5da6_experimenthub.sqlite3’
  reason: `method` must be a single string, not `NULL`. 
2: bfcadd() failed; resource removed
  rid: BFC7
  fpath: ‘https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3’
  reason: download failed

[lshep]

Thank you. I'll continue to investigate. It looks like there were recent updates to httr2 which may be the root cause

[lshep]

I'm still having trouble reproducing this. Would you mind seeing if the file will download manually so I can try and narrow down the issue.

download.file("https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3", file.path(tempdir(), "test.sqlite3"))

and also the results of

getOption("download.file.method")

[mavino]

> download.file("https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3", file.path(tempdir(), "test.sqlite3"))
trying URL 'https://experimenthub.bioconductor.org/metadata/experimenthub.sqlite3'
Content type 'application/octet-stream' length 9698304 bytes (9.2 MB)
==================================================
downloaded 9.2 MB

> getOption("download.file.method")
[1] "libcurl"