Unable to create Arrow Files (including from tutorial data)

[ew367]

I have updated Ar:chR using:
devtools::install_github("GreenleafLab/ArchR", ref="master", repos = BiocManager::repositories())

Attach your log file
I have attached the log file for when I try to create arrow files using the tutorial data
ArchR-createArrows-88a793f96bb-Date-2022-05-18_Time-13-34-37.log

Describe the bug
When I try and create the arrowfiles using the tutorial data (or my own) in R.4. I get various error messages. I have tried calling library(parallel) at the beginning of the script, and various parameters for addArchRThreads(). Strangely, if I try running the createArrowFiles() repeatedly (without first removing the exisitng tutorial files), I am occasionally able to successfully make 2/3 of the arrowFiles.

I am seemingly able to create arrowFiles using R3.6, but am then unable to continue with any analysis as I am unable to install the Seurat package for this version of R.

To Reproduce

library(ArchR)
library(stringr)
library(parallel)

set.seed(1)
addArchRThreads(threads = 14) 
addArchRGenome("hg38")

inputFiles <- getTutorialData("Hematopoiesis")

ArrowFiles <- createArrowFiles(
  inputFiles = inputFiles,
  sampleNames = names(inputFiles),
  filterTSS = 4, #Dont set this too high because you can always increase later
  filterFrags = 1000, 
  addTileMat = TRUE,
  addGeneScoreMat = TRUE
)

Expected behavior
I expected arrowFiles to be created.

Screenshots

Session Info
If you do not have a log file because the function that caused the error does not produce one, please paste the output of "sessionInfo()" here.

Additional context
Add any other context about the problem here.

[rcorces]

Hi @ew367! Thanks for using ArchR! Please make sure that your post belongs in the Issues section. Only bugs and error reports belong in the Issues section. Usage questions and feature requests should be posted in the Discussions section, not in Issues.

Before we help you, you must respond to the following questions unless your original post already contained this information:
1. If you've encountered an error, have you already searched previous Issues to make sure that this hasn't already been solved?
2. Can you recapitulate your error using the tutorial code and dataset? If so, provide a reproducible example.
3. Did you post your log file? If not, add it now.
4. Remove any screenshots that contain text and instead copy and paste the text using markdown's codeblock syntax (three consecutive backticks). You can do this by editing your original post.

[ew367]

Hi, yes I've searched previous issues and am still unable to solve the error, and believe I have specifically addressed all other questions in my original post.

Thanks

[rcorces]

Thanks for using ArchR. Before answering the issue, I just want to point you to Item number 4 above - please avoid screenshotting text - it is not searchable so any users having a similar error would not be able to find this issue.

This looks like a permissions issue to me. You can try using chmod to force permissions on this directory or umask as suggested here - #1183 (comment)

This is almost certainly an environment-specific problem which is challenging to troubleshoot on my end so you will have to do some exploring.

[ew367]

Thanks for getting back to me so quickly. I am able to get 2 out of the 3 arrow files created for the tutorial data by doing the following:

immediately before calling createArrowFiles() (in this specific order)

library(parallel)
library(magick)
addArchRThreads(threads = 1) 

Only running ArchR in one R session! I had been running it in mutiple sessions (on different datasets) while I was troubleshooting which was also causing problems.

However the PBMC file is still failing at the very end of the process:

2022-05-18 15:06:05 : Batch Execution w/ safelapply!, 0 mins elapsed.
2022-05-18 15:06:05 : (scATAC_BMMC_R1 : 1 of 3) Arrow Exists! Marking as completed since force = FALSE!, 0 mins elapsed.
2022-05-18 15:06:05 : (scATAC_CD34_BMMC_R1 : 2 of 3) Arrow Exists! Marking as completed since force = FALSE!, 0.001 mins elapsed.
2022-05-18 15:06:05 : (scATAC_PBMC_R1 : 3 of 3) Reading In Fragments from inputFiles (readMethod = tabix), 0.001 mins elapsed.
2022-05-18 15:06:05 : (scATAC_PBMC_R1 : 3 of 3) Tabix Bed To Temporary File, 0.001 mins elapsed.
2022-05-18 15:06:33 : (scATAC_PBMC_R1 : 3 of 3) Reading TabixFile 42 Percent, 0.467 mins elapsed.
2022-05-18 15:06:56 : (scATAC_PBMC_R1 : 3 of 3) Reading TabixFile 83 Percent, 0.865 mins elapsed.
2022-05-18 15:07:04 : (scATAC_PBMC_R1 : 3 of 3) Successful creation of Temporary File, 0.991 mins elapsed.
2022-05-18 15:07:04 : (scATAC_PBMC_R1 : 3 of 3) Creating ArrowFile From Temporary File, 0.991 mins elapsed.
2022-05-18 15:07:32 : (scATAC_PBMC_R1 : 3 of 3) Successful creation of Arrow File, 1.452 mins elapsed.
2022-05-18 15:08:36 : createArrowFiles has encountered an error, checking if any ArrowFiles completed..
ArchR logging successful to : ArchRLogs/ArchR-createArrows-88a5cbb46b6-Date-2022-05-18_Time-15-06-04.log
Warning messages:
1: In sprintf("%s Reading TabixFile %s Percent", prefix, round(100 *  :
  one argument not used by format '%s Reading TabixFile %s Percent'
2: In sprintf("%s Reading TabixFile %s Percent", prefix, round(100 *  :
  one argument not used by format '%s Reading TabixFile %s Percent'
>

I've tried specificying cleanTmp = FALSE, nChunk = 1 as suggested here: #1275 but it's not solved the issue

[rcorces]

try deleting the tutorial data and re-downloading it

[ew367]

Hi, I've tried deleting and redownloading the tutorial data as you suggest, but I'm still having the same problem. When I look in the QC directory I can see a folder called "scATAC_PBMC_R1" but it only contains the fragment distribution pdf (there is no tss plot or rds file). I have two other 10x scATAC datasets, one of them has successfully created 21/21 arrowFiles/QC metrics etc and I am able to continue to downstream analyses., The other dataset (~21 samples again) is also producing the createArrowFiles has encountered an error, checking if any ArrowFiles completed. error and only creating the fragment distribution plot when it tries to process sample 3/21 (the first two samples process as expected).

[rcorces]

Can you update ArchR to the current development release release_1.0.2 and try again?

devtools::install_github("GreenleafLab/ArchR", ref="release_1.0.2", repos = BiocManager::repositories())
detach("package:ArchR", unload=TRUE)
library(ArchR)

[ew367]

Hi, I've trired again using the current developmet release and am still having the same issue with the PBMC tutorial data not creating an arrow file. (I also have an additional error "WARNING: Error found with Cairo installation. Continuing without rasterization."

[rcorces]

Please always upload your log file and paste the text of the error from your console. Otherwise I cannot provide any help.

[ew367]

Apologies. Here is the log file, and the error message is still "createArrowFiles has encountered an error, checking if any ArrowFiles completed"
ArchR-createArrows-6d5319f1edd9-Date-2022-05-23_Time-10-21-16.log
"

[rcorces]

You must not be following the tutorial code. You are using the hg38 reference genome in this logfile. The tutorial is in hg19.

You logfile says:

2022-05-23 10:40:18 : Detected 2 or less cells pass filter (Non-Zero median TSS = 1.92, median Frags = 3313.5) in file!
       Check inputs such as 'filterFrags' or 'filterTSS' to keep cells! Exiting!

This is almost certainly because you are using the wrong reference genome.

[Jiahe-Sun]

Hello, I got the above error when I created ArrowFiles for BAM files, I tried to change the version, but still can't solve it, I hope to get your help, thank you

[Jiahe-Sun]

Here is the log file, thank you for your help!
Uploading ArchR-createArrows-792d7e2b80ef-Date-2022-05-27_Time-03-24-36.log…

[rcorces]

@sunjiahe-hub - unless you are encountering exactly the same problem. please start a new post. However, I think your issue has already been solved. see #1044 (comment)

I'm closing this as I believe the original issue has been addressed.