Running from Docker

[antortjim]

Hi there

My name is Antonio and I am studying Bioinformatics at University of Copenhagen.

I was directed to this tool by @Maux82 in CompOmics/moFF#26. I am trying to use ThermoRawFileParser through the Docker image and I am encountering some path issues when mapping the Docker volumes to files in my local filesystem.

I built the image using the command in your README.md

docker build --no-cache -t thermorawparser .
Then I tried running
docker run -v $DATA_INPUT:/data_input -i -t --user biodocker thermorawparser mono /home/biodocker/bin/bin/Debug/ThermoRawFileParser.exe -i=/data_input/$RAW_FILE -o=/data_input/output -m -c=$PXD
where

• DATA_INPUT is an absolute path to the directory where my RAW_FILE is
• RAW_FILE is the name of my raw file (no path prepended, i.e myfile.RAW)
• PXD is a custom string.

I get the following error (please click on the image to get the full picture):

The file I am trying to convert is available here:
https://www.ebi.ac.uk/pride/data/archive/2014/09/PXD000279/20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW
and it belongs to the dataset used in the MaxLFQ paper http://www.mcponline.org/content/13/9/2513.long

I am running this on an Ubuntu 16.04.
Could it be due to the fact that the path /data_input does not exist inside the container filesystem?

Thanks beforehand for your help and for developing such a useful tool!
Best
Antonio

[nielshulstaert]

Hi Antonio,

Thanks for trying out the library, it's still in development so any suggestions/improvements are more than welcome! I'm adding mzML export functionality for the moment.

I've tested the RAW file and it works on my computer. If the output directory exists in $DATA_INPUT, it should be mapped inside the container. I ran the following command (as root user)
sudo docker run -v /home/niels/Desktop/raw:/data_input -i -t --user biodocker thermorawparser mono /home/biodocker/bin/bin/Debug/ThermoRawFileParser.exe -i=/data_input/20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW -o=/data_input/test/ -m -c=PXD00001
I've added the docker file for biodocker usage. You can also use mono
mono ThermoRawFileParser.exe -i=/home/niels/Desktop/raw/20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW -o=/home/niels/Desktop/raw/test/

Best regards,

Niels

[antortjim]

Hi Niels,

Thanks for your fast answer. I am very happy to see someone addressing the need to get the raw->mgf/mzml in Linux working!

I tried creating a directory output inside $DATA_INPUT and running Docker with sudo, but still no luck getting it to work. I think it's due to some permission issues between Docker and my host. I can see that if I pass the -m flag:

2018-05-23 22:46:36,915 ERROR ThermoRawFileParser.MainClass - System.UnauthorizedAccessss to the path "/data_input/output/20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02_metadata" is denied.

while if I don't pass -m nor -c

2018-05-23 22:47:25,608 ERROR ThermoRawFileParser.MainClass - System.UnauthorizedAccessException: Access to the path "/data_input/output/20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.mgf" is denied.
i.e the error is the same just trying to create different files. Could it be that this permission issue is not present when running in your computer?

This is the output of ls when run on $DATA_INPUT
>ls -1 $DATA_INPUT
20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW
convert.sh
output

Thanks for your time and help

Best
Antonio

[chrishuges]

Hi Antonio,

Not to hijack here, but you could also try using Singularity if Docker is giving the issue. We use this setup on our machines here successfully (CentOS 7 system):

Navigate to the containing directory and mount the image (docker image here is fine):

singularity shell -B /projects/ptx_analysis/chughes/projects-current/data-pipeline-testing/mgf:/data_input thermorawparser-2018-04-30-a95c72004473.img

Process the file:

mono /src/bin/Debug/ThermoRawFileParser.exe -i=/data_input/mgf/ch_16Feb2018_SP3low-SP3high-SP3Mag-TMT10_hph_6.raw -o=/data_input/mgf/ 

As an aside, we have incorporated RawFileReader into a development version the RawQuant Python tool to allow for conversion of raw files to mgf on linux/mac (we have no plans to do mzML, so RawFileParser will be superior if that is what you desire). It only requires mono and python (no docker), so it could give you an alternate avenue for testing to see if Docker is the issue.

https://github.com/kevinkovalchik/RawQuant/tree/RawFileReader_dev

wget https://github.com/kevinkovalchik/RawQuant/archive/RawFileReader_dev.zip
unzip RawFileReader_dev.zip
cd RawQuant-RawFileReader_dev
pip install .
python -m RawQuant parse -f path/to/input/file.raw -o 0 -mgf

Cheers,
Chris

[antortjim]

Hi @chrishuges
Thanks for your answer! I am sorry I got back to you this late.

I still haven't been able to get it to work with Docker. It would be great if @nielshulstaert could share a more comprehensive MWE. Could these weird errors be due to my filesystem being NFS mounted?

Moreover, I just got to install the singularity shell, but I don't know how to generate the .img file you use in your command. Could you please share the command needed to create it from the thermorawparser Docker image?

The link to the Python module is broken. I tried this instead
https://github.com/kevinkovalchik/RawQuant/tree/qc_dev (I am guessing you just changed your branch name 😛 ). But whether I install from the contents or this folder, or running
sudo pip install RawQuant I get an error. Maybe I should open an issue in your repo?

Thanks for your input and your help!

Best

Antonio

[antortjim]

I just tried setting DATA_INPUT to a folder under /tmp, so it's not under the NFS system, and now I get the following error:

2018-05-30 13:55:06,846 INFO ThermoRawFileParser.CentroidedMgfExtractor - Started parsing /data_input/20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW
2018-05-30 13:55:07,369 ERROR ThermoRawFileParser.CentroidedMgfExtractor - Unable to access the RAW file using the RawFileReader class!

[chrishuges]

Hi Antonio,

I meant to use singularity as an alternative to docker, but based on your latest error it seems that this is not your issue. Sorry I couldn't be of more help here.

Yes for the error with our tool please open an issue on our repository with details of the problem. The link was broken because we just pushed that branch to master yesterday actually!

Chris

[nielshulstaert]

Hi Antonio,

My apologies for the late reply and @chrishuges thanks for the support. I'm pretty new to docker, my initial version of the docker file was (without biodocker user etc)

FROM mono:latest

RUN apt-get update
RUN apt-get install -y git
WORKDIR /src
RUN git clone -b master --single-branch https://github.com/compomics/ThermoRawFileParser /src
RUN msbuild

and to run it

sudo docker run -v /home/niels/Desktop/raw:/data_input -i -t thermorawparser mono /src/bin/Debug/ThermoRawFileParser.exe -i=/data_input/testFile.raw -o=/data_input/test/ -m -c=PXD0001

Does this work for you?

Best regards,

Niels

[antortjim]

Hi Niels

No worries! I tried building the image with the Dockerfile you wrote in the comment above and running the command. I got it to work!!
But it works only if the directory where the RAW file "lives" is not nfs (network filesystem) mounted. My home directory is nfs mounted, so I just placed the RAW in the /tmp folder.

Also, the output directory needs to exist, i.e there needs to be a test directory in /home/niels/Desktop/raw (as you said in a previous comment)

I get as output 2 files in test, the mgf file and the metadata file. It took ~15 seconds to run (pretty fast) 👍

The reason why I got the last error was that I was trying to read from a soft link I created using ln -s.
In other words, the program works only if passed the path to the true file, and it lives in the local filesystem.

Thanks for the support and for making this great tool!!

Thanks Chris also for the suggestions.

Best

Antonio