diff --git a/content/authors/det/_index.md b/content/authors/det/_index.md new file mode 100644 index 00000000..05ee5733 --- /dev/null +++ b/content/authors/det/_index.md @@ -0,0 +1,26 @@ +--- +# Display name +title: Deb Triant + +# Username (this should match the folder name) +authors: +- det + +# Is this the primary user of the site? +superuser: false + +# Role/position +role: Research Computing Scientist + +# Organizations/Affiliations +organizations: +- name: University of Virginia Research Computing + url: "https://www.rc.virginia.edu" + + +interests: +- Bioinformatics +- HPC +- Research + +--- diff --git a/content/notes/bioinfo-intro/02-intro.md b/content/notes/bioinfo-intro/02-intro.md new file mode 100644 index 00000000..2ac68231 --- /dev/null +++ b/content/notes/bioinfo-intro/02-intro.md @@ -0,0 +1,17 @@ +--- +title: Bioinformatics +date: 2025-08-23-03:19:53Z +type: docs +weight: 150 +menu: + bioinfo-intro: +--- + +The term _Bioinformatics_ first appeared in 1970s and exploded in the 1990s with the Human Genome Project and the rise of high-throughput sequencing technologies. Earlier roots include computational tools for analyzing molecular data developed in the 1960s, with methodological precedents in wartime cryptanalytic work from the 1940s. + +> [Read More](https://www.nature.com/articles/35042090) + +{{< figure src=/notes/bioinfo-intro/img/bioinformatics-ss.png caption="Bioinformatics sits at the intersection of biology, computer science, mathematics/statistics, engineering, and biochemistry." width=70% height=70% >}} + + + diff --git a/content/notes/bioinfo-intro/03-analys-types.md b/content/notes/bioinfo-intro/03-analys-types.md new file mode 100644 index 00000000..5aec4dea --- /dev/null +++ b/content/notes/bioinfo-intro/03-analys-types.md @@ -0,0 +1,59 @@ +--- +title: Types of Bioinformatics Analyses +date: 2025-08-23-03:19:53Z +type: docs +weight: 200 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +The following are common categories of analyses performed in modern genomics and systems biology. + +**1. Proteomics** + +Proteomics is the large-scale study of proteins. + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_6.png caption="Protein structure ribbon diagram" width=30% height=30% >}} + +**2. Metabolomics** + +Metabolomics focuses on the complete set of small molecules within a biological sample. + +**3. RNA-Seq** + +RNA Sequencing is used to quantify RNA molecules and gene expression. + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_7.png caption="RNA-seq protocol similarity heatmap" width=45% height=45% >}} + +**4. Single-cell Analysis** + +Single-cell analysis explores gene expression at the individual cell level. + +**5. Genome Assembly and Annotation** + +Genome assembly and annotation reconstructs complete genomes from short or long sequencing reads and labels genes, regulatory regions, and functional elements. + +**6. Regulatory Genomics** + +Regulatory genomics explores how DNA and other factors control gene expression patterns. + +**7. Variant Calling and Haplotype Analysis** + +Variant calling and haplotype analysis identifies base substitutions (Single Nucleotide Variants), helping identify mutations. + +Example SNV: + +C A GCTTA G + +T GCTTA T + +A GCTTA G + +A A GCTTACG G + +>Blue = reference base (G), red = alternate base (T). + +[Read More: RNA-Seq Methods](https://www.nature.com/articles/s41592-024-02298-3) + + diff --git a/content/notes/bioinfo-intro/04-databases.md b/content/notes/bioinfo-intro/04-databases.md new file mode 100644 index 00000000..950ea9c0 --- /dev/null +++ b/content/notes/bioinfo-intro/04-databases.md @@ -0,0 +1,62 @@ +--- +title: Databases +date: 2025-08-23-03:19:53Z +type: docs +weight: 250 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +**InterPro** + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_8.png width=45% height=45% >}} + + +[https://www.ebi.ac.uk/interpro/entry/pfam](https://www.ebi.ac.uk/interpro/entry/pfam) + +--- + +**National Library of Medicine** + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_9.png width=45% height=45% >}} + + +[https://www.ncbi.nlm.nih.gov](https://www.ncbi.nlm.nih.gov) + +--- + +**Ensembl** + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_11.png width=45% height=45% >}} + + +[https://www.ensembl.org/index.html](https://www.ensembl.org/index.html) + +--- + +**Fang** + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_10.png width=45% height=45% >}} + + +[https://data.faang.org/home](https://data.faang.org/home) + +--- + +**EMBL-EBI** + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_13.png width=65% height=65% >}} + + +[https://www.ebi.ac.uk](https://www.ebi.ac.uk) + +--- + +**RGD** + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_12.png width=75% height=75% >}} + + +[https://rgd.mcw.edu](https://rgd.mcw.edu) + diff --git a/content/notes/bioinfo-intro/05-technologies.md b/content/notes/bioinfo-intro/05-technologies.md new file mode 100644 index 00000000..db6bda9c --- /dev/null +++ b/content/notes/bioinfo-intro/05-technologies.md @@ -0,0 +1,21 @@ +--- +title: Sequencing Technologies +date: 2025-08-23-03:19:53Z +type: docs +weight: 300 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + + +**Illumina**: Illumina tools generate short read sequences (< 1kb). They are widely used for whole-genome and exome sequencing, small RNA/microRNA profiling, and many single-cell applications. + +**PacBio**: PacBio generates long read sequences (~ 25 kb). The PacBio Revio sequencer is available at UVA. + +**Nanopore**: Nanopore generates "ultra-long" sequences (up to 1Mb). + +**HiC**: HiC is a crosslinking technique used to capture interactions within a genome. + + + diff --git a/content/notes/bioinfo-intro/06-pacbio.md b/content/notes/bioinfo-intro/06-pacbio.md new file mode 100644 index 00000000..c4303a03 --- /dev/null +++ b/content/notes/bioinfo-intro/06-pacbio.md @@ -0,0 +1,19 @@ +--- +title: PacBio HiFi Reads +date: 2025-08-23-03:19:53Z +type: docs +weight: 350 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +The below figure compares how the different sequencing technologies map reads to the STRC gene. + +{{< figure src=/notes/bioinfo-intro/img/pacbio.png width=90% height=90% >}} + +This shows how PacBio produces reads that are both long and accurate. + +[Read More](https://www.pacb.com) + + diff --git a/content/notes/bioinfo-intro/07-fileformats.md b/content/notes/bioinfo-intro/07-fileformats.md new file mode 100644 index 00000000..94e6eb25 --- /dev/null +++ b/content/notes/bioinfo-intro/07-fileformats.md @@ -0,0 +1,70 @@ +--- +title: File Formats +date: 2025-08-23-03:19:53Z +type: docs +weight: 400 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_16.png caption="Source: https://xkcd.com/927/" width=80% height=80% >}} + +The format name usually denotes the file suffix. + +**FASTA** files (suffix: `.fasta`, `.fna`, `.fa`) store sequencing data. + +**FASTQ** files (suffix: `.fastq`) include sequencing data and quality scores. + +**SAM/BAM** files (suffix: `.sam`/`.bam`) were developed for next-generation sequencing (NGS) data. SAM stands for Sequence Alignment Map. These files are used to store alignment information. + +**VCF** (suffix: `.vcf`) stands for Variant Call Format. These files are used to store information about genetic variants. [Read More](https://samtools.github.io/hts-specs/VCFv4.2.pdf) + +**GFF3** (suffix: `.gff3`) stands for Generic Feature Format (version 3). These files are used to store information about genomic features. [Read More](https://github.com/The-Sequence-Ontology/Specifications/blob/master/gff3.md) + +**BED** (suffix: `.bed`) stands for Browser Extensible Data format. These files are used to store genomic regions. [Read More](https://github.com/arq5x/bedtools2) + +### FASTA Format + +A FASTA file begins with a header line, indicated by the `>` symbol, that contains an identifier and optional description The following lines contain the biological sequence itself. + + + __>__ NP_000552.2 Human glutathione transferase M1 (GSTM1) ``` +MPMILGYWDIRGLAHAIRLLLEYTDSSYEEKKYTMGDAPDYDRSQWLNEKFKLGLDFPNLPYLIDGAHKITQSNAILCYIARKHNLCGETEEEKIRVDILENQTMDNHMQLGMICYNPEFEKLKPKYLEELPEKLKLYSEFLGKRPWFAGNKITFVDFLVYDVLDLHRIFEPKCLDAFPNLKDFISRFEGLEKISAYMKSSRFLPRPVFSKMAVWGNK``` + + +### FASTQ Format + +FASTQ files are helpful for base calling, quality control, and trimming. + +Most sequencing tools return data in FASTQ format with quality scores included (ASCII code). + +FASTQ files contain four lines: +1. ID, beginning with `@` +2. Sequence +3. Description line (typically a `+`) +4. Base qualities in ASCII format + +```plaintext +@SEQ_ID +GATTTGGGGTTCAAAGCAGTATCGATCAAATA ++ +!''*((((***+))%%%++)(%%%%).1***- +``` + +**FASTQ File Example: Multiple Reads** + +```plaintext +@M00747:32:000000000-A16RG:1:1112:15153:29246 1:N:0:1 +TCGATCGAGTAACTCGCTGCTGTCAGACTGGTTTTTGGTCGATCGACTATTGTTTCAGTCGCAAGAATATTGTGTCCAGTCGATCGACTGAATTCTGCTGTACGGCCACGGCGGATGCACGGTACAGCAGGCTCAGACGGATTAAACTGTT ++ +5=9=9<=9,-5@<<55>,6+8AC>EE.88AE9CDD7>+7.CC9CD+++5@=-FCCA@EF@+**+*--55--AA---AA-5A<9C+3+<9)4++=E=+===7@FHHHHHHFG?EAFGF@EEDEHHDGHHCBDFFGDFHF)?))5?**0?:AE*A*0//:/*:*:**.0) +@M00747:32:000000000-A16RG:1:1112:15513:29246 1:N:0:1 +GCTAGTCTTGTGTTTAGTTTTATGTTTTGCATGTTGTAACGGATTCATAAACATAGGTGTTTGTTTCTTTTTATGGTTGTACAATTTGGCCCTAAGGCCCTACACTTACTTGTTTGTTTCTTTTATGGTACGACATTTGAGTGGTGGTTGA ++ +``` + diff --git a/content/notes/bioinfo-intro/08-qualityscores.md b/content/notes/bioinfo-intro/08-qualityscores.md new file mode 100644 index 00000000..724f4008 --- /dev/null +++ b/content/notes/bioinfo-intro/08-qualityscores.md @@ -0,0 +1,27 @@ +--- +title: Quality Scores +date: 2025-08-23-03:19:53Z +type: docs +weight: 450 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +`Q` (Quality) scores are defined as a property that is logarithmically related to the base calling error probabilities (`P`). + +### Calculating Phred Quality Scores - Base calling accuracy + +$$ +Q = -10 \log_{10} P +$$ + +`Q` represents the sequencing quality score of a given base Q + +`P` represents the probability of base call being wrong + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_21.png caption="Table source: https://www.illumina.com/Documents/products/technotes/technote_Q-Scores.pdf " width=90% height=90% >}} + +While next-generation sequencing metrics vary from those of Sanger sequencing (e.g., no electropherogram peak heights), the process of generating a Phred quality scoring scheme is largely the same. + +[More on Quality Scores](https://help.basespace.illumina.com/files-used-by-basespace/quality-scores) diff --git a/content/notes/bioinfo-intro/09-sambam.md b/content/notes/bioinfo-intro/09-sambam.md new file mode 100644 index 00000000..d4002b58 --- /dev/null +++ b/content/notes/bioinfo-intro/09-sambam.md @@ -0,0 +1,38 @@ +--- +title: SAM/BAM Sequence Alignment +date: 2025-08-23-03:19:53Z +type: docs +weight: 500 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +An alignment file provides context for raw data. It has 11 tab-delimited columns with one alignment record per line. + +`SAM` is plain-text (human readable), whereas `BAM` is in binary format. + +[SAMTools](http://samtools.sourceforge.net) is a suite of utilities for SAM/BAM files. [Picard](https://broadinstitute.github.io/picard/) is a set of tools for sequencing data. + +### Example SAM file + +```plaintext +D4ZHLFP1:53:D2386ACXX:6:2115:17945:68812 0 Mle_000001 18 42 108M * 0 0 + TCCCCCTGCATGGTCCGTCTGCGTGCAATCGCATGAGTATGCCTCCAGCATGAGTTACCGATCGTGGACACCTGCTTG +GCCAAGATGTACTGAGATGCAT +C@CFDEFFHHGHHFGBGFEGGDGGGEHGHGGGJJJJIIGIIB9BFBFHGHHICEAHGGEGEDHIGEEDBECCACBDDC@CCDBCDD< +?2+4>@4>>CCCAA@@ AS:i:-5 XN:i:0 XM:i:1 XO:i:0 XG:i:0 NM:i:1 MD:Z:0A107 + YT:Z:UU +D4ZHLFP1:53:D2386ACXX:7:2110:5214:83081 0 Mle_000001 18 42 108M * 0 0 +TCCCCCTGCATGGTCCGTCTGCGTGCAATCGCATGAGTATGCCTCCAGCATGAGTTACCGATCGTGGCAACCTGCTTGCCAA +GATGTACTGAGATGCAT +CCCFFFFHHHHHHHGGGEGIJIIGJFHJJJJIJIJJIJIJGIJJIJJIJFHJJJIJJHHFFCEEEEEDDDDDDDDDDDDD AS:i:-5 XN:i:0 XM:i:1 XO:i:0 XG:i:0 NM:i:1 MD:Z:0A107 + YT:Z:UU +D4ZHLFP1:53:D2386ACXX:7:2206:9985:31556 0 Mle_000001 18 42 108M * 0 0 +TCCCCCTGCATGGTCCGTCTGCGTGCAATCGCATGAGTATGCCTCCAGCATGAGTTACCGATCGTGGCAACCTGCTTGCCAA +GATGTACTGAGATGCAT +CCCEFFFFHHHHHJJIJHJJIJIJJIJIJJJJIJIJJJIJJIJJJIJJJGEFFEEEEDDDDDDDDDDDDDDDDDDDDDDD AS:i:-5 XN:i:0 XM:i:1 XO:i:0 XG:i:0 NM:i:1 MD:Z:0A107 + YT:Z:UU +``` + +Helpful site for looking up `SAM` flags: [https://broadinstitute.github.io/picard/explain-flags.html](https://broadinstitute.github.io/picard/explain-flags.html) \ No newline at end of file diff --git a/content/notes/bioinfo-intro/10-fastqc-qualityreads.md b/content/notes/bioinfo-intro/10-fastqc-qualityreads.md new file mode 100644 index 00000000..a93e4ea0 --- /dev/null +++ b/content/notes/bioinfo-intro/10-fastqc-qualityreads.md @@ -0,0 +1,21 @@ +--- +title: Checking Read Quality - FASTQC +date: 2025-08-23-03:19:53Z +type: docs +weight: 550 +menu: + bioinfo-intro: + parent: Bioinformatics +--- + +FASTQC provides an overview of sequencing read quality. + +Sample FASTQC reports displaying varying metrics: + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_24.png width=85% height=85% caption="FASTQC report showing per-base sequence quality, with most bases maintaining high quality across reads and slight drops at the read ends typical of Illumina data." >}} + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_25.png width=85% height=85% caption="FASTQC report showing per-base sequence quality, where read quality declines toward the end, indicating potential sequencing degradation or lower confidence in base calls at later positions." >}} + +{{< figure src=/notes/bioinfo-intro/img/readq3.png width=85% height=85% caption="FASTQC report showing a decline in per-base sequence quality toward the end of reads, indicating significant quality drop-off and potential sequencing errors in later positions." >}} + +[Read More](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) diff --git a/content/notes/bioinfo-intro/11-rcresources.md b/content/notes/bioinfo-intro/11-rcresources.md new file mode 100644 index 00000000..4933805d --- /dev/null +++ b/content/notes/bioinfo-intro/11-rcresources.md @@ -0,0 +1,16 @@ +--- +title: Research Computing Resources +date: 2025-08-23-03:19:53Z +type: docs +weight: 600 +menu: + bioinfo-intro: +--- + +Relevant Tutorials: + +[Using UVA's HPC System from the Terminal](https://learning.rc.virginia.edu/notes/hpc-from-terminal/) + +[HPC orientation session and office hours](https://www.rc.virginia.edu/support/#office-hours) + + diff --git a/content/notes/bioinfo-intro/12-loggingin.md b/content/notes/bioinfo-intro/12-loggingin.md new file mode 100644 index 00000000..10f7556b --- /dev/null +++ b/content/notes/bioinfo-intro/12-loggingin.md @@ -0,0 +1,28 @@ +--- +title: Logging In +date: 2025-08-23-03:19:53Z +type: docs +weight: 650 +menu: + bioinfo-intro: + parent: Research Computing Resources + +--- + +```bash +ssh user_id@login.hpc.virginia.edu +``` + +`pwd` is the command to print your working directory. + +UVA RC working directories: + +`$ /home/user_id` is your home directory, which contains up to 200GB of storage. + +`$ /scratch/user_id` is your scratch directory, which contains up to 10TB storage. The scratch directory is "temporary" storage, meaning files not accessed within 90 days are deleted. You can check files for purging through Open On Demand utilities. + +[Read More](https://www.rc.virginia.edu/userinfo/hpc/ood/) + + + + diff --git a/content/notes/bioinfo-intro/13-allocations-storage.md b/content/notes/bioinfo-intro/13-allocations-storage.md new file mode 100644 index 00000000..76a3f9c6 --- /dev/null +++ b/content/notes/bioinfo-intro/13-allocations-storage.md @@ -0,0 +1,17 @@ +--- +title: HPC allocations & storage +date: 2025-08-23-03:19:53Z +type: docs +weight: 700 +menu: + bioinfo-intro: + parent: Research Computing Resources + +--- + + +`$ hdquota` is the command for checking disk space. + +`$ allocations` is the command to print active allocations and balances. For example, for live workshop demos, the `hpc_training` allocation will be available and removed after the workshop ends. + +It is also important to understand the difference between a login node and a compute node: the login node is where you connect, prepare files, and submit jobs, while the compute node is where the actual job execution takes place. diff --git a/content/notes/bioinfo-intro/14-modules.md b/content/notes/bioinfo-intro/14-modules.md new file mode 100644 index 00000000..1c853a2d --- /dev/null +++ b/content/notes/bioinfo-intro/14-modules.md @@ -0,0 +1,37 @@ +--- +title: Modules and Commands +date: 2025-08-23-03:19:53Z +type: docs +weight: 750 +menu: + bioinfo-intro: + parent: Research Computing Resources +--- + +Modules are used to manage software packages. + +**Helpful Commands:** + +`module avail` lists all available modules (there may be a lot!). + +`module spider ` lists all available versions of ``. + +`module spider /` describes how to load `/`. There may be prerequisite modules. + +`module key ` provides a handy keyword search of all modules categorized into "application classes". + +`module list` lists modules loaded in the current shell. + +`module purge` removes all module modifications in your environment. + +`module load /[]` loads the module for (optionally) `` of ``. + +`module unload ` deletes the changes made by the `` module. + +`module swap / /` exchanges one version of a package for another. + +[Learn More](https://learning.rc.virginia.edu/notes/hpc-from-terminal/section5/18) + + + + diff --git a/content/notes/bioinfo-intro/15-running-jobs.md b/content/notes/bioinfo-intro/15-running-jobs.md new file mode 100644 index 00000000..1940959e --- /dev/null +++ b/content/notes/bioinfo-intro/15-running-jobs.md @@ -0,0 +1,14 @@ +--- +title: Running Jobs +date: 2025-08-23-03:19:53Z +type: docs +weight: 800 +menu: + bioinfo-intro: + +--- +There are 2 options for running FASTQC jobs: +- Interactively +- Batch/Slurm Scripts + + diff --git a/content/notes/bioinfo-intro/16-interactive.md b/content/notes/bioinfo-intro/16-interactive.md new file mode 100644 index 00000000..60372cd2 --- /dev/null +++ b/content/notes/bioinfo-intro/16-interactive.md @@ -0,0 +1,57 @@ +--- +title: Running FASTQC Interactively +date: 2025-08-23-03:19:53Z +type: docs +weight: 850 +menu: + bioinfo-intro: + parent: Running Jobs +--- + +Running FASTQC interactively is good for testing. + +Sample terminal command to run interactive jobs: + +```bash +$ ijob -c 1 -A hpc_training -p standard -v +``` + +### Copying workshop files + +```bash +$ pwd + +$ cp /project/rivanna-training/bioinformatics-hpc . +# . means copy it in your current working directory + +# check to make sure all files transferred: +$ ls -lh +``` + +### Running FASTQC + +```bash +$ module spider fastqc + +$ module load fastqc/0.12.1 + +$ module list + +$ fastqc -t 4 -o fastqc-out ./fastqc/ecoli-fastq/SRR258*fastq.* +``` + +Notes: +* You need to know the path to your FASTQ files. +* The `*` wildcard (`*fastq`) matches all FASTQ files in the folder. +* You need to specify an output directory (`fastqc-out`) where the results will be saved. +* `-t 4` sets the number of threads (4). + +### FASTQC Outputs + +The above command produces `.html` reports. + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_34.png caption="Job output - Quality Scores" width=70% height=70% >}} + + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_36.png caption="Job output - Adapter Content. The red X indicates that adapter contamination is detected." width=80% height=80% >}} + diff --git a/content/notes/bioinfo-intro/17-slurmscript.md b/content/notes/bioinfo-intro/17-slurmscript.md new file mode 100644 index 00000000..38025e67 --- /dev/null +++ b/content/notes/bioinfo-intro/17-slurmscript.md @@ -0,0 +1,76 @@ +--- +title: Running FASTQC with Slurm +date: 2025-08-23-03:19:53Z +type: docs +weight: 870 +menu: + bioinfo-intro: + parent: Running Jobs +--- + +Slurm is a resource manager. You can run your codes as Slurm scripts. + +Sample Slurm script: `fastqc_slurm_submit.sh` + +```bash +#!/bin/bash + +#SBATCH -A hpc_training           # account name (--account) +#SBATCH -p standard               # partition/queue (--partition) +#SBATCH --nodes=1                 # number of nodes +#SBATCH --ntasks=1                # 1 task – how many copies of code to run +#SBATCH --cpus-per-task=1       # total cores per task – for multithreaded code +##SBATCH --mem=3200         # total memory (Mb) *Note ##comment +#SBATCH -t 00:20:00               # time limit: 20 min +#SBATCH -J fastqc-test            # job name +#SBATCH -o fastqc-test-%A.out     # output file +#SBATCH -e fastqc-test-%A.err     # error file +#SBATCH --mail-user=dtriant@virginia.edu   # where to send email alerts +#SBATCH --mail-type=ALL                    # receive email when starts/stops/fails + +__module purge__   # good practice to purge all modules +__module load__ fastqc/0.12.1 + +cd /project/rivanna-training/bioinformatics-hpc + +mkdir fastqc-out-slurm + +__fastqc__ -t 4 -o fastqc-out-slurm  ecoli-fastq/SRR258*fastq # command for running software +``` +> Mail option output: The mail option notifies you when the job starts and completes. + +Job started: + + `Slurm Job_id=4972539 Name=busco-test Began, Queued time 00:00:46` + + Job ended: + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_38.png width=75% height=75% >}} + +### Operational Terminal Commands + +To submit your Slurm script from the terminal, use the following command: + +```bash +$ sbatch fastqc_slurm_submit.sh +``` + +Submitted batch job 4938712 + + +For job monitoring, use the following command: + +```bash +$ squeue -u user_id # checks job status +``` + +`PD` - pending + +`R` - running + +`CG` - exiting + +The Slurm script method should have the same output files as running the job interactively. + + + diff --git a/content/notes/bioinfo-intro/17a.md b/content/notes/bioinfo-intro/17a.md new file mode 100644 index 00000000..6ea3837f --- /dev/null +++ b/content/notes/bioinfo-intro/17a.md @@ -0,0 +1,12 @@ +--- +title: Sequence Processing & Alignment +date: 2025-08-23-03:19:53Z +type: docs +weight: 999 +menu: + bioinfo-intro: + +--- + + + diff --git a/content/notes/bioinfo-intro/19-trimming-sequences.md b/content/notes/bioinfo-intro/19-trimming-sequences.md new file mode 100644 index 00000000..b16bed2c --- /dev/null +++ b/content/notes/bioinfo-intro/19-trimming-sequences.md @@ -0,0 +1,104 @@ +--- +title: Trimming Sequences +date: 2025-08-23-03:19:53Z +type: docs +weight: 1000 +menu: + bioinfo-intro: + parent: Sequence Processing & Alignment + +--- + +Before aligning sequencing reads, it’s important to remove adapter sequences and low-quality bases. + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_44.png caption="Illumina documentation listing adapter and primer sequences commonly trimmed from raw reads." width=80% height=80% >}} + +[Read more about Illumina adapter sequences](https://support-docs.illumina.com/SHARE/AdapterSequences/Content/SHARE/AdapterSeq/Nextera/SequencesNextera_Illumina.htm) + + +### Cutadapt + +[Cutadapt](https://cutadapt.readthedocs.io/en/stable/) is a program that finds and removes adapter sequences, primers, poly-A tails and other types of unwanted sequence from your high-throughput sequencing reads. + +Essentially, Cutadapt trims adapters from short read Illumina data. + +[Documentation](https://cutadapt.readthedocs.io/en/stable/) + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_42.png width=85% height=85% >}} + +The above image shows a FASTQC `.html` report indicating that adapter sequences are still present in the reads. In this case, you would use **Cutadapt** to trim the adapters. + +### Using Cutadapt + +Check available versions on the cluster: + +```bash +$ module spider cutadapt +``` + +Output: + +`cutadapt: cutadapt/4.9` + + +This module can be loaded directly: + +```bash +module load cutadapt/4.9 +``` + + +Sample run command: + +```bash +$ cutadapt -a CTGTCTCTTATACACATCT -o SRR2584866-trimmed_1.fastq  SRR2584866_1.fastq +``` + +Output Summary (Cutadapt 4.9, Python 3.12.9): + +```text +Processing single-end reads on 1 core ... + +Done           00:00:07     2,768,398 reads @   2.8 µs/read;  21.79 M reads/minute + +Finished in 7.626 s (2.755 µs/read; 21.78 M reads/minute). + +=== Summary === + +Total reads processed:               2,768,398 + +Reads with adapters:                   683,198 (24.7%) + +Reads written (passing filters):     2,768,398 (100.0%) + +Total basepairs processed:   415,259,700 bp + +Total written (filtered):    377,219,215 bp (90.8%) +``` + + +### Impact + +After adapter trimming, run FASTQC again to check the quality of the results. + +```bash +$ module spider FASTQC + +$ module load fastqc/0.12.1 + +$ module list # check what’s loaded (did you remember to `module purge` first?) + +$ mkdir fastqc-out-trimmed + +$ fastqc -t 4 -o fastqc-out-trimmed  ecoli-fastq/SRR2584866-trimmed_1.fastq +``` + +Before trimming: + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_49.png width=70% height=70% >}} + +After trimming: + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_51.png width=70% height=70% >}} + +As you can see, adapter sequences have been successfully removed. diff --git a/content/notes/bioinfo-intro/21-cutadapt-slurm.md b/content/notes/bioinfo-intro/21-cutadapt-slurm.md new file mode 100644 index 00000000..d25d6f26 --- /dev/null +++ b/content/notes/bioinfo-intro/21-cutadapt-slurm.md @@ -0,0 +1,40 @@ +--- +title: Running Cutadapt with Slurm +date: 2025-08-23-03:19:53Z +type: docs +weight: 1100 +menu: + bioinfo-intro: + parent: Sequence Processing & Alignment +--- + +As an alternative to running Cutadapt interactively, you can submit a job on Rivanna using a Slurm script. + +Sample Slurm script: `cutadapt_slurm_submit.sh` + +```bash +#!/bin/bash +#SBATCH -A hpc_training           # account name +#SBATCH -p standard               # partition/queue +#SBATCH --nodes=1                 # number of nodes +#SBATCH --ntasks=1                # 1 task +#SBATCH --cpus-per-task=1       # total cores per task +#SBATCH -t 00:20:00               # time limit: 20 min +#SBATCH -J cutadapt-test            # job name +#SBATCH -o cutadapt-test-%A.out     # output file +#SBATCH -e cutadapt-test-%A.err     # error file + +module purge  # good practice to purge all modules + +module load cutadapt/4.9 + +cd /[your working directory] + +mkdir fastqc-out-trimmed + +# enter all sequences, loop around large dataset +__cutadapt__ -a CTGTCTCTTATACACATCT -o SRR2584866-trimmed_1.fastq  SRR2584866_1.fastq + +``` + + diff --git a/content/notes/bioinfo-intro/22-bowtie2.md b/content/notes/bioinfo-intro/22-bowtie2.md new file mode 100644 index 00000000..a9dbe116 --- /dev/null +++ b/content/notes/bioinfo-intro/22-bowtie2.md @@ -0,0 +1,171 @@ +--- +title: Bowtie2 - Sequence Alignment +date: 2025-08-23-03:19:53Z +type: docs +weight: 1150 +menu: + bioinfo-intro: + parent: Sequence Processing & Alignment +--- + +After trimming adapters and low-quality sequences with Cutadapt, the next step is to align the cleaned reads to a reference genome using Bowtie2. + +{{< figure src=/notes/bioinfo-intro/img/bowtie.png width=75% height=75% caption="After alignment with tools such as Bowtie2, the mapped reads can be visualized to assess coverage, variants, and alignment quality across sequencing platforms." >}} + +Alignment is often the time-limiting step. + +**Bowtie** can be used to align short reads. + +__End-to-end alignment example__ + +The following is an "end-to-end" alignment because it involves all the characters in the read (default). + +{{< figure src=/notes/bioinfo-intro/img/bowtieexample.png width=60% height=60% >}} + +__Local alignment example__ + +The following is a "local" alignment because some of the characters at the ends of the read do not participate. In this case, 4 characters are omitted (or "soft trimmed" or "soft clipped") from the beginning and 3 characters are omitted from the end. + +{{< figure src=/notes/bioinfo-intro/img/read2bowtie.png width=60% height=60% >}} + +[Bowtie2 Documentation](https://bowtie-bio.sourceforge.net/bowtie2/index.shtml) + +### Loading and Running the Module + + +```bash +$ module spider bowtie2 +$ module load bowtie2 # which version loaded? +$ module list # did you purge? +$ module spider bowtie2/2.5.4 # sometimes more detail about specific version +$ bowtie2 # to run +``` + +`$ bowtie2 -help` outputs a summary of command-line options. + +**Command Structure:** + +bowtie2 [options]* -x {-1 -2 | -U } [-S < sam >] + +`-x `  Index filename prefix (minus trailing `.X.bt2`). + +- NOTE: Bowtie 1 and Bowtie 2 indexes are not compatible. + +`-1 `       Files with #1 mates, paired with files in ``. + +- Could be `gzip`'ed (extension: `.gz`) or `bzip2`'ed (extension: `.bz2`). + +`-2 `       Files with #2 mates, paired with files in ``. + +- Could be `gzip`'ed (extension: `.gz`) or `bzip2`'ed (extension: `.bz2`). + +`-U `        Files with unpaired reads. + +- Could be `gzip`'ed (extension: `.gz`) or `bzip2`'ed (extension: `.bz2`). + +`-S `      File for SAM output (default: stdout) + +``, ``, `` can be comma-separated lists (no whitespace) and can be specified many times.  + +E.g. `-U file1.fq,file2.fq -U file3.fq`. + +### Using Bowtie2 + +First, build an index for your genome using sample data provided with the bowtie2 download. + +`$ ls example` + +Output: + +index/ +reads/ +example/ + +This command creates the index: + +`$ __ bowtie2-build __ lambda_virus.fa lambda_virus *.bt2 index files` + +These index files will now be available in your directory: + +`lambda_virus.4.bt2` +`lambda_virus.3.bt2` +`lambda_virus.1.bt2` +`lambda_virus.2.bt2` +`lambda_virus.rev.1.bt2` +`lambda_virus.rev.2.bt2` + +Next, align the reads using this command: + +`$ __bowtie2__ -x lamda_virus -1 lamda-reads_1.fastq -2 lamda-reads_2.fastq -S lamda.sam` + + +**Bowtie2 Output** + +``` +Building a SMALL index + +10000 reads; of these: + + __Concordant alignment__ + +10000 (100.00%) were paired; of these: + +834 (8.34%) aligned concordantly 0 times + +__9166 (91.66%) aligned concordantly exactly 1 time__ + +0 (0.00%) aligned concordantly >1 times + +---- + + __Discordant alignment__ + +__834 pairs aligned concordantly 0 times; of these:__ + +42 (5.04%) aligned discordantly 1 time + +---- + + __The rest of the reads either align as singles __ + +__792 pairs aligned 0 times concordantly or discordantly__ ; of these: + +1584 mates make up the pairs; of these: + +1005 (63.45%) aligned 0 times + +579 (36.55%) aligned exactly 1 time + +0 (0.00%) aligned >1 times + +__94.97% overall alignment rate__ +``` + +The result summary is divided into 3 sections: + +1. Concordant alignment + +This shows the reads that were aligned properly as pairs. In this example, 91.66% of reads were aligned concordantly. + + +2. Discordant alignment + +This shows the reads that did not align in the expected configuration. In this example, of the remaining 8.34% of reads, 792 reads align discordantly. That is to say, of the non-concordant fraction, 5.04% of reads (42 reads) align discordantly. + +3. Single or unpaired alignments + +Alignments, whether concordant or discordant, are aligned in paired-end mode. The rest of the reads either align as singles (i.e. `Read1` in one locus and `Read2` in a different locus, or one mate aligned and the other unaligned) or may not align at all. The reads in this section are + +$$Total - (Concordant + Discordant)$$ + +In this example, that is + +$$1000 - (9166 + 42) = 792$$ + +To reach the overall alignment, count the mates in total (i.e. mates aligned in paired and mates aligned in single fashion). In this example, that would be: + +$$(9166 * 2) + (42 * 2) + 579 = 18995$$ + +That is 18885 mates aligned of total (10000 x2) mates, which is 94.97%. + + diff --git a/content/notes/bioinfo-intro/23-bowtie-slurm.md b/content/notes/bioinfo-intro/23-bowtie-slurm.md new file mode 100644 index 00000000..60b8912e --- /dev/null +++ b/content/notes/bioinfo-intro/23-bowtie-slurm.md @@ -0,0 +1,37 @@ +--- +title: Running Bowtie2 - Slurm Script +date: 2025-08-23-03:19:53Z +type: docs +weight: 1250 +menu: + bioinfo-intro: + parent: Sequence Processing & Alignment +--- + +Sample slurm script: `fastqc_slurm_submit.sh` + +The script loads the Bowtie2 module, builds a genome index, and performs read alignment on paired-end FASTQ files. + +```bash +#!/bin/bash +#SBATCH -A hpc_training           # account name +#SBATCH -p standard               # partition/queue +#SBATCH --nodes=1                 # number of nodes +#SBATCH --ntasks=1                # 1 task +#SBATCH --cpus-per-task=1       # total cores per task +#SBATCH -t 00:20:00               # time limit: 20 min +#SBATCH -J bowtie2-test            # job name +#SBATCH -o bowtie2-test-%A.out     # output file +#SBATCH -e cutadap-test-%A.err     # error file + +module purge  # good practice to purge all modules + +module load bowtie2/2.5.4 + +cd /project/rivanna-training/bioinformatics-hpc + +__bowtie2-build __ lambda_virus.fa lambda_virus + +__bowtie2__ -x lamda_virus -1 lamda-reads_1.fastq -2 lamda-reads_2.fastq -S lamda.sam +``` + diff --git a/content/notes/bioinfo-intro/24-samtools.md b/content/notes/bioinfo-intro/24-samtools.md new file mode 100644 index 00000000..2dcce4b0 --- /dev/null +++ b/content/notes/bioinfo-intro/24-samtools.md @@ -0,0 +1,31 @@ +--- +title: Samtools +date: 2025-08-23-03:19:53Z +type: docs +weight: 1350 +menu: + bioinfo-intro: + parent: Sequence Processing & Alignment +--- + +After aligning reads with Bowtie2, the resulting output file is in **SAM format**. +Because SAM files are large and not indexed, we use **Samtools** to convert them into the compressed **BAM** format and sort them for downstream analysis. + +**Samtools** is a suite of programs for analyzing sequence data. + +Running Samtools interactively: + +```bash +$ module spider samtools + +$ module load samtools/1.21 + +# look at difference in file sizes +$ samtools view -S -b lamda.sam > lamda.bam + +$ ls -lh lamda.* + +$ samtools sort lamda.bam > lamda-sorted.bam # puts in same order as fastq files + +``` + diff --git a/content/notes/bioinfo-intro/25-smrtlink.md b/content/notes/bioinfo-intro/25-smrtlink.md new file mode 100644 index 00000000..cde51605 --- /dev/null +++ b/content/notes/bioinfo-intro/25-smrtlink.md @@ -0,0 +1,31 @@ +--- +title: PacBio - SMRTLink +date: 2025-08-23-03:19:53Z +type: docs +weight: 1400 +menu: + bioinfo-intro: + parent: Sequence Processing & Alignment +--- + +### Long-Read Sequencing: PacBio and SMRT Link + +A new **PacBio Revio** sequencing machine is available in the **School of Medicine Genomics Core**. + +To run SMRTLink on Rivanna: + +```bash +# check which versions are available and load the module +$ module spider SMRT +$ module load smrtlink/25.2.0 +``` + +Lots of tools are available for sequence analysis and data manipulation. + +For example, `bam2fasta` converts `BAM` to `FASTA`, and `bam2fastq` converts `BAM` to `FASTQ`. + + +{{< figure src=/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_67.png width=90% height=90% caption="PacBio Revio sequencing instrument interface" >}} + +[SMRT Tools Reference Guide](https://docslib.org/doc/175362/smrt%C2%AE-tools-reference-guide-v8-0) + diff --git a/content/notes/bioinfo-intro/26-future-directions.md b/content/notes/bioinfo-intro/26-future-directions.md new file mode 100644 index 00000000..1805aea2 --- /dev/null +++ b/content/notes/bioinfo-intro/26-future-directions.md @@ -0,0 +1,29 @@ +--- +title: Future Directions +date: 2025-08-23-03:19:53Z +type: docs +weight: 1450 +menu: + bioinfo-intro: +--- + +Recent work highlights the rapid integration of **machine learning** and **large language models (LLMs)** into **bioinformatics** and **genomics research**. + +These developments suggest a shift toward intelligent systems that can interpret, generate, and analyze biological data at unprecedented scale. + +Recent Work: + + +{{< figure src="/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_68.png" caption="**DrBioRight 2.0:** An LLM-powered bioinformatics chatbot for large-scale cancer functional proteomics analysis (Nature Communications, 2025)." width=80% height=80% >}} + +{{< figure src="/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_69.png" caption="**Review:** Deep learning applications in human genomics using next-generation sequencing data (Human Genomics, 2022)." width=90% height=90% >}} + +{{< figure src="/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_70.png" caption="**Perspective:** Advancing bioinformatics with large language models — components, applications, and future outlook (Computational and Structural Biotechnology Journal)." width=90% height=90% >}} + +{{< figure src="/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_71.png" caption="**Editorial:** Large language models and their applications in bioinformatics (Nature Portfolio, 2023)." width=95% height=95% >}} + +{{< figure src="/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_72.png" caption="**Collection:** Artificial intelligence in genomics (Nature, 2023)." width=95% height=95% >}} + + + + diff --git a/content/notes/bioinfo-intro/27-resources.md b/content/notes/bioinfo-intro/27-resources.md new file mode 100644 index 00000000..5ff7a904 --- /dev/null +++ b/content/notes/bioinfo-intro/27-resources.md @@ -0,0 +1,22 @@ +--- +title: More from RC +date: 2025-08-23-03:19:53Z +type: docs +weight: 1500 +menu: + bioinfo-intro: +--- + +UVA Research Computing Learning Portal: + +[https://learning.rc.virginia.edu](https://learning.rc.virginia.edu) + +Using UVA’s HPC system from the terminal: + +[https://learning.rc.virginia.edu/notes/hpc-from-terminal/](https://learning.rc.virginia.edu/notes/hpc-from-terminal/) + +HPC orientation session and office hours: + +[https://www.rc.virginia.edu/support/#office-hours](https://www.rc.virginia.edu/support/#office-hours) + + diff --git a/content/notes/bioinfo-intro/Intro-Bioinformatics-for-posting_20250604.pptx b/content/notes/bioinfo-intro/Intro-Bioinformatics-for-posting_20250604.pptx new file mode 100644 index 00000000..bf4199dd Binary files /dev/null and b/content/notes/bioinfo-intro/Intro-Bioinformatics-for-posting_20250604.pptx differ diff --git a/content/notes/bioinfo-intro/_index.md b/content/notes/bioinfo-intro/_index.md new file mode 100644 index 00000000..3ee6bc69 --- /dev/null +++ b/content/notes/bioinfo-intro/_index.md @@ -0,0 +1,32 @@ +--- +title: Introduction to Bioinformatics Tools for HPC +date: 2025-08-23-03:19:53Z +authors: [mab] +categories: ["Bioinformatics"] +tags: ["Bioinformatics"] +type: docs +weight: 30 +date: 2025-08-23-03:19:53Z + +menu: + bioinfo-intro: +--- + +**Workshop Outline** + +* What is bioinformatics? +* Types of bioinformatics analyses +* Databases +* Sequencing technologies +* File formats +* HPC overview - loading modules +* Software demos: + * - FastQC - quality control + * - Cutadapt - cleaning data + * - Bowtie2 - alignment + * - Samtools - format conversion + * - PacBio SMRTools +* Future Directions + + + diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_10.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_10.png new file mode 100644 index 00000000..08d1befd Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_10.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_11.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_11.png new file mode 100644 index 00000000..8d933166 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_11.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_12.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_12.png new file mode 100644 index 00000000..7734d8fc Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_12.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_13.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_13.png new file mode 100644 index 00000000..69717a83 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_13.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_14.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_14.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_14.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_15.tiff b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_15.tiff new file mode 100644 index 00000000..b53780e7 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_15.tiff differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_16.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_16.png new file mode 100644 index 00000000..5d383037 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_16.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_17.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_17.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_17.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_18.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_18.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_18.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_19.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_19.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_19.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_20.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_20.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_20.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_21.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_21.png new file mode 100644 index 00000000..d0afe604 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_21.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_22.wmf b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_22.wmf new file mode 100644 index 00000000..87d00e9a Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_22.wmf differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_24.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_24.png new file mode 100644 index 00000000..b17b274b Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_24.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_25.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_25.png new file mode 100644 index 00000000..7bbd8cf1 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_25.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_26.tiff b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_26.tiff new file mode 100644 index 00000000..67d840f2 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_26.tiff differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_34.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_34.png new file mode 100644 index 00000000..eb4eb63d Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_34.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_36.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_36.png new file mode 100644 index 00000000..0261a7be Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_36.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_38.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_38.png new file mode 100644 index 00000000..44a42688 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_38.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_42.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_42.png new file mode 100644 index 00000000..0261a7be Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_42.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_44.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_44.png new file mode 100644 index 00000000..78109f05 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_44.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_45.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_45.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_45.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_49.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_49.png new file mode 100644 index 00000000..0261a7be Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_49.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_50.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_50.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_50.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_51.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_51.png new file mode 100644 index 00000000..10e3ac28 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_51.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_52.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_52.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_52.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_54.tiff b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_54.tiff new file mode 100644 index 00000000..b53780e7 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_54.tiff differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_6.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_6.png new file mode 100644 index 00000000..2d680530 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_6.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_65.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_65.png new file mode 100644 index 00000000..15af6a2c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_65.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_67.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_67.png new file mode 100644 index 00000000..5f58efe8 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_67.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_68.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_68.png new file mode 100644 index 00000000..aa7a446c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_68.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_69.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_69.png new file mode 100644 index 00000000..793802d4 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_69.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_7.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_7.png new file mode 100644 index 00000000..59d7368c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_7.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_70.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_70.png new file mode 100644 index 00000000..1c30baba Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_70.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_71.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_71.png new file mode 100644 index 00000000..57885449 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_71.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_72.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_72.png new file mode 100644 index 00000000..81d90238 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_72.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_8.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_8.png new file mode 100644 index 00000000..0efcac1c Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_8.png differ diff --git a/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_9.png b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_9.png new file mode 100644 index 00000000..fa5867a2 Binary files /dev/null and b/content/notes/bioinfo-intro/img/Intro-Bioinformatics-for-posting_20250604_9.png differ diff --git a/content/notes/bioinfo-intro/img/bioinformatics-ss.png b/content/notes/bioinfo-intro/img/bioinformatics-ss.png new file mode 100644 index 00000000..fb4eb17b Binary files /dev/null and b/content/notes/bioinfo-intro/img/bioinformatics-ss.png differ diff --git a/content/notes/bioinfo-intro/img/bowtie.png b/content/notes/bioinfo-intro/img/bowtie.png new file mode 100644 index 00000000..9a5f0c18 Binary files /dev/null and b/content/notes/bioinfo-intro/img/bowtie.png differ diff --git a/content/notes/bioinfo-intro/img/bowtieexample.png b/content/notes/bioinfo-intro/img/bowtieexample.png new file mode 100644 index 00000000..26da9398 Binary files /dev/null and b/content/notes/bioinfo-intro/img/bowtieexample.png differ diff --git a/content/notes/bioinfo-intro/img/filepermissions.png b/content/notes/bioinfo-intro/img/filepermissions.png new file mode 100644 index 00000000..28cb7ec9 Binary files /dev/null and b/content/notes/bioinfo-intro/img/filepermissions.png differ diff --git a/content/notes/bioinfo-intro/img/pacbio.png b/content/notes/bioinfo-intro/img/pacbio.png new file mode 100644 index 00000000..9fee44a7 Binary files /dev/null and b/content/notes/bioinfo-intro/img/pacbio.png differ diff --git a/content/notes/bioinfo-intro/img/read2bowtie.png b/content/notes/bioinfo-intro/img/read2bowtie.png new file mode 100644 index 00000000..0d352d8c Binary files /dev/null and b/content/notes/bioinfo-intro/img/read2bowtie.png differ diff --git a/content/notes/bioinfo-intro/img/readq3.png b/content/notes/bioinfo-intro/img/readq3.png new file mode 100644 index 00000000..809105d2 Binary files /dev/null and b/content/notes/bioinfo-intro/img/readq3.png differ diff --git a/content/notes/rio-intro/01-hpcuva.md b/content/notes/rio-intro/01-hpcuva.md new file mode 100644 index 00000000..0293843f --- /dev/null +++ b/content/notes/rio-intro/01-hpcuva.md @@ -0,0 +1,25 @@ +--- +title: HPC at UVA +date: 2025-11-12-03:53:56Z +type: docs +weight: 150 +menu: + rio-intro: +--- + +High Performance Computing (HPC) systems are tailored to data sensitivity. + +**Rivanna/Afton (Standard Security)** + +These systems are used to process and store public data, internal use data, or sensitive data. Highly Sensitive Data (HSD) is not permissible. + +> More information on the SSZ HPC can be found in our [Intro to HPC slides](https://learning.rc.virginia.edu/notes/hpc-intro/). + +**Rio (High Security)** + +Rio is designed for processing and storing HSD such as HIPAA, FERPA, dbGAP, etc. + +Rio is not CUI certified yet; use the Ivy VM instead. + +> For more information, please see [RC Systems Data Sensitivity](https://www.rc.virginia.edu/userinfo/storage/data-sensitivity/). + diff --git a/content/notes/rio-intro/02-hsd.md b/content/notes/rio-intro/02-hsd.md new file mode 100644 index 00000000..e93b0469 --- /dev/null +++ b/content/notes/rio-intro/02-hsd.md @@ -0,0 +1,19 @@ +--- +title: Highly Sensitive Data +date: 2025-11-12-03:53:56Z +type: docs +weight: 200 +menu: + rio-intro: + parent: HPC at UVA +--- + +Highly Sensitive Data (HSD) is data that requires restrictions on access under the law or that may be protected from release in accordance with applicable law or regulation. + + +Examples: + +**1. Personally identifiable information (PII)** is any information that can be used to identify a person. Examples include social security number, passport number, driver’s license number, military identification number, or biometric records. + +**2. Health information (PHI)** is any information that reveals an individual’s health condition and/or medical history, including information defined by the Health Insurance Portability and Accountability Act (HIPAA). + diff --git a/content/notes/rio-intro/03-ivy-rio.md b/content/notes/rio-intro/03-ivy-rio.md new file mode 100644 index 00000000..4cd04e1f --- /dev/null +++ b/content/notes/rio-intro/03-ivy-rio.md @@ -0,0 +1,26 @@ +--- +title: Ivy and Rio (Overview) +date: 2025-11-12-03:53:56Z +type: docs +weight: 250 +menu: + rio-intro: + parent: HPC at UVA +--- + +**Ivy** + +Ivy is a secure computing environment, offering Virtual Machines (VM) with both Linux and Windows operating systems. + +Ivy VM is designed for interactive and small-scale analysis of highly sensitive data including HIPAA, FERPA, and CUI. + +Access to the Ivy platform is project based. Each project gets a VM. + +Project-specific storage volumes are mounted to the VM and available on the HPC system enabling a seamless transition between tasks on VM and HPC environment. + +Ivy Linux VMs can serve as a frontend for accessing the Rio HPC system (available upon request). + +**Rio** + +Rio is used for large-scale analysis of HIPAA, FERPA, and dbGaP data. + diff --git a/content/notes/rio-intro/04-terminology.md b/content/notes/rio-intro/04-terminology.md new file mode 100644 index 00000000..aabdfb08 --- /dev/null +++ b/content/notes/rio-intro/04-terminology.md @@ -0,0 +1,35 @@ +--- +title: Terminology +date: 2025-11-12-03:53:56Z +type: docs +weight: 300 +menu: + rio-intro: + parent: HPC at UVA +--- + +**Node** + +Nodes are a type of computer called a server. They generally have more power than a typical computer. + +They may have special hardware like graphical processing units (GPUs). + +There are two types of Nodes: +1. _Login node_ - a server used for logging in and submitting jobs. +2. _Compute node_ - a server that carries out the computational work. + + +**Core** + +A core is an individual processor on a computer. + +**Memory** + +In HPC, memory refers to the random-access memory on a node. + +**Storage** + +In HPC, storage refers to disk storage visible from a node. + + +{{< figure src=/notes/rio-intro/img/riodiagram.png caption="Overview of the Rio HPC user access workflow, from PI request to job execution within the cluster environment." alt="A flowchart showing the process for accessing the Rio high-security HPC system. It begins with a PI submitting a VM request, followed by Research Computing provisioning the VM and user accounts. Users then connect to the cluster environment, which includes modules for software, partitions for hardware, and Slurm as the job scheduler." width=70% height=70% >}} diff --git a/content/notes/rio-intro/05-access.md b/content/notes/rio-intro/05-access.md new file mode 100644 index 00000000..0e8ac048 --- /dev/null +++ b/content/notes/rio-intro/05-access.md @@ -0,0 +1,24 @@ +--- +title: Requesting Access to Rio +date: 2025-11-12-03:53:56Z +type: docs +weight: 350 +menu: + rio-intro: +--- + +Principal Investigators (PIs) can request access by submitting an Ivy Linux VM request through the [services app](https://services.rc.virginia.edu/). + +Research staff may apply for an exception. + +Students must be sponsored by a faculty or research staff. + +In addition to being a login node to Rio, a Linux VM can be used for preliminary computations as well. + +VMs comes in different sizes. Request a VM sized appropriately for your workflow. For larger groups or projects with computationally intensive tasks, choose a larger VM. We recommend Small and above. + +VMs exist in a private, secure network and cannot reach outside resources on the Internet. Most inbound and outbound data transfer is managed through the Data Transfer Node (DTN). + +More information on requesting a VM can be found [here on the RC website](https://www.rc.virginia.edu/userinfo/ivy/). + + diff --git a/content/notes/rio-intro/06-provisioning.md b/content/notes/rio-intro/06-provisioning.md new file mode 100644 index 00000000..18f3f0cf --- /dev/null +++ b/content/notes/rio-intro/06-provisioning.md @@ -0,0 +1,18 @@ +--- +title: VM Provisioning +date: 2025-11-12-03:53:56Z +type: docs +weight: 400 +menu: + rio-intro: + parent: Requesting Access to Rio +--- + +After access is requested, the Research Computing (RC) team will handle the provisioning of the Virtual Machine (VM). + +First, the RC team will create a Grouper group for Rio access. + +Next, the RC team will request access to the High Security VPN (HSVPN) on your group's behalf and provision 1 TB of High Security Research Standard storage for your project. This storage volume will also be available when using Rio HPC for your project. + +The PI and researchers will be notified upon the completion of the VM provisioning, and the associated IP address and group name will be provided to them. + diff --git a/content/notes/rio-intro/07-userprovisioning.md b/content/notes/rio-intro/07-userprovisioning.md new file mode 100644 index 00000000..0adb564c --- /dev/null +++ b/content/notes/rio-intro/07-userprovisioning.md @@ -0,0 +1,20 @@ +--- +title: User Provisioning +date: 2025-11-12-03:53:56Z +type: docs +weight: 450 +menu: + rio-intro: + parent: Requesting Access to Rio +--- + +Any users of the VM must be added to the project in the [services app](https://services.rc.virginia.edu/) by the PI. + +Once a project is approved, a PI and her/his researchers must sign a RUDA (one for every researcher on each project). + +Each researcher must also complete the _[High Security Awareness Training (HSAT)](http://in.virginia.edu/hsat-training)_. This must be completed to permit each researcher access to the HSVPN filter. + + +> More information on security training and HSVPN access can be found [here](https://www.rc.virginia.edu/userinfo/ivy/) on the RC website. + + diff --git a/content/notes/rio-intro/08-connecting.md b/content/notes/rio-intro/08-connecting.md new file mode 100644 index 00000000..d9d75395 --- /dev/null +++ b/content/notes/rio-intro/08-connecting.md @@ -0,0 +1,25 @@ +--- +title: Connecting to the System +date: 2025-11-12-03:53:56Z +type: docs +weight: 500 +menu: + rio-intro: +--- + +Before connecting to your VM, you must install the following on your personal machine/device. + * Install the Cisco AnyConnect Secure Mobility Client. + * Install Opswat. + * Install Duo MFA on personal smartphone. + * See [here on the RC website](https://www.rc.virginia.edu/userinfo/ivy/) for details. + +There are two ways to connect to your Linux VM: + +1. Accessing the VM from a browser window +2. Using SSH (Secure Shell) + * Use a terminal window with Mac/Linux + * Windows users will need to install an SSH client application. We recommend [MobaXTerm](https://www.rc.virginia.edu/userinfo/hpc/logintools/mobaxterm/). + +In either case, you need the IP address of your VM along with your Netbadge credentials to connect to the VM. + + diff --git a/content/notes/rio-intro/09-access-ssh.md b/content/notes/rio-intro/09-access-ssh.md new file mode 100644 index 00000000..04c85249 --- /dev/null +++ b/content/notes/rio-intro/09-access-ssh.md @@ -0,0 +1,28 @@ +--- +title: Accessing from SSH +date: 2025-11-12-03:53:56Z +type: docs +weight: 550 +menu: + rio-intro: + parent: Connecting to the System +--- + +Steps: + +1. Start the HSVPN + +2. Open your SSH client and type: +```bash +ssh [mst3k@10.xxx.xxx.xxx](mailto:mst3k@10.xxx.xxx.xxx) +``` +, where `mst3k` is replaced with your user ID and the `x`'s are replaced with your VM's IP address (given in the Services app). + +{{< figure src=/notes/rio-intro/img/rio-intro_5.png alt="Screenshot showing a terminal window in MobaXterm with the command ssh mst3k@10.xxx.xxx.xxx entered, demonstrating how a user connects to their Rio VM over SSH using their user ID and VM IP address. The command line interface displays a green and yellow bar with the date, time, and file path /home/mobaxterm." width=90% height=90% >}} + +When prompted for a password, use your EServices password. + +This login method provides only command line access to the VM. No graphical user interfaces (GUIs) are usable. + + + diff --git a/content/notes/rio-intro/10-ssh-clients.md b/content/notes/rio-intro/10-ssh-clients.md new file mode 100644 index 00000000..a2e8ab3e --- /dev/null +++ b/content/notes/rio-intro/10-ssh-clients.md @@ -0,0 +1,36 @@ +--- +title: SSH Clients +date: 2025-11-12-03:53:56Z +type: docs +weight: 600 +menu: + rio-intro: + parent: Connecting to the System +--- + +SSH clients mostly use instructions typed at the command prompt. + +You will need to become familiar with Linux/Unix commands. + +Some useful Unix/Linux commands: + +```bash +$ ls +``` +```bash +$ pwd +``` +```bash +$ cd _folder_name_ +``` +```bash +$ cp file_1 file_2 +``` +```bash +$ rm file_1 +``` +```bash +$ cd .. +``` + +See [learning.rc.virginia.edu/tutorials/unix-tutorial/](http://learning.rc.virginia.edu/tutorials/unix-tutorial/) to learn more. \ No newline at end of file diff --git a/content/notes/rio-intro/11-access-browser.md b/content/notes/rio-intro/11-access-browser.md new file mode 100644 index 00000000..168760e3 --- /dev/null +++ b/content/notes/rio-intro/11-access-browser.md @@ -0,0 +1,23 @@ +--- +title: Accessing from a Browser +date: 2025-11-12-03:53:56Z +type: docs +weight: 650 +menu: + rio-intro: + parent: Connecting to the System +--- + +Steps: + +1. Start the HSVPN +2. Open a web browser and enter the IP address for your VM (e.g., `https://10.xxx.xxx.xxx`). If you get a warning message, you may need to click on Advanced Settings and/or a Connect Anyway option, depending on your web browser. +3. Upon login, you will see a selection of different graphical applications that you can use. + +{{< figure src=/notes/rio-intro/img/rio-intro_6.jpg caption="Graphical application options available when accessing a Rio VM through a web browser: JupyterHub, RStudio, and FastX Desktop." alt="Screenshot showing the login interface for accessing a Rio virtual machine through a web browser. Three application icons are displayed — JupyterHub, RStudio Server, and FastX Desktop — representing the graphical tools users can launch on their VM after connecting via HTTPS." width=90% height=90% >}} + +These applications run on the VM itself and do not use HPC resources. + + + + diff --git a/content/notes/rio-intro/14-fastx.md b/content/notes/rio-intro/14-fastx.md new file mode 100644 index 00000000..fb1673e5 --- /dev/null +++ b/content/notes/rio-intro/14-fastx.md @@ -0,0 +1,24 @@ +--- +title: Working with Files Using FastX +date: 2025-11-12-03:53:56Z +type: docs +weight: 700 +menu: + rio-intro: +--- + +FastX Desktop is a great tool for managing files and launching applications with a graphical user interface. + +{{< figure src=/notes/rio-intro/img/rio-intro_7.jpg caption="FastX Dashboard" alt="Screenshot of the FastX dashboard showing the “My Sessions” window with options to launch a MATE desktop environment or a terminal session. A sidebar includes buttons for launching a session and managing bookmarks." width=80% height=80% >}} + +Once signed-in, you can select either a "MATE" (GUI Linux Desktop) or "Terminal" session + +MATE sessions provide a terminal (black box in menu bar) and a file browser (white box) in menu bar. The CAJA file browser can help with management and navigation of the filesystem. + +{{< figure src=/notes/rio-intro/img/rio-intro_8.jpg caption="Example of the MATE Desktop environment on a Rio VM, showing the CAJA file browser open to the user’s home directory." alt="Screenshot of the MATE Desktop environment with the CAJA file browser window open. The desktop background shows icons for “Computer,” “egg3xa’s Home,” and “Trash.” The file browser displays folders such as Desktop, Documents, Downloads, Music, Pictures, Public, R, Templates, and an untitled Jupyter notebook file." width=90% height=90% >}} + +More information on using FastX can be found on our [website](https://www.rc.virginia.edu/userinfo/hpc/logintools/fastx/) and in our [Intro to HPC slides](https://learning.rc.virginia.edu/notes/hpc-intro/connecting_to_the_system/connecting_fastx_desktop/). + + + + diff --git a/content/notes/rio-intro/15-file-editing.md b/content/notes/rio-intro/15-file-editing.md new file mode 100644 index 00000000..2f34aaf6 --- /dev/null +++ b/content/notes/rio-intro/15-file-editing.md @@ -0,0 +1,15 @@ +--- +title: File Editing in Linux VM +date: 2025-11-12-03:53:56Z +type: docs +weight: 750 +menu: + rio-intro: +--- + +Editing files in a Linux virtual machine can be done using several text editors, depending on whether you are working in a terminal window or within the desktop environment. + +When working in a Terminal Window, use command-line editors such as `vi`, `emacs`, or `pluma`. + +When working in the Desktop environment, you can use the Pluma Text Editor. + diff --git a/content/notes/rio-intro/16-storage.md b/content/notes/rio-intro/16-storage.md new file mode 100644 index 00000000..5e2dd1c2 --- /dev/null +++ b/content/notes/rio-intro/16-storage.md @@ -0,0 +1,22 @@ +--- +title: Storing Data +date: 2025-11-12-03:53:56Z +type: docs +weight: 800 +menu: + rio-intro: +--- + +Virtual machines do not come with any significant disk storage of their own. + +A PI specifies the storage space they would like to have when requesting access to Ivy. + +The default allocation is of 1TB of Research Standard Storage. + +This storage is mounted under `/data/ivy-hip-name`, where `ivy-hip-name` is replaced by the name of your Ivy project's Grouper group name. + +Storage can be resized upon request. Submit your request [here on the RC website](https://www.rc.virginia.edu/form/storage/). + +JupyterLab, RStudio, Rio, and FastX can all save data into the shared storage at `/data/ivy-hip-name`. + + diff --git a/content/notes/rio-intro/17-transferring-data.md b/content/notes/rio-intro/17-transferring-data.md new file mode 100644 index 00000000..6c8a70a4 --- /dev/null +++ b/content/notes/rio-intro/17-transferring-data.md @@ -0,0 +1,19 @@ +--- +title: Transferring Data +date: 2025-11-12-03:53:56Z +type: docs +weight: 850 +menu: + rio-intro: +--- + +**Globus** is the only permitted data-transfer protocol for highly sensitive data. + +**UVA IVY-DTN** is the the official collection for moving files into High-Security Research Standard Storage. + +To transfer data to High-Security Research Standard Storage, please see the special instructions [here on the RC website](https://www.rc.virginia.edu/userinfo/ivy/). + +> Ensure that you are __NOT__ connected to the HSVPN. Data transfer will not work if you are connected to the HSVPN. + +Additional instructions and information on using Globus can be found [here on the RC learning site](https://learning.rc.virginia.edu/tutorials/globus-data-transfer/). + diff --git a/content/notes/rio-intro/18-preinstalled-software.md b/content/notes/rio-intro/18-preinstalled-software.md new file mode 100644 index 00000000..54706995 --- /dev/null +++ b/content/notes/rio-intro/18-preinstalled-software.md @@ -0,0 +1,18 @@ +--- +title: Preinstalled Software +date: 2025-11-12-03:53:56Z +type: docs +weight: 900 +menu: + rio-intro: +--- + +Every Virtual Machine comes with a base installation of software by default. See [here on the RC website](https://www.rc.virginia.edu/userinfo/ivy/ivy-linux-sw/) for a list of preinstalled software and how to request new software installations. + +Additional software packages are pre-approved and available for installation upon request. Software not already approved will need to undergo a security evaluation to determine if it is suitable for a High Security environment. + +Python and R packages are available to users through the normal `pip`, `conda`, and `CRAN` library installation methods. + +Access to installed software is done using the `lmod` Module System. + + diff --git a/content/notes/rio-intro/19-modules.md b/content/notes/rio-intro/19-modules.md new file mode 100644 index 00000000..acb8d479 --- /dev/null +++ b/content/notes/rio-intro/19-modules.md @@ -0,0 +1,37 @@ +--- +title: Modules +date: 2025-11-12-03:53:56Z +type: docs +weight: 950 +menu: + rio-intro: + parent: Preinstalled Software +--- + +Any application software that you want to use will need to be loaded with the `module load` command. + +For example: +```bash +module load apptainer/1.3.1 +``` +You will need to load the module any time that you create a new shell. This includes every time that you log out and back in as well as every time that you run a batch job on a compute node. + +**Module Commands** + +Some basic commands to use `lmod`: + +`module avail` lists every module avaiable to load. + +`module key ` searches for modules in a specified category (i.e. `module key bio`). + +`module spider ` prints information about the software including different offered versions. + +`module load ` loads the desired software. + +`module load ` loads a specific version of a module. + +`module unload ` unloads the desired software. + +`module list` prints all currently loaded modules. + +`module purge` unloads all modules. diff --git a/content/notes/rio-intro/20-slurm.md b/content/notes/rio-intro/20-slurm.md new file mode 100644 index 00000000..d88cbfdf --- /dev/null +++ b/content/notes/rio-intro/20-slurm.md @@ -0,0 +1,72 @@ +--- +title: Slurm +date: 2025-11-12-03:53:56Z +type: docs +weight: 1000 +menu: + rio-intro: +--- + +**Slurm** manages the hardware resources on the cluster (e.g. compute nodes/cpu cores, compute memory, etc.). + +Slurm allows you to request resources within the cluster to run your code. It is used for submitting jobs to compute nodes from an access point (Ivy VM). + +Frontends are intended for editing, compiling, and very short test runs. + +Production jobs go to the compute nodes through the resources manager. + +#### Example Slurm Script + +A Slurm script is a bash script with Slurm directives (#SBATCH) and command-line instructions for running your program. + +```bash +#!/bin/bash +#SBATCH --nodes=1 #total number of nodes for the job +#SBATCH --ntasks-per-node=1 #how many copies of code to run +#SBATCH --time=1-12:00:00 #amount of time for the whole job +#SBATCH --partition=standard #the queue/partition to run on +#SBATCH --account=myGroupName #the account/allocation to use + +module purge +module load goolf R #load modules that my job needs +Rscript myProg.R #command-line execution of my job +``` + +#### Submitting a Slurm Job + +To submit the Slurm command file to the queue, use the `sbatch` command at the command line prompt. + +`/home` only exists on VM. Compute nodes do not have access to `/home`. + +Job submission needs to be done from `/standard/storage/`. + +It is recommended to include the complete file paths in the script. For example, if the script on the previous page is in a file named `job_script.slurm`, we can submit it as follows: +```bash +[jus2yw@ivy-tst-rc-1 ivy-tst-rc]$ sbatch job_script.slurm +``` + +Output: +```plaintext +Submitted batch job 18316 +``` + +#### Submitting an Interactive Slurm Job + + +To submit an interactive Slurm job to the queue, use the `ijob` command at the command line prompt. + +For example, if you want to run an interactive application on a compute node in the standard queue using one cpu, we can submit it as follows: +```bash +-bash-4.1$ ijob –c 1 –p standard –A MyGroupName -t 06:00:00 +``` +Output: +```plaintext +salloc: Pending job allocation 21640112 +salloc: job 21640112 queued and waiting for resources salloc: job 21640112 has been allocated resources +salloc: Granted job allocation 21640112 srun: Step created for job 21640112 +udc-aw34-21c0-teh1m$ +``` + +Slurm documentation: + * [https://www.rc.virginia.edu/userinfo/hpc/slurm/](https://www.rc.virginia.edu/userinfo/hpc/slurm/) + * [https://slurm.schedmd.com/](https://slurm.schedmd.com/) \ No newline at end of file diff --git a/content/notes/rio-intro/21-partitions.md b/content/notes/rio-intro/21-partitions.md new file mode 100644 index 00000000..c9785db5 --- /dev/null +++ b/content/notes/rio-intro/21-partitions.md @@ -0,0 +1,21 @@ +--- +title: Partitions (Queues) +date: 2025-11-12-03:53:56Z +type: docs +weight: 1050 +menu: + rio-intro: + parent: Slurm +--- + +Rio has several partitions (or queues) for job submissions. + +You will need to specify a partition when you submit a job. + +To see the partitions that are available to you, type `qlist` at the command-line prompt. + + +{{< figure src=/notes/rio-intro/img/rio-intro_9.png caption="Command output" alt="Screenshot of terminal output displaying Slurm partition information. The table lists four queues—neo, neo-gpu, standard, and gpu—along with their total and free cores, running and pending jobs (all zero), and time limits of seven days for most partitions and three days for the GPU queue." width=75% height=75% >}} + +> The `neo` partitions are exclusive to a specific group and are not for general users of Rio. + diff --git a/content/notes/rio-intro/22-jobstatus.md b/content/notes/rio-intro/22-jobstatus.md new file mode 100644 index 00000000..9ee4cf92 --- /dev/null +++ b/content/notes/rio-intro/22-jobstatus.md @@ -0,0 +1,58 @@ +--- +title: Checking Job Status +date: 2025-11-12-03:53:56Z +type: docs +weight: 1400 +menu: + rio-intro: + parent: Slurm +--- + +To display the status of only your _active_ jobs, type: `squeue –u `. + +The `squeue` command will show pending jobs and running jobs, but not failed, canceled or completed job. + +```bash +-bash-4.2$ squeue –u mst3k +``` +Output: + +{{< table >}} +| JOBID | PARTITION | NAME | USER | ST | TIME | NODES | NODELIST(REASON) | +| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | +| 18316 | standard | job_sci | mst3k | R | 1:45 | 1 | udc-aw38-34-l | +{{< /table >}} + +A job's status is indicated by: + +`PD` – pending + +`R` – running + +`CG` – exiting + + +To display the status of all jobs, type: `sacct –S `. + +```bash +[jus2yw@ivy-tst-rc-1 ivy-tst-rc]$ sacct –S 2025-01-01 +``` +Output: +{{< table >}} +| JobID | JobName | Partition | Account | AllocCPUS | State | ExitCode| +| :-: | :-: | :-: | :-: | :-: | :-: | :-: | +| 3104009 | RAxML_NoC+ | standard | hpc_build | 20 | COMPLETED | 0:0 | +| 3104009.bat+ | batch | | hpc_build | 20 | COMPLETED | 0:0 | +| 3104009.0 | raxmlHPC-+ | | hpc_build | 20 | COMPLETED | 0:0 | +| 3108537 | sys/dashb+ | gpu | hpc_build | 1 | CANCELLED+ | 0:0 | +| 3108537.bat+ | batch | | hpc_build | 1 | CANCELLED | 0:15 | +| 3108562 | sys/dashb+ | gpu | hpc_build | 1 | TIMEOUT | 0:0 | +| 3108562.bat+ | batch | | hpc_build | 1 | CANCELLED | 0:15 | +| 3109392 | sys/dashb+ | gpu | hpc_build | 1 | TIMEOUT | 0:0 | +| 3109392.bat+ | batch | | hpc_build | 1 | CANCELLED | 0:15 | +| 3112064 | srun | gpu | hpc_build | 1 | FAILED | 1:0 | +| 3112064.0 | bash | | hpc_build | 1 | FAILED | 1:0 | +{{< /table >}} + +The `sacct` command lists all jobs (pending, running, completed, canceled, failed, etc.) since the specified date. + diff --git a/content/notes/rio-intro/23-canceling-job.md b/content/notes/rio-intro/23-canceling-job.md new file mode 100644 index 00000000..40bea88e --- /dev/null +++ b/content/notes/rio-intro/23-canceling-job.md @@ -0,0 +1,25 @@ +--- +title: Canceling a Job +date: 2025-11-12-03:53:56Z +type: docs +weight: 1450 +menu: + rio-intro: + parent: Slurm +--- + + +To delete a job from the queue, use the `scancel` command with the job ID number at the command line prompt: `-bash-4.2$ scancel 18316` + +To cancel all your jobs, run this command: `-bash-4.2$ scancel –u $USER` + +We use `scontrol` to print information of a running job: `scontrol show job ` + +**More Commands:** + +`sacct` will return accounting information about your job. See [Slurm's](https://slurm.schedmd.com/sacct.html)[ ](https://slurm.schedmd.com/sacct.html)[docuementation](https://slurm.schedmd.com/sacct.html) for a full list of options. + * Use the option `–j ` to inspect a particular job. + + +`seff` will return information about the utilization (called the “efficiency”) of core and memory. + diff --git a/content/notes/rio-intro/24-support.md b/content/notes/rio-intro/24-support.md new file mode 100644 index 00000000..ade0181c --- /dev/null +++ b/content/notes/rio-intro/24-support.md @@ -0,0 +1,23 @@ +--- +title: Support +date: 2025-11-12-03:53:56Z +type: docs +weight: 1500 +menu: + rio-intro: +--- + +Need Help? + +__Research Computing Zoom Office Hours__ + +* Tuesdays: 3pm – 5pm + +* Thursdays: 10am – noon + +> To connect to the Zoom sessions, go to [https://www.rc.virginia.edu/support/#office-hours](https://www.rc.virginia.edu/support/#office-hours) and click the "Join us via Zoom" button. + +Alternatively, contact us through the forms at: [https://www.rc.virginia.edu/support/](https://www.rc.virginia.edu/support/) + + + diff --git a/content/notes/rio-intro/_index.md b/content/notes/rio-intro/_index.md new file mode 100644 index 00000000..5dbfa80b --- /dev/null +++ b/content/notes/rio-intro/_index.md @@ -0,0 +1,15 @@ +--- +title: Intro to High Security HPC (Rio) +date: 2025-11-12-03:53:56Z +authors: [as, pbo, Camden Duy] +categories: [HPC] +tags: [HPC] +type: docs +weight: 1 +date: 2025-11-12-03:53:56Z + +menu: + rio-intro: +--- + + diff --git a/content/notes/rio-intro/img/rio-intro_5.png b/content/notes/rio-intro/img/rio-intro_5.png new file mode 100644 index 00000000..90650111 Binary files /dev/null and b/content/notes/rio-intro/img/rio-intro_5.png differ diff --git a/content/notes/rio-intro/img/rio-intro_6.jpg b/content/notes/rio-intro/img/rio-intro_6.jpg new file mode 100644 index 00000000..73a45a5d Binary files /dev/null and b/content/notes/rio-intro/img/rio-intro_6.jpg differ diff --git a/content/notes/rio-intro/img/rio-intro_7.jpg b/content/notes/rio-intro/img/rio-intro_7.jpg new file mode 100644 index 00000000..527bebe4 Binary files /dev/null and b/content/notes/rio-intro/img/rio-intro_7.jpg differ diff --git a/content/notes/rio-intro/img/rio-intro_8.jpg b/content/notes/rio-intro/img/rio-intro_8.jpg new file mode 100644 index 00000000..5f5bb021 Binary files /dev/null and b/content/notes/rio-intro/img/rio-intro_8.jpg differ diff --git a/content/notes/rio-intro/img/rio-intro_9.png b/content/notes/rio-intro/img/rio-intro_9.png new file mode 100644 index 00000000..f39e2fc9 Binary files /dev/null and b/content/notes/rio-intro/img/rio-intro_9.png differ diff --git a/content/notes/rio-intro/img/riodiagram.png b/content/notes/rio-intro/img/riodiagram.png new file mode 100644 index 00000000..744741b7 Binary files /dev/null and b/content/notes/rio-intro/img/riodiagram.png differ diff --git a/content/notes/rio-intro/index.md b/content/notes/rio-intro/index.md deleted file mode 100644 index 9cfac339..00000000 --- a/content/notes/rio-intro/index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "Introduction to Rio" -date: "2025-01-30T00:00:00" -authors: [as, pbo, Camden Duy] ---- - -{{< slideshow folder="slides/rio-intro" ext="jpg" >}} diff --git a/content/notes/rio-intro/rio-intro.pdf b/content/notes/rio-intro/rio-intro.pdf new file mode 100755 index 00000000..9566be69 Binary files /dev/null and b/content/notes/rio-intro/rio-intro.pdf differ diff --git a/content/notes/rio-intro/rio-intro.pptx b/content/notes/rio-intro/rio-intro.pptx new file mode 100644 index 00000000..e5127c2b Binary files /dev/null and b/content/notes/rio-intro/rio-intro.pptx differ diff --git a/content/tutorials/bioinfo-intro/index.md b/content/tutorials/bioinfo-intro/index.md new file mode 100644 index 00000000..7c3bd509 --- /dev/null +++ b/content/tutorials/bioinfo-intro/index.md @@ -0,0 +1,11 @@ +--- +title: "Introduction to Bioinformatics Tools for HPC" +date: "2025-05-14T00:00:00" +authors: [mab] +categories: ["Bioinformatics"] +tags: ["Bioinformatics"] +summary: "An introduction to bioinformatics concepts, sequencing technologies, common analysis tools, and their use on high-performance computing systems." + +notes: bioinfo-intro + +--- \ No newline at end of file diff --git a/content/tutorials/rio-intro/index.md b/content/tutorials/rio-intro/index.md index ac90a9ad..1e399841 100644 --- a/content/tutorials/rio-intro/index.md +++ b/content/tutorials/rio-intro/index.md @@ -16,6 +16,7 @@ categories: ["HPC"] # Otherwise, set slides = "". url_slides: notes/rio-intro +notes: rio-intro weight: 12