Skip to content
/ LLM-PathwayCurator Public

Transform enrichment outputs into verifiable pathway claims via stability distillation, evidence modules, and mechanical PASS/ABSTAIN/FAIL audits.

llm-pathwaycurator.readthedocs.io/

License

MIT license
0 stars 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

kenflab/LLM-PathwayCurator

BranchesTags

Repository files navigation

LLM-PathwayCurator

LLM-PathwayCurator Enrichment interpretations → audit-gated, decision-grade claims.

Docs bioRxiv DOI RRID License: MIT Python

🚀 What this is

LLM-PathwayCurator is an interpretation quality-assurance (QA) layer for enrichment analysis.
It does not introduce a new enrichment statistic. Instead, it turns EA outputs into auditable decision objects:

  • Input: enrichment term lists from ORA (e.g., Metascape) or rank-based enrichment (e.g., fgsea, an implementation of the GSEA method)
  • Output: typed, evidence-linked claims + PASS/ABSTAIN/FAIL decisions + reason-coded audit logs
  • Promise: we abstain when claims are unstable, under-supported, contradictory, or context-nonspecific

Selective prediction for pathway interpretation: calibrated abstention is a feature, not a failure.

LLM-PathwayCurator workflow: EvidenceTable → modules → claims → audits

Fig. 1a. Overview of LLM-PathwayCurator workflow:
EvidenceTable → modules → claims → audits (bioRxiv preprint)

🧭 Why this is different (and why it matters)

Enrichment tools return ranked term lists. In practice, interpretation breaks because:

  1. Representative terms are ambiguous under study context
  2. Gene support is opaque, enabling cherry-picking
  3. Related terms share / bridge evidence in non-obvious ways
  4. There is no mechanical stop condition for fragile narratives

LLM-PathwayCurator replaces narrative endorsement with audit-gated decisions.
We transform ranked terms into machine-auditable claims by enforcing:

  • Evidence-linked constraints: claims must resolve to valid term/module identifiers and supporting-gene evidence
  • Stability audits: supporting-gene perturbations yield stability proxies (operating point: τ)
  • Context validity stress tests: context swap reveals context dependence without external knowledge
  • Contradiction checks: internally inconsistent claims fail mechanically
  • Reason-coded outcomes: every decision is explainable by a finite audit code set

🔍 What this is not

  • Not an enrichment method; it audits enrichment outputs.
  • Not a free-text summarizer; claims are schema-bounded (typed JSON; no narrative prose as “evidence”).
  • Not a biological truth oracle; it checks internal consistency and evidence integrity, not mechanistic truth.

🧩 Core pipeline (A → B → C)

A) Stability distillation (evidence hygiene)
Perturb supporting genes (seeded) to compute stability proxies (e.g., LOO/jackknife-like survival scores).
Output: distilled.tsv

B) Evidence factorization (modules)
Factorize the term–gene bipartite graph into evidence modules that preserve shared vs distinct support.
Outputs: modules.tsv, term_modules.tsv, term_gene_edges.tsv

C) Claims → audit → report

  • C1 (proposal-only): deterministic baseline or optional LLM proposes typed claims with resolvable evidence links
  • C2 (audit/decider): mechanical rules assign PASS/ABSTAIN/FAIL with precedence (FAIL > ABSTAIN > PASS)
  • C3 (report): decision-grade report + audit log (audit_log.tsv) + provenance

⚡ Quick start (library entrypoint)

llm-pathway-curator run \
  --sample-card examples/demo/sample_card.json \
  --evidence-table examples/demo/evidence_table.tsv \
  --out out/demo/

Key outputs (stable contract)

📊 Rank & visualize ranked terms (rank / plot-ranked)

LLM-PathwayCurator includes two small post-processing commands for ranking and publication-ready visualization of ranked terms/modules:

  • llm-pathway-curator rank — produces a ranked table (claims_ranked.tsv) for downstream plots and summaries.
  • llm-pathway-curator plot-ranked — renders ranked terms/modules as either:
    • bars (Metascape-like horizontal bars), or
    • packed circles (module-level circle packing with term circles inside).

A) Rank (produce claims_ranked.tsv)

Use rank to generate a deterministic ranked table from a run output directory.

llm-pathway-curator rank --help
# Typical workflow: point rank to a run directory and write claims_ranked.tsv
# (See --help for the exact flags supported by your installed version.)

B) Plot (bars or packed circles)

plot-ranked auto-detects claims_ranked.tsv (recommended) or falls back to audit_log.tsv under --run-dir.

Packed circles require an extra dependency: python -m pip install circlify

Bars (Metascape-like)

llm-pathway-curator plot-ranked \
  --mode bars \
  --run-dir out/demo \
  --out-png out/demo/plots/ranked_bars.png \
  --decision PASS \
  --group-by-module \
  --left-strip \
  --strip-labels \
  --bar-color-mode module

Packed circles (modules → terms)

llm-pathway-curator plot-ranked \
  --mode packed \
  --run-dir out/demo \
  --out-png out/demo/plots/ranked_packed.png \
  --decision PASS \
  --term-color-mode module

Packed circles (direction shading)

llm-pathway-curator plot-ranked \
  --mode packed \
  --run-dir out/demo \
  --out-png out/demo/plots/ranked_packed.direction.png \
  --decision PASS \
  --term-color-mode direction

Consistent module labels/colors across plots

plot-ranked assigns a single module display rank (M01, M02, ...) and a stable module color per module_id, so bars and packed circles can be placed side-by-side without label/color drift.

⚖️ Inputs (contracts)

EvidenceTable (minimum required columns)

Each row is one enriched term.

Required columns:

  • term_id, term_name, source
  • stat, qval, direction
  • evidence_genes (supporting genes; TSV uses ; join)

Sample Card (study context)

Structured context record used for proposal and context gating, e.g.:

  • condition/disease, tissue, perturbation, comparison

Adapters for common tools live under src/llm_pathway_curator/adapters/.

🔧 Adapters (Input → EvidenceTable)

Adapters are intentionally conservative:

  • preserve evidence identity (term × genes)
  • avoid destructive parsing
  • keep TSV round-trips stable (contract drift is treated as a bug)

See: src/llm_pathway_curator/adapters/README.md

🛡️ Decisions: PASS / ABSTAIN / FAIL

LLM-PathwayCurator assigns decisions by mechanical audit gates:

  • FAIL: auditable violations (evidence-link drift, schema violations, contradictions, forbidden fields, etc.)
  • ABSTAIN: non-specific, under-supported, or unstable under perturbations / stress tests
  • PASS: survives all enabled gates at the chosen operating point (τ)

Important: the LLM (if enabled) never decides acceptance. It may propose candidates; the audit suite is the decider.

🧪 Built-in stress tests (counterfactuals without external knowledge)

  • Context swap: shuffle study context (e.g., BRCA → LUAD) to test context dependence
  • Evidence dropout: randomly remove supporting genes (seeded; min_keep enforced)
  • Contradiction injection (optional): introduce internally contradictory candidates to test FAIL gates

These are specification-driven perturbations intended to validate that the pipeline abstains for the right reasons, with stress-specific reason codes.

♻️ Reproducibility by default

LLM-PathwayCurator is deterministic by default:

  • fixed seeds (CLI + library defaults)
  • pinned parsing + hashing utilities
  • stable output schemas and reason codes
  • run metadata persisted to run_meta.json (and runner-level manifest.json when used)

Paper-side runners (e.g., paper/scripts/fig2_run_pipeline.py) orchestrate reproducible sweeps and do not implement scientific logic; they call the library entrypoint (llm_pathway_curator.pipeline.run_pipeline).

📦 Installation

PyPI PyPI version Docker (GHCR) Jupyter

Option A: PyPI (recommended)

pip install llm-pathway-curator

(See PyPI project page: https://pypi.org/project/llm-pathway-curator/)

Option B: From source (development)

git clone https://github.com/kenflab/LLM-PathwayCurator.git
cd LLM-PathwayCurator
pip install -e .

🐳 Docker (recommended for reproducibility)

We provide an official Docker environment (Python + R + Jupyter), sufficient to run LLM-PathwayCurator and most paper figure generation.
Optionally includes Ollama for local LLM annotation (no cloud API key required).

  • Option A: Prebuilt image (recommended)

    Use the published image from GitHub Container Registry (GHCR).

    # from the repo root (optional, for notebooks / file access)
docker pull ghcr.io/kenflab/llm-pathway-curator:official

    Run Jupyter:

    docker run --rm -it \
  -p 8888:8888 \
  -v "$PWD":/work \
  -e GEMINI_API_KEY \
  -e OPENAI_API_KEY \
  ghcr.io/kenflab/llm-pathway-curator:official

    Open Jupyter: http://localhost:8888

    (Use the token printed in the container logs.)

    Notes:

    For manuscript reproducibility, we also provide versioned tags (e.g., :0.1.0). Prefer a version tag when matching a paper release.

  • Option B: Build locally (development)

    • Option B-1: Build locally with Compose (recommended for dev)
      # from the repo root
docker compose -f docker/docker-compose.yml build
docker compose -f docker/docker-compose.yml up

      B-1.1) Open Jupyter

      B-1.2) If prompted for "Password or token"

      • Get the tokenized URL from container logs: 
        docker compose -f docker/docker-compose.yml logs -f llm-pathway-curator
      • Then either:
        • open the printed URL (contains ?token=...) in your browser, or
        • paste the token value into the login prompt.
    • Option B-2: Build locally without Compose (alternative)
      # from the repo root
docker build -f docker/Dockerfile -t llm-pathway-curator:official .

      B-2.1) Run Jupyter

      docker run --rm -it \
  -p 8888:8888 \
  -v "$PWD":/work \
  -e GEMINI_API_KEY \
  -e OPENAI_API_KEY \
  llm-pathway-curator:official

      B-2.2) Open Jupyter

🖥️ Apptainer / Singularity (HPC)

  • Option A: Prebuilt image (recommended)

    Use the published image from GitHub Container Registry (GHCR).

    apptainer build llm-pathway-curator.sif docker://ghcr.io/kenflab/llm-pathway-curator:official

  • Option B: a .sif from the Docker image (development)

    docker compose -f docker/docker-compose.yml build
apptainer build llm-pathway-curator.sif docker-daemon://llm-pathway-curator:official

Run Jupyter (either image):

apptainer exec --cleanenv \
  --bind "$PWD":/work \
  llm-pathway-curator.sif \
  bash -lc 'jupyter lab --ip=0.0.0.0 --port=8888 --no-browser

🤖 LLM usage (proposal-only; optional)

If enabled, the LLM is confined to proposal steps and must emit schema-bounded JSON with resolvable EvidenceTable links.

Backends (example):

  • Ollama: LLMPATH_OLLAMA_HOST, LLMPATH_OLLAMA_MODEL
  • Gemini: GEMINI_API_KEY
  • OpenAI: OPENAI_API_KEY

Typical environment:

export LLMPATH_BACKEND="ollama"   # ollama|gemini|openai

Deterministic settings are used by default (e.g., temperature=0), and runs persist prompt/raw/meta artifacts alongside run_meta.json.

📄 Manuscript reproduction

paper/ contains manuscript-facing scripts, Source Data exports, and frozen/derived artifacts (when redistributable).

🧾 Citation

If you use LLM-PathwayCurator, please cite:

About

Transform enrichment outputs into verifiable pathway claims via stability distillation, evidence modules, and mechanical PASS/ABSTAIN/FAIL audits.

llm-pathwaycurator.readthedocs.io/

Topics

reproducible-research computational-biology quality-assurance pathway-analysis enrichment-analysis gsea bioinformatics-tool abstention selective-prediction metascape llm fgsea

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases 1

v0.1.0 — Paper release (decision-grade pathway claims) Latest
Feb 17, 2026

Packages

 
 
 

Languages