annopi-ts

YAML Pipelines → Shell Scripts → DAG Execution

TypeScript rewrite of annopi — a bioinformatics workflow engine that expands pipeline.yml and project.yml into shell scripts and tasks.yml, then runs them locally via ata or on SGE clusters via annotask.

Get Started Installation Guide

Why annopi-ts

Bioinformatics pipelines need reproducible orchestration without rewriting glue scripts for every project.

Common Pain Points

📝

Ad-hoc Shell Scripts

Every project reinvents sample loops, dependency ordering, and resume logic in brittle bash.

🔀

Local vs Cluster Divergence

The same task needs different wrappers for local execution and SGE submission — easy to get out of sync.

🐍

Python-Only Runtime

The original annopi works well but a TypeScript engine enables tighter integration with Node tooling and annovibe.

How annopi-ts Helps

📋

Declarative YAML

Define tasks, params, dependencies, and resources once. annopi conf expands them per sample and writes shell/*.sh.

⚙️

Unified Runners

local mode uses ata or annotask with -i/-l/-t (no local subcommand). qsubsge mode uses annotask qsubsge with cluster options.

🔄

Python ↔ TS Interop

Python annopi and annopi-ts can read the same configs and execute each other's tasks.yml.

Features

🧩

ParamResolver

Expands single, sample, and cmp parameter modes with cross-references and optional blocks.

📦

Task Modules

Load tasks via imports, path, or inline definitions with deps merge from pipeline and modules.

📜

Script Generation

Produces annotask-compatible shell/*.sh and tasks.yml with runcmd for each numbered task.

✅

.sign Checkpoints

Resume pipelines by writing .sign files — run skips completed tasks automatically.

🖥️

CLI

annopi conf, annopi run, annopi install — same surface as Python annopi for V1 scope.

📊

Report (planned)

Report rendering is out of V1 scope; design targets ../rst instead of Jinja2 + Sphinx.

How It Works

pipeline.yml + project.yml

name: scRNA_pipeline
tasks:
  cellranger:
    command: |
      cellranger count --id=${sample_id}
    resources:
      executor: qsubsge
  qc:
    command: qc_script.py --input ${input_dir}
    resources:
      executor: local
dependencies:
  qc: [cellranger]

Generated tasks.yml

tasks:
  1-0-cellranger:
    runcmd: annotask qsubsge -i shell/1-0-cellranger.sh ...
    depends: []
  2-0-qc:
    runcmd: ata -i shell/2-0-qc.sh -l 4 -t 10
    depends: [1-0-cellranger]

Runner paths (ata / annotask)

# local: no "local" subcommand
${localRunner} -i script.sh -l N -t M

# qsubsge: annotask subcommand
${annotaskRunner} qsubsge -i script.sh \
  -l N -t M --cpu C --queue Q

Global deps in pipeline

deps:
  ata: /Volumes/data/GOPATH/bin/ata
  annotask: /opt/annotask
  cellranger: /opt/cellranger-7.0.0

Workflow

pipeline.yml

→

conf

→

shell/*.sh

→

tasks.yml

→

run

→

.sign

Architecture

annopi-ts splits pure logic (core) from I/O and runtime (node), exposing a CLI for conf / run / install.

pipeline.yml + project.yml
        │
        ▼
  @seqyuan/annopi-core   (parse, validate, resolve, DAG)
        │
        ▼
  @seqyuan/annopi-node    (generate shell/*.sh, tasks.yml, run)
        │
        ▼
  annopi CLI              (conf / run / install)

Installation

Prerequisites

Node.js ≥ 18, pnpm, and ata for local execution. annotask is required for qsubsge mode.

Install

Requires Node.js ≥ 18. Use pnpm add -g @seqyuan/annopi if you prefer pnpm.

npm install -g @seqyuan/annopi

Configure a Pipeline

annopi conf \
  -p tests/fixtures/annopi/pipeline.yml \
  -c tests/fixtures/annopi/project.yml \
  -o /tmp/annopi-out

Run Tasks

annopi run -o /tmp/annopi-out

Completed tasks get a .sign file in shell/. Re-run resumes from pending tasks.

关注公众号获取更新

微信公众号搜索 seqyuan，查看 annopi、ata、annotask 与生信流程编排教程。

Quick Start Commands

# Install
npm install -g @seqyuan/annopi

# Generate scripts and tasks.yml
annopi conf \
  -p tests/fixtures/annopi/pipeline.yml \
  -c tests/fixtures/annopi/project.yml \
  -o /tmp/annopi-out

# Execute pipeline (resume via .sign files)
annopi run -o /tmp/annopi-out

# Optional: set global runner paths in pipeline.yml
# deps:
#   ata: /path/to/ata
#   annotask: /path/to/annotask