10x Genomics
Chromium Genome & Exome

Long Ranger2.2, printed on 12/26/2024

Pipestance Structure

The pipeline output directory, described in Understanding Output, contains all of the data produced by one invocation of a pipeline (a pipestance) as well as rich metadata describing the characteristics of each stage. This directory contains a specific structure that is used by the Martian pipeline framework to track the state of the pipeline as execution proceeds.

Pipeline Structure

Long Ranger's notion of a pipeline is very flexible in that a pipeline can be composed of stages that run stage code or sub-pipelines that may themselves contain stages or sub-pipelines.

Long Ranger pipelines follow the convention that stages are named with verbs (e.g., TRIM_READS, REPORT_COVERAGE, FILTER_SNPINDELS) and sub-pipelines are named with nouns and prefixed with an underscore (e.g., _ALIGNER, _SNPINDEL_PHASER, _STRUCTVAR_CALLER). Each stage runs in its own directory bearing its name, and each stage's directory is contained within its parent pipeline's directory.

For example, the longranger demux pipeline has the following process graph:

where

• BCL_PROCESSOR_CS is the top-level pipeline stage
• BCL_PROCESSOR is a sub-pipeline contained in BCL_PROCESSOR_CS
• BARCODE_AWARE_BCL2FASTQ, DEMULTIPLEX, GZIP_FASTQ, and ANALYZE_RUN are stages contained in the BCL_PROCESSOR sub-pipeline.

Directory Structure

Every pipestance operates wholly inside of its pipeline output directory. When the pipestance completes, this pipestance output directory contains three outputs: metadata files, the pipestance output file directory, and the top-level pipeline stage directory.

• Metadata files are files prefixed with an underscore (_) and usually contain unstructured text or JSON-encoded arrays and hashes.
• The pipestance output file directory is a directory called outs/ that contains the pipestance's output files.
• The top-level pipeline stage directory is a directory named according to the top-level pipeline stage (BCL_PROCESSOR_CS or PHASER_SVCALLER_CS) that contains the child stage directories that compose this pipestance.

The top-level pipeline stage directory (BCL_PROCESSOR_CS or PHASER_SVCALLER_CS) is a stage directory that contains any number of child stage directories as well as one stage output directory for each fork run by that stage. All of the Long Ranger pipelines only contain single-fork stages, so there will only ever be a fork0 stage output directory within each stage directory. Chunk output directories are a subset of stage output directories that additionally contain runtime information specific to the job or process being run by that chunk (e.g., a process ID or cluster job ID).

For example, the longranger demux pipeline's pipeline output directory contains the following directory structure:

Commonly Generated Metadata

The metadata contained in the pipeline output directory includes

Stage directories contain stage output directories, stage output files, and the stage directories of any child stages or pipelines.

Stage output directories typically contain:

Chunk output directories are a subset of stage output directories that, in addition to the aforementioned stage output, may contain:

These metadata files should be treated as read-only, and altering the contents of metadata files is not recommended.

Navigating Pipestances

Pipestance output directories can demonstrate very complicated structures, and re-attaching the Long Ranger UI is the easiest way to quickly navigate to a pipeline stage of interest and examine its metadata. In the absence of being able to access the UI, the standard find command can quickly return high-level information about a pipestance.

For example, to find the stages that resulted in the overall failure of a pipestance whose output directory is sample345/,

$ find sample534/ -name _errors
sample345/PHASER_SVCALLER_CS/PHASER_SVCALLER/_REPORTER/REPORT_BASIC/fork0/join/_errors

This tells us that the failed stage was REPORT_BASIC which is a part of the _REPORTER sub-pipeline. PHASER_SVCALLER is the parent pipeline, and PHASER_SVCALLER_CS is the trivial top-level pipeline that wraps PHASER_SVCALLER.

It can be helpful to view all _errors files' contents at once by piping to xargs cat:

$ find sample534/ -name _errors | xargs cat
 
Traceback (most recent call last):
  File "/home/jdoe/longranger-2.2.2/martian-cs/2.2.2/adapters/python/join.py", line 21, in 
    martian.run("martian.module.join(args, outs, chunk_defs, chunk_outs)")
  File "/home/jdoe/longranger-2.2.2/martian-cs/2.2.2/adapters/python/martian.py", line 380, in run
    exec(cmd, __main__.__dict__, __main__.__dict__)
  File "", line 1, in 
  File "/home/jdoe/longranger-2.2.2/pipelines-cs/2.2.2/mro/stages/reporter/report_basic/__init__.py", line 102, in join
    compute_summary_metrics( misc_sm, qual_sms )
  File "/home/jdoe/longranger-2.2.2/pipelines-cs/2.2.2/mro/stages/reporter/report_basic/__init__.py", line 562, in compute_summary_metrics
    fraction_fragments_on_target = float(targ_dists[0]) / total_reads
KeyError: 0

In the above case, the error is an unhandled exception whose cause is not obvious; these sorts of failures should be reported to the 10x software support team for assistance with diagnosis.

Stages whose stage code run external binaries (for example, the CALL_SNPINDELS stage which runs FreeBayes or GATK) often generate output to their stdout and stderr streams. These messages are captured in the _stdout and _stderr metadata files within the chunk output directories, and combining find and xargs cat to examine their contents can also assist with troubleshooting.