HOME  ›   pipelines
If your question is not answered here, please email us at:  ${email.software}

10x Genomics
Chromium Single Cell Multiome ATAC + Gene Exp.

Pipestance Structure

The pipeline output directory, described in Understanding Output, contains all of the data produced by one invocation of a pipeline (a pipestance) as well as rich metadata describing the characteristics of each stage. This directory contains a specific structure that is used by the Martian pipeline framework to track the state of the pipeline as execution proceeds.

Pipeline structure

Cell Ranger's notion of a pipeline is very flexible in that a pipeline can be composed of stages that run stage code or sub-pipelines that may themselves contain stages or sub-pipelines.

Cell Ranger pipelines follow the convention that stages are named with verbs (e.g., ALIGN_READS, MARK_DUPLICATES, FILTER_BARCODES) and sub-pipelines are named with nouns and prefixed with an underscore (e.g., _BCSORTER). Each stage runs in its own directory bearing its name, and each stage's directory is contained within its parent pipeline's directory.

For example, the cellranger-arc mkfastq pipeline has the following process graph:


Directory structure

Every pipestance operates wholly inside of its pipeline output directory. When the pipestance completes, this pipestance output directory contains three outputs: metadata files, the pipestance output file directory, and the top-level pipeline stage directory.

The top-level pipeline stage directory is a stage directory that contains any number of child stage directories as well as one stage output directory for each fork run by that stage. The top-level pipeline stages for Cell Ranger ARC are:

Most of the Cell Ranger ARC pipelines contain single-fork stages, which means there is one fork0 stage output directory within each stage directory. Chunk output directories are a subset of stage output directories that additionally contain runtime information specific to the job or process being run by that chunk (e.g., a process ID or cluster job ID).

For example, the cellranger-arc mkfastq pipeline's pipeline output directory contains the following directory structure:

_logMetadata file
outs/Pipestance output file directory
MAKE_FASTQS_CS/Top-level pipeline stage directory
MAKE_FASTQS_CS/fork0/Stage output directory
MAKE_FASTQS_CS/fork0/files/Stage output files
MAKE_FASTQS_CS/MAKE_FASTQS/fork0/Stage output directory
MAKE_FASTQS_CS/MAKE_FASTQS/fork0/files/Stage output files

Commonly generated metadata

The metadata contained in the pipeline output directory includes

File Name Description
Metadata cache that is populated when a pipestance completes to minimize re-aggregation of metadata
The MRO call used to invoke this pipestance
The log messages that are reported to your terminal window when running cellranger-arc commands
_mrosourceThe entire MRO describing the pipeline with all @include statements dereferenced
_perfDetailed runtime performance data for every stage in the pipestance
_timestampThe start and finish time for this pipestance
_vdrkillA list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted
_versionsVersions of the components used by the pipeline

Stage directories contain stage output directories, stage output files, and the stage directories of any child stages or pipelines.

Stage output directories typically contain:

File Name Contents
files/Directory containing any files created by this stage that were not considered volatile (temporary)
split/A special stage output directory for the step that divided this stage's input into parallel chunks
chnkN/A chunk output directory for the Nth parallel chunk executed
join/A special stage output directory for the step that recombined this stage's parallel output chunks into a single output dataset again
_completeA file that, when present, signifies that this stage has successfully completed
_errorsA file that, when present, signifies that this stage failed. Contains the errors that resulted in stage failure.
_invocationThe MRO call used to execute this stage by the Martian framework
_outsThe output files generated by this stage
_vdrkillA list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted

Chunk output directories are a subset of stage output directories that, in addition to the aforementioned stage output, may contain:

File Name Contents
_argsThe arguments passed to the stage's stage code
_jobinfoMetadata describing the stage's execution, including performance metrics, job manager jobid and jobname, and process ID
_jobscriptThe script submitted to the cluster job manager (cluster mode-only)
_stdoutAny stage code output that was printed to the stdout stream
_stderrAny stage code output that was printed to the stderr stream

These metadata files should be treated as read-only, and altering the contents of metadata files is not recommended.

Navigating pipestances

Pipestance output directories can demonstrate very complicated structures, and re-attaching the Cell Ranger ARC UI is the easiest way to quickly navigate to a pipeline stage of interest and examine its metadata. In the absence of being able to access the UI, the standard find command can quickly return high-level information about a pipestance.

For example, to find the stages that resulted in the overall failure of a pipestance whose output directory is sample345/,

$ find sample345/ -name _errors

This tells us that the failed stage was COUNT_CUT_SITES.

It can be helpful to view all _errors files' contents at once by piping to xargs cat:

$ find sample345/ -name _errors | xargs cat
Traceback (most recent call last):
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 659, in _main
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 618, in main
    self._run(lambda: self._module.main(args, outs))
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 589, in _run
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 618, in 
    self._run(lambda: self._module.main(args, outs))
  File "/home/jdoe/cellranger-arc-1.0.0/mro/atac/stages/processing/count_cut_sites/__init__.py", line 119, in main
    contig=args.contig, filename=args.fragments, index=args.fragments_index
  File "/home/jdoe/cellranger-arc-1.0.0/lib/python/atac/tools/io.py", line 136, in parsed_fragments_from_contig
    fragments = pysam.TabixFile(filename, index=index)
  File "pysam/libctabix.pyx", line 351, in pysam.libctabix.TabixFile.__cinit__
  File "pysam/libctabix.pyx", line 404, in pysam.libctabix.TabixFile._open
  File "pysam/libchtslib.pyx", line 516, in pysam.libchtslib.HTSFile.tell
NotImplementedError: seek not implemented in files compressed by method 1

In the above case, the error is an unhandled exception that results in a stack trace and whose cause is not obvious; these sorts of failures should be reported to the 10x Genomics software support team for assistance with diagnosis. (Note: this failure was induced by truncating the fragments file while the pipestance was running.)

Stages whose stage code run external binaries (for example, the ALIGN_AND_COUNT stage which runs STAR) often generate output to their stdout and stderr streams. These messages are captured in the _stdout and _stderr metadata files within the chunk output directories, and combining find and xargs cat to examine their contents can also assist with troubleshooting.