Long Ranger2.0, printed on 11/25/2024
Specifying the --uiport=3600 option when using the longranger run or longranger demux command will enable the pipeline's visual user interface (UI). This UI is accessible through a web interface that runs on the given port (3600 when using --uiport=3600) on the machine where the pipeline was started.
While a pipeline is running with the UI enabled, you can open http://localhost:3600/ in your web browser (assuming you are browsing from the same workstation that is running Long Ranger) to view the pipeline process graph:
Clicking on any of the graph's nodes will reveal more information about that stage in the right pane. This info pane is broken into several sections, and the topmost shows high-level details about the stage's execution. For example, the MARK_DUPLICATES stage would show:
This includes the state of the stage (running, failed, or complete), the fully-qualified stage name (FQName), the directory in which the stage is running (Path), the stagecode that is being run (the location of the Python package being run in the above example), and any information about parameter sweeping that may apply to this stage.
Clicking the arrow next to the stage name (← MARK_DUPLICATES in the above example) will return to the pipeline-level metadata view.
The Sweeping section allows you to view the MRO call used to invoke this stage (invocation) and information about the split and join components of the stage if it was parallelized.
| Martian, the pipeline framework used by Long Ranger, supports parameter sweeping for pipelines. The Forks and Permute fields in the Sweeping section would display information about different parameters being swept, but neither longranger run nor longranger demux perform parameter sweeping. These fields will always contain only trivial information as a result. |
The Argument Bindings and Return Bindings sections display the inputs and outputs of the stage:
In general, only top-level pipeline stages (those represented by rectangular nodes in the process graph) contain return bindings.
The Chunking section displays information about the parallel execution of the stage:
Many stages are automatically parallelized and process different pieces (chunks) of the same input dataset in parallel. In the above example, the input dataset (the BAM file from the SORT_BY_POS stage according to the Argument Bindings section above) was split into 22 chunks for this stage. Chunks 0-16 already completed, 16-19 are in flight, and 20-22 are waiting for CPU and/or memory to become available before running.
Clicking on an individual chunk exposes additional options for viewing metadata about that chunk's execution (Chunk Def) including how much RAM (__mem_gb) and how many cores (__threads) it requests. As with the Sweeping section, additional metadata associated with the chunk execution can also be viewed.
You can also examine pipestances that have already completed using the Long Ranger UI. Assuming your pipestance output directory was /home/jdoe/runs/sample534, simply re-run the longranger run command with the --noexit option passed along with --uiport to re-attach the UI:
$ longranger run --id=sample345 \ --sex=female \ --fastqs=/home/jdoe/runs/HAWT7ADXX/outs/fastq_path \ --indices=SI-GA-A1 \ --reference=/opt/refdata-hg19-2.0.0 \ --targets=/home/jdoe/runs/agilent_exome.bed \ --uiport=3600 \ --noexit Martian Runtime - 2.0.1 Serving UI at http://localhost:3600 2016-05-01 12:00:00 [runtime] Reattaching in local mode. Running preflight checks (please wait)... Pipestance completed successfully, staying alive because --noexit given.
Alternately, you can re-attach using the longranger run command's MRO-mode syntax detailed in the Multi-Flowcell Sample page. Because the _invocation metadata file contains the MRO used to invoke the stage, you can simply pass it to the longranger run command (e.g., sample345/_invocation) along with the pipestance output directory's name (which is the same as the sample name; e.g., sample345):
$ longranger run sample345/_invocation sample345 \ --uiport=3600 \ --noexit Martian Runtime - 2.0.1 Serving UI at http://localhost:3600 2016-05-01 12:00:00 [runtime] Reattaching in local mode. Running preflight checks (please wait)... Pipestance completed successfully, staying alive because --noexit given.
Because longranger run assumes it is resuming an incomplete pipeline job when re-attaching, the pipeline must be valid and preflight checks must still be passed. As such, relocating a complete pipeline may prevent the UI from re-attaching.
If you run pipelines on a server that blocks access to all ports except SSH, you can still access the Long Ranger UI using SSH forwarding. Assuming you have a longranger run pipeline running with --uiport=3600 on cluster.university.edu, you can type the following from your laptop:
$ ssh -NT -L 9000:cluster.university.edu:3600 [email protected] [email protected]'s password:
Upon entering your password (assuming you are [email protected]), the command will appear to hang. However, in the background it has mapped port 9000 on your laptop to port 3600 on cluster.university.edu through the ssh connection for which you just entered your password.
This allows you to go to http://localhost:9000/ in your web browser and access the UI running on cluster.university.edu:3600. Once you are done examining the UI, use Ctrl+C in your ssh -NT -L ... terminal window to terminate this SSH forwarder.
A full explanation of SSH forwarding is beyond the scope of this guide, but the OpenSSH Documentation explains this in greater depth.