10x Genomics

Data Transfers to 10x Genomics

This page describes how authorized customers and partners can quickly and efficiently send large datasets, such as BCL, FASTQ, and BAM files, to 10x Genomics. This approach makes it easy to use multi-stream transfers to ensure optimal and secure utilization of your Internet connection. Redstone uploads to Amazon Web Services S3 which is secured using SSL during transport.

Prepare Your Data

Illumina BCL Data

In the following example, replace the strings in red with the specifics of your run. /mnt/runs should be replaced with the parent folder that contains your Illumina run folders.

$ tar -czvf H3NVFBBXX.tar.gz -C /mnt/runs --exclude=Thumbnail_Images \
150929_ST-K00126_0004_AH3NVFBBXX

FASTQ or BAM Data

In the following example for a FASTQ directory, replace the strings in red with the specifics of your run. /fastq/directory should be replaced with actual directory that contains your FASTQ files.

$ tar -czvf compressed-fastqs.tar.gz /fastq/directory

In the following example for a FASTQ directory, replace the strings in red with the specifics of your run. /fastq/directory should be replaced with actual directory that contains your FASTQ files. The difference here is omitting a 'z' from the tar command.

$ tar -cvf allfastqs.tar /fastq/directory

Imaging Data

Example Slide: Lot# V19L22, ID# 001.
The four capture areas (from top to bottom) would be submitted as images named:
V19L22-001-A1.tif
V19L22-001-B1.tif
V19L22-001-C1.tif
V19L22-001-D1.tif

Download the 10x Genomics Upload Tool

10x Genomics provides an easy-to-use multi-stream upload tool called Redstone. If you installed Cell Ranger v4 or higher, the Redstone executable is already located in the Cell Ranger install path cellranger-x.y.z/lib/bin/redstone and need not be installed again. You may add this path to your .bashrc using export PATH=/path/to/cellranger-x.y.z/lib/bin:$PATH or type in the full path on your command prompt to call Redstone.

The download link below is for the Linux version, compatible with RedHat/CentOS 5.2 or later, and Ubuntu 8.04 or later.

Redstone is a single 9MB executable that can be run directly and requires no compilation or installation. Place the executable file in a directory that is on your PATH. For this Linux version please make sure to chmod 700 to make it executable.

Upload Your File

Make sure to perform this step from a computer with an unencumbered path to your Internet connection. That means hard-wired 1GbE or 10GbE connections, and not Wi-Fi.

Please also ensure the time/date are set correctly on your system, which is required for security reasons. If your clock is mis-set, you will encounter a RequestTimeTooSkewed error.

To run Redstone, just pass it your email address, the email address of the recipient at 10x Genomics, and the name of the file to be uploaded. Code in red must be customized before running.

$ redstone --from=[email protected] --to=[email protected] H3NVFBBXX.tar.gz
Uploading file...
Size: 10,773,244,733 bytes
 50% [=============>            ]  5,386,622,366   376.79Mb/s eta 1m 54s

Download the 10x Genomics Upload Tool

10x Genomics provides an easy-to-use multi-stream upload tool called Redstone. The link below is for the macOS version.

Download redstone-mac and Open a Terminal Window

After you click the link to download redstone, the executable will download to your Downloads folder. Right click on the file and select Open. You will likely see this message on a pop-up window:

macOS cannot verify the developer of “redstone-mac”. Are you sure you want to open it?

Click Open. It is not a .dmg file so the installer will try to open the application in your default text editor. This file is not human-readable so you can close it. However, this step is necessary because it changes how the macOS gatekeeper views the file, from unsafe to safe.

Next, use the terminal program on your Mac to open a terminal window. A window will open with a prompt that looks something like this:

user.name@computer:~$

Make redstone-mac Executable

Use the chmod command to make the redstone-mac file executable.

user.name@computer:~$ chmod 700 Downloads/redstone-mac

Move redstone-mac to Applications

Move the redstone program to the Applications folder on your computer using the mv command.

user.name@computer:~$ mv Downloads/redstone-mac /Applications

Add redstone-mac to the PATH

Next we need to add redstone to your PATH using the export command. This will allow you to type redstone-mac on the command line to run the program.

user.name@computer:~$ export PATH=/Applications:$PATH

Run redstone-mac to get the Usage

Now if you type redstone-mac on the command line and push enter it will print the usage statement. The usage statement will contain a pattern for building the command you will use to transfer your data.

user.name@computer:~$ redstone-mac
Usage:
    redstone --from=EMAIL --to=EMAIL  [options]
    redstone -h | --help | --version

Transfer a Single File

If you are sending a single file, such as a tissue optimization cDNA footrpint tiff, you would replace in the usage statement above with a "path" to the the file. A "path" is the series of folders that you would go through on your computer to get to the file. A trick that works on macOS is to find the folder you want to send in finder and drag and drop it into the terminal window. If I wanted to send an image downloaded from a shared drive into the Downloads folder, the command would look like this:

user.name@computer:~$ redstone-mac --from=[email protected] --to=[email protected] /Users/username/Downloads/V19L22-001-A1.tif
Uploading file...
Size: 10,773,244,733 bytes
 50% [=============>            ]  5,386,622,366   376.79Mb/s eta 1m 54s

Transfer a Compressed Directory of Files

If you are sending a compressed directory of files, like you would have if you compressed multiple images using the tar command described in "Prepare Your Data" above, the final command will have the same structure as sending a single file and may look like this:

user.name@computer:~$ redstone-mac --from=[email protected] --to=[email protected] /Users/username/Downloads/H3NVFBBXX.tar.gz
Uploading file...
Size: 10,773,244,733 bytes
 50% [=============>            ]  5,386,622,366   376.79Mb/s eta 1m 54s

Download the 10x Genomics Upload Tool

10x Genomics provides an easy-to-use multi-stream upload tool called Redstone. The link below is for the windows version.

Download redstone.exe and Open a Command Prompt Window

After you click the link to download redstone, the executable will download to your Downloads folder. Do NOT click on the downloaded file. Installation does not require the windows installer, it will give an error. Instead do the following:

Use the Command Prompt app to open a Command prompt. You can find Command Prompt by searching for it in the start menu search bar. A window will open with a prompt that looks something like this:

C:\Users\username>

Run redstone-exe from the command prompt to get the usage statement

Now if you type downloads\redstone.exe on the command line and push enter it will print the usage statement. The usage statement will contain a pattern for building the command you will use to transfer your data.

C:\Users\username> Downloads\redstone.exe
Usage:
    redstone --from=EMAIL --to=EMAIL  [options]
    redstone -h | --help | --version

Transfer a Single File

To transfer a file the first step is to change directories to the directory that contains the file you want to transfer using the cd command. If the file is in the Downloads directory the cd command would look like this:

C:\Users\username> cd \Users\username\Downloads

The redstone.exe command would then start with a path to the Redstone executable followed by the --to= and --from= email addresses and ending with the name of the file that you want to transfer. The command would look like this:

C:\Users\username> \Users\username\Downloads\redstone.exe --from=[email protected] --to=[email protected] V19L22-001-A1.tif
Uploading file...
Size: 10,773,244,733 bytes
 50% [=============>            ]  5,386,622,366   376.79Mb/s eta 1m 54s

Troubleshooting

Error Message: 'redstone.exe' is not recognized as an internal or external command, operable program or batch file.

This error indicates that redstone.exe program did not download properly, the full path to redstone.exe was not provided at the command prompt, or the name of the executable was misspelled. Please try one of these solutions:

  • Check the spelling of the executable name
  • Run redstone.exe using a full path (Downloads\redstone.exe)
  • Click on the download link again to download the executable again