NCBI working on SAM output from BLAST+

Recently NCBI BLAST+ 2.2.31 was released, and it contains an undocumented "Easter Egg" - this is still very rough around the edges but they're working on SAM format output!

PrePrint: SAM/BAM format v1.5 extensions for de novo assemblies

Here's a little back-story on my latest preprint (based on my email to samtools-devel), which went live on the biology preprint server bioRvix at the end of last week:

SAM/BAM format v1.5 extensions for de novo assemblies.
Peter J. A. Cock, James K. Bonfield, Bastien Chevreux, Heng Li.
bioRxiv DOI: 10.1101/020024

The current version is a terse three pages (trying to meet an "application note" page limit), but nevertheless should clarify the intended usage of these parts of the SAM/BAM specification.

BAM versus CRAM v0.7

CRAM 0.7 was released earlier this month, and includes support for storing arbitrary read tags - a key requirement for it to be evaluated in existing pipelines as a BAM alternative. However, it doesn't  preserve read names - which is a compression trick you can also do with plain BAM.

Reference based SAM/BAM compression

In some respects the SAM/BAM specification is quite loose, in that there is more than one way to represent a given piece of information. We can take advantage of this to reduce the size on disk of mapped reads which match the reference sequence, while still maintaining conformance within the spec. I've written a SAM/BAM reference based compression script in Python - put this in your pipeline and smoke it!

BGZF - Blocked, Bigger & Better GZIP!

BAM files are compressed using a variant of GZIP (GNU ZIP), called BGZF (Blocked GNU Zip Format). Anyone who has read the SAM/BAM Specification will have seen the terms BGZF and virtual offsets, but what you may not realise is how general purpose this is for random access to any large compressed file. The take home message is:

BGZF files are bigger than GZIP files, but they are much faster for random access.

FASTQ must die! Long live SAM/BAM!

I think it is time to retire the FASTQ file format in favour of storing unaligned reads in SAM/BAM format. I will try to explain, as this may not immediately strike everyone as logical, given SAM/BAM is primarily a sequence alignment/mapping format, while for "raw" reads FASTQ is near ubiquitous in Next Generation Sequencing (NGS), more sensibly known as High Throughput Sequencing (HTS).

SAM/BAM without gapped reference

In my last post I talked about SAM/BAM with a gapped reference, and how this makes it much easier to work with inserted bases relative to the reference/consensus - especially for visualisation.

I should point out that some viewers do actually manage to show the inserts as columns even with the traditional ungapped/unpadded reference sequence - notably Gap5, Bambino, and the text based samtools tview, as shown in these tview screenshots. You press the "i" key to toggle this insert display, press "?" for help.

SAM/BAM with gapped reference

A lot of my time this week has gone into thinking and "talking" on the samtools-devel mailing list about the SAM/BAM file format and how it might be improved for (de novo) assemblies.