[Seqan-dev] RazerS3: extreme malloc

[Uwe Appelt <uappelt@clcbio.com>]

Hi @All,

I am currently observing some maybe undesired behavior using razers3, 
trunk version (checked out 29/08/2012, 9am, Berlin time).

Within a Rabema scenario the following is observed (impatient readers go 
straight to (3)):

1) 100,000 reads from Illumina GAII, single-end, 100Nt
../bin/rabema/razers3 -v -tc 32 -of sam -rr 100 -i 90 -m 1000000 -ds -o 
./rabema/NA18507_1_1e5complex.sam ../ref_genomes/hg19.fa 
../reads/NA18507_1_1e5complex.fastq
=> ran entirely and finished successfully.

2) 100,000 454-Titanium reads, ranging from 50Nt up to 3575
../bin/rabema/razers3 -v -tc 32 -of sam -rr 100 -i 90 -m 1000000 -ds -o 
./rabema/NA12878_1e5complex.sam ../ref_genomes/hg19.fa 
../reads/NA12878_1e5complex.fastq
=> didn't yet finish, but steady progress is observed, while 
mem-consumption seems reasonable (below 20Gb).

3) 100,000 IonTorrent reads, ranging from 5Nt up to 387(less than 100 
reads are actually shorter than 50 nukes)
../bin/rabema/razers3 -v -tc 32 -of sam -rr 100 -i 90 -m 1000000 -ds -o 
./rabema/NS12911_1e5complex.sam ../ref_genomes/hg19.fa 
../reads/NS12911_1e5complex.fastq
=> this didn't finish and no progress is observed any more.

### the according STDERR-output is:
___SETTINGS____________
Genome file:                         ../ref_genomes/hg19.fa
Read file: ../reads/NS12911_1e5complex.fastq
Compute forward matches:             YES
Compute reverse matches:             YES
Allow Indels:                        YES
Error rate:                          0.1
Pigeonhole mode with overlap:        0
Shape:                               11111111111
Repeat threshold:                    1000
Overabundance threshold:             1
Program PID:                         29923

25014636 bps of 100000 reads loaded.
Loading reads took                   0.507097 seconds

Number of threads:                   32

Initialization took                  0.05828 seconds

Process genome seq #0[fwd]........1M.
[###EOF###]

### the according TOP-output says:
top - 12:41:08 up 49 days, 45 min,  2 users,  load average: 32.80, 
21.50, 10.65
Tasks: 980 total,   3 running, 976 sleeping,   1 stopped, 0 zombie
Cpu(s): 41.2%us,  0.1%sy,  0.0%ni, 58.7%id,  0.0%wa, 0.0%hi,  0.0%si,  
0.0%st
Mem:  132003720k total, 124116252k used,  7887468k free, 98572k buffers
Swap: 32764556k total,    31340k used, 32733216k free, 12365036k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+  COMMAND
29923 uappelt   25   0  136g  97g 1928 R 3197.9 77.3 141:03.96 razers3
29931 uappelt   16   0 11544 1772  780 R  1.0  0.0 0:02.60 top
[###EOF###]

### My according output is: *DOOOOH*[###EOF###]

So sample 3 kind of skrewed the machine. The poor thing is close to 
swapping and CPUs are under heavy loads, while progress didn't move a 
single period further (within minutes). I guess, extremely short reads 
(below 50 nukes) are the culprits to blame, since this phenomenon 
doesn't occur with longer reads in samples 1 & 2. Still, less than 100 
reads are in fact shorter than 50 nukes. And even a million 
reference-genome locations that were asked to be reported per read 
shouldn't cause this, should it.

Is there an easy way to prevent this behavior? Filtering for short reads 
would certainly be an option. Couldn't, however, long reads entirely 
lying within repetitive elements (e.g. SINES or LINES) cause similar 
malloc-behavior? This case couldn't be taken care of so easily from 
outside the read mapper, could it.

Let me know, if you want reads of sample 3 to be emailed somewhere 
(gzipped file size: 22,558 KB)

Cheers,
Uwe