Chromosomes, compressed: the computing race behind the genome

A finished human genome is, in the end, a text file. Getting there in 2000 took two of the largest computing efforts biology had ever attempted — one public, one private, racing each other to the same finish line.

A genome that fits on a CD

The number itself is almost deceptively small. The human genome runs to roughly three billion base pairs — and because one million base pairs of sequence works out to about one megabyte of plain text, the entire finished reference genome fits in roughly three gigabytes^cite — small enough to burn onto a single CD.

That tidy final number hides a much messier process. Nobody sequenced the genome in one pass, start to finish. Sequencing machines produced millions of short, overlapping fragments — and turning that pile of fragments into 23 continuous chromosomes was, in the end, a computing problem as much as a biological one.

Two clusters, one finish line

By the late 1990s the public consortium — the same one described in an earlier Note — had generated enormous volumes of raw fragment data, but lacked a way to assemble it into a working draft. In the spring of 2000, Jim Kent, a graduate student at UC Santa Cruz, wrote an assembly program in about a month, working long days in a converted garage office. It became known as GigAssembler, roughly 10,000 lines of code, run on a cluster of Linux workstations his UCSC colleague David Haussler had to secure emergency funding to buy.

Contemporary accounts put the cluster at up to 100 Pentium III machines^cite — though Wikipedia's own account gives a more modest 50; the records don't fully agree, and it's not clear anyone counted precisely at the time. Kent's first complete assembly came together on 22 June 2000, just days before the public announcement at the White House — and, narrowly, ahead of the other side.

That other side was Celera Genomics, the private company racing to sequence the genome independently. Celera's computing effort was on a different scale entirely: by its own IT department's account, the company ran about 800 processors' worth of Compaq ES40 servers^cite, built in clustered modules so capacity could shift between sequencing support and raw number-crunching as needed. It was, for its moment, one of the largest civilian computing installations outside a national lab.

The tools the project left behind

Assembling a genome from fragments needs more than raw processing power; it needs algorithms built for the job, and several that the project either created or pushed hard were general-purpose enough to outlive it by decades. BLAST, for comparing one sequence against millions of others, predates the Human Genome Project but became indispensable to it. Phred and Phrap, developed to score how trustworthy each base call was and then assemble reads accordingly, became close to a standard. Kent went on, after GigAssembler, to write BLAT and the UCSC Genome Browser — tools that are still in everyday use in genomics labs today, including this one.

Where the data went

Under the Bermuda Principles agreed in 1996, the sequence had to be made public almost as fast as it was produced — which meant the receiving end needed to scale too. GenBank in the US, the European Nucleotide Archive, and Japan's DNA Data Bank each expanded to hold and mirror the incoming sequence, laying down the public-database infrastructure that genomics still runs on, long after the project that forced the expansion was finished.

Then and now

It's a useful contrast to hold alongside the one in the Note on early sequencing chemistry: there, the method that read the first human genome was essentially 1977 chemistry, barely changed. Here, the computing wasn't borrowed from anywhere — biology was, by its own admission, setting the pace for an entire industry. Two clusters, a garage office, and a month of one graduate student's life turned out to matter about as much as any single sequencing centre.

Sources: PBS/NOVA's companion facts page for "Cracking the Code of Life" (2001); Dr Dobb's Journal profile of Jim Kent (2001); Computerworld's report on Celera's server infrastructure, quoting Celera VP John Reynders directly. The exact size of the UCSC assembly cluster (50 vs 100 machines) is reported inconsistently across sources and is presented here as uncertain rather than resolved.