Donald Becker, during the summer of 1997. View Slideshow Who’s afraid of the big bad Beowulf?
No one now, but 10 years ago the scientific community greeted the first Beowulf supercomputer cluster with fear and loathing. “The initial reaction of the supercomputer-oriented scientific community to the Beowulf project was very negative,” says Donald Becker, co-founder of the original Beowulf project.
“The reaction went far beyond the expected ‘prove it’ indifference to active opposition to any development that would make commodity clusters a better platform,” he says. “Curiously, it seems that everyone has now been transformed into Beowulf supporters.”
Becker was one of the attendees at a party held Wednesday night in San Francisco to celebrate the 10-year anniversary of Beowulf and to unveil the newly redesigned Beowulf project website.
Named for the hero of the epic poem written around 1000 about a man who slays a monster, Beowulf clusters are supercomputers that are built by linking individual, commodity (off-the-shelf) computers together, using an open-source architecture – developed in part by Becker – that enables the machines to work together.
In late 1993, Becker and Thomas Sterling, both working at NASA’s Goddard Space Flight Center, began thinking about the possibilities of cluster computing. In the summer of 1994, Wiglaf – the first Beowulf cluster, sporting 16 66-Mhz 486 DX4 processors connected by channel-bonded ethernet – was born. It cost about $40,000 to build.
“There were several converging ideas that triggered the start of the Beowulf project,” Becker says. “First was the observation that commodity PC-class machines were improving their price-performance at a much faster rate than other types, especially traditional supercomputers.
“Second was the recognition that the key to using PC-class machines to supplement supercomputers was developing a common, community-driven software system. And third, by late 1993 Linux was a reliable, network-capable operating system. As a Linux developer working on the first network code for Linux, I saw both the technical advantages of the design and the important advantages of Internet-centered collaborative development.”
At Beowulf’s birthday party, old photos of early Beowulf machines were projected on a screen and conversations ranged from debates about the ubiquitous “open-source beard” – that is, the open-source programmers’ preference for facial hair – and the sad demise of so many Linux startups during the dot-com bust. Sterling spoke about the first time he met Becker 22 years ago, when Becker was a freshman at MIT.
Becker was seeking a job and Sterling said his mission was to “get rid of Donald by asking him a series of difficult computing questions that would certainly make him go away.”
But Becker would not be deterred and he returned weeks later with his responses to Sterling’s questions, as well as schematics for a computing architecture intended to solve a problem that had stumped his more seasoned peers, including Sterling. The two have worked together ever since.
There were a few misty eyes at the party when the 10 candles on Beowulf’s birthday cake were blown out. There was also a brief indication that all that emotion might devolve into a food fight, but the desire to eat cake rather than fling it quickly prevailed.
Becker said time has proven that there are several obvious advantages to Beowulf clusters: He believes they offer the best performance bang for the buck, as well as approachable software that enables almost anyone to build his or her own cluster. And the ready availability and affordability of powerful off-the-shelf computers is a new benefit that allows Beowulf clusters to become significantly more capable.
“I see now that when we were initially thinking about the benefits Beowulf would provide we missed one of the most important elements – clusters are incrementally scalable,” said Becker. “Unlike custom-designed supercomputer systems that are designed as large machines, you can start with a small cluster and scale it as the demand grows.”
“This isn’t as important in R&D labs, where machines are typically bought as a complete installation in a long procurement cycle, but it’s a good fit to how people actually use scientific computers. They have an initial round of understanding what they are trying to accomplish, followed by increasingly complex problems.”
Not all Beowulf clusters are supercomputers – a Beowulf cluster can be as simple as two computers linked together. And not all clusters run open-source software – Apple-powered clusters are an option that some Mac fans are exploring.
But purists insist it isn’t a real Beowulf cluster if it’s not running open source. “Part of the original definition (of Beowulf) was ‘running open-source software,'” Becker said. “Over time we have relaxed that to ‘running an open-source infrastructure.’ But I still believe that open source is a critical element of a stable infrastructure.
“We can never hope to have all software tools built exclusively for clusters. Open source allows us to examine the software, not necessarily to change it but rather to verify that the unmodified software will work correctly in a cluster environment. With closed-source software tools we could only guess that software that superficially appears to work will actually work.”