Our Technological Future: Star Trek or Terminator? 
    A Warning from Bill Joy of Sun Microsystems
    (10/2000)

    by Douglas Dixon

In a controversial cover story in the April issue of Wired magazine, Bill Joy, corporate executive officer and chief scientist at Sun Microsystems, Inc., challenged technologists to consider the moral issues of their work. Joy's article, "Why the Future Doesn't Need Us," warns that the most powerful 21st century technologies - robotics, genetic engineering, and nanotechnology - have the potential to make humans an endangered species (www.wired.com/wired/archive/8.04/joy.html).

Joy's manifesto has made him the front man on this issue, as he calls for a broader public discussion of the risks of technology, and even the possibility of voluntary relinquishment of scientific investigation and technological progress in these areas. Since his article was published, he has continued to address these issues in follow-up articles and public talks, at a rate of at least one per week.

Joy's article has been compared to Einstein's 1939 letter to President Roosevelt alerting him of the possibility of a nuclear bomb. Joy points to the Pugwash Conferences, which have been held since 1957 to discuss arms control, as a model of how to address these issues. "It's unfortunate that the Pugwash meetings started only well after the nuclear genie was out of the bottle - roughly 15 years too late. We are also getting a belated start on seriously addressing the issues around 21st-century technologies - the prevention of knowledge-enabled mass destruction - and further delay seems unacceptable," he says in the Wired article.

Bill Joy

Joy is no average Joe. He received his B.S. in electrical engineering from the University of Michigan in 1975, and an M.S. in electrical engineering and computer science from the University of California, Berkeley, where he led the design of the Berkeley version of the Unix operating system. Berkeley Unix was an early example of the almost viral spread of software in the computer industry, since it was made widely available and in open source form for others to learn from and improve. As a result, it became the standard in education and research, and the common reference point for a generation of researchers and programmers.

Joy left Berkeley in 1982 as a cofounder of Sun Microsystems. He has spearheaded Sun's open systems philosophy and has lead its technical strategy in both software and hardware. He became chief scientist in 1998.

Called by Fortune magazine "The Edison of the Internet," Joy, was co-chair since 1997 of the Presidential Information Technology Advisory Committee; this group provides guidance on all areas of high-performance computing, communications and information technologies to accelerate development and adoption of technologies that will be important in the 21st century.

Corecipient of the Computerworld-Smithsonian award for Innovation in 1999, Joy is a member of the National Academy of Engineering and a fellow of the American Academy of Arts and Sciences.

Rate of Change

Joy was inspired in his thinking by Ray Kurzweil, a prolific inventor and entrepreneur in artificial intelligence technologies including character recognition, music synthesis, speech recognition, and reading machines for the blind (www.kurzweiltech.com). In his recent book, "The Age of Spiritual Machines," Kurzweil paints a glowing picture of the future evolution of mankind and its merger with technology. At the current rate of technological progress, he predicts that a personal computer will match the processing power of the human brain around the year 2020.

This issue of the exponential rate of change is the underlying theme of Kurzweil's predictions, and was the wake-up call to Joy. "We will see 1,000 times more technological progress in the 21st century than we saw in the 20th," says Kurzweil. "It's remarkable how people fail to internalize the implications of this."

But we live in linear time, living from day to day and year to year, and have difficulty appreciating how exponential growth will cause dramatic changes in only 10 or 20 years, changes that we can only imagine today as far-out science fiction. This is difficult even for technologists and scientists; the recent success in sequencing DNA is only one example of this, achieved much sooner than predicted by the experts, or even thought possible only a few years ago.

The rate of change in computing is described by Moore's law, named for Gordon Moore of Intel, which has correctly predicted the exponential rate of improvement of semiconductor technology for decades. And we have all experienced this improvement, as processor speed has continued to double and redouble every year or two, jumping from 5 to 20 to 1000 MHz (million operations per second) over the past two decades.

If you have forgotten, the original IBM PC, introduced in 1981, had an Intel 8088 processor running at 4.77 MHz. The IBM PC / AT introduced in 1994 had an Intel 286 processor that doubled the speed at 6 and then 8 MHz. By 1990, the Intel 386 had redoubled to 20 MHz, followed by the 486 in 1992 at 33 MHz. And then the exponential curve started really exploding: from the Pentium in 1994 at 100 MHz, to the Pentium MMX in 1997 at 200 MHz, the Pentium II in 1998 at 333 MHz, the Pentium III in 1999 at 733 MHz, the Pentium III Xeon in 2000 at 933 MHz, and finally to today's processors starting to cross the GHz boundary, at a billion operations a second.

This is nicely on target with the prediction, doubling every 2 years for 20 years: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024. Now, imagine this rate of change continuing for the next 20 years, with speeds for common personal computers growing from billions to trillions of operations a second. And it's not just processor speed changing at this rate, but all areas of technology, from storage capacity, to communications bandwidth, to shrinking component sizes. This kind of growth is not just seen in computer technology, but in other areas like biotechnology as well.

Some have argued that the end is in sight for this accelerating growth in performance and shrinking of components, that chips are reaching physical and atomic limits. But Kurzweil points out that computer design has reached physical limits before, yet has actually continued on this curve throughout the century because of a series of paradigm shifts to new technologies, from electro-mechanical, to relays, vacuum tubes, transistors, and silicon. And the technological basis for the next paradigm shift is in sight: molecular and optical computing.

Once you really accept the implications of this exponential rate of growth, as Joy has done, then it becomes clear that what were once thought of as wild science-fiction scenarios will happen, not only in our lifetime, but in the next decades.

"Because of the recent rapid and radical progress in molecular electronics - where individual atoms and molecules replace lithographically drawn transistors - and related nanoscale technologies," writes Joy in Wired, "we should be able to meet or exceed the Moore's law rate of progress for another 30 years. By 2030, we are likely to be able to build machines, in quantity, a million times as powerful as the personal computers of today."

Joy's Concerns

Kurzweil has a utopian view of these possibilities for this future in his book, optimistically subtitled "When computers exceed human intelligence." But Joy is less sanguine, arguing that concerns about these kinds of 21st century technologies are more serious than even the nuclear technology of the 20th century:

"The 21st-century technologies - genetics, nanotechnology, and robotics - are so powerful that they can spawn whole new classes of accidents and abuses. Most dangerously, for the first time, these accidents and abuses are widely within the reach of individuals or small groups. They will not require large facilities or rare raw materials. Knowledge alone will enable the use of them."

"Thus we have the possibility not just of weapons of mass destruction but of knowledge-enabled mass destruction, this destructiveness hugely amplified by the power of self-replication."

Genetics, Nanotechnology, Robotics

Joy's concern with genetic engineering seems more real to us today because of recent headlines. "Genetic engineering technology is already very far along," he writes. "The USDA has already approved about 50 genetically engineered crops for unlimited release; more than half of the world's soybeans and a third of its corn now contain genes spliced in from other forms of life."

The promise of genetic engineering seems wonderful, from increasing crop yields to creating cures for diseases, and even increasing our life span and our quality of life. But Joy's concern is that it will become too easy, even for individuals, to make new and dangerous life: "It gives the power - whether militarily, accidentally, or in a deliberate terrorist act - to create a White Plague," some kind of new and highly contagious plague that kills widely but selectively, like the disease in the book of the same name by Frank Herbert.

Concern with progress in robotics seems more far-out, given today's feeble progress in machine intelligence, but is an inevitable implication of the rate of progress. Joy describes the dream of robotics as developing "intelligent machines that can do our work for us, allowing us lives of leisure, restoring us to Eden," but once an intelligent robot exists, it is only a small step to a robot species, an intelligent robot that can make evolved copies of itself.

"Given the incredible power of these new technologies," he writes, "shouldn't we be asking how we can best coexist with them? And if our own extinction is a likely, or even possible, outcome of our technological development, shouldn't we proceed with great caution?"

Joy's third concern, nanotechnology, is based on manipulation of matter at the atomic level. Molecular-level "assemblers" could make possible low-cost solar power, augmentation of the human immune system, and almost complete cleanup of the environment.

Joy predicts that "the enabling breakthrough to assemblers seems quite likely within the next 20 years. Molecular electronics should mature quickly and become enormously lucrative within this decade, causing a large incremental investment in all nanotechnologies."

But, he warns, "unfortunately, as with nuclear technology, it is far easier to create destructive uses for nanotechnology than constructive ones. An immediate consequence of the Faustian bargain in obtaining the great power of nanotechnology is that we run a grave risk - the risk that we might destroy the biosphere on which all life depends."

This threat has become known as the "gray goo problem," envisioning the uncontrolled spread of masses of replicators able to obliterate life. "The gray goo threat makes one thing perfectly clear: We cannot afford certain kinds of accidents with replicating assemblers."

Star Trek or Terminator?

So, will the future be the bright hope of Star Trek, where earthling morality guides the stars, or the dismal gloom of The Matrix or Terminator, where machines control the planet and humanity is dead or oblivious to the truth?

Joy's primary purpose in writing the article was to start a discussion: "The new Pandora's boxes of genetics, nanotechnology, and robotics are almost open, yet we seem hardly to have noticed. Ideas can't be put back in a box; unlike uranium or plutonium, they don't need to be mined and refined, and they can be freely copied."

He does not offer much in the way of practical approaches to mitigating these problems, beyond a proposal for relinquishment of research and development in these areas: "If we could agree, as a species, what we wanted, where we were headed, and why, then we would make our future much less dangerous - then we might understand what we can and should relinquish. Otherwise, we can easily imagine an arms race developing over [these] technologies, as it did with the [nuclear] technologies in the 20th century. This is perhaps the greatest risk, for once such a race begins, it's very hard to end it."

This is Joy's challenge to technologists, and society at large:

"Perhaps it is always hard to see the bigger impact while you are in the vortex of a change. Failing to understand the consequences of our inventions while we are in the rapture of discovery and innovation seems to be a common fault of scientists and technologists; we have long been driven by the overarching desire to know that is the nature of science's quest, not stopping to notice that the progress to newer and more powerful technologies can take on a life of its own."

References

Institute for Advanced Study
    www.ias.edu

Sun Microsystems
    www.sun.com

Bill Joy
    www.sun.com/aboutsun/media/ceo/mgt_joy.html

"Why the Future Doesn't Need Us" - Bill Joy, Wired 8.04 , Apr. 2000
    www.wired.com/wired/archive/8.04/joy.html

Kurzweil Technologies, Inc. 
    www.kurzweiltech.com

Ray Kurzweil 
    www.kurzweiltech.com/aboutray.html

- The Age of Intelligent Machines, MIT Press, 1990
 - The Age of Spiritual Machines, When Computers Exceed Human Intelligence 
    (Viking hard cover, Penguin paperback, 1999)
    www.penguinputnam.com/kurzweil/index.htm