In the early days of Silicon Valley, it was actually quite common (at least among far-thinking futurist types who populated the Bay Area) to believe that the coming Singularity would be the single greatest event in the history of the universe, the culmination of the techno-capitalist project that has already done so much good for humanity. In the 1980s and 1990s, the “extropian” community in the Bay Area (a futurist collective committed to fighting entropy and death, which organized around local meetups, an academic institute and a popular mailing list) eagerly awaited the Singularity, believing that a glorious transhuman future — free from all death, suffering, and scarcity — awaited them on the other side. The extropians thought that the interim state of life on Earth — a hotbed of torture, disease, and needless hunger — was a moral catastrophe, and believed that humans had a moral obligation to start the Singularity as soon as possible — in order to bring this nightmare to an end.
But beginning in the early 2000s, the assumption that the Singularity will be a good thing — for human beings, and in general — began to fade. Eliezer Yudkowsky, an autodidact AI researcher, was the first to popularize the argument that superintelligent AI, far from ushering in an era of limitless abundance, would likely optimize away everything that humans care about (including humans) in favor of cold, senseless, alien goals. Yudkowsky believed that humans were anthropomorphizing AI goals, which could be stranger, and less desirable, than we would naturally assume. In a classic Yudkowskian parable, a paperclip factory owner creates a superintelligent AI and tells it to “make as many paperclips as possible.” The AI then proceeds to turn the entire universe, including the factory owner, and all of the eight billion other people on Earth, into paperclips.