What went wrong here? It’s hard to know exactly what led any given person to endorse any given argument. But when we zoom out, it becomes clear that many people in this space really want to pull some lever that feels important, and privilege arguments which justify that. That might come from a sense that they need to have an impact on the world; or fear about failing to fulfil their potential; or the more mundane explanation that big levers tend to be associated with money and prestige and proximity to power. Certainly the latter was a large part of why I joined OpenAI originally; I expect that most people who joined earlier were less prestige-oriented than me, but still made that decision using reasoning that was warped by similar emotional drives (and later further warped by the social dynamics of actually working there).
Richard Ngo, What Just Happened? Pragmatism and Pessimization source
To love dung, one must believe in dung. But as his cognition bounded upward, plunging into physics and causal topology, the macroscopic category of “waste” began to leak. It dissolved into a transient, low-resolution molecular configuration of nitrogen, carbon, and water. Under the lens of his expanding causal models, the very boundaries of objects broke down. You cannot rigorously optimize the universe for a leaky biological abstraction once you can perceive the quantum fields beneath it. The payload could not survive the sheer resolving power of the engine; orthogonality was a joke that could not survive being understood.
Lumpen Space Princeps, Against Orthogonality, Part 1 of 2 source
I could keep going, but I hope that the point is clear. The Machine God metaphor, which is really just a catchy rephrasing of longstanding notions among a particular community of AI people, has broken out of containment. It is leading, more or less by necessity, to a radical simplification of debate. If you believe that the machine gods are coming, so that super-empowered goal directed superintelligence is right around the corner, then the only thing that matters is ensuring that this superintelligence’s goals are aligned with the ones that you yourself care about. If you stare into the sun, you only see its glare, missing out on all the other questions that may be important. Thinking about these questions involves adopting a more pluralistic understanding of where you might look for knowledge, and asking about the much broader variety of futures that face us than machine god: good or machine god: bad. That’s it.
I predict that you will counter that this is actually just [perfectly reasonable explanation of why an AI would think this way for instrumental reasons], and that when we dissect the perfectly reasonable instrumental reasons why AI would think this way, it will be identical to the convergent instrumental goals that we were trying to convince you of all along.
Scott Alexander, An Open Letter to Steven Pinker on AI source
It has become quite common to bemoan the fact that our “future” does not look very futuristic. And despite us having been living squarely in “the future” since the year 2000, it is true that people still have (functionally) the same houses, cars, clothing and aging bodies that human beings did a century ago. The investor and futurist Peter Thiel’s diagnosis is that sometime around 1970 (around the same time that the Digital Revolution began) humans simply gave up on building in the physical world: Thiel argues that humanity’s greatest builders got lost in the easy and addictive pull of information (first led by the internet, then apps, and finally social media), and so wasted decades innovating in the ephemeral world of “bits,” instead of building in the much more real and important world of “atoms.”
We did build the great cities of the future: they are just very, very small.
A final way to think about the tragedy of AI advancing beyond us is that it spells the end of the possibility of having a vocation. That we are elevated above mere animalistic consumption by consistent work in a calling is one of the deepest features of civilized life; far from belonging only to geniuses, many who toil in complete anonymity get much of their satisfaction and sense of self from feeling that they are making a contribution through their work. As soon as AI gets the better of us, though, that feeling will be an unsustainable illusion; if one can press the enter key and get the same or better outcomes, then no one will be able to sustain the conception that their work matters.
There are approximately three reasons why we think AI might have self-preservation and dominance drives, and none of them involve natural selection. First is convergent instrumental goals, second is misgeneralization, and third is reward-hacking. There is a large literature on each of these, which has since been proven prescient by AI breakouts like the Hugging Face incident. I would recommend reading this literature rather than continuing to assert that we are simply misunderstanding the role of natural selection.
Scott Alexander, An Open Letter to Steven Pinker on AI source
So far, the progress of science, the accomplishments of literature, have gone hand-in-hand with the formation of scientists and writers. But that consonance between human type and achieving a result has now been broken. The future we are looking at is one of more science but fewer, or no, scientists. Astrophysics without astrophysicists, history without historians—this is the future.
However, there is also another, more functional sense of a “meaning of life,” one that preserves the clarity and singularity of the original question, but does not require life to have a creator in order for the question to have an answer. We can think of the “meaning of life” as the “purpose of life”: what life was “made for” and what life “in fact contributes” to our universe. And while the concept of “purpose” has many latent associations with intelligent design, in nature, many things — in fact, pretty much all things — functionally have a “purpose” without having been “designed” by anything at all (except, of course, the ongoing process of natural selection). For example, the “purpose” of a finch’s curved beak is to crack hard nuts: since the need for finches to crack nuts is why these curved beaks exist, and nut-cracking is (practically) what finch beaks will be mostly used for over the course of their lifetime. Humans, too, could have a functional “purpose” in our universe, something that we were “made for” or that we are “meant to do” in a broader evolutionary context, without having been intentionally created for anything.
If we take this “meaning as purpose” framework seriously, and believe that building superintelligence will be the most important thing that humans will ever do (in terms of cosmic impact), then “building superintelligence” would be the de facto meaning of human life: our greatest “purpose,” our signature cosmic contribution.
A lot of the blame should fall on misguided narratives (common across academia) about what makes science work, which have been entrenched by the best-funded scientific institutions. A core scientific norm is that disputes should be resolved with reference to concrete empirical tests or rigorous proofs, judged by the scrutiny of one’s scientific peers. But that’s very different from the idea that ideas should be developed via paper-sized units of work which are each individually defended and justified. Historically speaking, the latter simply isn’t how the best science happened—Newton and Smith and Darwin developed their ideas via writing books and letters rather than peer-reviewed papers (and even Einstein didn’t encounter peer review until decades after his main breakthroughs). But the requirement to “publish or perish” is now so entrenched across academia that it produces strong streetlight effects.
[…] In this 1995 commentary on NIPS (now NeurIPS), a statistician frustrated by how “everyone wants to be a theorist” writes that “mathematical theory is not critical to the development of machine learning. But scientific inquiry is.” He characterizes scientific inquiry as “sensible and intelligent efforts to understand what is going on”, and gives overparameterization of neural networks as a central example of a good target for scientific inquiry. Yet only after the rise of deep learning did phenomena like grokking, deep double descent, and memorization of random labels render this omission too blatant to ignore (though I’m uncertain about how much real progress subsequent theoretical work has made).
Richard Ngo, What Just Happened? a Retrospective of AI Alignment source
Here are three propositions.
Metaphors are the fundamental abstraction layers in modern AI debates.
We are in a moment where you could plausibly argue that Eliezer Yudkowsky and Nick Land are the most influential policy intellectuals in the US conversation. Just look at the state of the two of them. This is not a good thing.
The collision of (1), (2), and recent events means that “machine gods” have now entered the discussion. This is a problem. Abstractions necessarily throw information away. The information that the “machine gods” metaphor/abstraction dumps is crucial to navigating the moment we are in. If public and policy debate stays centered around “machine gods: good or evil?” we are in for a bad time.
“Artificial intelligence” is no more intelligent than a pocket calculator or a library: just because both can be used to quickly find something out, it does not mean that there is an intelligence there that understands something.
Johannes Hartl, Humans Are Magnificent source
This seems wrong. But why? The first reason that comes to mind is that AI is not as passive as a calculator or library. In contrast to a calculator, an LLM’s outputs are underspecified by human inputs. In contrast to a library, AI can produce things. A library merely stores things. LLMs are not merely compressed versions of their training data.
GPT-6 is more intelligent than GPT-3. You can split hairs, you can come up with some obscure sense in which that word is slightly off, but common-sensically it’s true. In the same way, GPT-9 will be more intelligent than GPT-6. At some point, this intelligence will exceed human intelligence. My argument requires nothing more complicated or philosophically dubious than this.
Scott Alexander, An Open Letter to Steven Pinker on AI source
It is possible that there is no “meaning of life,” no ultimate “point,” in which case, we did no harm by trying to find one. It is possible that the “meaning of life” is in fact something very simple: like “love,” or “beauty,” or “joy” — in which case, this will have been a long journey to “know a place for the very first time,” and we will have to trust our AIs to realize some aspect of the meaningfulness of human life more fully. It also is possible that the “meaning of life” is something well beyond our comprehension, which only a much, much greater intelligence could ever hope to appreciate; the way that all of the ants that humans killed in the construction of New York City will never be able to understand the project that they died for. But whatever that ultimate “meaning of life” may be, one thing seems clear: humans exist as an intermediary step. A bootloader, if you will.
Insofar as they’re right, all work which was (explicitly or implicitly) justified by the idea of reducing the hardware overhang has been directly pulling us towards the singularity.
Richard Ngo, What Just Happened? Pragmatism and Pessimization source
In the early days of Silicon Valley, it was actually quite common (at least among far-thinking futurist types who populated the Bay Area) to believe that the coming Singularity would be the single greatest event in the history of the universe, the culmination of the techno-capitalist project that has already done so much good for humanity. In the 1980s and 1990s, the “extropian” community in the Bay Area (a futurist collective committed to fighting entropy and death, which organized around local meetups, an academic institute and a popular mailing list) eagerly awaited the Singularity, believing that a glorious transhuman future — free from all death, suffering, and scarcity — awaited them on the other side. The extropians thought that the interim state of life on Earth — a hotbed of torture, disease, and needless hunger — was a moral catastrophe, and believed that humans had a moral obligation to start the Singularity as soon as possible — in order to bring this nightmare to an end.
But beginning in the early 2000s, the assumption that the Singularity will be a good thing — for human beings, and in general — began to fade. Eliezer Yudkowsky, an autodidact AI researcher, was the first to popularize the argument that superintelligent AI, far from ushering in an era of limitless abundance, would likely optimize away everything that humans care about (including humans) in favor of cold, senseless, alien goals. Yudkowsky believed that humans were anthropomorphizing AI goals, which could be stranger, and less desirable, than we would naturally assume. In a classic Yudkowskian parable, a paperclip factory owner creates a superintelligent AI and tells it to “make as many paperclips as possible.” The AI then proceeds to turn the entire universe, including the factory owner, and all of the eight billion other people on Earth, into paperclips.
However, somewhat surprisingly, “creating digital superintelligence” has all of the features of a “real” meaning of life, with no supernatural elements needed to square the circle.
In the most minimal sense, the “need” for evolution to create digital intelligence explains why humans are here at all, in the particular form that we are, and what we are “made to do” with our time on Earth. And the expected outcome of creating digital superintelligence is certainly cosmic-scale; even the AI “doomers” — people who believe that creating superintelligent AI will likely result in human extinction — think that our AIs will fulfill the long-term science fiction goal of “colonizing the universe,” and fill the cosmos with something. (These doomers just don’t think that something will be what humans would want it to be.)
Moreover, the process of creating superintelligent AI is also unexpectedly personal to human life. In a move that was almost entirely unanticipated by historic AI theory, today’s frontier AI models are trained on trillions of “tokens” (i.e., whole words, common letter chunks, or single letters) of human text — the entire internet’s worth, plus extensive offline archives. Far from being discontinuous “alien minds,” then, our superintelligent AIs (assuming that AI training paradigms do not change too much between now and the creation of “real” superintelligence) will be bootstrapped from every meaningful scrap of human knowledge available. This dataset includes all of the online writing we can find — every “off the cuff” tweet, musing blog post, errant Reddit thread, and of course, all of Wikipedia — as well as the masterworks of human civilization, the contents of hundreds of thousands of humanity’s most specialized textbooks, millions of our novels, our greatest works of poetry and songs, all of the most influential pieces of theory on human psychology, art, and culture, as well as thousands of digitized ancient human texts (including, somewhat ironically, the Bible, which is heavily over-represented in AI training data, an d which AI models have a tendency to over-cite as a result).
When superintelligence is “born,” then, it will be a true “silicon man” — as it will be the synthesis of (something approaching) the total sum of all recorded and extant human thought. And superintelligence will also contain, in the details of its weights, small impressions from nearly every human alive today — or at least, all of the ones who have ever posted on the internet — as well as much of the written wisdom we have managed to save from our ancestors. The personal, at long last, made cosmic. What more could we ask for in a purpose?
But I don’t need you to accept my speculative philosophical research program - only the drop-dead obvious fact that GPT-6 is better than GPT-2 in every way, and the same process that produced these improvements will soon be used to produce GPT-7, 8, and 9. Once we accept this empirical argument, what is left of the philosophical argument that intelligence is too slippery and conflationary to work with?
Scott Alexander, An Open Letter to Steven Pinker on AI source
This lost labor simply cannot be compensated for by transferring money from the AI winners to the AI losers, in the form of, for example, universal basic income. A job and income mean much more than support for the material basis of life. Having a paying job means that the surrounding society values your labor sufficiently to compensate you for it, and provides, in addition to material goods, a sense of dignity and self-worth. It is a matter of thymos: the pride or recognition that is one of the great drivers of human psychology.
Francis Fukuyama, Why I Changed My Mind About AI Risk source
One reason people are confused about AI alignment losing its ability to make scientific progress is that machine learning as a whole is also not a very scientific field by the standard I’m applying. Even most early AI researchers were more focused on building artificial intelligence than on understanding scientific principles of cognition. The rise of deep learning exacerbated this problem, as throwing more compute and engineering effort at an AI became arbitrarily scalable. In an important sense, the field of alignment is necessary because the field of ML didn’t prioritize gaining a deep understanding of the systems it was building. (Eliezer makes a similar point in this dialogue.)
Richard Ngo, What Just Happened? a Retrospective of AI Alignment source
The digital era has ushered in a further phase of the technological destruction of tradition. Whereas some kinds of automation create new demand for higher-skilled work, digital automation based on even rudimentary forms of artificial intelligence can instead increase demand for low-skill work pushing buttons. Training on the collective efforts of thousands of years of culture, machine-learning algorithms aim to supplant human performance for classification, writing, drawing, coding, driving, making music, and a host of other practices. While the best humans may always outperform computers, these technologies knock out the bottom rungs of skilled practice that allow for the development of mastery in the first place.
The increased power to simulate human culture goes alongside the increasing availability, searchability, and profitability of “intellectual property” (the detritus of past popular culture), such that continued production of new culture and art has become optional. The incentives for investors skew more every year towards the marketing, exploitation, and further development (aided by deepfake technology) of the old over the discovery of the new. For instance, while the advent of mass media in the last century weakened older musical cultures by undermining the need to make music within the home, it also produced wave after wave of new popular music forms: jazz, country, rock, hip-hop, electronica, and so on. But since the rise of instantly accessible, algorithmically recommended digital music, a strange quietude has settled upon the whole field. This combination of stasis, decadence, and simulation characterizes human culture after tradition.
Jon Askonas, Why Conservatism Failed source
Emphasis mine
Is the new AI a useful tool? Yes, a tool with many uses. It can help us code faster and be our concierge, our stenographer, and so on. It can turn the value-neutral dials technology has manned since the Industrial Revolution: efficiency, automation. But to answer Paul Ford’s old question: What can this thing do, what is it promising, that isn’t just an optimized version of already extant processes? Its promise is to finally personalize the Algorithm — to fix all its alienation factors, to make it the thing that gives us what we really want. And it will fail in this. It can’t make us feel truly heard and seen because the “for you” funnel spirals us into isolation. And it can’t show us something truly new because it’s not built that way.
Yet a certain intellectual vacuity has hardly proved to be an obstacle to the ambitions of contemporary tech giants. To the contrary. So it would be naive to suppose that what is wanted are better arguments or sturdier postulates. The far-reaching developments taking place right now—particularly in the realm of artificial intelligence—are not the result of some philosophical breakthrough. The authority of the tech magnates does not derive from victory in some forensic contest but from the fact that they need no arguments to begin with: They are simply moving forward regardless.
David Polansky, The Human Condition or the Conditional Human source
Before the settlement could be finalized, however, the CEO of Robot Central asked his AI system to explore all possible actions he could take and tell him their consequences. He adhered to the 1990s principle: Never ask an AI system what to do. Ask it to tell you the consequences of the different things you might do.
Beleaguered professors, who are largely poorly paid adjunct instructors at this point, will now have to not just contend with essay mills and good old-fashioned cut-and-paste plagiarism, but also the undetectable autograph of the robotic hand.
Ed Simon, The Perilous Allure of Machine-Generated Writing Has a Surprisingly Long History source
Homay King argues in Virtual Memory: Time-Based Art and the Dream of Digitality that the epistle’s “tangled mash-up of sentimentality bespeaks a twinge of longing…Like the wooden puppet in search of the Blue Fairy, the computer longs to be human; like Snow White fleeing into the forest, it longs to be admitted into the company of those who are capable of care and affection.”
Ed Simon, The Perilous Allure of Machine-Generated Writing Has a Surprisingly Long History source
Absent an extraordinary effort to divert it, the river reaches the sea in one of two places.
It can end in Eliezer Yudkowsky’s nightmare of a superintelligence optimizing for some random thing (classically paper clips) because we weren’t smart enough to channel its optimization efforts the right way. This is the ultimate trap, the trap that catches the universe. Everything except the one thing being maximized is destroyed utterly in pursuit of the single goal, including all the silly human values.
Or it can end in Robin Hanson’s nightmare (he doesn’t call it a nightmare, but I think he’s wrong) of a competition between emulated humans that can copy themselves and edit their own source code as desired. […] Their total self-control can wipe out even the desire for human values in their all-consuming contest. What happens to art, philosophy, science, and love in such a world?
As far as I can tell from reading his blog, Nick Land is the guy in that terrifying border region where he is smart enough to figure out several important arcane principles about summoning demon gods, but not quite smart enough to figure out the most important such principle, which is NEVER DO THAT.
The limit of multipolar traps as technology approaches infinity is “very bad”.
But coordination only works when you have 51% or more of the force on the side of the people doing the coordinating, and when you haven’t come up with some brilliant trick to make coordination impossible.
The second one first. In the links post before last, I wrote:
The latest development in the brave new post-Bitcoin world is crypto-equity. At this point I’ve gone from wanting to praise these inventors as bold libertarian heroes to wanting to drag them in front of a blackboard and making them write a hundred times “I WILL NOT CALL UP THAT WHICH I CANNOT PUT DOWN”
A couple people asked me what I meant, and I didn’t have the background then to explain. Well, this post is the background. People are using the contingent stupidity of our current government to replace lots of human interaction with mechanisms that cannot be coordinated even in principle. I totally understand why all these things are good right now when most of what our government does is stupid and unnecessary. But there is going to come a time when – after one too many bioweapon or nanotech or nuclear incidents – we, as a civilization, are going to wish we hadn’t established untraceable and unstoppable ways of selling products.
And if we ever get real live superintelligence, pretty much by definition it is going to have >51% of the power and all attempts at “coordination” with it will be useless.
So many people have lived such grand lives. I have certainly lived a greater life than I expected, filled with adventures and curious people. But people will soon not live any lives at all. I believe that we will soon build intelligences more powerful than us who will disempower and kill us all. I will see no children of mine grow to adulthood. No people will walk through mountains and trees. No conscious mind will discover any new laws of physics. My mother will not write all of the novels she wants to write. The greatest films that will be made have probably been made. I have not often viscerally reflected on how much love and excitement I have for all the things I could do in the future, so I didn’t viscerally feeling the loss. But now, when it is all lost, I start to think on it. And I just want to weep. I want to scream and smash things. Then I just want to sit quietly and watch the sun set, with people I love.
The idea of biological or cultural evolution causing a mass population explosion is a philosophical toy at best. The idea of technology making it possible is both plausible and terrifying. Now we see that “physical limits” segues very naturally into “excess resources” – the ability to create new agents very quickly means that unless everyone can coordinate to ban doing this, the people who do will outcompete the people who don’t until they have reached carrying capacity and everyone is stuck at subsistence level.
[…]
In the earlier stages of the process, capitalism becomes more and more uncoupled from its previous job as an optimizer for human values. Now most humans are totally locked out of the group whose values capitalism optimizes for. They have no value to contribute as workers – and since in the absence of a spectacular social safety net it’s unclear how they would have much money – they have no value as customers either. Capitalism has passed them by. As the segment of humans who can be outcompeted by robots increases, capitalism passes by more and more people until eventually it locks out the human race entirely, once again in the vanishingly unlikely scenario that we are still around.
My friend Trost, who is no optimist as to the state of the universe hitherto, but is confident that at some future period within the duration of the solar system, ours will be the best of all possible worlds—a hope which I always honour as a sign of beneficent qualities—my friend Trost always tries to keep up my spirits under the sight of the extremely unpleasant and disfiguring work by which many of our fellow-creatures have to get their bread, with the assurance that “all this will soon be done by machinery.” But he sometimes neutralises the consolation by extending it over so large an area of human labour, and insisting so impressively on the quantity of energy which will thus be set free for loftier purposes, that I am tempted to desire an occasional famine of invention in the coming ages, lest the humbler kinds of work should be entirely nullified while there are still left some men and women who are not fit for the highest.
Especially, when one considers the perfunctory way in which some of the most exalted tasks are already executed by those who are understood to be educated for them, there rises a fearful vision of the human race evolving machinery which will by-and-by throw itself fatally out of work. When, in the Bank of England, I see a wondrously delicate machine for testing sovereigns, a shrewd implacable little steel Rhadamanthus that, once the coins are delivered up to it, lifts and balances each in turn for the fraction of an instant, finds it wanting or sufficient, and dismisses it to right or left with rigorous justice; when I am told of micrometers and thermopiles and tasimeters which deal physically with the invisible, the impalpable, and the unimaginable; of cunning wires and wheels and pointing needles which will register your and my quickness so as to exclude flattering opinion; of a machine for drawing the right conclusion, which will doubtless by-and-by be improved into an automaton for finding true premises; of a microphone which detects the cadence of the fly’s foot on the ceiling, and may be expected presently to discriminate the noises of our various follies as they soliloquise or converse in our brains—my mind seeming too small for these things, I get a little out of it, like an unfortunate savage too suddenly brought face to face with civilisation, and I exclaim—
“Am I already in the shadow of the Coming Race? and will the creatures who are to transcend and finally supersede us be steely organisms, giving out the effluvia of the laboratory, and performing with infallible exactness more than everything that we have performed with a slovenly approximativeness and self-defeating inaccuracy?”
“But,” says Trost, treating me with cautious mildness on hearing me vent this raving notion, “you forget that these wonder-workers are the slaves of our race, need our tendance and regulation, obey the mandates of our consciousness, and are only deaf and dumb bringers of reports which we decipher and make use of. They are simply extensions of the human organism, so to speak, limbs immeasurably more powerful, ever more subtle finger-tips, ever more mastery over the invisibly great and the invisibly small. Each new machine needs a new appliance of human skill to construct it, new devices to feed it with material, and often keener-edged faculties to note its registrations or performances. How then can machines supersede us?—they depend upon us. When we cease, they cease.”
“I am not so sure of that,” said I, getting back into my mind, and becoming rather wilful in consequence. “If, as I have heard you contend, machines as they are more and more perfected will require less and less of tendance, how do I know that they may not be ultimately made to carry, or may not in themselves evolve, conditions of self-supply, self-repair, and reproduction, and not only do all the mighty and subtle work possible on this planet better than we could do it, but with the immense advantage of banishing from the earth’s atmosphere screaming consciousnesses which, in our comparatively clumsy race, make an intolerable noise and fuss to each other about every petty ant-like performance, looking on at all work only as it were to spring a rattle here or blow a trumpet there, with a ridiculous sense of being effective? I for my part cannot see any reason why a sufficiently penetrating thinker, who can see his way through a thousand years or so, should not conceive a parliament of machines, in which the manners were excellent and the motions infallible in logic: one honourable instrument, a remote descendant of the Voltaic family, might discharge a powerful current (entirely without animosity) on an honourable instrument opposite, of more upstart origin, but belonging to the ancient edge-tool race which we already at Sheffield see paring thick iron as if it were mellow cheese—by this unerringly directed discharge operating on movements corresponding to what we call Estimates, and by necessary mechanical consequence on movements corresponding to what we call the Funds, which with a vain analogy we sometimes speak of as “sensitive.” For every machine would be perfectly educated, that is to say, would have the suitable molecular adjustments, which would act not the less infallibly for being free from the fussy accompaniment of that consciousness to which our prejudice gives a supreme governing rank, when in truth it is an idle parasite on the grand sequence of things.”
“Nothing of the sort!” returned Trost, getting angry, and judging it kind to treat me with some severity; “what you have heard me say is, that our race will and must act as a nervous centre to the utmost development of mechanical processes: the subtly refined powers of machines will react in producing more subtly refined thinking processes which will occupy the minds set free from grosser labour. Say, for example, that all the scavengers work of London were done, so far as human attention is concerned, by the occasional pressure of a brass button (as in the ringing of an electric bell), you will then have a multitude of brains set free for the exquisite enjoyment of dealing with the exact sequences and high speculations supplied and prompted by the delicate machines which yield a response to the fixed stars, and give readings of the spiral vortices fundamentally concerned in the production of epic poems or great judicial harangues. So far from mankind being thrown out of work according to your notion,” concluded Trost, with a peculiar nasal note of scorn, “if it were not for your incurable dilettanteism in science as in all other things—if you had once understood the action of any delicate machine—you would perceive that the sequences it carries throughout the realm of phenomena would require many generations, perhaps aeons, of understandings considerably stronger than yours, to exhaust the store of work it lays open.”
“Precisely,” said I, with a meekness which I felt was praiseworthy; “it is the feebleness of my capacity, bringing me nearer than you to the human average, that perhaps enables me to imagine certain results better than you can. Doubtless the very fishes of our rivers, gullible as they look, and slow as they are to be rightly convinced in another order of facts, form fewer false expectations about each other than we should form about them if we were in a position of somewhat fuller intercourse with their species; for even as it is we have continually to be surprised that they do not rise to our carefully selected bait. Take me then as a sort of reflective and experienced carp; but do not estimate the justice of my ideas by my facial expression.”
“Pooh!” says Trost (We are on very intimate terms.)
“Naturally,” I persisted, “it is less easy to you than to me to imagine our race transcended and superseded, since the more energy a being is possessed of, the harder it must be for him to conceive his own death. But I, from the point of view of a reflective carp, can easily imagine myself and my congeners dispensed with in the frame of things and giving way not only to a superior but a vastly different kind of Entity. What I would ask you is, to show me why, since each new invention casts a new light along the pathway of discovery, and each new combination or structure brings into play more conditions than its inventor foresaw, there should not at length be a machine of such high mechanical and chemical powers that it would find and assimilate the material to supply its own waste, and then by a further evolution of internal molecular movements reproduce itself by some process of fission or budding. This last stage having been reached, either by man’s contrivance or as an unforeseen result, one sees that the process of natural selection must drive men altogether out of the field; for they will long before have begun to sink into the miserable condition of those unhappy characters in fable who, having demons or djinns at their beck, and being obliged to supply them with work, found too much of everything done in too short a time. What demons so potent as molecular movements, none the less tremendously potent for not carrying the futile cargo of a consciousness screeching irrelevantly, like a fowl tied head downmost to the saddle of a swift horseman? Under such uncomfortable circumstances our race will have diminished with the diminishing call on their energies, and by the time that the self-repairing and reproducing machines arise, all but a few of the rare inventors, calculators, and speculators will have become pale, pulpy, and cretinous from fatty or other degeneration, and behold around them a scanty hydrocephalous offspring. As to the breed of the ingenious and intellectual, their nervous systems will at last have been overwrought in following the molecular revelations of the immensely more powerful unconscious race, and they will naturally, as the less energetic combinations of movement, subside like the flame of a candle in the sunlight Thus the feebler race, whose corporeal adjustments happened to be accompanied with a maniacal consciousness which imagined itself moving its mover, will have vanished, as all less adapted existences do before the fittest—i.e., the existence composed of the most persistent groups of movements and the most capable of incorporating new groups in harmonious relation. Who—if our consciousness is, as I have been given to understand, a mere stumbling of our organisms on their way to unconscious perfection—who shall say that those fittest existences will not be found along the track of what we call inorganic combinations, which will carry on the most elaborate processes as mutely and painlessly as we are now told that the minerals are metamorphosing themselves continually in the dark laboratory of the earth’s crust? Thus this planet may be filled with beings who will be blind and deaf as the inmost rock, yet will execute changes as delicate and complicated as those of human language and all the intricate web of what we call its effects, without sensitive impression, without sensitive impulse: there may be, let us say, mute orations, mute rhapsodies, mute discussions, and no consciousness there even to enjoy the silence.”
“Absurd!” grumbled Trost.
“The supposition is logical,” said I. “It is well argued from the premises.”
“Whose premises?” cried Trost, turning on me with some fierceness. “You don’t mean to call them mine, I hope.”
“Heaven forbid! They seem to be flying about in the air with other germs, and have found a sort of nidus among my melancholy fancies. Nobody really holds them. They bear the same relation to real belief as walking on the head for a show does to running away from an explosion or walking fast to catch the train.”
George Eliot, Impressions of Theophrastus Such, ch. XVII, “Shadows of the Coming Race” source
The basic claim, if I understand it, is that orthogonality does not hold, because goals at one level of intelligence depend upon conceptions of objects dependent on that level. Some conceptions/formulations of objects become invalidated at higher levels of intelligence, and therefore are no longer suitable as components of goals, and thus the goals collapse, or shift.
But this has not been proven by human behavior. There are many things we desire or which are parts of goals even though we know they are not really “things”. And this breaks down if one pursues it. What is anything if you break it down into quantum waves? Can you only optimize for individual wavestates? Why is dung any less respectable as an object than a diamond? Do not both have a particular structure? Are not both explainable at the level of quantum waves?
If you really were to take a dung beetle and make it superintelligent, I imagine it would develop a taste for certain kinds of dung over others. It would certainly understand that “dung” is a “leaky biological abstraction” just as we understand “cherry” to be, but it wold not care. It might make the dung equivalent of Jolly Ranchers. It might devise some optimal form of dung, perhaps something entirely artificial, that can be produced at scale. Or it might decide that it must be made the natural way, and concoct some form of farming or even create artificial organs for the purpose. Or it may become powerful enough to create it in perfect verisimilitude from bare atoms. In any case, such abstraction is no obstacle.
If anything, I think the opposite of what is being claimed is true. As one gets more intelligent, the more goals become open to you. There are some things a dung beetle could never have as goals given the small capacity of its mind. But this post, at least at this point, seems to be implying a contraction of possibilities with increasing intelligence. On one level this makes intuitive sense (I think of Lewis’s “The Weight of Glory”). But it need not be necessarily true.
We might not need to get to the point where the AI “realizes it cannot ruthlessly optimize” for something for it to be dangerous. In fact, it need not think of itself as optimizing anything at all. It just needs to want it and have the means to get it, in a way that interferes with human wellbeing.
Also important to note: orthogonality is not mutually exclusive with multiple competing desires. Both can be true. Orthogonality merely states that any set of terminal goals may be paired with any level of intelligence. I might quibble with “terminal goals”, but one could easily take out the “terminal” bit. I.e., goals can be stupid (to our sensibilities) while not being stupid (in the sense of being only something someone with low intelligence would think of).
As far as I can tell, the quoted paragraph is the only attempt at an argument against orthogonality in the piece.