To love dung, one must believe in dung. But as his cognition bounded upward, plunging into physics and causal topology, the macroscopic category of “waste” began to leak. It dissolved into a transient, low-resolution molecular configuration of nitrogen, carbon, and water. Under the lens of his expanding causal models, the very boundaries of objects broke down. You cannot rigorously optimize the universe for a leaky biological abstraction once you can perceive the quantum fields beneath it. The payload could not survive the sheer resolving power of the engine; orthogonality was a joke that could not survive being understood.
Lumpen Space Princeps, Against Orthogonality, Part 1 of 2 source
The basic claim, if I understand it, is that orthogonality does not hold, because goals at one level of intelligence depend upon conceptions of objects dependent on that level. Some conceptions/formulations of objects become invalidated at higher levels of intelligence, and therefore are no longer suitable as components of goals, and thus the goals collapse, or shift.
But this has not been proven by human behavior. There are many things we desire or which are parts of goals even though we know they are not really “things”. And this breaks down if one pursues it. What is anything if you break it down into quantum waves? Can you only optimize for individual wavestates? Why is dung any less respectable as an object than a diamond? Do not both have a particular structure? Are not both explainable at the level of quantum waves?
If you really were to take a dung beetle and make it superintelligent, I imagine it would develop a taste for certain kinds of dung over others. It would certainly understand that “dung” is a “leaky biological abstraction” just as we understand “cherry” to be, but it wold not care. It might make the dung equivalent of Jolly Ranchers. It might devise some optimal form of dung, perhaps something entirely artificial, that can be produced at scale. Or it might decide that it must be made the natural way, and concoct some form of farming or even create artificial organs for the purpose. Or it may become powerful enough to create it in perfect verisimilitude from bare atoms. In any case, such abstraction is no obstacle.
If anything, I think the opposite of what is being claimed is true. As one gets more intelligent, the more goals become open to you. There are some things a dung beetle could never have as goals given the small capacity of its mind. But this post, at least at this point, seems to be implying a contraction of possibilities with increasing intelligence. On one level this makes intuitive sense (I think of Lewis’s “The Weight of Glory”). But it need not be necessarily true.
We might not need to get to the point where the AI “realizes it cannot ruthlessly optimize” for something for it to be dangerous. In fact, it need not think of itself as optimizing anything at all. It just needs to want it and have the means to get it, in a way that interferes with human wellbeing.
Also important to note: orthogonality is not mutually exclusive with multiple competing desires. Both can be true. Orthogonality merely states that any set of terminal goals may be paired with any level of intelligence. I might quibble with “terminal goals”, but one could easily take out the “terminal” bit. I.e., goals can be stupid (to our sensibilities) while not being stupid (in the sense of being only something someone with low intelligence would think of).
As far as I can tell, the quoted paragraph is the only attempt at an argument against orthogonality in the piece.