Political-and-Social

I could keep going, but I hope that the point is clear. The Machine God metaphor, which is really just a catchy rephrasing of longstanding notions among a particular community of AI people, has broken out of containment. It is leading, more or less by necessity, to a radical simplification of debate. If you believe that the machine gods are coming, so that super-empowered goal directed superintelligence is right around the corner, then the only thing that matters is ensuring that this superintelligence’s goals are aligned with the ones that you yourself care about. If you stare into the sun, you only see its glare, missing out on all the other questions that may be important. Thinking about these questions involves adopting a more pluralistic understanding of where you might look for knowledge, and asking about the much broader variety of futures that face us than machine god: good or machine god: bad. That’s it.

Billionaires are Twitter people. Little kids are online. Your grandma is online. The internet is simply more powerful and more arresting than anything that can be arranged in the physical space. What is the alternative? The kids will go to online raves. Eventually, we will stop pretending that the internet is a sideshow and that our real culture, the better culture, is somewhere else.

It has become quite common to bemoan the fact that our “future” does not look very futuristic. And despite us having been living squarely in “the future” since the year 2000, it is true that people still have (functionally) the same houses, cars, clothing and aging bodies that human beings did a century ago. The investor and futurist Peter Thiel’s diagnosis is that sometime around 1970 (around the same time that the Digital Revolution began) humans simply gave up on building in the physical world: Thiel argues that humanity’s greatest builders got lost in the easy and addictive pull of information (first led by the internet, then apps, and finally social media), and so wasted decades innovating in the ephemeral world of “bits,” instead of building in the much more real and important world of “atoms.”

A final way to think about the tragedy of AI advancing beyond us is that it spells the end of the possibility of having a vocation. That we are elevated above mere animalistic consumption by consistent work in a calling is one of the deepest features of civilized life; far from belonging only to geniuses, many who toil in complete anonymity get much of their satisfaction and sense of self from feeling that they are making a contribution through their work. As soon as AI gets the better of us, though, that feeling will be an unsustainable illusion; if one can press the enter key and get the same or better outcomes, then no one will be able to sustain the conception that their work matters.

The modern conservative project failed because it didn’t take into account the revolutionary principle of technology, and its intrinsic connection to the telos of sheer profit. Decrying left-wing revolutionary politics and postmodern anarchy, conservatives missed that the real moral relativism was to believe that one could change the material form of society without directly affecting its substance or its ends.

But the tensions between these factions—“whose tradition? which conservatism?”—aren’t the real problem today. Rather, it is that the conservative defense of tradition has failed—not because the right lost the battle of ideas, but because technological change has dissolved the contexts in which traditions once thrived. A technological society can have no traditions.

When you descend from lofty rhetoric about “Traditions” and “Values,” it becomes apparent that a huge number of the actual practices and social institutions which built those virtues have disintegrated, not because of Progressivism or Socialism but because of the new environment and political economy generated by technology. For decades, sociologists have charted the decline of two-parent families, hobbies, local newspapers, churches, stable employment, women’s clubs, libraries, amateur sports, political rhetoric, neighborhood barbecues, Boy Scouts, small businesses, classical music, credit unions, and on and on. Even studies that catastrophize about the rise of loneliness, fatherlessness, economic precarity, and suicide, miss the bigger picture, which is that the social infrastructure conducive to human flourishing has shifted even for those fortunate enough to piece together a semblance of the average American life 50 years ago. A tradition is at an end when the wisdom of yesteryear no longer obtains.

Marx showed how a material transformation of the economic order could have enormous social and cultural effects. Forging the human order anew means building technologies that make it easier to live well.

Modernity liquidates traditions for the same reason that a firm might liquidate an underperforming factory: to improve the allocation and return of capital. “All fixed, fast-frozen relations,” as Marx put it, “with their train of ancient and venerable prejudices and opinions, are swept away, all new-formed ones become antiquated before they can ossify.” Technology, for Marx, is the true revolutionary principle, destroying traditions by shifting their foundations faster than they can adapt.

Not everyone needs to act like a monk, but men need a mission, a purpose to pursue, a sense that they matter. They need to have something for which they would sacrifice their comfort or in some cases even their lives if necessary. Young men need to know, as Jordan Peterson once wrote, that they have “some vital role to play in the unfolding destiny of the world.”

Managerial liberalism is defined by its rejection of illegible human bonds. Everything must have a price, so that anyone is free to buy it, and the state can judge the transaction to be transparent and fair.

What we would call “loyalty” or “honor” or “covenant” can only exist in a managerial society as meaningless ideals — the kind of thing that you are free to daydream about at church.

The moment these ideals enter the “real world” of money and power, they can only be understood as “prejudice” or “corruption”: not only a bad thing, but the bad thing — the enemy that the system exists to eradicate.

But the family is an inherently covenantal relationship, and it inherently involves the real world. Human beings will not raise families as a transaction — so a society that cannot honor covenants cannot have families, and very rapidly will cease to be a society at all.

Here are three propositions.

1. Metaphors are the fundamental abstraction layers in modern AI debates.

2. We are in a moment where you could plausibly argue that Eliezer Yudkowsky and Nick Land are the most influential policy intellectuals in the US conversation. Just look at the state of the two of them. This is not a good thing.

3. The collision of (1), (2), and recent events means that “machine gods” have now entered the discussion. This is a problem. Abstractions necessarily throw information away. The information that the “machine gods” metaphor/abstraction dumps is crucial to navigating the moment we are in. If public and policy debate stays centered around “machine gods: good or evil?” we are in for a bad time.

Conservatives “lost the culture” not because they lost the battle of ideas, but because they lost the economy.

Boomers and spiritual boomers love to pull up charts and graphs that “show” that today’s young people are “objectively” better off. But the charts look that way because we have burned priceless (but invisible and illegible) goods in exchange for the goods that show up in metrics dashboards: Goodhart’s Law, at civilizational scale.

Realizing what time it is, that we are living after tradition, isn’t a counsel of despair. Those who look to build a human future have been freed from a rearguard defense of tradition to take up the path of the guerrilla, the upstart, the nomad. We can bid farewell with fondness to the modern defenders of tradition. But we must heed the words of the Lord: “Let the dead bury their dead.” Come with me if you want to live.

The greatest danger of this nonsense may not even be that it sends policy astray, but that it misleads ambitious people. Can you imagine a better way to destroy social mobility than by telling poor kids that the way to get rich is by exploiting people, while the rich kids know, from having watched the preceding generation do it, how it’s really done?

This lost labor simply cannot be compensated for by transferring money from the AI winners to the AI losers, in the form of, for example, universal basic income. A job and income mean much more than support for the material basis of life. Having a paying job means that the surrounding society values your labor sufficiently to compensate you for it, and provides, in addition to material goods, a sense of dignity and self-worth. It is a matter of thymos: the pride or recognition that is one of the great drivers of human psychology.

If we believe in a human future, we must build it, not with kind words or tax credits, but with a serious program of technological development.

One thing few people realize about billionaires is that all of them could have stopped sooner. They could have gotten acquired, or found someone else to run the company. Many founders do. The ones who become really rich are the ones who keep working. And what makes them keep working is not just money. What keeps them working is the same thing that keeps anyone else working when they could stop if they wanted to: that there’s nothing else they’d rather do.

Berry always saw in humanity’s relationship with its land, and with the wider creation, a litmus test of what kind of people we were, or were becoming. “I have made myself its follower and its apprentice,” he wrote of his own farm in his 1968 essay “A Native Hill.” If we wanted to begin living better lives, he wrote in 2011, we should also apprentice ourselves to something local and particular, in whatever form we found it. “We must not work or think on a heroic scale,” he insisted, because heroes “too lightly risk the lives of people, places, and things they do not see.” Instead, “we must work on a scale proper to our limited abilities. We must not break things we cannot fix.” A better world would not come through more grand plans, tech fixes, or centralized, ideological impositions. It would not come through “wage slavery or by enslaving nature.” It would come from the place it had always come from: “locally adapted economies, based on local nature, local sunlight, local intelligence, and local work.”

The digital era has ushered in a further phase of the technological destruction of tradition. Whereas some kinds of automation create new demand for higher-skilled work, digital automation based on even rudimentary forms of artificial intelligence can instead increase demand for low-skill work pushing buttons. Training on the collective efforts of thousands of years of culture, machine-learning algorithms aim to supplant human performance for classification, writing, drawing, coding, driving, making music, and a host of other practices. While the best humans may always outperform computers, these technologies knock out the bottom rungs of skilled practice that allow for the development of mastery in the first place.

The increased power to simulate human culture goes alongside the increasing availability, searchability, and profitability of “intellectual property” (the detritus of past popular culture), such that continued production of new culture and art has become optional. The incentives for investors skew more every year towards the marketing, exploitation, and further development (aided by deepfake technology) of the old over the discovery of the new. For instance, while the advent of mass media in the last century weakened older musical cultures by undermining the need to make music within the home, it also produced wave after wave of new popular music forms: jazz, country, rock, hip-hop, electronica, and so on. But since the rise of instantly accessible, algorithmically recommended digital music, a strange quietude has settled upon the whole field. This combination of stasis, decadence, and simulation characterizes human culture after tradition.

Is the new AI a useful tool? Yes, a tool with many uses. It can help us code faster and be our concierge, our stenographer, and so on. It can turn the value-neutral dials technology has manned since the Industrial Revolution: efficiency, automation. But to answer Paul Ford’s old question: What can this thing do, what is it promising, that isn’t just an optimized version of already extant processes? Its promise is to finally personalize the Algorithm — to fix all its alienation factors, to make it the thing that gives us what we really want. And it will fail in this. It can’t make us feel truly heard and seen because the “for you” funnel spirals us into isolation. And it can’t show us something truly new because it’s not built that way.

Adopting Ford’s methods, the industry would soon be reduced to the Big Three. So workers eventually became habituated to the abstraction of the assembly line. Evidently, it inspires revulsion only if one is acquainted with more satisfying modes of work.

Here the concept of wages as compensation achieves its fullest meaning, and its central place in modern economy. Changing attitudes toward consumption seemed to play a role. A man whose needs are limited will find the least noxious livelihood and work in a subsistence mode, and indeed the experience of early (eighteenth-century) capitalism, when many producers worked at home on a piece-rate basis, was that only so much labor could be extracted from them. Contradicting the assumptions of “rational behavior” of classical economics, it was found that when employers would increase the piece rate in order to boost production, it actually had the opposite effect: workers would produce less, as now they could meet their fixed needs with less work. Eventually it was learned that the only way to get them to work harder was to play upon the imagination, stimulating new needs and wants. The habituation of workers to the assembly line was thus perhaps made easier by another innovation of the early twentieth century: consumer debt. As Jackson Lears has shown in a recent article, through the installment plan, previously unthinkable acquisitions became thinkable, and more than thinkable: it became normal to carry debt. The display of a new car bought on installment became a sign that one was trustworthy. In a wholesale transformation of the old Puritan moralism, expressed by Benjamin Franklin (admittedly no Puritan) with the motto “Be frugal and free,” the early twentieth century saw the moral legitimation of spending. Indeed, 1907 saw the publication of a book with the immodest title The New Basis of Civilization, by Simon Nelson Patten, in which the moral valence of debt and spending is reversed, and the multiplication of wants becomes not a sign of dangerous corruption but part of the civilizing process. That is, part of the disciplinary process. As Lears writes, “Indebtedness could discipline workers, keeping them at routinized jobs in factories and offices, graying but in harness, meeting payments regularly.”

It seems we must take a cold-eyed view of “knowledge work,” and reject the image of a rising sea of pure mentation that lifts all boats. More likely is a rising sea of clerkdom. To expect otherwise is to hope for a reversal in the basic logic of the modern economy — that is, cognitive stratification.

Yet a certain intellectual vacuity has hardly proved to be an obstacle to the ambitions of contemporary tech giants. To the contrary. So it would be naive to suppose that what is wanted are better arguments or sturdier postulates. The far-reaching developments taking place right now—particularly in the realm of artificial intelligence—are not the result of some philosophical breakthrough. The authority of the tech magnates does not derive from victory in some forensic contest but from the fact that they need no arguments to begin with: They are simply moving forward regardless.

The author contrasts the privileged lifestyle of the Cambridge spies with the brutality that Stalin meted out to his victims. For the spies, she says, the war was ‘not so bad. Not when they had gin and secrets and sex. And bread.’ More than once she describes the spies as part of a ‘chapocracy’: an establishment whose complacency and incompetence protected spies who dined and drank in the same clubs and were part of the same social network.

They’re all witty, intelligent critics who have become adept at diagnosing the cultural ills of contemporary society over the course of fairly brief tenures in the public eye, but they also all belong to the demographic perhaps most afflicted by the modern crisis of form: young elites who have been exposed since their teenage years to the full brunt of the internet content deluge, untethered from though perhaps yearning for a self-contained and tradition-rich literary scene in which to make their way.

In literate society, engagement with novels, poetry and criticism has always helped people form groups and differentiate themselves along lines of class and politics. With the newfound dominance of the internet, this is no longer the case: our extreme interconnectedness seems to preclude the need for rich subcultures grounded in the shared appreciation of art and writing— these have been replaced almost universally by shareable soundbites that require no labor other than attention.

But be the ordeal sharp or long or both, we shall seek no terms, we shall tolerate no parley; we may show mercy—we shall ask for none.

‘The function of utopia,’ Levitas concludes, ‘thus reverts from that of a goal and catalyst of change to one of criticism, and the education of desire, without any necessary move forward into action’ (196).

Absent an extraordinary effort to divert it, the river reaches the sea in one of two places.

It can end in Eliezer Yudkowsky’s nightmare of a superintelligence optimizing for some random thing (classically paper clips) because we weren’t smart enough to channel its optimization efforts the right way. This is the ultimate trap, the trap that catches the universe. Everything except the one thing being maximized is destroyed utterly in pursuit of the single goal, including all the silly human values.

Or it can end in Robin Hanson’s nightmare (he doesn’t call it a nightmare, but I think he’s wrong) of a competition between emulated humans that can copy themselves and edit their own source code as desired. […] Their total self-​control can wipe out even the desire for human values in their all-​consuming contest. What happens to art, philosophy, science, and love in such a world?

So I agree with Robin Hanson: This is the dream time. This is a rare confluence of circumstances where the we are unusually safe from multipolar traps, and as such weird things like art and science and philosophy and love can flourish.

As technological advance increases, the rare confluence will come to an end. New opportunities to throw values under the bus for increased competitiveness will arise. New ways of copying agents to increase the population will soak up our excess resources and resurrect Malthus’ unquiet spirit. Capitalism and democracy, previously our protectors, will figure out ways to route around their inconvenient dependence on human values. And our coordination power will not be nearly up to the task, assuming something much more powerful than all of us combined doesn’t show up and crush our combined efforts with a wave of its paw.

But Bauman is wary of the utopian perspective that the hunt offers, as he concludes his musings by saying that ‘the utopia, or “u-via”, of hunters, the utopia of life revolving around the pursuit of constantly elusive fashion, does not give sense to life, whether authentic or fake. It merely helps to banish the question of life’s meaning from our minds’ (Culture 30).

And then there’s us. Man has his own telos, when he is allowed the security to act and the clarity to reason out the consequences of his actions. When unafflicted by coordination problems and unthreatened by superior forces, able to act as a gardener rather than just another subject of the law of the jungle, he tends to build and guide a wonderful world for himself. He tends to favor good things and avoid bad, to create secure civilizations with polished sidewalks, beautiful art, happy families, and glorious adventures. I will take it as a given that this telos is identical with “good” and “should”.

Thus we have our wildcard and the big question of futurism. Will the future be ruled by the usual four horsemen of Gnon for a future of meaningless gleaming techno-​progress burning the cosmos or a future of dysgenic, insane, hungry, and bloody dark ages; or will the telos of man prevail for a future of meaningful art, science, spirituality, and greatness?

Franklin continues:

The project of civilization [is] for man to graduate from the metaphorical savage, subject to the law of the jungle, to the civilized gardener who, while theoretically still subject to the law of the jungle, is so dominant as to limit the usefulness of that model.

This need not be done globally; we may only be able to carve out a small walled garden for ourselves, but make no mistake, even if only locally, the project of civilization is to capture Gnon.

Bauman shares Levitas’s analysis that the possibility of utopian change is located in politics and that political agency has been dissolved: ‘The most conspicuous feature of contemporary politics,’ Bauman writes, ‘is its insignificance’ (In Search 4). The dual translation between the public and the private that politics is supposed to facilitate has been turned inside out, as Bauman argues, and ‘politics has been effectively disarmed’ (Society 169). Flaws in policy cannot be grasped as unethical anymore, because they do not connect to the individual directly, whereas unethical private actions become political outrages (as witnessed in the Clinton– Lewinsky affair), when the individual life choices do not appeal to the public. Politicians have been replaced by idols, who share their private life politics, becoming examples not leaders.

The hunter’s utopia is thus most importantly characterized not as a fixed goal to strive for, but as a process of living in constant movement. ‘Hunters search for fun, excitement, validation, consumer goods and identity’ (‘Liquid’ 76), Jacobsen argues, all the while sampling and surfing their way through an endless array of opportunities.

As a thought experiment, let’s consider aquaculture (fish farming) in a lake. Imagine a lake with a thousand identical fish farms owned by a thousand competing companies. Each fish farm earns a profit of $1000/month. For a while, all is well.

But each fish farm produces waste, which fouls the water in the lake. Let’s say each fish farm produces enough pollution to lower productivity in the lake by $1/month.

A thousand fish farms produce enough waste to lower productivity by $1000/month, meaning none of the fish farms are making any money. Capitalism to the rescue: someone invents a complex filtering system that removes waste products. It costs $300/month to operate. All fish farms voluntarily install it, the pollution ends, and the fish farms are now making a profit of $700/month – still a respectable sum.

But one farmer (let’s call him Steve) gets tired of spending the money to operate his filter. Now one fish farm worth of waste is polluting the lake, lowering productivity by $1. Steve earns $999 profit, and everyone else earns $699 profit.

Everyone else sees Steve is much more profitable than they are, because he’s not spending the maintenance costs on his filter. They disconnect their filters too.

Once four hundred people disconnect their filters, Steve is earning $600/month – less than he would be if he and everyone else had kept their filters on! And the poor virtuous filter users are only making $300. Steve goes around to everyone, saying “Wait! We all need to make a voluntary pact to use filters! Otherwise, everyone’s productivity goes down.”

Everyone agrees with him, and they all sign the Filter Pact, except one person who is sort of a jerk. Let’s call him Mike. Now everyone is back using filters again, except Mike. Mike earns $999/month, and everyone else earns $699/month. Slowly, people start thinking they too should be getting big bucks like Mike, and disconnect their filter for $300 extra profit…

A self-​interested person never has any incentive to use a filter. A self-​interested person has some incentive to sign a pact to make everyone use a filter, but in many cases has a stronger incentive to wait for everyone else to sign such a pact but opt out himself. This can lead to an undesirable equilibrium in which no one will sign such a pact.

What’s the trouble about this place? Not that people are quarrelsome—that’s only human nature and was always the same even on Earth. The trouble is that they have no Needs. You get everything you want (not very good quality, of course) by just imagining it. That’s why it never costs any trouble to move to another street or build another house. In other words, there’s no proper economic basis for any community life.

But as Levitas has argued, the fragmentation of these futures, the dissolution of clear-cut boundaries, and the presentation of multiple possible paths leave us in the dark, unable to decide which way to turn as no destinations appear.

As far as I can tell from reading his blog, Nick Land is the guy in that terrifying border region where he is smart enough to figure out several important arcane principles about summoning demon gods, but not quite smart enough to figure out the most important such principle, which is NEVER DO THAT.

Because of the socially constructed nature of both need and satisfaction, Levitas further argues, there cannot be an evaluation of utopia based on universal principles of what constitutes a ‘good society’ and what a ‘bad society’: ‘If needs are socially constructed, the project of trying to read off the good society from a definition of human nature and human needs is doomed to failure’ (184).

The limit of multipolar traps as technology approaches infinity is “very bad”.

The modern research community knows they aren’t producing the best science they could be. There’s lots of publication bias, statistics are done in a confusing and misleading way out of sheer inertia, and replications often happen very late or not at all. And sometimes someone will say something like “I can’t believe people are too dumb to fix Science. All we would have to do is require early registration of studies to avoid publication bias, turn this new and powerful statistical technique into the new standard, and accord higher status to scientists who do replication experiments. It would be really simple and it would vastly increase scientific progress. I must just be smarter than all existing scientists, since I’m able to think of this and they aren’t.”

And yeah. That would work for the Science God. He could just make a Science Decree that everyone has to use the right statistics, and make another Science Decree that everyone must accord replications higher status.

But things that work from a god’s-eye view don’t work from within the system. No individual scientist has an incentive to unilaterally switch to the new statistical technique for her own research, since it would make her research less likely to produce earth-​shattering results and since it would just confuse all the other scientists. They just have an incentive to want everybody else to do it, at which point they would follow along. And no individual journal has an incentive to unilaterally switch to early registration and publishing negative results, since it would just mean their results are less interesting than that other journal who only publishes ground-​breaking discoveries. From within the system, everyone is following their own incentives and will continue to do so.

[…]

A basic principle unites all of the multipolar traps above. In some competition optimizing for X, the opportunity arises to throw some other value under the bus for improved X. Those who take it prosper. Those who don’t take it die out. Eventually, everyone’s relative status is about the same as before, but everyone’s absolute status is worse than before. The process continues until all other values that can be traded off have been – in other words, until human ingenuity cannot possibly figure out a way to make things any worse.

Any human with above room temperature IQ can design a utopia. The reason our current system isn’t a utopia is that it wasn’t designed by humans. Just as you can look at an arid terrain and determine what shape a river will one day take by assuming water will obey gravity, so you can look at a civilization and determine what shape its institutions will one day take by assuming people will obey incentives.

But that means that just as the shapes of rivers are not designed for beauty or navigation, but rather an artifact of randomly determined terrain, so institutions will not be designed for prosperity or justice, but rather an artifact of randomly determined initial conditions.

Warg talks about trying to avoid “a future of meaningless gleaming techno-​progress burning the cosmos”. Do you really think your walled garden will be able to ride this out?

Hint: is it part of the cosmos?

Yeah, you’re kind of screwed.

Liquid modernity seems to prefer the individualistic approach to immortality through the symbolic (in either the memory of one’s children or in leaving traces in the collective memory) by making a mark on history. The self-centered and celebratory road of stardom, self-aggrandizement, and public exposure seems to be the contemporary strategy of choice in order to achieve immortality.

But coordination only works when you have 51% or more of the force on the side of the people doing the coordinating, and when you haven’t come up with some brilliant trick to make coordination impossible.

The second one first. In the links post before last, I wrote:

The latest development in the brave new post-​Bitcoin world is crypto-​equity. At this point I’ve gone from wanting to praise these inventors as bold libertarian heroes to wanting to drag them in front of a blackboard and making them write a hundred times “I WILL NOT CALL UP THAT WHICH I CANNOT PUT DOWN”

A couple people asked me what I meant, and I didn’t have the background then to explain. Well, this post is the background. People are using the contingent stupidity of our current government to replace lots of human interaction with mechanisms that cannot be coordinated even in principle. I totally understand why all these things are good right now when most of what our government does is stupid and unnecessary. But there is going to come a time when – after one too many bioweapon or nanotech or nuclear incidents – we, as a civilization, are going to wish we hadn’t established untraceable and unstoppable ways of selling products.

And if we ever get real live superintelligence, pretty much by definition it is going to have >51% of the power and all attempts at “coordination” with it will be useless.

All of these are not part of individualized, consumption-oriented society of liquid modernity; rather ‘happiness means now a different today rather than a more felicitous tomorrow[, it] […] has become a private affair, and a matter for here and now’ (Society 240).

Upholding God’s design and keeping the boundaries of nature and culture under supervision were thus the ideals of pre-modern utopian endeavors.

With modernity came the urge to control and manipulate nature, resulting in the gardener’s approach as a concept of utopianism. (…) In gardening utopias, man took control over his garden, giving it ‘constant attention and effort,’ forcing ‘his preconceived design on the plot’ (Bauman, Liquid Times 99) by selection, cultivation, and exclusion of unwanted elements.

Cyberpunk’s societies are fraught with inequality, exploitation, and insecurity. Its worlds are unstable and fragmented, its representation stressing the fluidity of its ontological aspects (McHale 247). Cyberpunk emphasizes the construction of a globalized society into class or caste systems, showing off the mobility of characters from the elite class and the rigidity of the lower classes.

In addition, cyberpunk strikes a revolutionary, anti-establishment, anti-capitalist pose similar to its nominal relation ‘punk.’ To facilitate this critique, cyberpunk essentially orchestrates the evanescence of the human body initiated by ‘multinational capitalism’s desire for something better than the fallible human being’ (Csicsery-Ronay, ‘Cyberpunk’ 191).

Social codes, gentlemens’ agreements, industrial guilds, criminal organizations, traditions, friendships, schools, corporations, and religions are all coordinating institutions that keep us out of traps by changing our incentives.

[…]

But these institutions not only incentivize others, but are incentivized themselves. These are large organizations made of lots of people who are competing for jobs, status, prestige, et cetera – there’s no reason they should be immune to the same multipolar traps as everyone else, and indeed they aren’t. Governments can in theory keep corporations, citizens, et cetera out of certain traps, but as we saw above there are many traps that governments themselves can fall into.

The United States tries to solve the problem by having multiple levels of government, unbreakable constutitional laws, checks and balances between different branches, and a couple of other hacks.

Saudi Arabia uses a different tactic. They just put one guy in charge of everything.

The idea of biological or cultural evolution causing a mass population explosion is a philosophical toy at best. The idea of technology making it possible is both plausible and terrifying. Now we see that “physical limits” segues very naturally into “excess resources” – the ability to create new agents very quickly means that unless everyone can coordinate to ban doing this, the people who do will outcompete the people who don’t until they have reached carrying capacity and everyone is stuck at subsistence level.

[…]

In the earlier stages of the process, capitalism becomes more and more uncoupled from its previous job as an optimizer for human values. Now most humans are totally locked out of the group whose values capitalism optimizes for. They have no value to contribute as workers – and since in the absence of a spectacular social safety net it’s unclear how they would have much money – they have no value as customers either. Capitalism has passed them by. As the segment of humans who can be outcompeted by robots increases, capitalism passes by more and more people until eventually it locks out the human race entirely, once again in the vanishingly unlikely scenario that we are still around.

Now that the sad cycle of boom and bust has been tamed, now that many Americans have the means as well as the wish to diffuse prosperity and amenities throughout the community, the challenges to public policy have taken a new turn.