Transformational Alignment: The Economy as Singleton

1. A singleton is an alignment accomplishment

A singleton, coined by Nick Bostrom, refers to a decision-making agency at the highest level of the world order. A singleton is able to control what happens in its domain and prevent challenges to its supremacy.

Bostrom defines possible singletons very broadly and abstractly—it could refer to a democratic global republic, or a dictatorship, or even a set of shared values that effectively run the world at the highest level. The singleton also need not be monolithic. It could contain many agents each pursuing their own goals. It could deliberately increase diversity among these agents if that helped the singleton achieve its own goals. And it might operate subtly and out of sight by simply enforcing certain background conditions. The one essential property all singletons must share is the ability to solve global coordination problems.

But in order to solve global coordination problems, the singleton must itself be the solution to a global coordination problem: there must be some mechanism by which the higher-order solution constrains the behavior of the participants in the global order.

The question then becomes: what kinds of alignment processes can actually create and maintain singleton-scale organization? What keeps all the heterogeneous parts inside the boundary functioning as one system? What architectures make such an object dynamically possible?

It’s noteworthy that an agent might be thought of as a singleton over some domain. An animal like a gibbon is a singleton over its internal milieu and perhaps its immediate environmental niche. Then consider a very simple empirical question: why hasn’t an organism become a planet-sized singleton already? It’s not a question of creating a planet-sized gibbon. It’s a question of why a gibbon can’t extend the control it has over its internal environment and its immediate niche to the rest of the world.

In other words: every agent might be considered a singleton at some scale. The question of a singleton, then, is a question of scale: why are some singletons small and others large? What are the scaling rules?

Dominance is not enough. An army can conquer the world, but what keeps it together? You need ongoing alignment, not just temporary victory, as many an empire has learned.

One approach has been described in different works by Carl Shulman and Nick Bostrom: remove agency problems by aligning purposes.

A different kind of possible singleton to consider is the economy. It operates at an extraordinarily high organizational level, coordinating production and consumption across billions of people around the world. The economy does the main task that Bostrom identifies for a singleton: solve global coordination problems. Whether this makes it literally a Bostromian singleton is less important than the fact that it confronts many of the same problems of maintaining large-scale coordinated order.

As a singleton scales, it must maintain coherence across an increasingly large and changing set of components. The economy indicates how this might be accomplished at global scale because the economy does not continually force its components to align to one substantive purpose. Instead, much of the economy’s higher-level structure changes to suit what the components are doing.

This flexibility about its own structure, enabling it to maintain concordance among its members, may be precisely what allows the economy to function as a world-sized singleton. This makes the economy a case study in an important answer to the question of what architecture makes a singleton dynamically possible.

2. Fixed-target alignment and the problem of scale

As a useful contrast to transformational alignment, one imaginable architecture for a singleton is more like a typical organism. In comparison to the economy, an organism has a relatively stable purpose. Lower-scale components (muscles, cells, etc.) are recruited toward this purpose; new components are assimilated into the existing purpose. The whole system works to correct deviations from the purpose.

Organisms achieve strong integration by repeatedly pulling heterogeneous activity back toward a relatively narrow target. This becomes harder as the governed domain grows.

The problem is not just size. There are many dimensions that can limit scale: the number of components, the heterogeneity of components, the geographic extent of components, the rate at which novelty is introduced, the degree of internal conflict, the dispersion of relevant information, and the diversity of operative timescales.

So the scaling problem is this: for an organism to scale its governance, it needs to compile its objective across an increasingly large and heterogeneous system. This means incorporating new components (not just cells now, but animals, plants, geography, weather, etc.) and coordinating those components despite their different and even conflicting goals. It must anticipate and adapt to new and larger shocks. The larger the organism’s governance scales, the more biting Hayekian knowledge problems become—the organism must constrain the very components whose local knowledge and competencies it relies on.

Those are all alignment problems.

These alignment problems require flexibility. To continually create and maintain alignment toward a particular objective across increasingly numerous and capable components, the organism has to somehow pull a changing and growing set of interactions back into a narrow attractor state. It has to somehow do this while relying on its components to supply the local knowledge and competencies the organism needs to solve Hayekian knowledge problems and do complex, specialized work across the volume over which its governance extends.

This is hard.

A more scalable solution is to let go of the fixed target.

3. Transformational alignment

The challenge the organism faces in scaling up its governance is that it must preserve a macro-target despite changing and competent parts, and new dynamics between and among those parts as scale increases. The economy, which operates at a global scale, does something very different. The economy changes its own macro-order so that the parts can continue to belong to it.

Unlike an organism, the economy doesn’t need to unite consumers, workers, firms, entrepreneurs, etc., around one purpose. Rather, the economy flexibly incorporates changes in their own plans—changes in the conditions of supply and demand—into changes in relative prices, which regulate people’s plans into mutual alignment with other people’s plans, not with the economy’s higher-level fixed target.

This is transformational alignment: the higher-level organization changes so that the heterogeneous components remain incorporated into the common order. The higher-level organization maintains governance by giving up a fixed target. It goes with the flow so that the flow remains internally coherent, at the cost of not being able to choose where the flow is going.

These are not mutually exclusive architectures. Real systems combine both. The distinction concerns how much alignment is maintained by pulling components toward a relatively stable macro-target versus allowing the macro-order itself to transform around changes in the components.

To a relatively fixed-target system, novelty is often a cost—an alignment burden. Each shock, each new event, each surprise raises a challenge: how do I make this new thing serve my goal? To a relatively transformational system, novelty can be received as an opportunity: how can the order change so that this thing becomes another participant in it?

This is the scaling strategy: don’t force things into a predetermined purpose. Instead, find a way to create space for another voice.

This gives an alignment hypothesis for why economies can scale across spectacular diversity in a way that’s challenging for organisms. The maximum scale of an aligned system depends in part on how much transformation of the higher-level target or organization it permits.

A human can deal with a lot of novelty and diverse internal and external demands because we can flexibly shift our targets to incorporate them. A brick building cannot because its organization is relatively fixed: once something challenges that, it resists or collapses in a relatively binary manner.

This means that there are tradeoffs between kinds of alignment as a system scales up. Relatively small systems, or systems that face less novelty, may be able to maintain strong goal alignment. As a system scales up or faces more novelty, it may need to exhibit more transformational alignment to maintain governance.

4. But transformational alignment has characteristic failure modes

Giving up alignment to a fixed target has some characteristic failure modes, many of which are subsumed under the heading of externality. The failure mode is a direct consequence of the mechanism. The economy generates its higher-order pattern through the interactions of the participating components. As a result, events that fail to enter the alignment relation will fail to matter to the economy’s governance.

A simple example is pollution. Pollution is relevant to people’s welfare and the efficiency of the economy, but it often doesn’t enter into the interactions between people because the architecture governing those interactions—prices, property rights, contracts, etc.—fail to register it. As a result, the system acts like pollution isn’t happening or doesn’t count, even when events like excess carbon emissions are causing severe problems.

Externality gives transformational alignment its characteristic danger: it can be extraordinarily accommodating toward whatever can participate in steering the higher-level order, while being extraordinarily destructive toward whatever lacks an effective channel for steering the higher-level order.

The same openness that permits transformational systems to scale also enlarges the importance of deciding what receives a channel into the transformation process. The more the system scales, the vaster the capacity it exhibits for ignoring things.

So while the economy’s relation with its higher-level target is very different from how an AI superintelligence is traditionally envisioned, a similar kind of danger remains: things that the economy does not “care” about, in the sense of being able to participate in the transformation of its order, may be crushed as if they do not matter.

5. Implications for AI singletons

AIs may form increasingly integrated societies, potentially using systems analogous to markets, protocols, firms, and so on to govern and organize interactions, just like humans do. The analysis of singletons in terms of transformational vs fixed-goal alignment allows us to ask some important questions.

What holds the higher-scale system together? It could be a shared commitment to a set of values, but it could also be the ability to rearrange the higher-scale system to be compatible with heterogeneous values. What can steer the transformation of the system? Sometimes it is perceptions and expectations relevant to the fixed goal, but it can also be the components competing and cooperating to reallocate scarce resources toward their own interests. And perhaps most importantly, what remains outside the system’s alignment relations? In a fixed-goal system, this would be anything irrelevant to the goal; in a transformational system, this would be anything that fails to enter into the transformation-inducing interactions between the system’s components—i.e., an externality.

One possible scaling principle is that the attainable scale of an aligned system depends partly on how much transformation of its higher-order organization it can tolerate. As heterogeneity, novelty, and dispersion increase, preserving a narrow macro-target may impose increasing alignment costs. Systems that permit more transformation at the higher level may therefore scale farther.

Consequently, there may be a tradeoff between purposive rigidity and organizational scale. Systems that insist on preserving a narrow macro-target may achieve intense local integration but face increasing difficulty expanding their governance. Systems that permit the macro-order itself to transform may be able to align far more heterogeneous components, with more tolerance for perturbation and surprise, across much larger scales.

There is a surprising implication: how safe a future singleton is cannot be inferred from the singleton’s values alone. It may depend instead on the alignment architecture through which its components become, and remain, one larger system.

Two systems might both count as singletons while being held together in radically different ways. A singleton could depend on commands, common goals, incentives, shared signals, constraints, relational dependence, transformational adjustment, or some combination of all of these. None are preferred or to be avoided in the abstract. Each architecture generates different capacities, scaling properties, vulnerabilities, and safety properties.

The economy teaches us about the alignment dynamics by which singleton-like systems become possible. Pursuing a fixed target is powerful, but hard to scale. By enabling its components to transform its macro-structure, the economy induces people to solve the problems of maintaining, growing, repairing, and regenerating it.

Will AI form a global-scale singleton? Any system that approaches that scale must survive the alignment problems of getting there. The more interesting question is therefore: what alignment architecture can survive the scaling process required to govern at a global level?

Further reading

• Nick Bostrom, “What is a Singleton? “
• Nick Bostrom, Superintelligence (in particular, chapter 11 and the section on superorganisms and scale economies)
• Carl Shulman, “Whole Brain Emulation and the Evolution of Superorganisms”
• F. A. Hayek, “The Use of Knowledge in Society”
• Vernon Smith, “Markets as Economizers of Information: Experimental Examination of the ‘Hayek Hypothesis’”
• See also Alexey Turchin, David Denkenberger, and Brian Patrick Green, “Global Solutions vs. Local Solutions for the AI Safety Problem,” for their discussion of a multipolar singleton