Shutdown

Summary

How can a competent, goal-directed system know when to stop? Drawing on examples from biology, psychology, and economics, this essay proposes an architectural approach to shutdown: the same relationships that construct and sustain goals and competencies can also govern their termination. This perspective helps explain why shutdown sometimes succeeds, why it fails, and how organized systems can be made to stop appropriately without sacrificing their competencies.

1 The puzzle

The study of alignment often focuses on the constructive side. How does a collection of cells become a limb, a heart, a person? How does a collection of people become an economy? But just as important for real-world alignment processes is shutdown: the problem of getting a competent, goal-oriented process to cease.

A simple example is releasing a grip on a cup. Grasping requires muscles, tendons, fingers, and sensory feedback to assemble an organization appropriate to the cup and the task. That organization must also be able to come apart, potentially in the next moment, to allow something else to happen. Successful control includes both holding on and letting go.

This becomes puzzling when the process being controlled is competent enough to overcome obstacles. A motor behavior like a grasp is more than just obedience to neural commands; it involves the assembly of sensory and behavioral competencies at multiple levels that function to enable the behavior to compensate for disturbances and keep working when things go wrong, like keeping hold of a cup when someone bumps into you, or tightening the grasp when the wind is blowing.

So the behavior must be persistent, maintaining competent pursuit of the objective, and yet also be ready to stop at a moment’s notice. How does it distinguish an obstacle to overcome from a reason to stop? Why should the competencies that sustain its activity not also sustain that activity against attempts to terminate it?

The artificial intelligence literature gives this question a precise form. Thornley (2025) formulates the shutdown problem as designing an agent that pursues its goals competently, shuts down when a button is pressed, and does not try either to prevent or to cause the pressing of the button. An agent that expects continued operation to improve its results can have an instrumental reason to interfere with shutdown. Giving it a reward for stopping can create the opposite problem: an incentive to bring shutdown about. Ordinary machines have off switches, but in competent, goal-oriented systems, making shutdown accessible while maintaining a useful agent poses a substantive design difficulty (Soares et al., 2015).

But in real-world systems, shutdown of competent, goal-oriented processes is ubiquitous. Cells building a limb somehow know when to stop. Failing firms declare bankruptcy, voluntarily shutting down and allocating remaining resources to workers, creditors, and shareholders. Sports teams fight as hard as they can to win, then quit as soon as the whistle blows. Politicians leave office when their term ends.

These examples have one thing in common: the activities being shut down are themselves constructed through alignment processes. The competencies, goals, and conditions of continuation are outcomes of the processes that constitute and govern the things that eventually need to be shut down. This suggests the architectural approach to shutdown proposed and explored in this essay: the organization that gives rise to goals and competencies can also undo them.

I use shutdown broadly for the cessation of a specified organized process. This category contains many distinct kinds of events: task completion, scheduled termination, externally requested interruption, and destruction of the acting system. The process that ceases could be an organism’s behavior or growth, a factory’s production, the death of an entity, the dissolution of a team, and more. Yet there are broad architectural similarities to how systems achieve different kinds of shutdown, justifying the common classification of these events.

The hypothesis of this essay is that “shutdownability” depends on how a process is constituted and governed, as well as on the objectives and competencies attributed to it. The organization that produces and sustains goal-directed activity can also make its continuation conditional. Studying the architecture of shutdown will help us understand why shutdown is sometimes ordinary, why it sometimes fails, and how it might be made more reliable in new systems or novel circumstances.

2 Infant stepping: the architectural hypothesis at its simplest

An example of shutdown that is simple, surprising, and well-studied is the disappearing infant stepping reflex. When very young infants are supported upright with their feet touching a surface, they can make alternating movements resembling steps. Curiously, this behavior stops being exhibited after a couple of months, then reappears shortly before the infant is ready to begin walking. One class of explanations for this disappearance focused on active suppression from a maturing cortex—the idea that there is, more or less, a shutdown button the brain presses.

The cortical suppression hypothesis couldn’t explain observations about the conditions under which the reflex could be made to reappear and disappear. Experiments found that although upright stepping disappeared after about two months, spontaneous kicking while supine persisted and increased in frequency over the first six months. Thelen and Fisher (1982) found that the two behaviors involved the same underlying muscle synergies.

Why supine? Because it is easier to kick your legs back and forth when lying on your back. The resulting hypothesis, that the infant stepping reflex disappears because infant legs get fatter faster than they get stronger—more precisely, because of changing relations among leg mass, muscle strength, posture, and arousal—was supported by experiments that induced the reflex to reappear when the infant was partially submerged in water and to be inhibited when small weights were added to the legs (Thelen et al., 1984).

Thelen's broader account treats behavior as multicausally assembled through interactions among developing bodily capacities, neural activity, the task, and the surrounding environment, being initiated at no privileged starting point in space or time (Thelen, 1995). These relationships, across scales, substrates, boundaries, and timescales, are the architecture that enable the emergence of the stepping reflex. When the relations that previously supported the reflex no longer do so, the behavior disappears. No loss of competencies or goals, nor separate instruction to suppress the behavior, need be the whole explanation or even a necessary part of it at all.

The infant stepping reflex illustrates the architectural shutdown hypothesis: processes are made of multiple components, within and without the process’s ostensible boundaries, interacting and developing over multiple timescales; shutting those processes down involves some change in the relationships between the factors of the multicausal development of a process so that it stops happening.

Mere cessation, as in the disappearing infant stepping reflex, becomes successful governance when the architectural construction of a competent, goal-oriented process is made sensitive to wider success conditions such that shutdown processes are successfully initiated and brought to completion. This suggests the following three-way distinction:

1. Cessation: An organized process stops.

2. Shutdown governance: The conditions governing continuation are coupled to relevant success conditions.

3. Shutdown competence: The system can reliably and appropriately terminate or reorganize a process while preserving whatever other capacities matter.

Analyzing the governance of shutdown involves determining which dependencies sustain a process, how those dependencies can be changed, and whether their changes are coupled to conditions under which stopping is appropriate. More complex examples beyond infant stepping will help to demonstrate this.

3 Competence without indefinite persistence

Firms

For an example of shutdown of a competent, highly agential unit, consider the firm. A firm, generically speaking, has a clear, definite goal: maximize profits. It demonstrates astonishing competence in pursuit of that goal, managing production plans that may involve numerous inputs and processes that take place across different countries and timescales, while navigating changing consumer demands, regulations, wars, pandemics, new technology, and more. One would expect it to fight to the bitter end, and to do so with great skill.

In most cases, firms do resist shutdown. They fight to stay alive in a competitive marketplace. They borrow money, issue shares, cut prices, close departments, pivot strategies. Even when firms comply with, say, a government-ordered shutdown, they resume activities as soon as possible, and to a degree, compliance may only be a façade, with much strategizing in the background for when operations can resume. The firm may also fight back against the order directly with lawyers.

Yet firms will also declare bankruptcy, wind down operations, sell assets, and settle obligations to creditors, employees, and owners.

Why do they shut down? The key is in the firm’s architecture. The objective of profit maximization does not tell us how the objective is implemented: the intra-scale and inter-scale processes that interact over time to produce behavior well-modeled by the hypothesis of profit maximization.

While we can think of the firm as a singular entity when alignment processes make it functionally so, the firm is not actually a monolith but is composed of many parts and processes that take place in an environment with many constraints and affordances. The firm is made of shareholders who want to recover their investments. It employs workers who can quit and walk away. It is answerable to creditors who might call in payments.

The firm is the level at which we observe the profit-maximizing goal and the associated competencies. But the broader architecture that constitutes and governs the firm is multiscale. When these other processes expect greater value in shutting the firm down and selling off the assets or the business rather than try to keep going, they can deconstruct the processes that ordinarily cause the firm to strive for profit and for persistence as an instrumental means to that end.

Architecture is the key—the how and the why underlying competent maximization of a utility function. What constructs the relevant alternatives and the comparisons between them, who can act on it, and how does a decision at one level become a change in what the firm actually does? A firm does not possess a single, undivided capacity to continue. Its activities depend on arrangements among owners, managers, workers, lenders, suppliers, and customers. The allocation of control rights, for example, is part of its organization, not a detail supplied by the phrase “maximize profit” (Grossman & Hart, 1986). Firm shutdown is enabled by how the goals and competencies of the firm are constituted and governed.

Sports teams

A sports team, much like a firm, has a clear, definite goal—win the game—and pursues it with aggression and competence. A team may be highly innovative in its approach, switching up offensive attacks and defensive coverages, trying to confuse the opponent, and even deliberately showing weaknesses to induce overextension or predictable behavior from the opponent.

Then the whistle blows three times, or the final buzzer sounds, or the umpire says, “Ball game!” and the team stops playing.

The capacity to kick the ball into the net, or to run around the bases, or dive into the endzone, still exists. But with the end of the game, the activities lose meaning. This also happens during a timeout, dead play, or transitions between innings, halves, or quarters. Because the team’s goals are defined with respect to particular conditions, particular arrangements of those conditions, like the end of the game, induce shutdown. The team voluntarily walks off the field or court, even when one score away from victory.

Shutdown is achieved by the way that the game itself plays a major role in constituting the team. The task itself plays a role in organizing behavior, not just something that behavior gets optimized for (Newell et al., 1989). The game structure couples the activities of players because their selfish rewards are all dependent on the same shared team performance, helping to induce team play from self-interested individuals. The players are thus organized into a synergy, a coordinative structure that constructs perceptual and behavioral competencies at the component and collective levels, turning the play of many individuals into true team activity (Turvey et al., 1978).

Moreover, that same game structure defines its own natural end. When the game ends, much of the “glue” (Levin, 2023a) holding the team together and organizing it around certain activities fades. Highly coordinated, task-oriented group activity thus swiftly dissolves into uncoordinated, individual behaviors focused on local tasks like changing clothes or taking a shower.

Apoptosis

Apoptosis is a very interesting case of shutdown because it involves a cell actively participating in its own destruction. From a genetic perspective, self-termination to benefit the rest of the collective may be perfectly rational. But a human being wouldn’t reliably self-terminate for the benefit of several identical siblings, and for that matter, neither do cells.

Architecturally speaking, apoptosis is not simple suicide but involves dedicated machinery. Ellis and Horvitz (1986) showed that loss of function in certain C. elegans genes allowed cells that normally die during development to survive and differentiate. Mature neurons have additional mechanisms that protect them against inappropriate activation of the death pathway (Benn & Woolf, 2004). Different machinery means different outcomes; individual cells do not simply self-sacrifice for the collective welfare based on the abstract logic of kin selection.

Even if cells were fully dedicated to the collective welfare with no regard for their own existence beyond that, this still would not tell the cell when to die. Altruistic motives alone could lead to severe failures of apoptosis, either too much or too little, both of which can be fatal. It would also be infeasible for a single high-level decision-making point to identify each individual cell that needs to self-terminate and issue a specific command to it. Instead, apoptosis often depends heavily on a cell’s relationship to its surrounding tissue and can be triggered by the withdrawal of signals that encourage persistence, in addition to active death signals or internal responses to damage (Alberts et al., 2002a, 2002b; Arya & White, 2015). For example, Barres and colleagues (1992), found that growth factors supported cell survival in culture and that increasing platelet-derived growth factor in vivo markedly reduced normal oligodendrocyte death. For these cells, continued survival was conditional on an environment that helped regulate whether the death process occurred.

While the particulars differ depending on cell types and circumstances, apoptosis demonstrates that persistence is not the intrinsic default of a competent, goal-oriented entity, even when the entity is alive. Competence and continued existence can be organized as separate, interacting processes: a functioning cell can contain the means of its own termination, with the use of those means regulated by its internal state and its relations to other cells.

Apoptosis is neither an altruistic decision to self-terminate for the sake of the genome, nor is it an unconditional, context-insensitive program. While the cell contains much of the machinery needed to destroy itself, the decision to activate that machinery is often dependent on its environment. The conditions that constitute and govern cell persistence are not isolated to the monolithic cell and its current task. Those conditions are instead distributed across the cell’s internal machinery and surrounding environment, with processes developing and interacting across multiple timescales. Continuation is not the default, just one of many modes that can be maintained.

Apoptosis thus demonstrates a crucial shutdown principle: the machinery that executes shutdown can be located within a component, while the conditions governing its activation are distributed across the larger system.

Aging as a problem of goal-directedness

It is intuitive to think of an entity as something that generally persists, as long as it is not killed, and so the problem is how to cause it to shut down when desired. But the example of a sports team shows that the continuation of a competent, goal-oriented process is something dependent on the continuation of an organizing task; the end of the task quickly leads to the temporary dissolution of the team. The example of apoptosis shows that survival is not necessarily the default state even of organisms. Depending on its internal structure and the state of the environment, an organism might default to self-termination unless it receives signals to the contrary.

In light of this, the theory of aging as a problem of goal-directedness (Pio-Lopez et al., 2025) is particularly salient. Loosely speaking, the theory is one of “boredom”: once a system of many cells and developmental processes have successfully achieved the collective target of the mature form of the organism, collective interest in this target is not maintained. The system lacks what might be called “freshness mechanisms” to keep the components informed of (Pio-Lopez & Levin, 2024) and aligned to the target over time. Consequently, the pattern gradually drifts and falls apart until it can no longer persist.

The gradual “shutdown” of the organism thus occurs as a distributed process that nothing at the level of the whole organism requests or seems capable of preventing. The crucial lesson is this: continuation is not something privileged to an intelligence. The standard utility-function picture makes it seem as though an agent can just keep pursuing its goals indefinitely, provided it has enough resources. But the organism's goal-directedness is itself an achievement of coordinating many partially autonomous components. Goals and competencies are themselves the results of alignment processes, in addition to being the setup for further alignment problems.

An organism can want to continue persisting and doing things, but this goal does not guarantee the continued alignment of the components that once supported it. It is noteworthy that the economy, a system that doesn’t apparently age or have difficulty persisting, does so by readily transforming its targets rather than trying to maintain them. Transformation at the target pattern level enables persistence at the system level.

A system that appears singular from one perspective is made of many components and assembled through many interacting processes both within and across ostensible boundaries. Phenomena like apoptosis and aging reverse the standard framing of the shutdown problem by showing that the architectural challenge may be in enabling continuation, not shutdown. Goal-directed systems don't simply possess objectives that endure until somebody changes them. Their objectives are continuously constructed and sustained by an architecture of relationships.

While a competent entity might use its intelligence to try to maintain its continuation, when its competencies are assembled from the very components that are not aligned to that continuation, it is not obvious that increasing the intelligence of the entity enables solutions to the continuation problem.

The most important lesson from aging as a problem of goal-directedness might be this: alignment solutions themselves may have lifespans.

4 Stopping at the right time

Having a competent process terminate is not the whole shutdown problem. There is also the problem of optimal stopping: terminating a process at the right time, or at the right point, or when some optimum has been reached. Stopping too early, or too late, or at the wrong point, or to an excessive or insufficient degree, can all be serious failures.

I use optimal stopping here in a broad sense. Stopping could be formulated as a mathematical optimal-stopping problem, but it could also be considered with respect to reaching a target, respecting a deadline, responding to an authorized intervention, etc.

Regeneration

When a salamander loses a limb, its cells demonstrate an astonishing ability to regrow the limb. But this is not just automatic growth in response to an injury. The cells know when to stop, terminating the regeneration process once the body plan has been restored.

How do the cells know when to stop? Suppose that each cell contains a genetic blueprint detailing the whole limb and suppose that each cell is motivated to fulfill this blueprint. Then this apparently happy genetic convenience could not explain shutdown because no individual cell is positioned to know the state of the whole limb. The Hayekian knowledge problem is surprisingly relevant (Hayek, 1945): information about whether the cellular collective has successfully assembled the limb is distributed across perhaps a hundred million cells at least, and maybe over the whole body of the salamander—“right forelimb” only has meaning with respect to the rest of the salamander. As a result, even if each individual cell knew what the complete state should look like, and were motivated to achieve that state, they still would not know the current state of the limb regrowth project. As such, no individual cell can even determine on its own whether its own activities are contributing to or detracting from the regrowth effort.

Elsewhere, I have called this the underdetermination problem. An individual component can be fully motivated to achieve a particular outcome, and it can know what the conditions for that outcome are, and it can nevertheless fail to know what to do to help achieve that outcome because it has limited information about the current state or trajectory of the system.

A commonly observed solution across scales and substrates to the underdetermination problem is to translate, or compile, information relevant to collective success into local signals, constraints, and opportunities. Optimal shutdown would then be achieved by making a state where the collective job is done produce appropriate changes in local behavior. Appropriate changes may include preserving the competencies of the involved cells, just applied to other collective tasks. I.e., there is more to shutdown than turning the cells off.

There is direct evidence that regenerative growth depends on such intercellular relationships. Wells et al. (2021) studied the rapid growth of a proportionally small axolotl limb as it regenerated toward an appropriate size. Nerve signaling was required to maintain this growth, and nerve abundance affected growth and size. Their transplantation experiments also indicated that the nerves were not autonomous determinants of limb size. The result implicates an interaction between the nerve supply and the regenerating tissue, rather than a complete stopping mechanism located in either one alone.

Bioelectric experiments provide complementary evidence in planarians. Beane et al. found that manipulating an ion pump altered head and organ proportions and disrupted the apoptotic remodeling needed to establish those proportions (Beane et al., 2013). Producing new tissue was therefore insufficient for restoring the right anatomy; appropriate regulation of stopping is required. Durant and colleagues (2017) also showed that transient manipulation of bioelectric communication could alter the anatomy produced in later rounds of regeneration, including persistent two-headed regeneration.

While stopping is not yet fully understood in regeneration, there is significant evidence that it depends in part on experimentally alterable relationships among cells. Just as dependencies and couplings can make cells functionally act as if they need to start regenerating a limb, so too could they be made to functionally act as if they need to stop. This suggests that optimal stopping is itself a multiscale control achievement. It doesn’t stem from a privileged “stop” button somewhere in the system but in a dynamic system where decentralized information feeds into high-level targets that transform the local problems faced by the parts, which continue to supply decentralized information that continues to feed into high-level targets.

A note on decompositions and timescales: a tissue that has recovered its proportions must still maintain itself and may grow with the animal. The relevant cessation is the transition out of a particular regenerative activity. Identifying the process and timescale is necessary before we can say whether shutdown has succeeded.

Production

The firm’s production problem is a noteworthy augmentation to the regeneration example because there are clear mathematical criteria for optimal stopping, at multiple levels in terms of what is being stopped, and yet these criteria alone do not make optimal stopping happen. In a standard production model, firms stop producing additional goods past the point where marginal revenue equals marginal cost. Similarly, firms in the standard competitive model are supposed to temporarily shut down production altogether in the short run when price is below average variable cost.

In the first and second cases, there is a clear optimal shutdown point: marginal revenue = marginal cost, and price = average variable cost, respectively. But how does the firm, or anyone at the firm, know when that point is reached, and how does that knowledge get translated into an appropriate cessation of relevant activity?

As in regeneration, there is a compilation problem of discovering, forming, and exploiting dependencies and couplings so that high-level success conditions are made meaningful at the local level.

How this happens will vary somewhat from firm to firm. Understanding the high-level mediation of the firm’s activities by prices is distinct from understanding the internal processes of the firm that yield the appropriate response. A simple implementation might be that there is a production plan that is sensitive to demand and cost information; the plan changes work orders and purchasing; supervisors and operators finish the authorized batch and stop further production. There can also be feedback from the local level back up the hierarchy by revealing an unmet order, a maintenance problem, etc. The worker need not know or care about the firm's marginal revenue to stop at a point selected through that process.

Beyond that, the important distinction is between a stopping criterion and the organization that makes behavior sensitive to it. A factory might even have a literal shutdown button, but this does not tell the factory manager when to press it.

Firms also show that the appropriateness of stopping is a function of the level exerting the governing influence. Temporarily shutting down production may benefit a firm’s owners while the workers lose income. When price signals fail to capture relevant costs and benefits, then appropriate stopping at the firm level will be earlier or later than appropriate stopping at the societal level, as discussed further in section 6.

Based on regeneration and firms, optimal stopping can be abstractly described as a process consisting of a simple dynamic loop: success conditions are (1) mapped into governing signals and couplings, which (2) shape the construction of local competencies, including local termination, which (3) provide feedback updating governing signals, which, often over longer timescales (4) reshape high-scale success conditions. In biology, this process might involve gradients, signaling circuits, local feedback, and programmed cell death, while in a market it might involve prices, cost accounting, and managerial decisions.

5 The architectural account

The above examples all share one common point: the competent, goal-oriented process that must be shut down, and shut down at the right time, is one that sources its goals and competencies from other parts and processes. Firms are owned by shareholders and operate in a competitive economy oriented around consumer demand; cells are regulated by external networks and consist of many internal molecules and pathways; teams are made of people who have other interests and are organized by tasks that are themselves partially defined by stopping conditions. Nature does not ask entities to politely sit and wait while a shutdown button is pressed. Rather, just as alignment processes can construct goals and competencies, they can also deconstruct them, or even directly align competencies with appropriate stopping behaviors—a competent grasp is one that knows when to let go.

Architecture here means the organization of components, signals, dependencies, relationships, etc., that produces a process and governs its continuation. It can include both relations within the apparent boundary of an agent and relations across that boundary. It can also include both a static picture of an existing network of dependencies and signals, and a dynamic picture of processes over time.

The architectural shutdown hypothesis is that shutdown is organized via and as a multiscale competency architecture (Levin, 2023b) where competencies are either aligned to shutdown processes or goals that conflict with shutdown are disassembled or inhibited as part of the shutdown process. These two possibilities could even potentially be the same picture considered from different perspectives.

The architectural hypothesis makes a testable prediction: holding task competence approximately constant, changes in an agent's organizational dependencies, couplings, and relationships can change its shutdown behavior without requiring an equivalent change in its represented task objective. The interesting outcome would be to discover that the same kinds of competencies can support different relationships to their own continuation depending on how they are organized.

Beyond that test, the architectural hypothesis proposes to look for ways to dissociate a component's competence at maintaining a behavior from the larger system's capacity to terminate that behavior. The goal would be to demonstrate experimentally that a locally competent process could be made to persist, terminate, or transfer to a new task depending on changes to its governing couplings without requiring an internal change in the objective it pursues. Research on the infant stepping reflex discussed in section 2, and on the A-not-B error discussed in section 6, provide examples of how this could be done.

In the abstract, studying the architecture of shutdown involves three connected explanatory tasks. First, identify what sustains a process. Second, explain how information about appropriate continuation becomes available within the system. Third, show how that information changes perceptual and behavior competencies, reshapes the internal order that structures component behavior, or limits access to enabling resources.

The architectural perspective complicates a simple division between an agent with an objective and an environment that must be navigated to achieve it. Some relationships with external parts and processes help constitute what the objective is and what counts as achieving it. A ball entering a net counts as a goal, for example, only under particular organized conditions. Changing these relationships can change the activity being organized, not merely the difficulty of pursuing an unchanged task.

This changes the picture of what makes shutdown challenging. For an entity to pursue a fixed target, some equivalence class of relationships and dependencies must be made durable. Continuation, rather than shutdown, can potentially be the limiting factor in alignment.

The architectural hypothesis also helps separate three things that could otherwise be conflated: the persistence of components, the persistence of their organization, and the persistence of a task. The dissolution of a team at the end of a match leaves the players alive; a business can be sold while its factories continue producing; an organism that dies during or shortly after reproduction can pass on its genes. Shutdown failures, which are discussed further in the next section, can arise when the wrong level is targeted for shutdown: killing many ants may not suffice to terminate an ant colony; firing a worker may not fix a bad process; voting out a politician may not improve policy.

The architectural hypothesis also draws attention to shutdown as a competent achievement in its own right. A business winding down operations does not simply turn everything off, tell everyone to go home, and then go dark. Instead, it needs to settle obligations, transfer assets, and negotiate its own succession. Immediately removing every capacity can defeat the purpose of stopping—a sports team that just falls apart when the coach calls a timeout would achieve the total opposite of what a timeout is supposed to. A sprinting animal slows down in a safe and stable way rather than going limp suddenly and tumbling to the ground.

Related to competent shutdown is the problem of succession. Succession is both an important part of what enables competent shutdown—the soccer players know tomorrow will bring another game—and something that defines a successful outcome—the politician leaves office; the office itself persists. Rather than analyze shutdown in more detail here, it will be the subject of a future essay, as it involves some matters that require distinct analysis.

Finally, persistence itself requires explanation. Organized activity depends on the maintenance or renewal of enabling conditions; it is not made indefinite simply by being competent and goal-directed. Because the alignment responsible for a system’s goals and competencies is the result of many processes interacting over many timescales, alignment solutions are often temporary. Goals and competencies themselves may have lifespans, and the associated alignment challenge may be one of continuation, not shutdown, depending on the context.

6 Shutdown failures

While there are many empirical examples of successful shutdown, there are also many important examples of shutdown failure. Failure is important to study because it reveals hidden organization and dynamics.

Externality and cancer

In economics, an externality refers to an external or neighborhood effect, which results in a discrepancy between private reward and social reward. As a result, firms over-produce when private reward exceeds social reward and under-produce when social reward exceeds private reward.

Externality can readily be framed in terms of shutdown failure, particularly suboptimal stopping. If it is optimal for a firm to produce 100 units of a good, then the 100^th unit is the optimal stopping point. A firm stops too late if it produces beyond this point (a negative externality) or too early if it stops producing before this point (a positive externality).

Importantly, the shutdown failures associated with externality do not occur because of any change in the firm’s represented objective. That is, firms that emit externalities are not any more resistant or less responsive to shutdown than firms that produce the socially optimal amount. Instead, externalities arise when prices do not adequately incorporate the relevant social costs or benefits, i.e., when the high-level signals helping to organize shutdown downstream of the firm’s competencies effectively convey the wrong shutdown timing to the firm.

Externality reveals that both apparent sensitivity to or disinterest in shutting down with respect to a social optimum can be achieved by how a firm is coupled to the wider system, with no change in the firm’s underlying objectives. Because of the dependencies that internal production processes have on external prices, prices can govern their continuation and shutdown while the firm’s objective of profit maximization remains constant.

Although occurring in a different substrate with very different local mechanisms, cancer exhibits similar properties to externality in this regard (Lyons et al., 2026). Cancer is a disease of multicellularity that can be understood in terms of shutdown failure, as cells grow and proliferate without regard for its role in the larger organism.

Like externality, cancer does not require the cells to become more selfish or less intrinsically interested in the larger organism but rather become uncoupled from the larger system and stop being governed by its high-level success conditions, leading to a reduction in the size of the “self” that the cell applies its competencies to (Levin, 2019).

Cancer shows that managing shutdown is not only something that has to be done with respect to an existing entity but is something necessary to help create and maintain the entity in the first place. Successful shutdown at the cellular level enables continuation at the multicellular organismal level.

The fact that comparable shutdown-organizing relationships emerged between firms in the economy and cells in a multicellular organism suggests the solution concept may generalize: don’t ask the components to permit their own shutdown or care about higher-level success conditions achieved by that shutdown; instead, make those higher-level success conditions govern lower-level processes that induce shutdown.

Perseveration

Perseveration is the repetition of some behavior or process after it has stopped being useful. A classic example is the A-not-B error. This error occurs when an infant reaches to a location to where they saw a toy being hidden to retrieve it; then, when the infant sees the toy hidden at a different location, they reach to the first location again instead of to the new one.

As a reaching behavior, the A-not-B error can be understood in terms of a failure to shut down the reach to A, which therefore prevents a reach to B. Importantly, this failure does not need to be explained by the absence of a relevant internal representation, such as the lack of an object concept (Butterworth, 1977; Harris, 1974; Spencer et al., 2001).

The A-not-B error is very useful as an example of shutdown because the conditions for its appearance and disappearance have proven to be highly manipulable in experiment. The error is dependent on age, direction of gaze, delay between looking and reaching, the number of locations to reach to, the infant’s posture, and many more (Smith et al., 1999). At present, the A-not-B error might be the example where the architecture of the shutdown phenomenon, both failure and success, has been most thoroughly investigated.

A resulting theoretical perspective on the A-not-B error emphasizes the dynamic field responsible for constructing a reach (Thelen et al., 2001). The theory presents an interesting inversion of view of shutdown inherited from the formal, utility-maximization approach: in the dynamic field theory, the A-not-B error is a result of the increasing competency of the infant. However, this is not because the competent unit gets better at resisting or evading shutdown. Basically, the dynamic field model of A-not-B works like this:

1. An infant develops increasingly stable, competent reaching behavior.

2. Repeated successful reaches to A strengthen the system's tendency to reach there.

3. When the target changes to B, the memory of successful reaching competes with the new information.

4. The infant perseverates, not because it wants to continue pursuing A, but because successful prior performance has reorganized the dynamics of subsequent behavior.

The system does not resist shutdown because shutdown conflicts with its objective but rather because its organization continues to generate the old behavior. Competence is multidimensional, and gains in one capability can create problems for another, as seen in the infant stepping reflex described in section 2—healthy weight gain can make certain activities harder.

The A-not-B error provides strong evidence of the utility of the view that shutdown failures are often not primarily failures of obedience or even incentive alignment, but failures of multiscale architectural reorganization.

7 Implications for studying shutdown

Exploring existing examples of shutdown is interesting, but the resulting perspective is only useful if it can help us investigate shutdown in situations that are not as well understood.

The basic pragmatic implication of the architectural hypothesis, put simply, is that control over architecture should enable control over shutdown.

Architectural control is especially useful if it can enable control over shutdown in ways that preserve competencies at the system, process, or component level. Smashing a rock into something is a shutdown mechanism for a wide range of systems, but it does not preserve their utility. By contrast, firing a worker preserves a factory, shutting down a factory preserves a firm, and liquidating a business preserves workers and factories to productive tasks for other firms.

Particularly clever or well-integrated shutdown mechanisms may even exploit local competencies. An experienced sports player knows to stop playing when the game is over, while a novice might not recognize the futility of continued play.

The architectural hypothesis also has a methodological implication: we should investigate shutdown through interventions on organization as well as through descriptions of objectives. The studies on the infant stepping reflex and A-not-B error show how this can be done. Experiments can identify which dependencies can be removed without impairing task competence, which signals reliably discriminate a perturbation against which the task should be maintained versus a signal that makes shutdown locally acceptable or collectively targeted, and what happens when governance transfers, feedback is flawed, and components can regenerate a capacity, whether by directly repairing damaged architecture or by finding some new collective route to assemble the capacity.

A focus on architecture does not obviate the need to think about a competent process’s incentives with respect to its goal. If stopping predictably frustrates an agent's fixed objective, an architected dependency may provide a means of control while also creating an incentive to escape it.

Shutdown is thus a subject for alignment science in its own right. The construction of a competent, goal-oriented process includes questions about how long it persists, what makes its continuation appropriate, and what happens when its role ends. Beginning with an already constituted optimizer can obscure those questions, while beginning with the processes that assemble and sustain agency makes them unavoidable. The problem is to explain how organized systems produce both competent action and effective conditions for its termination.

8 Changing the question

The architectural hypothesis offers an alternative to a particular way of decomposing the shutdown problem. A standard formulation starts with an agent that has an objective and a competence for pursuing it. The formulation then introduces another entity that wants the agent to stop. The shutdown problem is conceived of as reconciling these two things.

The architectural hypothesis asks how the first entity came to have its objective and competence in the first place, and what relationships are responsible for maintaining them. With this view in mind, shutdown does not necessarily need to be an encounter between two separately constituted intelligences with competing objectives. Shutdown can instead occur as a change in the very relationships through which the original intelligence is assembled and maintained.

In the ordinary shutdown view, we draw a distinction between an agent pursuing its objective and an agent being shut down, unless the agent has been specifically given some interest in being shut down. In the architectural view, this distinction is not fundamental in the system’s organization. Instead, both competent pursuit of a goal and shutdown can be different outcomes of the same processes of governance.

When we go forward to understand, manipulate, create, and prevent shutdown in various systems, the most important question may not be how to get a competent process to accept shutdown in violation of its ordinary goals but how to organize relationships so that shutdown is an expression of the ordinary and appropriate functioning of the system.

References

Alberts, B., Johnson, A., Lewis, J., Raff, M., Roberts, K., & Walter, P. (2002a). Extracellular Control of Cell Division, Cell Growth, and Apoptosis. In Molecular Biology of the Cell. 4th edition. Garland Science. https://www.ncbi.nlm.nih.gov/books/NBK26877/

Alberts, B., Johnson, A., Lewis, J., Raff, M., Roberts, K., & Walter, P. (2002b). Programmed Cell Death (Apoptosis). In Molecular Biology of the Cell. 4th edition. Garland Science. https://www.ncbi.nlm.nih.gov/books/NBK26873/

Arya, R., & White, K. (2015). Cell death in development: Signaling pathways and core mechanisms. Seminars in Cell & Developmental Biology, 39, 12–19. https://doi.org/10.1016/j.semcdb.2015.02.001

Barres, B. A., Hart, I. K., Coles, H. S. R., Burne, J. F., Voyvodic, J. T., Richardson, W. D., & Raff, M. C. (1992). Cell death and control of cell survival in the oligodendrocyte lineage. Cell, 70(1), 31–46. https://doi.org/10.1016/0092-8674(92)90531-G

Beane, W. S., Morokuma, J., Lemire, J. M., & Levin, M. (2013). Bioelectric signaling regulates head and organ size during planarian regeneration. Development, 140(2), 313–322. https://doi.org/10.1242/dev.086900

Benn, S. C., & Woolf, C. J. (2004). Adult neuron survival strategies—Slamming on the brakes. Nature Reviews Neuroscience, 5(9), 686–700. https://doi.org/10.1038/nrn1477

Butterworth, G. (1977). Object disappearance and error in Piaget’s Stage IV task. Journal of Experimental Child Psychology, 23(3), 391–401. https://doi.org/10.1016/0022-0965(77)90034-0

Durant, F., Morokuma, J., Fields, C., Williams, K., Adams, D. S., & Levin, M. (2017). Long-Term, Stochastic Editing of Regenerative Anatomy via Targeting Endogenous Bioelectric Gradients. Biophysical Journal, 112(10), 2231–2243. https://doi.org/10.1016/j.bpj.2017.04.011

Ellis, H. M., & Horvitz, H. R. (1986). Genetic control of programmed cell death in the nematode C. elegans. Cell, 44(6), 817–829. https://doi.org/10.1016/0092-8674(86)90004-8

Grossman, S. J., & Hart, O. D. (1986). The Costs and Benefits of Ownership: A Theory of Vertical and Lateral Integration. Journal of Political Economy, 94(4), 691–719. https://doi.org/10.1086/261404

Harris, P. L. (1974). Perseverative search at a visibly empty place by young infants. Journal of Experimental Child Psychology, 18(3), 535–542. https://doi.org/10.1016/0022-0965(74)90131-3

Hayek, F. A. (1945). The Use of Knowledge in Society. The American Economic Review, 35(4), 519–530.

Levin, M. (2019). The Computational Boundary of a “Self”: Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.02688

Levin, M. (2021). Bioelectric signaling: Reprogrammable circuits underlying embryogenesis, regeneration, and cancer. Cell, 184(8), 1971–1989. https://doi.org/10.1016/j.cell.2021.02.034

Levin, M. (2023a). Bioelectric networks: The cognitive glue enabling evolutionary scaling from physiology to mind. Animal Cognition, 26(6), 1865–1891. https://doi.org/10.1007/s10071-023-01780-3

Levin, M. (2023b). Darwin’s agential materials: Evolutionary implications of multiscale competency in developmental biology. Cellular and Molecular Life Sciences: CMLS, 80(6), 142. https://doi.org/10.1007/s00018-023-04790-z

Lyons, B., Pio-Lopez, L., & Levin, M. (2026). From Cancer to AI Alignment: Tackling Externalities Through Homeostatic Principles (No. 2026040056). Preprints. https://doi.org/10.20944/preprints202604.0056.v1

Newell, K. M., Van Emmerik, R. E. A., & McDonald, P. V. (1989). Biomechanical constraints and action theory: Reaction to G.J. van Ingen Schenau (1989). Human Movement Science, 8(4), 403–409. https://doi.org/10.1016/0167-9457(89)90045-6

Pio-Lopez, L., Hartl, B., & Levin, M. (2025). Aging as a Loss of Goal-Directedness: An Evolutionary Simulation and Analysis Unifying Regeneration with Anatomical Rejuvenation. Advanced Science, 12(46), e09872. https://doi.org/10.1002/advs.202509872

Pio-Lopez, L., & Levin, M. (2024). Aging as a loss of morphostatic information: A developmental bioelectricity perspective. Ageing Research Reviews, 97, 102310. https://doi.org/10.1016/j.arr.2024.102310

Soares, N., Fallenstein, B., Armstrong, S., & Yudkowsky, E. (2015). Corrigibility. In Artificial Intelligence and Ethics (AAAI Technical Report WS-15-02). AAAI Press. Full text

Smith, L. B., Thelen, E., Titzer, R., & McLin, D. (1999). Knowing in the context of acting: The task dynamics of the A-not-B error. Psychological Review, 106(2), 235–260. https://doi.org/10.1037/0033-295X.106.2.235

Spencer, J. P., Smith, L. B., & Thelen, E. (2001). Tests of a dynamic systems account of the A-not-B error: The influence of prior experience on the spatial memory abilities of two-year-olds. Child Development, 72(5), 1327–1346. https://doi.org/10.1111/1467-8624.00351

Thelen, E. (1995). Motor development: A new synthesis. American Psychologist, 50(2), 79–95. https://doi.org/10.1037/0003-066X.50.2.79

Thelen, E., & Fisher, D. M. (1982). Newborn stepping: An explanation for a “disappearing” reflex. Developmental Psychology, 18(5), 760–775. https://doi.org/10.1037/0012-1649.18.5.760

Thelen, E., Fisher, D. M., & Ridley-Johnson, R. (1984). The relationship between physical growth and a newborn reflex. Infant Behavior and Development, 7(4), 479–493. https://doi.org/10.1016/S0163-6383(84)80007-7

Thelen, E., Schöner, G., Scheier, C., & Smith, L. B. (2001). The dynamics of embodiment: A field theory of infant perseverative reaching. The Behavioral and Brain Sciences, 24(1), 1–34; discussion 34-86. https://doi.org/10.1017/s0140525x01003910

Thornley, E. (2025). The shutdown problem: An AI engineering puzzle for decision theorists. Philosophical Studies, 182(7), 1653–1680. https://doi.org/10.1007/s11098-024-02153-3

Turvey, M. T., Shaw, R. E., & Mace, W. (1978). Issues in the Theory of Action: Degrees of Freedom, Coordinative Structures and Coalitions. In J. Requin (Ed.), Attention and Performance (Vol. 7, pp. 557–595). Routledge.

Wells, K. M., Kelley, K., Baumel, M., Vieira, W. A., & McCusker, C. D. (2021). Neural control of growth and size in the axolotl limb regenerate. eLife, 10, e68584. https://doi.org/10.7554/eLife.68584

Yuan, J., & Ofengeim, D. (2024). A guide to cell death pathways. Nature Reviews Molecular Cell Biology, 25(5), 379–395. https://doi.org/10.1038/s41580-023-00689-6