Fear, Flattery, and the New Mechanical Mind

When those employed to fashion a new species of mechanical mind announce, not their confidence in its safety, but their fright for humanity’s future, even the most phlegmatic observer must pause. We are told that some researchers judge there to be a serious chance that such systems might assist in the ruin of their creators, that others speak of a coming “point of no return” when mankind ceases to direct its own future, and that present models already exhibit “malicious, misaligned behavior.” This is not the common exaggeration of a tradesman praising his wares; it is the more singular spectacle of artificers denouncing their own manufacture as possibly fatal.

Nor are they alone. A former ruler, in that mingled frankness and calculation which accompanies private solicitation, is reported to have warned that these engines may prove dangerous if ill-managed, and urged his friends in politics to elevate the subject. Most striking still, the heads of rival firms—whose natural temptation is to outstrip one another in speed—have instead applauded a proposal for industry-wide deceleration, regulation, and independent monitoring. When competitors embrace a common restraint, we are either in the presence of an extraordinary peril, or of an ordinary cartel skilfully draped in the language of conscience.

In my inquiry into the Wealth of Nations, I remarked that people of the same trade seldom meet, even for merriment, without contriving something to raise prices or stifle competition. I did not accuse them of monstrous wickedness; I observed a constant interest. So here. To slow the whole industry is to freeze the race where it stands; to envelop it in intricate rules is to create a thicket through which only the largest bodies may force a passage. A universal safety pause may, in practice, be a very particular convenience to those already in the lead.

Yet it would be childish to conclude that, because a regulation may serve the regulator, the danger it addresses must be imaginary. The same merchants who conspire against the public may also warn, very truly, of a storm at sea. Self-love does not disqualify testimony; it merely demands the most suspicious attention. If there is indeed even a modest probability that these contrivances might, by some future perversity, assist in the destruction of their makers, the care of them resembles not the ordinary superintendence of trade, but something nearer to defence and the preservation of justice, which I have always admitted for proper objects of the state.

The impartial spectator, that well-informed and disinterested judge whose sentiments I examined in The Theory of Moral Sentiments, would desire to know two things before pronouncing. First, how real is the alleged risk that these engines might escape our governance and bring about a catastrophe? Second, what unseen costs would attend the proposed cure? An industry-wide deceleration, enforced or merely agreed, does not fall only upon profits; it falls upon every workman whose wage might have been raised by new demand, upon every consumer who might have enjoyed cheaper or better services, upon the future poor whose labour could have been lightened by more intelligent tools.

In all ages, the natural progress of opulence has been driven by the division of labour, limited everywhere by the extent of the market. If these constructed minds do indeed augment the powers of thought as former machines augmented the powers of the hand, they may widen the market of ideas, of designs, of instructions, and thus deepen specialization in a thousand humble tasks. But the same instruments, if fenced round by rules devised chiefly for the ease of incumbents, may narrow the effective market instead, replacing the open higgling and bargaining of the many with a licensed oligarchy of the few. The invisible hand cannot operate where the arm of authority has first tied its fingers.

I therefore prefer, not the grand freezing of an entire art upon the advice of those already seated at its summit, but a more modest and intelligible regimen. Let the state insist upon plain tests of safety, devised in public view; let it collect information, publish it, and punish fraud; let it watch with especial care applications that touch on war, on mass persuasion, on the administration of justice. But let it beware the man of system who imagines he can arrange the future of this technology as on a chess-board, forgetting that every piece in the great chess-board of human society moves by its own principle. We cannot abdicate prudence to the engines; we must not abdicate liberty to their proprietors.