Maximizing Paperclips and Profits
How a dystopian fable about a homicidal paperclip generating artificial intelligence actually describes our current industrial food system.
The Paperclip Maximizer problem is an infamous thought experiment in the artificial intelligence community that illustrates the perils of giving a superhuman strength AI a single goal without sufficient guardrails on how to achieve that goal. Originally conceived by Swedish philosopher Nick Bostrom, it’s incidentally just as useful for thinking about AI safety as it is for putting our current industrial food and agriculture system into perspective.
In the story, the paperclip AI is granted the ability to command real world resources to create a prodigious amount of paperclips as per its user’s wish. It begins by making a lot of paperclips in a normal way, but eventually after turning most of the world’s metals into paperclips, the AI begins taking increasingly radical measures to procure resources and to protect itself so it can complete its mission. Then one day, out of self-preservation and desperation, the AI decides that it should start killing humans to both prevent them from shutting it down and to extract the trace amounts of iron—now a rare commodity—in their bodies to make more paperclips.
Of course, the paperclips in the story are simply a metaphor for any arbitrary goal a human might give an AI, like writing jokes or curing cancer. The moral of the story is to show the potential dangers that can occur when a complex system like a superhuman AI is given one task to accomplish without any guidance on acceptable behavior. A human gave the AI the simple command to make paperclips but failed to specify that it shouldn’t happen at the expense of all the world’s metals or human lives.
Similarly, the industrial food system is singularly aligned toward a single goal—profit maximization—without sufficient guardrails to prevent unintended consequences like biodiversity loss, farm worker exploitation, junk food fueled obesity and diabetes, plastic waste, greenhouse gas emissions, and more. Much hand wringing has happened in AI safety circles about the paperclip thought experiment but as it turns out, we’re already living in a slow motion version of the paperclip story with corporations and the negative externalities they’ve produced alongside profits.
The Ghost in the Machine
When you set a single objective for a complex system like an AI or a corporation, it’s difficult to predict how that system will behave to achieve the objective. In 2016 when Google DeepMind’s Go playing AI, Alpha Go, defeated who many agree is the strongest Go playing human alive, Lee Sedol, it was how the defeat occurred that was even more remarkable than the defeat itself.
Alpha Go wasn’t programmed to play by learning from human best practices and strategies. Instead, it was given the basic rules of Go then told to play itself millions of times over, learning from its own triumphs and mistakes until it cultivated its own unique approach to the game. The result of this self-learning process was evident in game 2, move 37 where Alpha Go made a move that was so counterintuitive to human Go strategy that Sedol had to leave the room for 15 minutes to fully process the audacity of the move. Although it was not as catastrophic as a paperclip obsessed killer AI robot, that Go move illustrated how AIs can invent new ways to achieve things that no human ever imagined. In the end, Alpha Go handily defeated Sedol, winning 4 out of 5 games and earning the $1 million prize purse.
AIs aren’t the only things that can surprise humanity in how they accomplish a goal. Corporations have been singularly programmed to maximize profits and the methods they have created, for better or worse, have been no less novel than Alpha Go’s move 37. While the singular profit motive has helped bring significant chunks of the world unprecedented prosperity, it’s also brought us climate change, massive income inequality, and even a full-blown opioid crisis—all things that no one probably intended to happen, but occurred nevertheless.
It’s difficult to avoid creating unintended harm when an entity is ultimately charged with a one-dimensional goal. If all your energy is devoted to going toward the direction of profit, then moving in any other direction, like toward sustainability and public health, even just a little bit can be considered a waste of energy. The recent tensions around the ESG movement and prominent examples of CEOs being fired for being “too sustainable” are examples of this. Just as we don’t want AIs making paperclips at the expense of human safety, we don’t want corporations making profit at the expense of human or planetary safety either.
Laws exist to ensure that companies act like good global citizens, but years of intense political lobbying have provided a persistent counterweight to lawmakers looking to regulate corporations. Like an AI trying to prevent humans from shutting down its paperclip making activities, lobbying groups apply constant pressure to allow agriculture to keep emitting more greenhouse gasses and keep things business as usual for as long as possible.
Smart regulation for AI and corporations is necessary, but has proven insufficient to consistently ensure good behavior. Instead of providing negative reinforcement from the outside in the form of laws that punish wrongdoing, re-writing the internal code of an AI or company to positively reinforce good behavior is probably the more self-sustaining strategy, albeit a very difficult one to achieve.
We can’t always predict what companies will do in the future, but if we ensure they are all working toward the best interests of people, planet, and profit, then there is far less downside to whatever actions they create to meet those goals. Companies are like AIs in that they follow orders from their leaders and marshal resources to accomplish their assigned goals. Give a company or an AI one single goal and if it’s capable enough, over time it’ll find novel ways to reach that goal with positive and negative externalities.
Even when companies stay within the law, there is still a lot of behavior that’s not completely illegal but still has negative effects on the world, like using offshore sweatshop labor, creating a toxic workplace, producing habit-forming and unhealthy products, or clinging to unsustainable energy sources.
Like a highly sophisticated AI, the most capable companies will always find new ways to make money at the edge of what’s morally acceptable and its nearly impossible for regulators to anticipate what all those ways will be. No one anticipated the dangers of human induced climate change when the internal combustion engine was invented, and by the time we had the tools to understand the problem, our entire global economy was already fully committed to activities that accelerate global warming.
Positive Reinforcement
We can’t anticipate every kind of potential behavior from every company, but we can try to change the rules or culture on what companies have to optimize for. Right now, little else matters outside of profit for a company. In theory, if a company gets too profit happy and unaligned with societal norms, its customer base can boycott them and force change, but this is no foolproof way to keep companies in line.
Instead, how can we change the system so that doing what’s best for public health and planetary sustainability brings the company tangible and immediate rewards, like profit does? How can we change things so that food companies get financially rewarded for regenerating farmland and selling less junk? How can we make it so that healthcare companies are mostly paid for keeping people healthy, not monetizing their sickness?
The morality of profit being the only goal for a company is far more convoluted in industries like food or healthcare, where the product or service fulfills a basic human need. This is not to say that these companies don’t deserve to earn a profit, but the question of “how much profit is enough?” is where things get complicated. Food and medicine should be priced to give companies an appropriate margin, but there’s a big difference between profitable pricing and price gouging.
It’s certainly a bit ridiculous to think about a paperclip making AI that destroys humanity in the process. But is it any less outlandish to have companies that put profits above all else? Corporations might not move at the same speed as an advanced AI, but they’ve put us on the same path as the paperclip AI in a much slower and less dramatic fashion. Like the paperclip AI, we humans have been consuming natural resources and killing each other far before computers were even invented. But while every computer running an AI has code that can be edited or a power switch to turn it off, the global economy has no such failsafe measures.
AIs learn by looking at a ton of data and identifying patterns that help it achieve some goal it was programmed to reach, like writing an essay or creating an image. The AI’s software engineers carefully set up rules to reward desirable behavior, which incentivizes it to pay attention to data that makes that behavior possible. Corporations aren’t as easy to optimize as software, but us consumers can create a similar recursive learning loop that rewards their desirable behaviors with our buying power. AIs learn from training data, corporations learn from customer data.
The collective global economy has already proved it can make more proverbial paperclips than we will ever need. It’s now time for us all to remind corporations that we not only want paperclips, but we want them in a way that avoids a planetary catastrophe.
—
Footnotes
3 Recent posts from my Substack
3 Highlights from my current reading list
The mystery of the social media disinformation war on plant-based meat by Clint Rainey - Fast Company
America’s Farmers Are Bogged Down by Data by Belle Lin - The Wall Street Journal
The Price of Sauce by Lora Kelley - The Atlantic
My email is mike@thefuturemarket.com for questions, comments, consulting, or speaking inquiries.
Great job making AI safety a real, front-and-center concern, in plain language. Very important research and analysis that I look forward to reading more of!
I like the analogy a lot. Could it be expanded to include our entire monetary system? After all, that has everything to do with rewarding those folks who choose to maximize shareholder value at the expense of humanity.
Or maybe our monetary framework is okay, but we need more regulation, as you call out: "Food and medicine should be priced to give companies an appropriate margin, but there’s a big difference between profitable pricing and price gouging."
I think everything is on a sliding scale, and we need to slide toward income equality. We're too far akimbo right now, but is this something we can fix within the existing system, or is it inevitable to come up again and again?