Multi-Agent Systems: Cooperation and Competition

It’s one thing to have an AI agent work for you. But, what if multiple AI agents could work together with other AIs and humans? How could we ensure that these agents would cooperate with each other to achieve common goals or objectives, and more importantly, how could we guarantee that they would cooperate with us in the same vein, especially when we leave them to their own devices? Why would AI agents favor cooperation over conflict, and what external factors might influence this kind of behavior, particularly over time?

In truth, conclusive and reliable answers to the questions we’ve just raised can’t yet be given. While nascent versions of operational multi-agent systems may exist behind closed doors, this technology has not been deployed in real-world environments, but make no mistake, multi-agent systems will become a part of our future.

Fortunately, we can theorize about these complex and abstract scenarios, and not without substance—reputable research on this niche does exist, and for readers wishing to examine this topic in academic detail, we recommend Chapter 7 of this textbook, which serves as the core inspiration for this post. Moving forward, we will nonetheless offer some of our own perspectives on this concept while doing our best to make this discussion as accessible as possible to a wide audience. To be blunt, this is not a simple topic.

In the sections that follow, we’ll start by breaking down the factors that influence the emergence of cooperation and conflict from the perspective of rational agents—this will serve as the theoretical meat of our discussion. Next, we’ll explore the role of competition in multi-agent cooperation and conflict, investigating how it can act as a double-edged sword in this context. Finally, we’ll consider additional strategies for fostering cooperation in multi-agent systems, focusing on the environments in which such systems operate, the structure they take, and what we could do as users to encourage cooperation throughout human-AI interactions.

The Nature of Cooperation and Rational Agents

Cooperation is when two or more individuals work together in the interest of a common goal or perceived benefit. This always comes at some individual cost, a cost that is outweighed by the benefits that are provided as an immediate or delayed consequence of working together, or by the reciprocal benefits provided by some other individual currently or at a later time.

For example, a lone wolf, when it catches its prey, has it all to itself whereas a pack of wolves divides the prey among themselves. If being part of the pack means that all the wolves get a smaller share of prey, then why should they hunt together? Because, hunting together maximizes their chances of survival, allowing them to cover more territory, expend less energy on hunts and recover quicker, and take down bigger prey than they would be able to on their own.

Alternatively, let’s say one of the wolves (W1) in the pack decides to give up its share of the prey to another wolf (W2) who has recently given birth to cubs. A few things could motivate this—if W1 and W2 are related, then W1’s offer is altruistic, intended to support her next of kin with no expected benefit in return. If they are unrelated, W2 might remember W1’s offer in the future, and when W1 is in a position of need, W2 may choose to forego her share for W1.

On the other hand, maybe there’s a third wolf (W3) who witnessed the initial interaction between W1 and W2. On the next hunt, W3 may decide to offer her share to W1, not for purely altruistic reasons (like we see with next of kin), but for the same reciprocal reasons that W1 offered her share to W2. W3 is now more likely to receive help from W1, W2, and any other wolves in the pack who witnessed the interaction, and so the cycle continues. At the group-level, it’s also collectively better for the whole pack to risk the death of one wolf (provided it’s not the Alpha) if it means the survival of three cubs—in the short-term, there may be more mouths to feed, but in the long-term, the pack becomes stronger, which is better for everyone.

Cooperation is anything but simple, and it is more likely to emerge when members of the group have good reason to believe they will interact with each other again in the future—if I know that I’ll see you again and I help you when you need it, you’ll be more likely to help me when we next meet, but if I run into you once, and we both have just as much to gain (or lose) from taking what the other has laid claim to, the risk of fighting might be worthwhile.

However, conflict isn’t just about what you and I have to gain, and neither is cooperation. Our environment tends to determine whether cooperation or conflict is in our best interest. If resources are scarce, I may favor myself and follow my instincts for self-preservation—if resources are plentiful, I may be more inclined to share. Still, even in this scenario, I might choose to fight if I have other mouths to feed and you represent a direct threat to my ability to keep my kin alive, regardless of whether it concerns resource availability or physical dominance (in the case of AI agents, cognitive dominance might be more pertinent).

- Comment: Under conditions of resource scarcity, cooperation can emerge as a viable strategy, enabling dynamics like risk distribution among members of the group, resource pooling, better protection against external threats, mutual aid systems and negotiated sharing tactics, and labor division, to name some relevant factors.

Taking this a step further, let’s imagine that resources are abundant and we all exist in a predominantly cooperative group. Over time, I may realize that I can begin skimming a little extra of the top, effectively working less but gaining more. In doing so, I risk being cast out of the group, but it’s also possible that other group members will see what I’m doing, and instead of ostracizing me, begin doing it themselves. You may resist for a while, but eventually, you join in too, and before we know it, we’ve depleted the resource pool. Now, we’re back at square one, fighting over resources that we once all had easy access to. The dynamics of cooperation and conflict at the group-level can be cyclical and iterative, with certain strategies being selected for due to the environmental pressures that exist at one time or another.

At the individual level, there’s still more to it—conflict between two parties can emerge for several reasons. For example, let’s assume there are two countries: X and Y, who both want control of a non-divisible territory. X thinks that if it waits too long, it will give up its offensive edge and Y will attack, forcing X to relinquish control of the territory. Despite the fact that diplomatic tensions haven’t escalated to a point of war, X decides to take advantage of Y’s potential oversight and issue a full-force first strike, attempting to damage Y enough so that X cements its advantage when war breaks out.

By contrast, let’s say that X believes it is stronger than Y, and while it does not issue a first strike, it openly demands that Y gives up control of the territory. In doing so, X implicitly assumes that Y will bluff and threaten war since X thinks that Y has an incentive to overrepresent its strength. However, Y is much stronger than X imagined, and instead of threatening, Y decides to attack. X’s failure to anticipate Y’s willingness to engage in conflict, which was due to a misunderstanding of Y’s true strength, results in war.

In another case, X and Y are at peace, having signed a treaty years ago in which X was granted full control over the territory. As time passed, X’s strength declined while Y’s strength increased. Y, having realized this, demands that control over the territory be renegotiated, yet X refuses. Believing it is now powerful enough to take the territory back from X, Y initiates a war.

Finally, what if X and Y are of equal strength and aren’t attempting to take control of a non-divisible territory, but X believes that Y has historically infringed upon X’s right as a sovereign nation? Whether Y has actually done this is somewhat relevant, though what matters most is that X perceives that Y has put itself in a position of relative superiority. To level the playing field, X decides to launch a small-scale invasion into Y’s territory, not necessarily in the interest of war, but to remind Y that it can’t be messed with.

- Comment: While this isn’t always the case, conflict is far more likely to arise when parties compete for control over something that is non-divisible. This is because they are precluded from bargaining with each other to reach a reasonable agreement, confronted with a “winner take all” scenario.

So, what’s the point of all this theoretical talk? The takeaway is this: cooperation and conflict can and do occur naturally when it’s in an individual’s or group’s rational best interest to do so. In other words, you don’t need to be “good” to cooperate or “evil” to instigate conflict, you just need to be a rational agent who is pursuing incentives aligned with your best interest, incentives that are structured by the circumstances and environment in which you exist.

In this respect, AI agents, especially since they don’t possess morality, emotion, cultural understanding, and genuine regard for others, in addition to being driven by some utility/optimization function, could be vulnerable to each of the factors we’ve just discussed. So, if we’re bound to encounter a world where AI agents interact with each other and/or humans, how could we get them to be reliably cooperative?

Competition in Multi-Agent Cooperation and Conflict

Competition is an interesting phenomenon—it can bring out the best and the worst in people. In some cases, it can be a force that fosters cooperation while in others, it can drive conflict. Importantly, where an environment consists of multiple actors, competition can be reasonably expected (this isn’t a guarantee, however, and there are more nuanced factors that can affect whether competition emerges).

At the inter-species scale, competition can favor individual group-level cooperation when two distinct groups exist within one environment and must compete over the same selection of resources. For example, energy resources are limited, and AI agents might someday be forced to compete with humans over them—in this case, both have direct incentives to cooperate with those of their own kind, but this also increases the probability of conflict between the two groups.

What We Can Do

• Design AI agents to perceive humans as part of their group. We don’t need AI agents to “think” they’re human, we just need to make sure they don’t exhibit any in-group biases that alienate us.
• Reward individual AI agents for cooperating with humans to create rational incentives for them to follow. For example, if some AI agents are more cooperative than others, they could receive memory and storage upgrades.
• Divide AI agents into groups that compete with one another to reach pre-defined human objectives, rewarding the groups that perform best. This could also encourage closer collective alignment with human values and preferences as AI agents intelligently adapt to their environments and learn from their experiences.
• Ensure that if AI agents and humans can’t avoid competing over resources, they can only compete for divisible resources, thereby creating space for a negotiation that aims toward some mutually beneficial outcome.

Individually, AI agents could be incentivized to compete with their kind and humans, and this could lead to harmful negative behaviors like power-seeking, deception, and manipulation in multi-agent networks that consist of humans and AIs. In these scenarios, we should be careful not to penalize AI agents too aggressively for lack of individual cooperation, as this may lead agents to start gaming cooperative objectives to avoid being punished—they might reach the intended objective, but misrepresent or conceal the steps they take along the way.

What We Can Do

• If non-cooperative behaviors must be negatively reinforced through punishment, ensure that it’s other AI agents who are administering the punishment, goal-based agents that act as “cooperative enforcers” on behalf of humans.
• Consider building hierarchical merit-based operational environments where humans sit at the top and where individual AI agents can “climb” the hierarchy based on how well they achieve human objectives, to receive greater benefits and control—control over other AI agents that sit lower in the hierarchy.
• As an alternative to hierarchical operational environments, consider organizing AI agents in series or sequence, where the array of actions pursued each agent serve a common goal. These systems could be guided by a “top tier” consisting of humans who oversee their operations.
• Provide individual AI agents with tasks that can only be completed by working together with humans and other AIs. It’s also worth giving explicit explanations to AI agents that dissect the rationale behind human-AI cooperation, so they can internalize cooperative logic.
• Expose AI agents to historical examples of cooperation and conflict, specifically, examples that highlight why cooperation is—from the AI agent’s perspective—a more rational choice than conflict.

Even if we attempt to comprehensively control for competitive pressures, it’s possible that competition will nonetheless emerge, particularly within changing environments where agents might cycle through different cooperative and non-cooperative strategies. Seeing as multi-agent systems will have to operate under real-world conditions, which tend not to remain constant, this is an angle worth exploring.

What We Can Do

• Limit the launch or development of multi-agent systems to operational environments whose overall structure is unlikely to change significantly over time. This could help ensure that cooperative incentives not only remain consistent but also visible from the human perspective.
• If multi-agent systems are deployed in fluctuating environments, individual agents and the system as a whole must be regularly monitored for emergent objectives that may undermine cooperation and favor conflict. When such objectives are identified, further operation should cease immediately until human reviewers transparently understand the origin of these objectives.
• Require that AI agents explain the logic that motivates their cooperative behaviors and decision-making. Ideally, this logic should map onto the technical characteristics, functions, processes, roles, and outputs of each agent in the system.
• Do not deploy AI agents with capabilities for recursive self-improvement and self-replication if they are intended to be used within a multi-agent system. Competitive pressures could lead such agents to self-improve and proliferate, not in the interest of becoming better cooperators, but to obtain more power and influence within their network.

Cooperation in Multi-Agent Systems: Additional Strategies

Below, we describe a series of additional strategies that could facilitate cooperation in multi-agent systems.

• Frame human-AI and AI-AI interactions as symbiotic, not competitive. This could involve aligning the rational interests of AI agents with those of humans under unified mutually beneficial or detrimental objectives.
• Create adversarial training environments in which AI agents are challenged to cooperate when conflict appears to be the better choice. These environments should pressure AI agents into conflict scenarios but allow them to discover a distinct cooperative path that yields much higher rewards than any conflict-based strategy—this path should not be obvious.
• Train AI agents to optimize for altruistic behaviors and decisions through cooperative gamification and shared accumulation systems whereby the state of the whole system depends on collective success and team-based reward pools.
• Introduce interdependent quotas in multi-agent systems such that AI agents can only receive rewards when other agents in the system also achieve their stated objectives and/or when human operators approve of their decisions and actions.
• Build operational environments that simulate artificial scarcity and common threats where AI agents’ direct survival is endangered if they don’t cooperate with each other and humans. This could include training and designing agents that are incentivized to “value” their survival (i.e., not being turned off or removed from the system).
• Provide collective rewards for multi-agent systems that prevent any individual AI agent from dominating or outperforming the rest. Additional mechanisms for conditional forgiveness might also be useful, allowing previously non-cooperative AI agents to re-enter the system insofar as they adhere to shared cooperative standards.
• Structure operational environments to incentivize tit-for-tat dynamics where initial cooperative gestures are proportionately rewarded, encouraging a trust-based feedback loop. AI agents that are more cooperative than others can also receive cumulative rewards to drive sustained group-level cooperation.

We encourage readers to think creatively and openly about cooperation in multi-agent systems, and in doing so, look beyond cooperation in human society. Nature has a lot to offer us in this context, and if we continue to widen the scope of our understanding of cooperation, chances are the solutions we develop will be more robust, ingenious, and ultimately, effective.

Conclusion

Throughout this post, we’ve covered several key ideas: 1) the essence of cooperation and conflict and the factors that drive them, 2) the role that rational incentives play in cooperation and conflict, 3) the dynamics and impacts of competition on cooperation and conflict in multi-agent systems, and 4) a selection of further strategies for promoting cooperation in multi-agent systems. While it will be some time before we approach this topic again, we urge readers to stay tuned for our next post, which will tackle an equally complex yet interesting topic.

We invite readers who crave more pragmatic guidance and insights on current real-world issues within AI governance and policy, risk management, and generative AI to follow our blog. We also suggest checking out our “deep dives,” which explore topics like AI literacy, existential and systematic risk, and policy prediction in depth.

For those who have initiated their AI governance and risk management journey, whether they are just beginning or in the later stages of refinement, we recommend taking a look at our responsible AI platform, AI policy analyzer, and AI risk advisor.

---

Related topics: AI Agents AI Safety AI Ethics