Open-ended multi-agent coordination benchmark

Alem

Can LLM agents coordinate in long-horizon, open-ended tasks?

Alem is a JAX benchmark for testing multi-agent coordination in long-horizon, procedurally generated open-ended worlds. Across nine levels with controllable coordination demands, agents must explore, communicate, trade resources, craft tools, build structures, and fight mobs. Alem supports LLMs, VLMs, RL agents, and human play.

Read paper Code Models Leaderboard Submit ▶ Watch reel Contact

Kale-ab Abebe Tessera · Andras Szecsenyi · Cameron Barker · Alexander Rutherford · Davide Paglieri · Aidan Scannell · Henry Gouk · Elliot J. Crowley · Tim Rocktäschel · Amos Storkey

Finding 1On Hard difficulty, a zero-shot frontier LLM approaches our 1B-step MARL baseline.
Finding 2Coordination is a distinct axis of difficulty, separate from base-task skill.
Finding 3Communication has the largest impact on coordination in our harness ablations.
01 · Leaderboard

Model leaderboard

Open the standalone leaderboard page ↗

Protocol3-agent zero-shot teams

Zero-shot homogeneous LLM teams are baselined against a trained MARL reference policy.

Key scoreCoord.%

Coordination reward is reported independently from base reward and total return.

Hard leaderGemini 3.1 Pro

17.5% Coord.% on Hard difficulty, close to our 1B-step MARL baseline on this metric (different interface, same underlying environment).

# Model Type / date Harness Base% Coord.% Total%

Click any LLM score for the run config and how-to-run command. Bars show 95% bootstrap confidence intervals; dots show means. Type / date reports the selected difficulty's evaluation date; Harness links to the evaluation harness implementation. Base%, Coord.% and Total% are normalised separately, so Total% is not the sum of the other columns. MARL 100M / 1B are environment training steps, not model parameters. On Hard, mixed-model teams (marked mix) join the table unranked; click one for the member breakdown. Submit a result →

Reference point · MARL baselines

Trained multi-agent RL provides a reference point for the LLM table above, using the same underlying environment through a symbolic interface. It improves with compute and degrades as coordination difficulty rises, while most LLMs fail before difficulty strongly affects aggregate scores. IPPO, HyperMARL-IPPO, MAPPO and PQN-VDN are shown at their training difficulty; the best 1B run appears as a reference row in the main table.

Show the MARL baseline table
# Algorithm Base% Coord.% Total%

Environment training steps, not model parameters (~25M-parameter recurrent policies). Means with 95% bootstrap CIs across 5 seeds. The primary leaderboard above includes the best MARL budget as a reference row.

02 · Gameplay

Watch each model play, and read what it says

The same world (Easy, seed 0) for all 13 language agents, sped up 8×, paired with the most interesting broadcast that agent sent to its teammates. The messages are verbatim from the logs, so coordination emerges, and breaks down, in the models' own words.

or click any model below · expand for all 13 ↓

Clips are the world view of each model's Easy seed-0 episode, sped up 8×. Messages are unedited agent broadcasts from the same runs (lightly trimmed). Click a card for the synced replay.

Below is a sample of the complete per-step debug log we publish for every model, including observations, reasoning traces, actions, messages and scratchpad memory, every step of the episode. Browse all 13 →

Live sampleQwen 3.6 27B · Open-weight · Easy · seed 0
Open in new tab ↗
Loading a live debug page…
03 · In action

Watch teams coordinate in Alem.

Trained MARL teams and zero-shot language agents tackle the same underlying open-ended world. Watch them split work, broadcast intent, and synchronize their actions, or play the full reel.

RL · symbolic A team of three, forager, miner and warrior, each acting from its own egocentric view and trained end-to-end in JAX.
LLM agents coordinating via broadcast messages in Alem
LLM · text Through the text interface, each agent broadcasts intent to teammates and keeps a private scratchpad every step.
04 · Environment

One world, three interfaces, diverse coordination demands.

Alem isolates coordination by pairing a rich open-ended Craftax-style world with explicit synchronization, handover, specialization, and construction tasks.

Controllable difficulty The same world under increasing coordination pressure. A single parameter α sweeps from Easy (0.30) through Medium (0.60) to Hard (0.90), adding agents per task, tightening timing windows, and reducing slack.

What makes Alem different

01

Long horizon

Episodes last up to 10,000 steps across nine procedurally generated levels.

02

Open ended

Agents pursue a web of goals: collect, craft, explore, trade, fight, descend, and build.

03

Explicit coordination

Tasks require shared timing, resource transfer, role division, and multi-stage convergence.

04

Controllable difficulty

A single α parameter scales incentive misalignment and coordination pressure.

Three interfaces onto one world

Symbolic

For MARL agents

  • 9,730-dimensional observation vector: local maps, inventory, teammate directions.
  • 60 discrete actions with legal-action masks (~20 valid per step).
  • IPPO, HyperMARL-IPPO, MAPPO and PQN-VDN baselines, trained end-to-end in JAX.
Pixel

For humans & vision agents

Pixel render of an agent's egocentric view
  • An egocentric rendered view of the same world state, including map, HUD, inventory and teammates.
Text

For language agents

Step: 0/10000    Role: forager
Position: (x=25, y=24)
You see:
- tree 3 steps north (x=25, y=21)
- construction_site 1 step north
- water 5 steps east (x=30, y=24)

Available: Move North · Do · Rest ·
  Request Wood · Request Stone · ...

> <action>Move North</action>
> <communication>A1: I'll grab wood,
  you mine the stone</communication>
  • Reason-then-act harness with scratchpad memory and broadcast messages.
View the full system prompt 
Loading…

Diverse coordination demands

Three agents converging on a shared tree to coordinate
Three agents converge on a shared target (boxed) to satisfy a coordination demand: divide roles, meet and act together.

Soft specialization

Any agent can act, but specialists are more effective; encourages division of labour across foraging, mining, and combat.

Handover

One agent begins, another completes within a time window; temporal slack makes it the most accessible.

Synchronous

Agents converge on a shared target and act on the same timestep; the hardest demand.

Construction

Gather resources, climb a tech chain, converge, and execute a multi-step build.

Agents mining
Mining
Agents crafting
Crafting
Agents in combat
Combat
Agents building
Construction

Scoring structure

Base66individual progress achievements derived from single-agent Craftax
Coordination27explicit multi-agent achievements
Total93normalised episode return, scored per category
05 · Paper results

What the benchmark reveals

13 LLMs in zero-shot three-agent teams, compared against trained MARL baselines. All comparisons use 95% bootstrap confidence intervals; Base%, Coord.% and Total% are normalised independently.

Current LLM agents remain far from solving Alem, averaging only ~6% normalised total return, yet their failures are not uniform.

Click any result to read the detail.

Q1.1 · zero-shot

Alem is unsolved, but separates models

Alem remains far from saturated by both zero-shot LLMs and trained MARL, while separating models across a broad range. On Easy, Total% spans 0.8% (Llama-3.1-8B) to 18.1% (Gemini 3.1 Pro); on Hard, LLMs average only 6.2% Total%. The top LLM reaches 17.5% Coord.% on Hard difficulty, close to the 17.6% 1B-step MARL baseline.

Q1.2 · decomposition

Base competence does not guarantee coordination

Decomposing performance reveals failures hidden by aggregate scores. GPT-5.4 achieves the second-highest Base% across difficulties, yet its Coord.% lags behind smaller models such as Gemma-4-26B-A4B on Hard, with non-overlapping 95% CIs. General environment competence does not guarantee coordinated multi-agent behaviour.

Q1.3 · difficulty

Most LLM agents do not exploit the difficulty axis

Trained MARL degrades as coordination difficulty increases, but aggregate LLM Total% remains nearly flat at ≈6% across Easy, Medium, and Hard. Most LLM agents do not coordinate reliably enough for the difficulty parameter to strongly affect aggregate performance.

Q1.4 · task structure

Coordination failures differ across task structure

Coordination is not a single axis. Handover tasks are comparatively more accessible because they allow temporal slack. Synchronous-Hard tasks require agents to identify a shared target, navigate to compatible positions, communicate intent, and act in the same timestep. Gemini 3.1 Pro leads LLMs here with only 18% coverage.

Q2 · ablations

Communication is critical for coordination

Removing communication produces the largest drop in coordination for both models tested: Gemini 3.1 Pro falls from 17.5 to 5.3 Coord.%, and Gemma-4-31B from 8.8 to 3.8. Reasoning supports both base and coordination reward; scratchpad memory helps mainly when used to plan ahead.

Q3 · composition

Heterogeneous teams perform near the homogeneous-team average

Mixed teams land close to the average of their corresponding homogeneous teams: the same-family Gemma team is slightly above (+0.1 Total%), the cross-family team slightly below (−0.1 Total%). Adding a stronger model is not sufficient to lift team performance.

Diagnostics

Click any plot to read the detail.

Coordination coverage by typeCoordination coverage by type.

Coordination is not a single axis. Handover tasks are comparatively more accessible because they allow temporal slack: one agent can initiate an opportunity and another complete it within a bounded time window. By contrast, Synchronous-Hard tasks require agents to identify a shared target, navigate to compatible positions, communicate intent, and act in the same timestep. Gemini 3.1 Pro leads LLMs on synchronous-hard coordination with only 18% coverage, while the next-best reach 9%.

Survival vs rewardSurvival vs reward.

Many models survive for long episodes despite low returns. Mean episode length versus Total% reward shows that longer survival does not necessarily translate into progression or coordinated achievements. This suggests the main bottleneck is not basic survival or interface parsing, but converting interaction time into coordinated progress.

Model size vs performanceScale vs performance.

Across open-weight models, performance is not monotonic in parameter count. Even within model families such as Gemma-4 and Qwen-3.5/3.6, coordination performance does not scale linearly across dense and mixture-of-experts variants. This suggests coordination depends on factors beyond raw scale, including post-training, reasoning, and how effectively agents use communication.

Harness ablations

Click either plot to read the detail.

Legend: coordination and base reward
Gemini ablationGemini 3.1 Pro ablations.

For coordination, communication matters most, then reasoning, then memory. Removing communication produces the largest drop, from 17.5 to 5.3 Coord.%. Reasoning is the only ablation that lowers both Base% and Coord.%. Scratchpad memory helps mainly when used as a forward-looking planner.

Gemma ablationGemma 4 31B ablations.

Gemma-4-31B shows the same ordering: removing communication is the largest drop, from 8.8 to 3.8 Coord.%. Without reasoning, Gemma writes longer scratchpads that partly substitute for it, but the compensation is incomplete, and both Base% and Coord.% still fall.

Heterogeneous teams (Hard)

Paper figure: mixed teams land close to the average of their homogeneous members.

Legend for heterogeneous team plots
Gemma-family heterogeneous team results
Gemma-family team: heterogeneous performance is near the homogeneous-team average.
Cross-family heterogeneous team results
Cross-family team: adding a stronger model does not directly transfer its performance to the whole team.

Citation

@article{tessera2026alem,
  title   = {Benchmarking Open-Ended Multi-Agent Coordination in Language Agents},
  author  = {Tessera, {Kale-ab} Abebe and Szecsenyi, Andras and Barker, Cameron and
             Rutherford, Alexander and Paglieri, Davide and Scannell, Aidan and
             Gouk, Henry and Crowley, Elliot J. and Rockt\"{a}schel, Tim and
             Storkey, Amos},
  year    = {2026},
  url     = {https://arxiv.org/abs/2606.08340}
}