Zero-shot homogeneous LLM teams are baselined against a trained MARL reference policy.
Alem leaderboard
Model leaderboard
Zero-shot 3-agent LLM teams and trained MARL baselines on the Alem benchmark, separating base-task competence from explicit coordination. ← Back to the full overview
Coordination reward is reported independently from base reward and total return.
18.8% Coord.% on Hard difficulty, the first open-weight model to lead this metric. It edges our 1B-step MARL baseline (17.6%), though the intervals overlap and the interface differs.
| # | Model | Type / date | Harness | Alem | Base% | Coord.% | Total% |
|---|
Click any LLM score for the run config and how-to-run command. Bars show 95% bootstrap confidence intervals; dots show means. Alem is the environment version the row was evaluated on: the only change between 0.1 and 0.2 is a fix to a rare ladder-generation collision that could block one of the three ladders on a level; levels that did not collide are bit-identical, and comparing results across the two versions found no statistically significant difference in performance. Base%, Coord.% and Total% are normalised independently, so Total% is not the sum of the other columns. MARL 100M / 1B / 10B are environment training steps, not model parameters. On Hard, switch on Mixed teams to add mixed-model teams (marked mix) to the table unranked; click one for the member breakdown. Submit a result →
Show the MARL baseline table
| # | Algorithm | Base% | Coord.% | Total% |
|---|
Trained MARL baselines (IPPO, HyperMARL-IPPO, MAPPO, PQN-VDN), ~25M-parameter recurrent policies. Means with 95% bootstrap CIs across 5 seeds.