Hugging Face published a survey on Sept. 11, 2026, of 15 reports from 13 labs on how agentic reinforcement learning environments are built. The post counts only environments labs say they train in, not evaluation-only harnesses.
The survey's central claim is that agentic RL needs a real computer per attempt: a filesystem, a shell, and processes that survive between calls. Short rollouts are destroyed after one use. Long rollouts are checkpointed and resumed, with high-end stacks running hundreds of thousands of concurrent machines.
Independent accounts match the scale pattern. Cursor says training Composer required hundreds of thousands of concurrent sandboxed coding environments and a scheduler rewritten for bursty training. SemiAnalysis describes RL environments as sandboxes ranging from Firecracker micro-VMs to QEMU VMs and reports that each rollout needs at least one sandbox, with scaling from 96 to 960 concurrent rollouts exposing initialization failures and about one-hour startup stragglers.
Paper Instruments' account on E2B describes thousands of concurrent isolated rollouts, each with its own filesystem and persistent state, started from snapshots. Modal says reinforcement learning can burst hundreds of thousands of sandboxes and reported a demonstration of one million concurrent sandboxes created in under a minute. Treat those vendor scale figures as reported demonstrations, not independently audited benchmarks.
Across the surveyed stacks, Hugging Face says labs converge on four pieces: a task with a verifier, an agent-facing harness or contract, a sandbox, and a trainer that consumes rollouts without waiting for the slowest attempt. Implementations remain lab-specific. The post frames asynchronous generation and environment infrastructure as becoming as important to scaling agentic RL as the algorithm itself.
What remains open is how far each lab's private environment data and compute differ from the public four-layer stack, and how third parties will reproduce concurrency and latency claims from vendor and partner reports. The Hugging Face synthesis and the independent systems accounts should be weighed separately until those points are settled.