If a model can learn to generate the training data most useful for its own improvement, pretraining stops being bottlenecked by human-curated corpora and becomes limited mainly by compute. On Sept. 24, 2026, Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, and Yoav Levine posted Self-Play Pretraining with Zero Data (arXiv:2609.30063) as an initial proof-of-concept for that path, taking inspiration from Solomonoff induction. Affiliations span independent research, Tel Aviv University, Stanford University, and LAPTh, USMB.
In plain terms, two transformers start from random weights. A generator proposes programs for a minimal universal Turing machine. The machine runs those programs and emits byte sequences. A learner trains on those bytes with ordinary next-token cross-entropy. The generator is updated with reinforcement learning so that new programs sit near the frontier of what the learner can still learn. Neither model is trained on natural text, images, or other real-world corpora. That setup is different from Absolute Zero-style post-training self-play, which targets reasoning with verifiable task loops, and from G-Zero, a separate paper on open-ended generation self-play. This work is tabula-rasa pretraining over computable structure.
Authors report a clean transfer test: after self-play only, zero-shot loss on held-out natural datasets improves predictably with self-play compute. Figure 1 shows compute-optimal frontiers across modalities, and Table 2 lists author-reported power-law compute exponents such as 0.123 on DCLM text and 0.249 on Mutopia MIDI melodies. The fuller scoreboard stays in the paper.
Authors also report that the learner shows in-context learning on held-out tasks with no further gradient updates, reaching nearly perfect greedy exact-match on reverse-string, stack, and associative-recall after enough examples. During training, the generator discovers recognizable mathematical sequences. Table 1 places Fibonacci-like programs by round 512, against an expected first appearance beyond 53,000 rounds under a fixed universal program prior.
The discussion keeps the frame honest. Authors say the runs stay below 25 million parameters at a 4K context, and that universal pretraining is not a replacement for contingent facts about a particular world. Authors report that natural data never enters the gradient updates. A validation loss averaged across DCLM and DNA was used only to choose hyperparameters, so the only contact with those corpora is that selection step. As of Sept. 25, 2026, Hugging Face Papers had no card for 2609.30063, and there is no project-page weight drop in the materials reviewed for this article. What remains open is whether the same adaptive curriculum keeps transferring at larger scale once natural data is scarce or expensive.