The everyday workloads of frontier AI labs: pretraining, data curation for text and video, evals on every checkpoint, interpretability, and AI-driven science.

Frontier AI

Tutorials for the workloads of frontier AI labs.

The headline training run is a small part of what a frontier AI lab does. Most of the work around it falls into three kinds:

  • Data factories: turning raw web text, video, audio, or lab measurements into training data, with every filter versioned and every document or clip traceable to the stage that kept it.
  • Decisions from many small runs: which checkpoint to ship, which data to keep, which candidate to send to an expensive experiment.
  • Continuous measurement: benchmarks on every checkpoint, and interpretability tools that look inside the model.

These tutorials follow those workloads at a scale that runs in minutes, each tested end to end on a Union cluster. Each one names the labs and papers it is modeled on.

Tutorial Who does this What makes it hard at scale
Distributed LLM pretraining Every foundation model lab Multi-GPU training that has to survive failures and stream data
Refine a pretraining corpus Pretraining data teams, as in FineWeb-Edu, Essential-Web, and Zyda-2 Billions of documents, CPU and GPU stages, and LLM labels that cost too much to run on everything
Evaluate every checkpoint Post-training and evals teams, as in OLMo and OLMES Evals that must keep up with training without slowing it, and never re-run when nothing changed
Curate a video dataset for generative video Generative video and world-model labs, as in Open-Sora and Stable Video Diffusion Hours of footage through decoders, scoring models, and captioners on different hardware
Find features in a language model with sparse autoencoders Interpretability teams, as in Scaling Monosemanticity and Gemma Scope Terabytes of activations, a grid of training runs, and thousands of LLM calls to label features
Closed-loop molecular discovery AI-for-science labs that run autonomous experiments A slow, failure-prone oracle, a loop that runs for days, and a scientist who approves each batch