A search router sits in front of several search providers and decides, per query, which one answers. The naive way to train it is on the answers it chose, and that fails for a structural reason: the router never sees what the provider it skipped would have said, so every label it collects is biased by its own past decisions, and fixing that means a permanent exploration budget and careful counterfactual bookkeeping.
We start instead from a router that fans out. Every query goes to every provider, every answer is stored, and a judge grades them afterwards. Each request is then a fully labeled example: no bandit, no bootstrap phase, no exploration to account for. The price is a provider bill linear in the number of providers and a latency equal to the slowest of them, so fan-out is a phase rather than the product. The lifecycle has three eras: fan out and collect; distill a query-only student from the judged traces; then route most traffic with the student while a small fan-out slice keeps producing labels. This guide is the second era.
Figure 1: the two phases of the router's life, played in a loop. Fan-out: a query goes to every provider at once, the judge grades the answers without seeing which provider wrote which, and the graded episode is pushed to the trace dataset that training reads. Routed: the adapter that training published reads the query and sends it to one provider, and that answer goes back to the caller. A fixed fraction of traffic keeps fanning out so the dataset keeps growing.
We train on Polygramme, our hosted continual-learning stack. Underneath it is SkyRL for the training loop and Tinker-compatible API, Megatron for the LoRA workers, vLLM for sampling, Harbor's task format for environments, and Modal for the GPUs and the sandboxes. Four properties of that stack matter for a router and we will come back to each: it is multi-tenant LoRA, one base model on each GPU and one adapter per run, so what you train is a small file you can download and serve anywhere; the sampler and the trainer are separate machines, so rollouts and optimizer steps never wait on each other; every model call a harness makes goes through a relay that keeps exact token accounting; and two training methods are available, on-policy self-distillation for when the only signal is a judge's verdict on recorded answers, and reinforcement learning for when the router can be run against a grader.
The policy in this guide is a 9B model with a rank-32 LoRA adapter. Nothing below depends on that size except the timings.
1.0 What a trace holds
One fan-out request produces one trace: the query, one candidate per provider with its answer text, citations, latency and cost, and the signals the router computed at selection time. Of those, citation overlap between providers is the one that matters later:
Where independent providers cite the same sources, that is evidence about the world, not just about the models. Where they diverge, the query is contested, and contested queries are where a judge's verdict carries information. Keep this signal: it decides what gets judged, and it is free.
1.1 Judging offline
The judge is a batch job over stored traces. It never runs on the request path, and nothing on the request path waits for it. Three rules are load-bearing, and each exists because we watched the alternative fail.
The judge is not one of the providers. Self-preference bias in LLM judges is well documented, and with N arms it compounds: one provider gets systematically flattered and the router learns the flattery. The judge model must have no arm in the race; the job refuses a configuration where it does.
Provider names never reach the judge. Candidates are relabeled A, B, C and shuffled per call. A judge that knows which answer is which vendor's is grading a brand. The letter-to-provider map is kept as an audit trail and the verdict is mapped back before it is stored, so the grades, the ranking and the reward are per provider name. Only the judge call is blind; the router trains on, and outputs, provider names.
Grade per result, then rank listwise, with ties. A 0 to 3 relevance grade per candidate on the UMBRELA scale, so it composes with standard IR metrics, plus one ordering. Ties are allowed on purpose: forcing an order between two equally good answers manufactures a label the data does not contain.
Judging every trace is unaffordable at fan-out volume and unnecessary. Where the overlap signal says the providers agreed, the judge mostly confirms it. We judge contested traces at 100% and the rest at a sample rate.
1.2 Calibrate before anything trains
Export a hundred or two contested traces, rank them by hand, and compute Kendall's tau and top-1 agreement between the judge and the humans, per provider. If the judge does not agree with you, everything downstream optimizes noise very efficiently. Below about 0.7 tau we fix the rubric and re-run. Keep collecting the number once live: an agree/disagree pair under every verdict in your trace viewer is the cheapest labeled set you will ever build.
Suggestion. Treat the agreement number as a gate you re-check whenever the rubric, the judge model or the provider set changes. It is the one number in this guide that, if wrong, makes every other number wrong.
1.3 Traces become episodes
Polygramme's unit of training data is the episode: a conversation, a map of named reward components, and, when the episode was sampled through the relay in Step 2, the exact token IDs. The router's episode is the query-only formulation of a trace: the student must decide from the query alone, so the assistant turn is the judge's ordering and the candidates' text stays out of the prompt. The reward is NDCG of the recorded ordering against the judge's grades, which rewards getting the top right and does not punish a tie it cannot resolve.
Two paths get episodes in. The importer in rlcli reads OpenAI, Anthropic, LangSmith, Vercel AI SDK, CSV and plain message dumps, redacts what you tell it to, and joins scores from a telemetry file by trace id. Or write the JSONL yourself; it is one object per line:
On push, each line becomes a trajectory row with its reward map, and the conversation is assembled into a file in Harbor's Agent Trajectory Interchange Format (ATIF v1.7): the system and user prompt as steps, one agent step per assistant turn, the reward under final_metrics.extra. Recorded rollouts and imported conversations share this format, so every downstream reader treats them the same. The server answers per item and returns 207 on partial failure rather than dropping the batch. Datasets are interval-versioned: --as-of 2026-09-01T00:00:00Z reproduces last week's run after another week of traces has landed. Reward components are named and idempotent by id, so re-judging under a new rubric adds a component beside the old one instead of overwriting it.
Get the environment and the harness right before any RL. Failed rollouts waste GPU hours and bias the reward; harness quirks teach the model to route around your harness instead of solving the task; and inexact token accounting makes training silently off-policy. Distillation in Step 6 does not need any of this. RL does.
2.0 The environment: replay tasks
A router needs somewhere to run and something to grade it, and a fan-out trace already contains every provider's answer for its query. So the environment is replay: a container that serves the recorded candidates, an instruction, and a test that scores whatever ordering the policy proposes against the judge's grades. No provider is called during training. One judged trace becomes one task in Harbor's format, and a forty-line script emits them:
The reward contract is the only thing the sandbox has to honor. After the agent exits, tests/test.sh runs from /root with the task's tests/ copied to /tests. If it writes /logs/verifier/reward.txt (a number) or reward.json, that is the reward; otherwise its exit status is, 0 for 1.0 and anything else for 0.0. A rollout that never reaches the test is not dropped: context exhausted, timed out and harness crashed each have a configured terminal reward, zero by default, so a failing harness is a low reward rather than a missing sample. A grader that crashes scores 0 and flags the episode. Tasks are content-addressed, so re-pushing an unchanged task changes nothing and an edited task is a new one.
Suggestion. Tag tasks by the judge's sampled_because. Contested tasks are the ones with reward variance, and Step 5 will need them.
2.1 The harness: the router itself
The agent that runs in the sandbox is the router, not a generic chat loop. It ships as an image and a command, and the platform launches it non-interactively with stdin closed, with the instruction at /root/instruction.md. Inside the container it finds PG_BASE_URL, PG_TRACE_TOKEN and PG_MODEL, plus the OpenAI and Anthropic base-URL and key variables pointed at the same place, so whatever client the router already uses works unchanged. The presets for OpenCode, Pi, OpenClaw, mini-swe-agent, Claude Code and Codex do the same wiring for those harnesses; a router is the custom case.
Before RL, run the untrained policy through the harness on a few hundred tasks and read the traces. How many failures are the model's, and how many are the harness's? We found the usual things: an output parser that rejected a valid JSON list because the model wrapped it in a code fence; a container image missing a package the router imported lazily; a harness that retried a failed call by rewriting its own history, which matters in 2.2. Fix these first. In Mercor's guide the harness fixes alone were worth about an epoch of training on the unfixed harness, and our experience matches: the untrained policy's pass rate moved before a single gradient step.
Suggestion. Put a timeout on everything the harness does, and let the platform's own limits (sandbox_timeout per rollout, command_timeout per tool call) be the backstop, not the first line. Anything without a timeout will eventually hang a rollout for its entire budget.
2.2 Token-in, token-out
When the harness's model calls are turned into the trainer's input, the engine's text output must not be re-tokenized. Re-tokenizing causes two misalignments: between what the model actually generated and what the trainer thinks it generated, and, in multi-turn settings, between the output token IDs of turn N and the input token IDs of turn N+1. Chat templates rewrite earlier turns when later ones are appended. We measured this on our own tokenizer while building the relay: a naive re-render of a two-turn conversation silently changed one token, and one token is enough to break an importance ratio.
There are three ways to get exact accounting. Rewrite the harness to speak a raw completions API in token IDs, which is the most control and the most work per harness. Ask the chat endpoint to echo token IDs, which fixes the first misalignment but not the second, so every turn splinters into its own training sequence. Or put a proxy between the harness and the engine that translates chat requests into token-ID completions and tracks the tokens across turns. We built the third one as a service, the relay, so that no harness has to change.
Per call, the relay does five things. It accepts the harness's OpenAI or Anthropic request unchanged. It renders the conversation to token IDs with the policy's own renderer. It samples on the sampling GPU through the raw completions endpoint and keeps the sampled IDs and their logprobs. It decides how this call relates to the previous one in the same episode. And it records all of it as one call of the trace, with the disposition on the record:
This is the retry case from 2.1: a harness that rewrites its history on retry produces new_segment:history_rewrite on that call. The episode still trains, as two segments instead of one, and the run page shows how often it happened. A harness that never rewrites history gets one contiguous segment per episode.
When the run ends, the calls fold into segments: contiguous token streams with a mask that is 1 on sampled tokens and 0 on everything the harness added, and the sampler's logprob under each sampled token. A segment becomes a training datum by the next-token shift, with weights, logprobs and advantages as full-length arrays that are zero off the mask. Housekeeping calls a harness makes, titles, summaries, compaction, are served by the relay and never enter a segment.
Every episode also carries a one-episode credential. The relay URL for a rollout is https://api.polygramme.com/tr/<trace_id> and its token opens that episode and nothing else: it cannot list, train or read anything. That is the credential handed to a container running someone else's code.
Figure 2: What runs where for one rollout. Left: your data is a task dataset of Harbor task directories (prompt, Dockerfile, verifier) and a trace dataset of ATIF episodes; each rollout consumes one task directory, and the harness is your image. Middle: every rollout gets a sandbox built from the task's Dockerfile; the harness process inside it talks only to the relay, and after it exits the verifier runs in the same sandbox. Right, top: the relay, a CPU web container; it translates each harness call into token-in, token-out against the sampling GPU and records both sides. Right, below it: the GPU cluster. The sampling GPU runs vLLM and serves every adapter the trainer publishes; the training GPU runs SkyRL's Tinker-compatible API over Megatron LoRA workers, one adapter per model id. Below: the training and rollout loops run in a CPU container with no GPU. It launches the sandboxes, turns the calls into datums, publishes every step, and runs the judge.
Two layout decisions deserve their reasons. The sampler is its own machine. A harness rollout takes tens of seconds to minutes; an optimizer step takes one to four. In a single-container design, the training GPU idles while sandboxes work and the sandboxes idle while it steps. With the sampler separate, rollouts keep sampling while the trainer steps, and the trainer takes the first finished groups (Step 4 covers the staleness rule). It also let us snapshot the sampler: Modal restores its memory image in about 30 seconds after idle, where a full vLLM boot with compile and graph capture is several minutes. The trainer, a Ray cluster with a state database, restores badly from a snapshot, so it boots the slow way, about three and a half minutes, and stays up for ten idle minutes to cover the gap between a bench and a run.
Adapters move over HTTP, not a shared disk. SkyRL registers a LoRA adapter on vLLM by file path and assumes a shared filesystem. A separate container has none, and a network volume only shows a write after the writer commits and the reader reloads, which races the load request. So the sampler pulls the two adapter files (about 350 MB) from the training GPU on every publish and loads the local copy; if it ever meets an adapter it does not know, after a restore or an eviction, it fetches on demand and retries the call. Publish-to-serving is 9 to 17 seconds per step.
Multi-tenant LoRA: one base model, many adapters
Both GPUs hold a single copy of the 9B base model. Everything a run trains is a rank-32 LoRA adapter on top of it, one adapter per model id, and the sampling GPU keeps four adapters resident with more in CPU memory, swapping them in per request. That is what makes the cluster multi-tenant: a run costs the adapter's share of a GPU, not a GPU, and several routers, or several versions of one, sample from the same vLLM process under different model ids. The adapter is also the artifact you own. It is about 350 MB, it is a PEFT directory, and it loads into any vLLM with --enable-lora or into transformers on your own hardware; a merged full-model export is one command if you would rather run it as a plain checkpoint. Nothing about the router is locked to the trainer that produced it.
What you write for a router is small, and none of it is training code:
With the environment and harness settled, the RL systems. The defaults are set for this shape of workload, but four knobs are worth understanding before a long run.
1. Rollout concurrency. max_parallel is the number of sandboxes in flight (default 8). Two ceilings bound it. The systems ceiling is the sampling GPU's KV cache: capacity in tokens divided by the average episode length. A router episode is short, a few thousand tokens, so this ceiling is high. The algorithmic ceiling is staleness: with async rollouts, at most (max_steps_off_policy + 1) × groups_per_batch × group_size episodes can be usefully in flight, for us (2 + 1) × 8 × 4 = 96. Past that, extra rollouts are too stale for the trainer to accept. Set concurrency to the lower of the two; for a router it is the second.
2. Staleness. Each group is stamped with the adapter version it started under. The trainer takes the first groups_per_batch finished groups, requeues any more than max_steps_off_policy steps behind (default 2), steps, and publishes. The loss is importance sampling by default, which is what makes a two-step lag safe: the sampler's logprobs are recorded per token by the relay, so the ratio between the policy that sampled and the policy being updated is exact, not estimated. Set max_steps_off_policy: 0 to get the classic synchronous loop for the de-risking run in Step 5.
3. Batch shape and kernel compile. The trainer compiles kernels the first time it sees a batch shape, 40 to 110 seconds per shape on a fresh boot, and then runs that shape at 1.0 s for 8×9 tokens, 1.45 s for 8×256 and 4.2 s for 64×256. Sampled completions have a new length every step, so fix max_tokens and let the trainer's token-based packing bucket the rest. Expect the first few steps of any boot to be slow while the buckets compile; a run with fixed shapes settles at one to two seconds per step on router-sized batches.
4. Train-inference mismatch. Every run reports the mean absolute difference between the logprob the sampler recorded and the logprob the trainer computes for the same token. A healthy run sits under 0.03. Above it, something between the relay's rendering and the trainer's rendering disagrees, and the reward curve will not tell you until much later. This check is how we caught a renderer that stripped thinking tokens from earlier turns on one side and not the other.
Suggestion. Run one evaluation pass over the full task set at the concurrency you intend to use, with the untrained policy, before any training. Drive the non-model error rate, sandbox failures, grader crashes, harness timeouts, to zero there. Masking errors during RL hides them; it does not remove their cost.
Before the real runs, confirm the tasks are learnable and the loop is closed, cheaply.
The one-step smoke. One task, a group of two, one step, synchronous. This is the run that exercises everything in Steps 2 and 3 at once: the sandbox builds from the Dockerfile, the harness starts, its calls reach the relay, the relay samples on the sampling GPU, the verifier runs, the calls come back with token IDs, a datum is built. Ours takes about 90 seconds end to end on a warm trainer and costs cents. It is also where we found our own worst bug: a network rule at the edge that returned "Your request was blocked" to the official OpenAI and Anthropic SDKs' user agents. The harness ran, the verifier ran, the run "succeeded" with zero relay calls and zero training signal. The fix was one setting; the lesson was to keep the harness's own output on the trace, which the runner now does, so a rollout that made no calls says why.
The overfitting run. A 32-task subset of contested tasks, group size 8, synchronous, so every step is one epoch. If the policy cannot overfit a handful of tasks, an end-to-end run has no chance, and the cause is almost always upstream. Watch for one pattern in particular: a group whose rollouts all score the same has no advantage and produces no gradient. Our first smoke logged exactly that, "every group uniform, nothing to train on", because both rollouts passed a trivial task. On a real task set that is why contested tasks and a group size of at least four matter more than the learning rate.
6.0 The run kinds
A training run is a run kind over a dataset, and the platform has four. They differ in what they read and where the trained tokens come from.
Supervised fine-tuning (--kind sft) reads a trace dataset and trains on the recorded assistant text, all assistant messages or only the last turn, with an optional reward window to pick which episodes count. The trained tokens are the ones in the file.
On-policy self-distillation (--kind opsd) reads the same trace dataset but trains on samples from the current adapter. Each episode becomes a prompt, the recorded answer or its reward becomes a hint that only the teacher sees, and the loss pulls the student's samples toward what the same model says with the hint. Section 6.1 covers the mechanics.
Reinforcement learning reads a task dataset instead, runs the policy against each task and trains on the sampled tokens with the grader's reward. With --kind train the hosted harness rolls out the task; with --kind byoh your own harness image runs in the sandbox and talks to the relay from Step 2. Section 6.2 covers the BYOH run.
Any run can start from the base model or from an adapter a previous run published. In this guide the OPSD run read the trace dataset from Step 1 and the RL run read the replay dataset from Step 2 and started from the OPSD run's adapter.
6.1 On-policy self-distillation, as implemented
Each episode becomes a prompt row: the conversation up to its last user turn, plus a hint the teacher will see and the student will not. With --hint-mode reward the hint is generated from the episode's reward map and its recorded answer:
Per step, the student samples group_size orderings per prompt from the current adapter, on the sampling GPU. The teacher is the same adapter, scored on the same sampled tokens with the hinted prompt substituted for the student's prompt; the implementation wraps the sampling client so that a logprob request for student prompt + completion is answered by scoring hinted prompt + completion and re-aligning the completion positions. The per-token advantage is the negative reverse KL, −(log p_student − log q_teacher) on sampled positions, scaled by kl_penalty_coef, and the update is an importance-sampling policy-gradient step on those advantages. After the step the adapter is republished, so the next batch is sampled from the weights just trained. That is what makes it on-policy rather than a replay of recorded text, and it is why there is no exposure bias between training and serving.
Two knobs do most of the work. --group-size is how many orderings the student samples per query; two is enough for an output as short as a provider list. --min-reward filters which episodes get a teacher's attention; the reward hint carries the judge's verdict either way, so a low-reward episode is still feedback, and we keep the floor at 0.5. The teacher-student KL should fall and flatten. If it flattens in the first few steps the hint is not informative enough, and --hint-mode custom with a rubric of your own is the next thing to try. One rule to know: a run makes at most one pass over its dataset, so with N episodes and groups_per_batch G it stops after ⌈N/G⌉ steps even when --steps is larger.
6.2 Reinforcement learning against the grader
The run is Step 5's overfitting command with the full dataset, the async defaults and an eval split:
Each step takes groups_per_batch tasks and rolls out group_size copies of each in separate sandboxes. The grader scores them; the advantage of each rollout is its reward minus its group's mean; the datums from Step 2 carry those advantages on the sampled tokens, and one optimizer step runs. On the run page, the number to watch besides reward is the sampler lag chart: the mean and max steps behind the trainer that each batch's rollouts were sampled, and how many groups were requeued as stale. A lag that grows step over step means the sandboxes cannot keep up and max_parallel should rise; a requeue count that grows means the opposite.
On loss functions: importance sampling, PPO, CISPO, GSPO and DPPO are all available and all use the relay's recorded logprobs directly, so none of them needs a second forward pass to obtain the sampling policy's logprobs. For a router, whose outputs are a few dozen tokens, we saw no difference between them at the scale of a 40-step run and kept the default.
Three numbers describe a run, and they answer different questions.
The training reward says whether rollouts pass their own graders. It is what the policy optimizes, and it is only as good as the fixtures and the judge label behind judge.json.
The eval sidecar says whether the policy is improving on tasks it did not train on. With --eval-every set, the held-out split is rolled out once per task at step 0, every N steps and at the end, with the same grader, and the results land under test/ separate from the training metrics. A training reward that climbs while test/ stays flat is a router memorizing its fixtures.
A judge run says whether the Step 1 judge agrees with what the router did. It takes a finished run's episodes, converts each trajectory to messages, and scores it with an LLM against a rubric; the judge is your own served model by default, or an external model with your key. Scores are stored per grader and version, so two rubrics on the same episodes sit side by side:
Running the judge on the eval split gives a held-out number that does not depend on the fixtures at all. When the grader's test/ reward and the judge's score disagree, either the fixtures are stale or the rubric is wrong, and the run does not promote until we know which. The comparison view puts a new run beside the deployed one on both numbers; a run is promoted when it beats the deployed one on both, and the baseline it must also beat is "always use the best single provider", which is a strong router and is reported beside every run.
The promoted adapter is served on an OpenAI-compatible endpoint under its model id, on the sampling GPU, and the router's request path calls it with the same system prompt it trained on. The same adapter can leave the platform at any time:
A fixed fraction of traffic stays on full fan-out. That slice is the only source of labels a routed request cannot produce, the first place drift appears, and the teacher data for the next retrain. From there the loop is a schedule: judge the new fan-out traces, push them into the same dataset and rebuild the replay tasks, distill nightly from the dataset as of that night, continue RL from it against the refreshed fixtures, evaluate, compare, promote.