ThunderSyncRL

For a group $\mathcal G$ of$\,m$ trajectories, GRPO weights each trajectory’s gradient by how its reward compares with the group. The group mean $\mu$ and standard deviation $\sigma$ need every reward:

$H_i$ is the gradient of trajectory$\,i$’s weighted action-token log probabilities, evaluated at the fixed policy $\theta_k$. It does not depend on the other rewards. Once trajectory$\,i$ and its reward are ready, ThunderSync runs that backward pass and adds to two buffers:

When the group closes, its mean and standard deviation turn those buffers into the final group gradient:

This is the same weighted sum, regrouped so the backward passes happen earlier. The policy stays at $\theta_k$ until the batch’s single optimizer step.

For each sampled action token, the frozen teacher and current student score the same token in its realized context. Let$\,\ell^{\mathrm T}_{j,t}$ and$\,\ell^{\mathrm S}_{j,t}$ be their log probabilities. Their difference gives the token’s weight:

Stop-gradient ($\operatorname{sg}$) holds this weight constant during differentiation. Once the teacher scores arrive, turn $j$ contributes a gradient while its tool call can keep running:

$\mathcal M_j$ contains the turn’s scored action tokens. ThunderSync accumulates these unnormalized contributions and completes any deferred gradients through earlier context. At batch closure, it divides once by the total scored-token count:

That normalization happens before clipping and the single optimizer step, giving the same batch-synchronous OPD update.