Repeated data
Mixture-of-experts models start overfitting repeated data at half a dense model’s budget
Stanford and Washington report dense models tolerating 8x data repetition against 4x for matched MoE models. The sparse advantage is gone entirely by 32x. Router assignments go near-stationary early, so each expert memorises one fixed slice of tokens.
Masking-based regularisation pushes the break-even past 64x, though the authors write that “no method fully matches the performance of all-unique training data”.
What it changes Halve the repetition budget you would assume from dense experience before continuing to train an MoE on a fixed corpus.
Primary source read
Mixture-of-experts models start overfitting repeated data at