Replaced Token Detection › Key Parameters

When implementing RTD training, several parameters affect performance:

Several parameters interact to determine how well RTD training proceeds.

• mask_prob is the fraction of tokens to mask before generating replacements. The default 0.15 (15%) balances training signal against context preservation. Higher rates provide more training examples but degrade generation quality: when too many tokens are masked, the generator has little context to work with and produces poor replacements. The 15% rate, which also works well for MLM, turns out to be well-calibrated for this reason.

• disc_weight ($\lambda$) is the weighting factor for the discriminator loss in the combined objective. ELECTRA uses 50, heavily prioritizing discriminator learning since it is the model used for downstream tasks. If $\lambda$ is too small, the generator dominates gradient updates and the discriminator underfits. If $\lambda$ is too large, the generator receives insufficient gradient and produces poor replacements, making the discriminator's task trivially easy.

• generator_size_ratio is the ratio of generator to discriminator model size. Optimal values lie between 0.25 and 0.33. Smaller generators produce easily-detected replacements that do not challenge the discriminator enough, while larger generators waste parameters that could go toward discriminator capacity.

• d_model (discriminator) is the hidden dimension of the discriminator transformer. Larger values increase capacity but require more compute. ELECTRA-Base uses 256, ELECTRA-Large uses 1024. Matching the discriminator's hidden dimension to the target task complexity is important: tasks requiring fine-grained semantic understanding benefit from larger hidden dimensions.

• d_model (generator) is the hidden dimension of the generator, typically 1/4 to 1/3 of the discriminator's dimension. This asymmetry ensures the generator produces challenging but not impossible replacements.

• n_layers (discriminator) is the number of transformer layers in the discriminator. More layers increase representational capacity. ELECTRA-Base uses 12 layers, ELECTRA-Large uses 24. The layer count can be kept equal between generator and discriminator even when the hidden dimension differs, as in the original ELECTRA design. This gives the generator enough depth to model context while keeping its parameter count low due to the smaller hidden dimension.

• learning_rate can be set separately for generator and discriminator to improve training stability. Typical values range from 1e-4 to 5e-4, with some implementations using slightly lower rates for the generator to prevent it from improving too quickly and producing over-challenging replacements early in training.

• temperature controls randomness when sampling from the generator's output distribution. Higher temperatures produce more diverse replacements by flattening the probability distribution, while lower temperatures favor the most likely tokens and produce less varied replacements. A temperature of 1.0 (the default, corresponding to unmodified softmax sampling) works well in practice.