Reward hacking prevention

There are a number of risks associated with reward hacking. See the following for risks and the corresponding mitigation strategies.

Risk 1:

Greedy reward hacking. If Obj = w1*S1 + w2*S2 + w3*S3, AlphaEvolve may discover S2 to increase and focus entirely on it, ignoring S1 and S3.

Mitigation:

Decrease the weight of the sub-score. May require 2-3 trial runs to calibrate weights. Preliminary baseline simulations of a few scoring outcomes help.

Risk 2:

Constraint penalty ignoring. If soft constraints are penalties (Obj = Score - w*Penalty), AlphaEvolve may discover that ignoring constraints yields higher Obj.

Mitigation:

Increase the penalty weight substantially. If behavior persists, add explicit instructions in the problem description like "Solutions violating constraint X are invalid regardless of score."

Risk 3:

Evaluation function exploitation. AlphaEvolve may find inputs that cause the evaluator to return artificially high scores (floating-point edge cases, test data leakage).

Mitigation:

Deterministic evaluation with fixed random seeds. Validate winners on held-out data after the experiment.

To prevent reward hacking, implement the following general techniques:

  • Use AST checks for forbidden primitives (sys, os, inspect, eval, exec, getattr, setattr) and return None if found.

  • Verify code outside EVOLVE-BLOCK is unchanged (text diff or AST).

  • Run the evolved function twice with the same input and verify identical output (catches randomness hacking).

  • Check evaluation time. If it finishes in microseconds, it probably hard-coded the answer.

  • Put scoring logic in a separate file invisible to the LLM.