Evaluator implementation patterns

When designing evaluation metrics for an AlphaEvolve evaluation harness, establish the following three distinct testing tiers.

Tier 1: Solution validation tests

These tests verify basic code execution safety and policy compliance before full execution.

  • Execution safety: Verify if the code executes. This includes reviewing syntax, checking compile time, performing static analysis, and checking infrastructure compatibility.

  • Execution policy: Determine if the code should be executed even if it is functional. This includes linting, general code quality checks, InfoSec reviews, risk and compliance validation, and API supply chain checks.

Considerations:

  • If validation fails, return a high negative score, such as -10000 or -100000. AlphaEvolve ignores these programs.

  • Include the failure type and logs for debugging. While AlphaEvolve does not use this metadata as a search signal, it is useful for post-facto analysis of failure modes.

Tier 2: Solution verification tests

These tests verify algorithmic correctness and adherence to constraints using unit and functional tests.

  • Functional accuracy: Verify if the code behaves correctly using unit and functional tests.

  • Constraint adherence: Verify solution feasibility or constraint satisfaction.

  • Reward hacking mitigation: Apply verification scores as negative penalties to the fitness function to prevent reward hacking. Asserting a score of -10000 or -100000 serves as an effective penalty.

Considerations:

  • Return the number or percentage of tests passed. This approach provides a gradient signal; a program passing 4 out of 5 tests is recognized as closer to correct than one passing 0 out of 5.

  • Treat verification as soft-constraint satisfaction. Even for hard system constraints, expressing them as soft penalties based on proximity to the feasible space gives AlphaEvolve a more actionable search signal than a binary pass or fail.

Tier 3: Solution evaluation tests (performance tests)

These tests measure objective performance metrics depending on the type of optimization target.

  • Direct measurement: Track objectives directly where instrumentation permits.

  • Deterministic estimates: Compute objectives analytically for classical operations research styles.

  • Out-of-sample validation: Test against validation datasets for use cases involving machine learning pipeline optimization or agent harness tuning.

  • Simulation-based estimates: Use Markov Chain Monte Carlo (MCMC) simulations, Bayesian Optimization type proxy functions, or other model-based approaches.

Considerations:

  • Return both granular and aggregate scores.

  • Don't offload score calculation to the LLM; compute it deterministically within your evaluator harness. AlphaEvolve hill climbs against these pre-calculated aggregate performance scores.

  • Partial credit provides a gradient signal that accelerates search convergence. For example, a program passing 4 out of 5 tests should receive a score of 0.8 rather than None. Returning None causes AlphaEvolve to ignore the program entirely, eliminating useful feedback. A score of 0.8 signals that the candidate is close to correct and holds viable potential for optimization.