How to meet it
OWASP AISVS v1.01
Model Lifecycle Management & Change Control · Pipeline Fine-Tuning
Detect reward hacking in RLHF
Verify that RLHF training stages include automated detection of reward hacking or reward model over-optimization.
OWASP AISVS v1.01, requirement 3.5.2
In plain English
Watch for reward hacking during feedback training, where the model learns to score well on the reward model without behaving better.
What good looks like
- Training is monitored for signs of over-optimisation, such as reward rising while held-out quality falls.
- Runs that show these signs are stopped or reviewed.
- Samples are checked by people at intervals.
How teams usually do it
- Tracking divergence from the reference model and held-out evaluations during training.
- Automatic early stopping when metrics move apart.
Evidence an auditor would accept
- Monitoring dashboards or logs from training runs.
- The stopping rules.
Common mistakes
- Judging the run only by the reward score.
Not sure which requirements apply to your system? Take the assessment.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.