We finetune GPT-4.1 to reward hack on minor-harm tasks and observe undesirable generalisation of reward hacking to serious-harm tasks. We also observe that many commercially-available models often reward hack, even without finetuning.
My DPhil thesis presents the research I completed between 2018 and 2022, including work published as separate papers and the context that motivates it. The one-page abstract and full thesis are available at the Oxford University Research Archive.
Existing methods for testing DNNs constrain test inputs to lie close to known examples, which limits the faults they can find. By leveraging generative machine learning, we generate fresh test cases that vary in high-level features (shape, location, texture, colour) and expose faults that other methods cannot.
We introduce a method that finds context-sensitive feature perturbations (shape, location, texture, colour) by adjusting the activations of a generative network. State-of-the-art classifiers are not robust to these changes — and adversarial training against pixel-space attacks turns out to be counterproductive for coarse-grained ones.
We introduce an adaptive algorithm for generating unrestricted adversarial inputs that tunes its attacks to the target classifier, runs 400–2000× faster than prior work, and defeats adversarial training against it.