Back to Basics: Why Naive Erasers Fail

With my pseudo-identities in place, I finally trained my base ResNet-18 facial classifier. The model performs wonderfully on the retain data. Now comes the fun part: making it forget.

I started with the naive baselines: Gradient Ascent, Random Relabeling, and Fine-Tuning.

Gradient Ascent is the most intuitive approach. You take the images you want to forget, and instead of taking a gradient step down the loss curve to learn, you intentionally take a step up to unlearn.

The result? Absolute catastrophe.

Naive gradient ascent doesn’t just make the model forget User 42. It violently wrecks the model’s core utility. The weights explode, and the model suddenly forgets everything. It’s the neural network equivalent of a lobotomy.

It became clear that unlearning is a delicate balance of erasing the target while protecting the rest of the network. We need a Gold Standard to measure against: the Oracle. The Oracle is a model retrained from scratch on the remaining data, representing perfect amnesia. If an unlearning algorithm can’t match the Oracle’s accuracy on the retain set, it’s essentially useless.

Next week, I am diving into the State-of-the-Art (SOTA) methods. Let’s see if the academic literature actually holds up.