The Iterative Reality Check

Academic unlearning benchmarks almost universally evaluate “single-shot” deletion. You delete 5% of the data once, measure the accuracy, and publish a paper.

But that’s not how the real world works. A deployed system gets deletion requests sequentially. User A leaves today. User B leaves tomorrow.

I shifted my evaluation paradigm to a cumulative mode protocol, testing how these SOTA models handle 15 to 20 consecutive deletion requests. It was an absolute bloodbath.

Models that performed beautifully on a single deletion started to drift, degrade, and eventually collapse entirely after the 10th deletion. For methods like MSG, the static saliency masks they calculated at step 1 were completely stale by step 10, meaning they were erasing the wrong parameters.

Single-shot benchmarks are lying to us. The stability of an unlearning algorithm can only be measured sequentially. And right now, we are failing.