The Ethics Dilemma and Finding SFHQ

Following up on my last post, I hit a massive roadblock early on. To test machine unlearning properly, I need a facial recognition dataset.

The standard academic approach would be to grab something off the shelf like MUFAC (a benchmark explicitly built for identity unlearning), CelebA, or Labeled Faces in the Wild. It’s what everyone does. You download a massive zip file of real human faces, slap it into a PyTorch DataLoader, and get to work.

But there’s an irony here that I couldn’t shake. I am building a project entirely focused on the “Right to be Forgotten” and protecting user privacy. Downloading thousands of real photos of people—many of whom never consented to their faces being used to test machine unlearning algorithms—felt deeply hypocritical.

More pragmatically, running a research project that handles real biometric data requires dragging the entire proposal through a university ethics review board. That process can take months. I don’t have months. I have a tight project deadline.

The “Aha!” Moment

I spent days trying to figure out how to bypass this. Then I realized something obvious: the neural network doesn’t care if a face is real.

To a ResNet-18, a face is just a grid of pixels, edges, and gradients. As long as the visual complexity is there, the unlearning algorithm will be stressed exactly the same way.

That’s when I stumbled onto SFHQ (Synthetic Faces High Quality). SFHQ is a massive dataset of 1024x1024 faces generated entirely by AI (StyleGAN2 and Stable Diffusion). There are absolutely no real people in this dataset. Every pore, every wrinkle, every strand of hair belongs to someone who doesn’t exist.

By pivoting to entirely AI-generated faces, I hit a massive win-win: I can simulate realistic privacy requests and model deletion, and I can completely sidestep the ethics review board because there is zero risk to any real human’s privacy.

The Catch

There’s always a catch, right?

SFHQ is incredible, but it’s just a giant folder of one-off fake faces. In order to test machine unlearning, I need a dataset of users (i.e., multiple photos of the same person) so I can say, “Hey model, forget User 42.”

Right now, I just have a sea of 400,000 disconnected generations. I don’t have “identities.”

I’m currently writing a script to try and artificially group these faces together. If it works, I’ll have my dataset. If it doesn’t, I might be back to square one. I’ll share how that goes next time.