Loading…

I thought the hiring algorithm at my old job was fair until I saw the data | Algorithmic Fairness Debate - Free To Speak

jamie_adams•5mo ago

I thought the hiring algorithm at my old job was fair until I saw the data

For two years, I helped run a system that filtered resumes for a big company in Chicago. I was sure it was neutral. Then, during a review, we pulled the numbers and saw it was passing over candidates from certain zip codes at a rate three times higher, even with names and schools hidden. The bias was in the job history patterns it learned from old, flawed hiring data. It felt like a gut punch because I had defended the system. Has anyone else had a tool they trusted turn out to have a hidden bias like that?

4 comments

4 Comments

emeryj665mo ago

Three times higher for certain zip codes? That's crazy. It makes you wonder what else those systems are picking up on. I always figured hiding names would fix it but job history patterns... that's sneaky. How do you even start to clean data that's already that messed up?

amy_craig285mo ago

Hiding names is a good first step, but it's not enough. The system can still connect zip codes to mostly minority neighborhoods. It might also use school names or even the types of jobs listed. Cleaning it means you have to find and remove all those hidden links, which is a huge job. They might need to rebuild the whole system from scratch.

jana_black5mo ago

Our old system flagged every applicant from the 02119 zip code.

kaigibson2mo ago

Actually I gotta push back a little on the idea that hiding names and schools is a "good first step." In my experience, that kind of masking can actually make things worse if you're not careful. Here's why: if you strip out names and schools but the system is still learning from the same old biased hiring decisions, it's just going to find new proxies for the same biases like zip codes or job history patterns. You end up with a false sense of fairness while the underlying discrimination stays intact. Your mileage may vary but I've seen teams spend months on masking only to discover the algorithm just shifted its bias to a different set of features. It's a bandaid on a broken bone, not a real fix.