I thought the hiring algorithm at my old job was fair until I saw the data

For two years, I helped run a system that filtered resumes for a big company in Chicago. I was sure it was neutral. Then, during a review, we pulled the numbers and saw it was passing over candidates from certain zip codes at a rate three times higher, even with names and schools hidden. The bias was in the job history patterns it learned from old, flawed hiring data. It felt like a gut punch because I had defended the system. Has anyone else had a tool they trusted turn out to have a hidden bias like that?

3 comments

3 Comments

emeryj6611d ago

Three times higher for certain zip codes? That's crazy. It makes you wonder what else those systems are picking up on. I always figured hiding names would fix it but job history patterns... that's sneaky. How do you even start to clean data that's already that messed up?

amy_craig2811d ago

Hiding names is a good first step, but it's not enough. The system can still connect zip codes to mostly minority neighborhoods. It might also use school names or even the types of jobs listed. Cleaning it means you have to find and remove all those hidden links, which is a huge job. They might need to rebuild the whole system from scratch.

jana_black11d ago

Our old system flagged every applicant from the 02119 zip code.