Saw a hiring algorithm in Phoenix toss out every resume from a certain community college

I was helping a friend's small tech firm in Phoenix set up a new resume filter last month, and we found it was automatically rejecting every single applicant from the local community college, about 200 people. The system's training data was full of resumes from big name schools, so it just learned to ignore anything else. Has anyone else seen a filter that basically built its own unfair zip code rule?

3 comments

3 Comments

michaelgrant18d ago

Did you try adding a bunch of resumes from smaller schools to the training data?

wyatt13518d agoMost Upvoted

That's not a zip code rule, it's a bias from bad training data.

robertcarr18d ago

Man, that's so messed up. It's scary how these filters just copy our worst habits. I saw something similar where a system kept picking guys named "John" for sales jobs. @michaelgrant has the right idea about fixing the data, but how many good people get passed over before anyone notices? It's like the algorithm decided that one school's name meant "not good enough" without ever looking at a single skill.