c/algorithmic-fairness-debate

Back in 2015, I argued with a coworker that our new hiring algorithm was just a faster resume sorter, but now I see it was quietly deciding who even got a chance.

People keep saying 'the algorithm is just a tool' like it's neutral, but I realized after reading the ProPublica report on COMPAS that the real problem is we never even debated what 'fair' meant before we built the thing, so how can we fix it now without starting over?

I had to pick between two job apps that used different hiring algorithms

I was applying for a customer service job and two companies had different online forms. One asked for my work history and a short video, the other just scanned my resume for keywords. I went with the video one because it felt more fair, but I never heard back. The keyword scan place sent a rejection in two days, saying I didn't match their 'profile'. Has anyone else felt like the way you apply changes your whole shot at a job?

Read a study showing a hiring algorithm in Chicago was 40% less likely to recommend women for tech roles

I found it in a research paper from a university lab last year. Does this mean we need stricter audits, or is the bias just reflecting real-world hiring patterns that already exist?

Update: the new hiring system at my old job in Cincinnati flagged my resume as low quality

I applied for a part time role at the warehouse I worked at three years ago. The new automated system sent a rejection email in under two minutes, saying my resume lacked 'key terms'. I know they use a scoring algorithm from a company called HireScore. Has anyone had luck getting a human to review after a fast rejection like this?

Just saw a loan approval system flag a clear 'yes' as a 'no' and it's worrying...

I was helping my friend apply for a small business loan online last week. Her credit score is 720, she's been in business for 3 years, and her paperwork was perfect. The automated system instantly denied her. We called, and a human reviewer approved it in 10 minutes. The rep said the algorithm might have been thrown by her business address being in a zip code the system associates with higher risk. It learned a bad pattern. How many good applications get auto-rejected like this without anyone noticing? Has anyone else dug into a denial and found a weird data point that tanked it?

Showerthought: We spent 3 months testing a 'fair' hiring algorithm against basic resume screening. The old way was way more fair.

Went to the county courthouse to pay a parking ticket and the automated kiosk kept flagging my ID as 'suspicious'. The clerk said it happens a lot to people from my neighborhood.

I thought the hiring algorithm at my old job was fair until I saw the data

For two years, I helped run a system that filtered resumes for a big company in Chicago. I was sure it was neutral. Then, during a review, we pulled the numbers and saw it was passing over candidates from certain zip codes at a rate three times higher, even with names and schools hidden. The bias was in the job history patterns it learned from old, flawed hiring data. It felt like a gut punch because I had defended the system. Has anyone else had a tool they trusted turn out to have a hidden bias like that?

Visiting my old college campus and seeing the new AI admissions chatbot made me think about how much has changed since we just had paper forms.

It was weirdly efficient but I couldn't help wondering if it was screening out applicants from certain high schools without anyone noticing, which is the kind of thing we'd debate in our ethics class 10 years ago.

Remember when credit score algorithms didn't use rent payments?

About five years ago, I was trying to get a car loan in Phoenix and got denied because my score was low, even though I'd paid rent on time for a decade. The old system just ignored that data. Now, after a lot of public pressure and new laws, some major scoring models finally factor in rental history. It took years of arguing that the old way unfairly hurt younger people and renters. Has anyone here actually seen their score go up because of this change?

PSA: Our school's new grading algorithm flagged my son's essay as 'low effort' because of his dyslexia

My son, who's 12 and dyslexic, worked for three hours on an essay about Boston. The new automated system his school uses gave him a 40% for 'repetitive language and short sentences.' I printed the essay and showed his teacher the creative ideas he had about the Freedom Trail. She agreed the algorithm missed the point and manually overrode the score. Has anyone else seen these systems punish kids for learning differences instead of their actual work?

Our hiring algorithm in Austin flagged a perfect candidate as 'high risk' because of their zip code

My city's job application bot thought I was a cat for 3 weeks

I applied for a permit to open a small cafe in Phoenix, and their automated system flagged my application as 'fraudulent' because my last name is Cooper and my business name was 'Cooper's Corner'. The algorithm decided it was a duplicate identity, lol. I spent 21 days calling and emailing before a human finally looked at it and cleared it in 5 minutes. Has anyone else had an algorithm just completely misunderstand basic info like that?

Serious question, has anyone else had a resume scanner reject them for a job they were clearly qualified for?

Mine got flagged last week for a data analyst role requiring 3 years of SQL experience, which I have from my 4 years at my current firm, because the algorithm parsed my resume differently than a human would.

Lost a $1500 grant because a scoring algorithm flagged my application as 'low priority'

I spent weeks on a small business grant application for my shop. The form was long, I gave them all the numbers they asked for, even got letters from my bank. Got a denial email yesterday that just said my score was too low for the next round. No human looked at it. I called and they said the algorithm looks at zip code and business age first. My shop is in a good area now, but the zip code covers some rough spots a few blocks over. The business is only 18 months old, so it got marked as 'high risk' automatically. That $1500 was for a new refrigeration unit I really need. How is that fair? It feels like the system is built to help people who don't need it as much. Has anyone else been knocked out by a zip code filter?

DAE get flagged by a credit scoring algorithm for a weird reason?

My score tanked 40 points after I paid off my car loan early, the system saw it as closing an account and reducing my credit mix. Anyone know a fair way to appeal this kind of automated decision?

Saw a hiring algorithm in Phoenix toss out every resume from a certain community college

I was helping a friend's small tech firm in Phoenix set up a new resume filter last month, and we found it was automatically rejecting every single applicant from the local community college, about 200 people. The system's training data was full of resumes from big name schools, so it just learned to ignore anything else. Has anyone else seen a filter that basically built its own unfair zip code rule?

I just found out my old college's admissions algorithm was flagged for bias

I was talking to a friend who still works there, and she told me back in 2019 they had to scrap a whole new ranking system they'd built. It was supposed to predict student success, but after a year they found it was unfairly lowering scores for applicants from certain high schools, basically penalizing kids from less wealthy areas. They only caught it because a professor in the sociology department demanded to see the data. Now they have a totally different review process that includes a human committee looking at the algorithm's suggestions. Has anyone else seen a case where catching bias meant throwing out a whole system and starting over?

My credit line got cut by an algorithm that only saw my zip code

Three days ago, my bank sent a notice saying they were lowering my credit limit from $8,000 to $500. I called, and the agent said it was an automatic review based on 'area risk factors.' I live in a neighborhood that's mostly renters, and the system flagged it as high risk without looking at my 10-year payment history. I feel like I'm being judged by my address instead of my actions. Has anyone fought a decision like this and gotten a real person to look at it?

My cousin's loan got denied by a bank algorithm and it felt completely wrong

He's a great mechanic in Tampa with steady work, but the system flagged him for a small car loan because his credit file is 'thin' from using cash. I saw the letter and it just used generic risk scores, not his actual income or local reputation. Has anyone else seen a good person get a hard NO from a machine that clearly didn't have the full picture?

I spent $80 on a resume scanner that said I was a perfect fit, but the real hiring algorithm never even saw my application.

I mean, it's wild that a tool can promise fairness by checking keywords, but then a completely different system with its own hidden rules just filters you out before a human even looks, has anyone else had a company's public and private algorithms totally disagree on you?

Remember when people said facial recognition would never be used for small stuff?

I was at a concert in Austin about five years ago and a friend argued it would stay in airports and police work. Then my local grocery store quietly put up cameras for a 'loyalty program' that tracks your mood based on your face to show ads. That convinced me. It went from a big scary idea to a normal shopping trip thing real fast. Has anyone else seen these systems pop up in places you didn't expect?

A client's loan denial letter made me question the whole 'unbiased algorithm' thing

I was helping a couple look at houses in Akron last fall, and they were perfect on paper. Good jobs, solid down payment, the works. Their loan got flagged by the bank's automated system for 'high risk', with no real reason given. The wife showed me the letter, and she said, 'It feels like the computer just looked at our zip code and said no.' That stuck with me. I've seen a dozen similar cases since. If the system is trained on old, biased lending data, it's just going to keep making the same bad calls forever. Has anyone else seen a clear case where a 'fair' algorithm seemed to repeat a human bias?

Warning: My friend in Tampa got flagged for a 'high-risk' loan algorithm last week because his address matched a 3-year-old police report for a different person.

How do we even start fixing a system that punishes people for data they can't control?

Shoutout to the ER in my city for showing me how triage software can get it wrong

I saw a person with a clearly broken wrist wait over an hour while the system kept bumping them for 'lower risk' cases flagged by the algorithm, which made me think about who writes the rules for these things and if they've ever worked a real emergency scene.