When AI Starts Shaping a Person’s Future, Who Ensures Fairness?

By Lingya (Lynn) Ke

Rethinking AI's decision-making power, from hiring and lending to healthcare

Imagine you've spent weeks preparing your résumé, carefully filled out the application, and finally submitted it to a company you really want to work for.

A few seconds later, you receive a rejection email.

You don't know why you were rejected, or whether a real person ever looked at your résumé. All you know is that the company's AI hiring system decided you weren't a suitable candidate.

Now imagine another scenario.

You apply for a loan, and the bank's system flags you as a high-risk customer. You have a stable income and have never missed a payment, yet the bank can't clearly explain why the algorithm decided you weren't trustworthy.

Both scenarios point to the same question:

When organizations use AI to make decisions that affect people, are those decisions reasonable and fair, and should AI really be the one making them?

As artificial intelligence (AI) is widely adopted in hiring, finance, healthcare, and education, more and more decisions once made by people are being handed over to algorithms.

For organizations, AI means efficiency, lower costs, and the ability to process vast amounts of information. For the individuals affected by those decisions, things may not be so simple.

An algorithm can calculate a person's risk, but it may not truly understand that person's circumstances.

I. Is AI Really Fairer Than Humans?

When efficiency becomes an organization's top priority

Why are more and more organizations letting AI take part in decision-making?

The answer is not complicated: efficiency.

Suppose a large company receives 100,000 job applications a year. Reviewing each one by hand would take the HR department enormous amounts of time and money. AI, by contrast, can quickly analyze applicants' education, work experience, skills, and other relevant information to help the company screen candidates.

At a bank, AI can predict the risk of loan default by analyzing customers' credit histories, income, and financial behavior. At a hospital, AI can help doctors spot abnormalities in medical images and make diagnosis more efficient.

From a management perspective, these applications clearly have value.

More importantly, human decision-making is not necessarily fair either.

A hiring manager may be biased by a candidate's appearance, accent, gender, or alma mater. A loan officer may be swayed by personal experience and subjective impressions.

AI, on the other hand, won't reject a résumé because it's in a bad mood, and it won't slow down because it's tired.

But does that mean AI is fairer than humans?

The answer is: not necessarily.

AI has no emotions, but it can inherit human bias

AI decisions depend on data, and that data often comes from past human behavior and decisions.

If past decisions were biased, AI may inherit those biases as it learns from the data.

A well-known example comes from Amazon.

In 2018, news reports revealed that Amazon had developed a machine-learning tool to screen job applicants' résumés. But because the system learned mainly from résumés submitted over the previous ten years, and the tech industry had long been dominated by male applicants, the system gradually began to disadvantage female candidates.

For example, the system gave lower ratings to résumés containing certain words associated with women.

Amazon eventually abandoned the project.

This case reveals an easily overlooked problem:

AI may have no intention of discriminating against anyone, and still produce discriminatory results.

From the algorithm's point of view, it is simply looking for features that resemble those of successful candidates in the historical data. But if the people who got opportunities in the past came mostly from certain groups, the standard of success that AI learns may simply carry past inequality forward.

This is algorithmic bias.

So judging whether AI is fair means looking beyond whether it applies a uniform standard. We also need to ask whether that standard is reasonable, and whether the outcomes place certain groups at a disproportionate disadvantage.

II. When an Algorithm Says "No," Do We Have the Right to Know Why?

A decision that can't be explained

In 2019, Apple Card customer David Heinemeier Hansson publicly questioned the card's credit limit decisions on social media.

He said that although he and his wife filed joint tax returns and had similar financial backgrounds, he had been given a far higher credit limit than she had.

The incident quickly sparked public debate over whether algorithmic credit decisions involved gender discrimination.

The New York State Department of Financial Services then opened an investigation. Its findings, released in 2021, did not find unlawful gender discrimination in the Apple Card credit algorithm, but regulators noted that consumers' limited understanding of credit decisions and the lack of transparency remained concerns.

What makes this controversy important is not proving that a particular algorithm discriminates, but the question it raises:

When two people receive clearly different outcomes, can the organization provide an explanation that is clear, reasonable, and verifiable?

The algorithmic "black box"

Many complex AI models can identify relationships among large numbers of variables, but their decision-making process is not always easy for ordinary people to understand.

This is often called the black box problem.

For example, an AI system might use hundreds of variables to conclude that an applicant has a high risk of default.

But if all the bank can tell the applicant is:

"The system has determined that your risk is too high."

Is that explanation enough?

For the organization, this may be a data-driven risk judgment. For the applicant, the decision may mean being unable to buy a home, pay tuition, or even lose a chance to improve their life.

Even more worrying, when people don't know why they were rejected, it becomes very hard for them to tell whether the decision was wrong.

If the input data is incorrect, who is responsible for fixing it?

If the algorithm used an unreasonable standard, who has the right to challenge it?

If the system's judgment clearly doesn't match reality, does the individual have any chance to ask for a review?

This involves three important concepts:

• Transparency: Can the organization explain how AI is involved in the decision and which main factors it uses?

• Explainability: Can the people affected understand the main reasons behind the decision?

• Accountability: When AI makes a wrong or unfair decision, who should be held responsible?

A decision that is highly accurate in statistical terms is not necessarily reasonable at the level of the individual.

If a person must accept an algorithm's decision with no opportunity to understand or challenge it, then the real issue is no longer just technology. It is power.

III. AI Can Predict Risk, but Can It Understand a Person?

The person in the data and the person in real life

In finance, risk assessment usually relies on historical data.

For example, a bank may use a credit scoring model to predict how likely a borrower is to default in the future.

Suppose an applicant was once unemployed for six months, which lowered their credit score. AI may conclude that this person is at higher risk of not repaying on time.

But reality may be more complicated.

Perhaps the applicant left work temporarily to care for a sick family member, and has since returned to work with a stable income.

AI can identify the statistical relationship between a history of unemployment and the risk of default, but it may not fully understand the reasons behind that unemployment.

Of course, human decision-makers may also overlook such context, or be influenced by even stronger subjective biases.

So the question is not whether humans are always more empathetic than AI. It is this: has the organization left enough room for special circumstances, corrections to information, and human review?

Does high accuracy mean a decision is reasonable?

Suppose an AI credit model achieves 95% predictive accuracy on test data.

That sounds quite reliable.

But if the system evaluates 100,000 applicants a year, even a 5% error rate could still mean 5,000 wrong predictions.

And those errors are not necessarily spread evenly across everyone.

Some groups may be more likely to be wrongly classified as high-risk, while others may be more likely to have their loans approved.

In machine learning, this involves different types of errors, such as:

• False positive: The system wrongly classifies a customer who would not actually default as high-risk.

• False negative: The system wrongly classifies a customer who would actually default as low-risk.

For a bank, false negatives can mean financial losses.

For applicants, false positives can mean losing opportunities they should have had.

So organizations can't just ask:

"How accurate is our model?"

They should also ask:

"Whom does our model wrongly harm?"

"Who bears the consequences of these errors?"

"Are we willing to make some people bear greater risk in exchange for a little more efficiency?"

Especially in high-impact decisions such as hiring, healthcare, and lending, overall accuracy is only one part of evaluating how a system performs.

Truly responsible AI decision-making must also consider how errors are distributed, how serious their impact is, and whether they can be corrected.

IV. Which Decisions Can Be Left to AI, and Which Should Not?

AI taking part in a decision doesn't mean AI gets the final say

AI does not have to be completely excluded from important decisions.

In fact, in some cases, AI can help organizations make judgments that are more consistent and accurate than those based on human experience alone.

In healthcare, for example, AI can help doctors analyze medical images and detect abnormalities that might otherwise be missed. In finance, AI can help banks identify unusual transactions and potential fraud.

The question is what role AI should play.

We can divide the use of AI in organizational decision-making into three levels.

Level 1: AI provides information (AI-assisted decision-making)

AI collects, organizes, and analyzes data, and offers recommendations to people.

For example, AI helps a hiring manager identify applicants with relevant skills, but the final decision is still made by the hiring team.

Level 2: AI makes recommendations, and humans review them (human-in-the-loop)

AI can form a preliminary judgment, but important decisions must be reviewed by a person with real authority.

For example, a bank's AI system flags a customer as high-risk, but a loan officer can examine the data further, consider special circumstances, and decide whether to approve the loan.

Level 3: AI makes decisions automatically (fully automated decision-making)

AI makes decisions directly based on preset rules or models, without a person approving each one.

This approach suits some low-risk tasks whose results are easy to correct, such as sorting ordinary spam email.

But when a decision involves job opportunities, medical treatment, credit eligibility, or a person's basic rights, full automation calls for much stricter limits.

The most important criterion here should not be AI's technical capability alone, but the consequences a decision may have.

The greater a decision's impact on people, and the harder it is to reverse, the more an organization needs to strengthen oversight, explanation, and appeal mechanisms.

At the same time, human review cannot be a mere formality.

If employees simply click "approve" without ever questioning the algorithm's output, so-called human-in-the-loop oversight accomplishes nothing.

Effective human oversight means that reviewers understand the system's limitations, have access to the information they need, and have the authority to overrule AI's recommendations.

V. When AI Gets It Wrong, Who Is Responsible?

Algorithms must not become an excuse to avoid responsibility

Suppose a company uses AI to screen job applicants.

One day, the company discovers that the system has long given lower scores to applicants from a particular group, causing many qualified people to miss out on interviews.

Who should be held responsible?

The tech company that developed the algorithm?

The team that supplied the training data?

The HR department that decided to adopt the system?

Or the senior managers who approved the purchase?

In reality, each of these parties may bear a different share of responsibility.

But one thing should be clear:

An organization cannot claim it is not responsible for the outcome just because the decision was made by AI.

AI is a tool that an organization chooses to adopt, not an entity that can independently bear legal and moral responsibility.

So before adopting AI, organizations should not only test a model's accuracy, but also assess its fairness, transparency, data quality, and potential social impact.

From "Can we use it?" to "Should we use it?"

In recent years, regulators in many countries have begun paying closer attention to the potential risks of AI.

For example, the EU Artificial Intelligence Act (EU AI Act) takes a risk-based approach, setting different requirements for AI systems depending on how they are used.

AI applications involving hiring, education, and certain essential services may be classified as high-risk systems, which must meet corresponding requirements for risk management, data governance, record-keeping, and human oversight.

This reflects an important principle of governance:

Not all AI applications should be regulated to the same degree.

Using AI to recommend movies and using AI to decide whether someone gets a job clearly cannot be held to exactly the same standard.

For organizations, responsible AI governance should include at least the following:

First, assess risks before deployment.

Identify which groups AI might disadvantage, and test whether the model shows significant bias.

Second, continuously monitor model performance.

An AI system's performance can change as data and conditions change, so testing once before launch is not enough.

Third, establish a clear division of responsibility.

Make clear who is responsible for model development, data quality, compliance review, final decisions, and incident response.

Fourth, provide an effective appeals process.

When individuals believe an AI decision is wrong, they should be able to object and, where appropriate, receive a genuine human review.

These measures do not mean AI will never make mistakes.

Their purpose is to ensure that when mistakes happen, the organization is able to detect, explain, and correct them, rather than shifting responsibility onto an algorithm that cannot answer any questions.

VI. What Can College Students and Job Seekers Do?

For college students, especially those about to enter the workforce, AI decision-making is not a distant technical issue.

It may already be part of applying for internships, looking for jobs, applying for credit cards, or even being evaluated in school.

In hiring, some companies use automated tools to analyze résumés, schedule interviews, or assess candidates' abilities.

This means a résumé may be screened by an automated system before a hiring manager ever sees it.

For job seekers, understanding how AI hiring tools work certainly matters, but it matters even more to recognize that being rejected by a system doesn't necessarily mean you lack ability.

An algorithm's judgment is shaped by its training data, evaluation criteria, and model design.

If a hiring system relies too heavily on particular keywords or historical hiring patterns, it may overlook applicants from different backgrounds who are just as capable of doing the job.

So when we face an automated decision, we can try to find out:

• Was AI or an automated system used in this decision?

• Has the organization explained its main evaluation criteria?

• If my personal information contains errors, can I request a correction?

• If I believe the outcome is unreasonable, is there a human review or appeals process?

These questions apply not only to job seekers, but to anyone who may be affected by algorithmic decisions.

At the same time, students who go on to work in finance, risk management, human resources, or technology may themselves become the designers, users, or managers of these AI systems.

When that time comes, we will need to think not only about how to use AI to improve efficiency, but also about how to make sure technology does not harm the very people organizations are supposed to protect.

Conclusion: Do We Need Smarter AI, or More Responsible Decisions?

Let's return to where we began.

When a job seeker is rejected by AI, when a customer is labeled high-risk by an algorithm, when a patient's medical needs are assessed by an automated system, the question we really need to ask is not only:

"Is the AI's judgment accurate?"

But also:

"Is this decision reasonable?"

"Is it fair?"

"Can the people affected understand and challenge it?"

"And should this decision really be made by AI?"

AI can help organizations process large amounts of data, uncover patterns humans would struggle to see, and make decisions more efficiently.

But efficiency should not be an excuse to ignore fairness, and accuracy should not be an excuse to avoid responsibility.

AI can calculate probabilities, but it cannot decide on its own how society should weigh fairness, dignity, and opportunity.

AI can make recommendations, but it should not become a tool organizations use to shift responsibility.

What is truly worth pursuing is not letting AI replace all human judgment, but building a decision-making system in which technological capability and human responsibility work together.

Because when a decision can change the course of a person's life, we cannot care only whether the algorithm produced an answer. We must also care whether that answer deserves to be trusted.

References

Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters.

European Commission. (n.d.). AI Act: Regulatory framework for artificial intelligence.

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0).

Organisation for Economic Co-operation and Development. (2019, updated 2024). OECD AI Principles.

Patnaik, S. (2019, November 11). Apple co-founder says Apple Card algorithm gave wife lower credit limit. Reuters.