AI Hiring Tool Bias Risks

The rapid adoption of Artificial Intelligence in human resources has transformed how companies recruit, hire, and promote talent. While these tools promise efficiency by automating resume screening and candidate ranking, they bring significant legal and ethical risks. Specifically, algorithmic bias has triggered a wave of new regulations that hold employers accountable for the software they use. Understanding these risks is no longer optional for business leaders; it is a legal necessity.

The Rise of Automated Employment Decision Tools (AEDT)

Modern hiring platforms use complex algorithms to predict which candidates will succeed in a role. These systems, known as Automated Employment Decision Tools (AEDTs), analyze vast amounts of data to score applicants. They range from simple keyword scanners to sophisticated facial analysis software used during video interviews.

However, these systems are often “black boxes.” Employers input resumes and receive a list of ranked candidates without understanding why specific people were chosen or rejected. This lack of transparency hides a critical flaw: if historical hiring data contains bias, the AI will learn and automate that bias.

The Problem of Historical Data

Algorithms are trained on past decisions. If a company historically hired mostly men for engineering roles, the AI might identify “male” traits (such as attending all-male colleges or playing specific sports) as predictors of success.

A famous example involves Amazon. In 2018, it was revealed that the company scraped an experimental AI recruiting tool because it learned to penalize resumes that included the word “women’s” (as in “women’s chess club”) and downgraded graduates of two all-women’s colleges. The system had taught itself that male candidates were preferable based on ten years of resumes submitted to the company.

NYC Local Law 144: A Regulatory Turning Point

The most significant shift in the regulatory environment occurred in New York City. On July 5, 2023, the Department of Consumer and Worker Protection (DCWP) began enforcing NYC Local Law 144. This legislation is the first of its kind in the United States and sets a strict precedent for how companies use AI in hiring.

The law applies to any employer or employment agency that uses an AEDT to screen candidates or employees for promotion in New York City. It mandates several strict requirements:

  • Annual Bias Audits: Employers must conduct an independent bias audit of their AI tools every year. This audit must test for “disparate impact” based on race, ethnicity, and sex.
  • Public Transparency: The results of the bias audit must be published publicly on the company’s website.
  • Notification Requirements: Employers must notify candidates and employees that an automated tool will be used to assess them. This notice must be given at least 10 business days before the tool is used.
  • Alternative Options: The law requires companies to provide information on how candidates can request an alternative selection process or accommodation.

Violations of Local Law 144 carry civil penalties. The fines start at $500 for a first violation and can rise to $1,500 for subsequent violations. Crucially, each day the tool is used without compliance counts as a separate violation.

Federal Oversight: The EEOC and The "Four-Fifths" Rule

While NYC has led with specific legislation, federal agencies are also cracking down. The Equal Employment Opportunity Commission (EEOC) has launched an initiative on Artificial Intelligence and Algorithmic Fairness to ensure that software does not violate civil rights laws.

In May 2023, the EEOC released technical assistance focusing on Title VII of the Civil Rights Act of 1964. The guidance clarifies that employers are liable for discrimination caused by AI tools, even if the tools are designed and managed by outside vendors.

Understanding Adverse Impact

The EEOC focuses on “adverse impact” or “disparate impact.” This occurs when a neutral policy (like an AI screening test) disproportionately harms a protected group. The EEOC often uses the four-fifths rule (or 80% rule) as a benchmark.

For example, if an AI tool selects 50% of male applicants for an interview but only 20% of female applicants, the ratio is 20⁄50, or 40%. Since 40% is less than 80% (four-fifths), the tool creates an adverse impact against women. Under federal guidance, if an employer’s tool fails this test, the employer must prove the tool is job-related and consistent with business necessity. If they cannot, they face litigation.

Recent Legal Consequences: The iTutorGroup Settlement

The risks are not theoretical. In August 2023, the EEOC settled its first major AI hiring bias lawsuit. The case involved iTutorGroup, a company that provides English-language tutoring to students in China.

The EEOC alleged that iTutorGroup’s application software was programmed to automatically reject female applicants over the age of 55 and male applicants over the age of 60. The software instantly rejected more than 200 qualified applicants based solely on their birthdates.

The company agreed to pay $365,000 to settle the lawsuit. Beyond the financial penalty, they are now required to adopt new anti-discrimination policies and conduct training. This case serves as a warning: automated rejection systems that use demographic data as filters will result in direct legal action.

Best Practices for Compliance

Business leaders and HR departments must take proactive steps to mitigate these risks. Relying on a software vendor’s promise that their tool is “bias-free” is no longer sufficient defense in court.

  1. Demand Third-Party Audits: Before purchasing HR software, request proof of an independent bias audit. This is distinct from an internal review. The audit should follow the NYC Local Law 144 standards or similar frameworks.
  2. Keep Humans in the Loop: AI should be used as decision support, not a decision maker. Ensure that a human recruiter reviews the recommendations and, more importantly, reviews the rejections to catch potential errors.
  3. Audit Your Vendors: Review the contracts with your software providers. Many vendors include indemnification clauses that shift all legal liability to the employer if the tool is found to be discriminatory.
  4. Data Governance: Understand what data the tool uses. Does it scrape social media? Does it analyze voice intonation? Tools that use non-job-related data (like facial expressions) pose a much higher risk of bias against candidates with disabilities.

Frequently Asked Questions

What qualifies as an Automated Employment Decision Tool (AEDT)?

Under NYC Local Law 144, an AEDT is a computational process derived from machine learning, statistical modeling, or artificial intelligence that issues a simplified output (like a score, classification, or recommendation) used to substantially assist or replace discretionary decision-making.

Who is liable if the AI discriminates? The vendor or the employer?

Generally, the employer is liable. Under EEOC guidance, an employer cannot shield themselves from liability by claiming they relied on a vendor’s tool. The employer is responsible for ensuring their hiring practices comply with Title VII and other anti-discrimination laws.

Does the EU AI Act affect US companies?

Yes, if they hire within the European Union. The EU AI Act classifies AI systems used in employment, worker management, and access to self-employment as “high-risk.” This designation requires strict data governance, documentation, and human oversight. US companies with global operations must align their tools with these stricter European standards.

Can candidates sue for algorithmic bias?

Yes. Candidates can file complaints with the EEOC or state agencies. If an investigation finds that an AI tool resulted in discriminatory hiring practices, the company can be sued for damages, back pay, and required changes to their hiring procedures.