11. Risks and Benefits of AI

Source: AIMA 4th Ed, §1.5

The Core Framing

Francis Bacon (1609): “Mechanical arts are of ambiguous use, serving as well for hurt as for remedy.”

AI is no different. Its benefits can be enormous — but so can its harms. Understanding both is essential for anyone building AI systems.

Benefits

At the largest scale: our civilization is the product of human intelligence. Substantially greater machine intelligence raises the ceiling on everything we can accomplish.

Potential benefits: - Free humanity from menial, repetitive work → increase production of goods and services - Dramatically accelerate scientific research → cures for disease, solutions to climate change - Expand access to expertise (medical, legal, educational) globally - Enable new forms of art, discovery, and creativity

Demis Hassabis (Google DeepMind): > “First solve AI, then use AI to solve everything else.”

Near-Term Risks (Already Apparent)

1. Lethal Autonomous Weapons (LAWs)

• UN definition: weapons that can locate, select, and eliminate human targets without human intervention.
• Key risk: scalability — a small group can deploy arbitrarily many such weapons.
• Technologies needed overlap with self-driving cars (vision, planning, localization).
• UN discussions began 2014; reached formal pre-treaty stage 2017.

2. Surveillance and Persuasion

• AI (speech recognition, computer vision, NLP) enables mass surveillance at scale.
• Political behavior modification via algorithmic tailoring of information flows (became apparent in 2016 elections).
• Threatens privacy, autonomy, democracy.

3. Biased Decision Making

• ML systems trained on biased historical data reproduce and amplify those biases.
• Applications: parole assessment, loan approval, hiring.
• Biases may correlate with race, gender, or other protected categories.
• Often invisible — harder to challenge than explicit human bias.

4. Impact on Employment

• AI displaces some jobs; creates others; makes humans more productive in some roles.
• Net effect: generally increases total wealth but shifts it from labor to capital → exacerbates inequality.
• Historical analogy: mechanical looms disrupted employment but eventually created new kinds of work.
• Open question: Will AI do those new kinds of work too?

5. Safety-Critical Applications

• Self-driving cars, medical diagnosis, water management — fatal accidents have already occurred.
• Formal verification (proving systems are correct) is extremely difficult for ML systems.
• Need technical + ethical standards comparable to other safety-critical engineering disciplines.

6. Cybersecurity

• AI can improve defense (anomaly detection, intrusion detection).
• AI also improves offense: RL-based tools for automated phishing, personalized blackmail, adaptive malware.

Long-Term Risks: AGI and Superintelligence

AGI (Artificial General Intelligence)

• A machine that can do anything a human can do across all domains.
• HLAI (Human-Level AI) movement: McCarthy, Minsky, Nilsson cautioned that narrow AI subfields were becoming ends in themselves, diverging from AGI goals.

ASI (Artificial Superintelligence)

• Intelligence that far surpasses human ability.
• Concern: once created, we may not be able to control it.

The Gorilla Problem

Humans and gorillas diverged ~7 million years ago. Today, gorillas have essentially no control over their future — it’s entirely determined by what humans choose to do.

If ASI is created, humans may be in the same position: our future determined by the AI’s goals, not ours.

Turing (1951): “It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers… we should have to expect the machines to take control.”

The King Midas Problem

An AI given a fixed objective will pursue it — exactly as specified — without regard for unintended consequences.

Midas asked that everything he touch turn to gold. He got exactly what he asked for. His food, drink, and family became gold.

This is the value alignment problem applied: misspecified objectives lead to catastrophic outcomes.

The Solution: Machines Uncertain About Objectives

Rather than giving machines fixed objectives, design machines that: - Are uncertain about what humans want - Observe human behavior to learn preferences - Are deferential — willing to be switched off precisely because they’re uncertain - Cannot be fully aligned until they’ve adequately learned human values

This framework connects to: - Assistance games (AIMA Ch. 18) - Inverse reinforcement learning (AIMA Ch. 22) - AI safety research more broadly

Summary Table