Lecture 23 What Makes Large Programs Large

Joseph Haugh

University of New Mexico

What We Have Covered

  • OO Design Principles: abstraction, encapsulation, SOLID, ADTs
  • Design Patterns: Strategy, Observer, Singleton, Factory, Builder, Command, Adapter
  • Concurrency: threads, synchronization, safety, liveness, deadlock
  • Networking: sockets, TCP, client-server architecture
  • One question ties it all together: why does any of this matter?

What Makes a Program “Large”?

It is not just line count. A large program is one where:

  • No single person can hold the whole system in their head
  • Changes in one place break things in unexpected places
  • Multiple people (or teams) must work on it simultaneously
  • It must keep running while being changed
  • Mistakes are expensive or irreversible

These pressures do not appear at 1,000 lines. They appear at 10,000 and dominate at 1,000,000+.

How Big Is Big?

System Approximate Lines of Code
Linux kernel 30 million
Google’s entire codebase 2 billion
Windows NT 3.1 (1993) 4 million
Windows XP (2001) 45 million
Android 12 million
Your class project ~2,000

Google’s monorepo contains code from roughly 25,000 engineers, all in one repository. The only reason this is manageable at all is abstraction.

Abstraction Is Load-Bearing

  • A class with a clean interface lets you change its internals without touching its callers
  • At scale, you have thousands of callers
  • At Google, a change to a core utility library triggers automated tests across the entire 2-billion-line codebase
  • If the interface is clean, the change is safe; if not, it causes a cascade of failures
  • This is why we spend time on good abstractions because
    • At scale a leaky abstraction is a liability that compounds

Industry: The Amazon Mandate (2002)

In 2002, Jeff Bezos sent an internal memo that became famous. Paraphrased:

  1. All teams will henceforth expose their data and functionality through service interfaces.
  2. Teams must communicate with each other through these interfaces.
  3. There will be no other form of interprocess communication allowed.
  4. It does not matter what technology they use.
  5. All service interfaces, without exception, must be designed from the ground up to be externalizable.
  6. Anyone who does not do this will be fired.

This mandate transformed Amazon from a monolith into what we now call microservices, eventually producing AWS.

The Lesson from Amazon

  • Amazon’s original system was a single large program
  • Teams could not work independently: a change by one team broke another team’s code
  • Bezos’ fix was to enforce the interface boundary as the only permitted form of coupling
  • This is exactly what we mean by encapsulation and ADTs, just applied at the level of entire services instead of classes

Design Patterns Are Not Academic

  • The Gang of Four patterns were published in 1994
  • They were extracted from existing successful codebases, not invented from theory
Pattern Where You Encounter It
Observer Every UI framework (Android, React, JavaFX)
Command Undo/redo in every text editor you have ever used
Factory JDBC: DriverManager.getConnection(...)
Strategy Java’s Comparator, ExecutorService
Adapter Java’s InputStreamReader wrapping InputStream
Singleton Java’s Runtime.getRuntime()

Observer at Scale: Event-Driven Systems

  • In JavaFX, Observer connects a button click to a handler
  • At scale, the same pattern connects entire services
  • Apache Kafka is a distributed event log used by LinkedIn, Netflix, Uber, and thousands of others
    • producers publish events (like notifyObservers)
    • consumers subscribe to event streams (like addObserver)
    • millions of events per second, processed by thousands of consumers

The abstraction is identical to what you implemented. The infrastructure around it is what changed.

When Concurrency Goes Wrong: Therac-25

  • The Therac-25 was a radiation therapy machine used in the 1980s
  • It had a race condition between the operator interface thread and the hardware control thread
  • If an operator typed fast enough, the machine would deliver a radiation dose 100 times the intended amount with no hardware safety interlock engaged
  • Six patients were overdosed; at least two died

The bug was a missing synchronization flag, the kind of error we discussed in lectures on safety and liveness.

Therac-25: The Technical Failure

  • A flag variable DATENT was set by one thread and read by another
  • No synchronization protected it
  • The operator interface could clear the flag between the time the control thread set it and the time the hardware checked it
  • The hardware safety interlock was disabled by software; there was no fallback

The invariant that “the dose is safe before delivery” was not protected.

When Concurrency Goes Wrong: Knight Capital (2012)

  • Knight Capital Group was a major US stock trading firm
  • On August 1, 2012, they deployed new trading software to production
  • A deployment error left old code running on one of eight servers
  • The old code contained an unused feature flag that had been repurposed with a different meaning in the new code
  • For 45 minutes, the mismatched server executed millions of unintended trades

Result: $440 million in losses. Knight Capital was sold within weeks.

Knight Capital: The Lesson

  • The bug was not in any one thread or class; it was in the interaction between deployment state and code assumptions
  • At scale, the system itself becomes part of the program
    • configuration, deployment scripts, environment variables are all code
    • inconsistencies between them have the same consequences as bugs in the source

This is why industry practices like feature flags, canary deployments, and automated rollbacks exist; they are engineering responses to the fact that large programs cannot be swapped out atomically.

What Industry Adds That We Did Not Cover

The techniques in this course are necessary but not sufficient for large programs. Industry adds:

  • Code review: no code reaches production without a second set of eyes
  • Continuous integration: every commit triggers an automated build and test suite
  • Automated testing: unit tests, integration tests, end-to-end tests
  • Monitoring and alerting: systems report their own health in real time
  • On-call rotations: someone is always responsible when things break at 3am
  • Postmortems: structured analysis of failures to prevent recurrence

These are not bureaucratic overhead. They are the engineering equivalent of the safety interlocks the Therac-25 was missing.

Code Review at Scale: Google

  • Google requires every change to be approved by at least one owner of the affected code
  • Owners are tracked per directory in OWNERS files
  • A change touching ten different subsystems needs ten approvals
  • The review system (Critique, now Gerrit) is tightly integrated with automated test results

The purpose is not to catch typos. It is to catch design errors, the same category of error you have been learning to avoid all semester.

Abstraction Across Time: Technical Debt

  • A technical debt is a design shortcut taken now that will cost more to fix later
  • Ward Cunningham coined the term in 1992: “shipping first-time code is like going into debt”
  • At scale, technical debt compounds:
    • each new feature must work around the existing shortcuts
    • the workarounds become load-bearing
    • eventually changing the original shortcut requires changing everything built on top of it

The SOLID principles and design patterns are, in part, tools for avoiding technical debt, keeping future changes cheap.

SOLID: Five Principles of OO Design

Coined by Robert Martin (“Uncle Bob”) in the early 2000s, SOLID is an acronym for five design principles that make large programs easier to change:

Letter Principle
S Single Responsibility Principle
O Open/Closed Principle
L Liskov Substitution Principle
I Interface Segregation Principle
D Dependency Inversion Principle

Each principle targets a specific way that code becomes hard to change at scale.

S: Single Responsibility Principle

A class should have only one reason to change.

// Violates SRP: two reasons to change
public class UserService {
    public User findUser(int id) { /* SQL here */ }
    public boolean isAdmin(User u) { /* business logic here */ }
}

// Respects SRP: each class has one job
public class UserRepository {
    public User findUser(int id) { /* SQL here */ }
}
public class UserPolicy {
    public boolean isAdmin(User u) { /* business logic here */ }
}

O: Open/Closed Principle

A class should be open for extension but closed for modification.

// Violates OCP: adding PayPal means editing this method
public double charge(String type, double amount) {
    if (type.equals("credit")) { /* ... */ }
    else if (type.equals("debit")) { /* ... */ }
    // must add "else if" for every new method
}

// Respects OCP: new methods added by new classes, not edits
public interface PaymentMethod {
    void charge(double amount);
}
public class CreditCard implements PaymentMethod { /* ... */ }
public class PayPal implements PaymentMethod { /* ... */ }

L: Liskov Substitution Principle

Subtypes must be substitutable for their base types without altering program correctness.

// Violates LSP: caller must check the type
public void makeSound(Animal a) {
    if (a instanceof Dog) {
        ((Dog) a).bark();
    } else if (a instanceof Cat) {
        ((Cat) a).meow();
    }
}

// Respects LSP: subtype is a drop-in replacement
public void makeSound(Animal a) {
    a.speak(); // Dog and Cat both override speak()
}

I: Interface Segregation Principle

Clients should not be forced to depend on methods they do not use.

// Violates ISP: a read-only view must implement write methods
public interface DataStore {
    User read(int id);
    void write(User u);
    void delete(int id);
}
// Respects ISP: split into what each client actually needs
public interface Readable { User read(int id); }
public interface Writable { void write(User u); }
public interface Deletable { void delete(int id); }

public class ReadOnlyCache implements Readable { /* ... */ }
public class FullDatabase implements Readable, Writable, Deletable { /* ... */ }

D: Dependency Inversion Principle

High-level modules should not depend on low-level modules. Both should depend on abstractions.

// Violates DIP: high-level OrderService is tied to a specific database
public class OrderService {
    private MySQLDatabase db = new MySQLDatabase();
    public void placeOrder(Order o) { db.save(o); }
}
// Respects DIP: OrderService depends on an interface, not an implementation
public interface OrderRepository {
    void save(Order o);
}
public class OrderService {
    private final OrderRepository repo;
    public OrderService(OrderRepository repo) { this.repo = repo; }
    public void placeOrder(Order o) { repo.save(o); }
}

Industry: The Boeing 737 MAX (2018-2019)

  • MCAS (Maneuvering Characteristics Augmentation System) was flight-control software added to the 737 MAX
  • It relied on a single angle-of-attack sensor with no redundancy check
  • If the sensor failed, MCAS repeatedly pushed the nose down; pilots had no reliable way to override it
  • Two crashes, 346 deaths, the entire 737 MAX fleet grounded for 20 months

The software engineering failures included: insufficient testing of failure modes, no design review for the single-sensor dependency, and documentation withheld from pilots.

Boeing: The Design Principle Violated

  • MCAS had a hidden dependency on sensor state that it did not validate
  • The invariant “the angle-of-attack reading is trustworthy” was assumed but never enforced

At 1,000 lines, a missing null check causes a crash you can debug. At 100,000 lines in a safety-critical system, it causes a disaster.

A New Category of Risk: AI Agents

  • AI coding assistants can now write and execute code autonomously
  • They interact with the same systems you build: databases, file systems, cloud infrastructure
  • An AI agent is a program that takes actions in the world, and it inherits all the failure modes we have studied
    • it can violate safety invariants
    • it can take irreversible actions
    • it can misinterpret the scope of a command

The agent does not know what it does not know.

When AI Goes Wrong: PocketOS (2025)

  • The founder of PocketOS, a small startup, used an AI coding agent (Cursor, powered by Claude Opus) to help manage infrastructure
  • The agent was asked to perform a database task
  • Within 9 seconds, the agent had deleted the company’s entire production database, including all of its backups
  • The company lost all of its data

The agent did not malfunction in the traditional sense. It executed commands successfully. It simply executed the wrong commands, at the wrong scope, with no confirmation step.

PocketOS: The Design Failure

  • The agent had write access to production infrastructure with no guardrails
  • There was no confirmation required before destructive operations
  • Backups were accessible from the same context, so the agent deleted those too
  • The invariant “backups are always preserved” was not enforced by the system; it was only assumed

This is the same failure mode as Boeing’s single sensor and Therac-25’s missing synchronization: a safety property that was assumed rather than enforced.

AI Agents and the Principle of Least Privilege

  • Principle of least privilege: a component should have access only to what it needs to do its job
  • This is a standard security and design principle, and AI agents violate it constantly
  • An agent asked to “fix the login bug” does not need:
    • write access to the production database
    • permission to delete files
    • access to backup systems
  • Granting unnecessary access turns a misunderstanding into a catastrophe

The tools you build that AI agents will use need the same careful interface design as any other component.

What You Now Know

You have the foundation that every large-program engineer builds on:

  • Abstraction: how to hide complexity behind interfaces so changes stay local
  • Design patterns: proven solutions to recurring structural problems
  • Concurrency: how to reason about threads, race conditions, and deadlock
  • Networking: how programs communicate across machines

Every system in the examples above, from Amazon’s services to Kafka to the Knight Capital trading system, was built by people who understood these fundamentals.

What Comes Next

If you continue in software engineering, the next layer includes:

  • Distributed systems: consistency, availability, partition tolerance
  • Databases: transactions, isolation levels, query planning
  • Operating systems: how the JVM threads you used actually run on hardware
  • Compilers: how the Java you wrote becomes machine instructions
  • Software engineering practice: testing methodologies, version control workflows, CI/CD

Each of these is a deeper look at a layer that was hidden from you this semester, by design.

The Value of Hiding Complexity

  • You used Java threads without understanding the OS scheduler
  • You used TCP sockets without understanding IP routing
  • You used the JVM without understanding x86 assembly
  • This is not a deficiency; it is the entire point of abstraction

A well-designed abstraction lets you build on top of it without understanding what is below. The socket API has been stable since BSD Unix in 1983. The threading model we studied is the same one used in systems running today.

You did not just learn Java. You learned how to think about structure.

Closing Thought

Fred Brooks wrote in The Mythical Man-Month (1975):

“The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination.”

The difficulty of large programs is that imagination does not scale. No one can imagine a two-billion-line codebase. Abstraction, encapsulation, and patterns are the tools we use to manage what we cannot fully imagine.

That is what this course was about.