# The Machine That Reads — A History of Web Scraping & Automation

> From 1970s green screens to autonomous agents: the long arc of machines that read and act on other machines’ interfaces.

**URL:** https://athree.dev/the-machine/  
**Canonical markdown:** https://athree.dev/the-machine.md  
**HTML experience:** interactive timeline essay on the live page.

## Thesis

Web scraping and browser automation did not begin with the web. They descend from terminal screen-scraping, crawlers, bots, headless browsers, anti-bot arms races, and finally LLM-driven agents. Offense and defense co-evolved for decades; neither side permanently “won.”

## Arc (compressed)

1. **Pre-web (1970s–1980s)** — IBM 3270 / VT terminals; EHLLAPI “PC robots”; the term *screen scraping*.
2. **Early web & crawlers (1990s)** — Full-text crawlers; politeness norms; `robots.txt` (Martijn Koster); search engines industrialize fetching.
3. **Scriptable HTTP (2000s)** — curl, wget, scrapers, early frameworks; CAPTCHAs and ToS become defenses.
4. **Browser automation (2010s)** — Selenium, PhantomJS, Puppeteer, Playwright; fingerprinting and residential proxies escalate the arms race.
5. **LLM era (2020s)** — Agents that *think* while browsing; publishers block AI crawlers; tools that emit LLM-ready Markdown/JSON; stage-based systems (like A3) separate expensive learning from cheap deterministic replay.

## Why A3 links this essay

A3’s learn-then-run model is a deliberate answer to the 2020s cost/reliability problem: use models where they shine (exploration, repair), use code where it shines (repeatable execution).

## Related

- How we built A3: https://athree.dev/how.md
- Overview: https://athree.dev/index.html.md