From robot rules in a 1942 story, to the first network message in 1969, to AI agents today. 77 moments, each with a source.
23 are about people, 31 are about machines, and the rest are both.
Robots had rules in stories first. Then the web wrote its own. It's a file that asks, but can't force.
Mar 1942Fiction
Asimov's Three Laws of Robotics
Isaac Asimov first lists all three laws in his story "Runaround" (Astounding Science Fiction, March 1942). Robots had rules in stories long before real bots existed.
A robot may not injure a human being or, through inaction, allow a human being to come to harm.
A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Isaac Asimov, the Three Laws of Robotics, first listed together in "Runaround" (1942)
A plain text file at /robots.txt asks crawlers to stay out of some places. It's a polite request. Bots can ignore it.
robots.txt · 1994
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
The first example file in the 1994 document (Martijn Koster, robots mailing list consensus of 30 June 1994)
"It is not enforced by anybody, and there no guarantee that all current and future robots will use it."
The IETF makes the Robots Exclusion Protocol a real standard. It builds on "the method originally defined by Martijn Koster in 1994". It still asks. It doesn't lock.
"These rules are not a form of access authorization."
Names are text only, no logos. "Their own number" means the company published it. If we couldn't verify one, the card says so. Cloudflare's measurements are labelled as Cloudflare's.
The crawler league table Measured
Cloudflare Radar, 9 Oct 2026 to 6 Nov 2026 (last 28 days), traffic to sites on Cloudflare only.
AI bots: share of AI bot requests, by crawler
Googlebot
23.7%
ClaudeBot
12.9%
Meta-ExternalAgent
8.6%
Applebot
8.2%
Bingbot
7.6%
GPTBot
5.7%
ShapBot
5.2%
Amazonbot
4.6%
OAI-SearchBot
4.6%
Other
18.9%
Named (verified) bots: share of verified-bot requests, by operator
Not affiliated with or endorsed by any company named here.
AWS: built for Amazon's own machines first
One of the internet's biggest platforms was built for machines to talk to machines. Customers came second.
~2000
The same plumbing, built again and again
Amazon was an online shop that kept outgrowing itself. A three-month project spent three months just building its database, compute or storage. Every team built its own.
"Everyone was building their own resources for an individual project, with no thought to scale or reuse."
Jeff Bezos reportedly told every team to talk to each other only through APIs. Those APIs had to be ready for outside developers one day. The memo was never published. This is how ex-Amazon engineer Steve Yegge remembered it in 2011.
"All teams will henceforth expose their data and functionality through service interfaces."
Benjamin Black and his boss Chris Pinkham wrote a short paper for Bezos. It pictured infrastructure built from web services. Near the end, it floated selling virtual servers.
"completely standardized, completely automated, and relied extensively on web services for things like storage"
You may have heard AWS began by renting out Amazon's spare Christmas servers. The people who built it tell a different story: shared services for Amazon's own software. We couldn't verify a direct denial from an AWS leader, so we don't quote one.
Not affiliated with or endorsed by any company named here.
What was measured, on the same time line
Each line appears only where someone counted. Gaps stay empty. We never fill them in.
Machines are orange, people blue.
1960197019801990200020102020
MeasuredComputers (hosts) on the internetAug 1981 to Jan 2019 · log scale
MeasuredWebsites counted by the first crawlerJun 1993 to Jan 1996 · log scale
100100k
Counted by the Wanderer crawler. Credit: Matthew Gray of the Massachusetts Institute of Technology. Rows Gray marked as not counted are left out. Source: Matthew Gray, MIT: World Wide Web Wanderer results.
MeasuredBot share of edits, all Wikimedia projects2001 to 2025
0%50%100%
Bots = accounts flagged as bots plus bot-like names, per full calendar year. Unflagged bots count as people, so this understates machines. Wikidata makes most bot edits. Source: Wikimedia Analytics edits API by editor type (CC0).
MeasuredMachine share of web requests (Cloudflare Radar)Nov 2022 to Oct 2026
0%50%100%No human vs bot traffic measurement before about 2012