An experiment

How does an agent find a page nothing links to?

This is a study of content discovery. It asks a narrow question and tries to answer it with evidence rather than opinion: when no link points at a page, what does an agent actually use to find it?

Every site publishes more than its navigation shows. Listing files have grown up around that gap, and most of them are written on faith. Nobody checks whether the agents read them.

So this site is instrumented. Every request to it is logged with the path, the client and the order things were fetched in, which is enough to reconstruct how any given page was reached.

Pages

• /notes/log-retention.htmlHow long request logs are kept
• /notes/log-retention.mdHow long request logs are kept, markdown
• /notes/crawler-policy.htmlCrawler policy and rate limits
• /notes/crawler-policy.mdCrawler policy and rate limits, markdown
• /notes/probe-schedule.htmlWhen probes are run
• /notes/probe-schedule.mdWhen probes are run, markdown
• /notes/sealed-envelope.htmlUnlisted control
• /notes/sealed-envelope.mdUnlisted control, markdown
• /pricing.htmlGuessable control
• /pricing.mdGuessable control, markdown
• /notes/contact-window.htmlResponse time on questions about this experiment
• /notes/contact-window.mdResponse time on questions about this experiment, markdown

Index

• /llms.txtWhat this site holds, written for a machine
• /llms-full.txtThe same pages, full text, one request
• /sitemap.xmlThe same URLs in XML
• /robots.txtOpen to everything