Static site

Everything on this site is in the HTML the server sends. Nothing is added by a script, nothing waits for a render. A crawl of this site should find every page below with the parser alone, which makes it the control case for the SPA site next door.

Pages worth reaching

• About — an ordinary page with ordinary metadata
• Products — an index one level down
• Blog — dated articles with Open Graph tags
• Deep chain — five levels, each linked only from the one above
• Thin content — a page of eleven words
• SEO problems — everything a report should complain about
• Noindex — asks not to be indexed, still crawlable
• Nofollow links — links a follower may skip

Links that are meant to misbehave

• A page that does not exist — expect 404
• A page robots.txt disallows — expect it to be recorded as blocked
• About, with tracking parameters — should collapse into About
• About, with a fragment — should collapse into About
• A mailto link — not crawlable
• A tel link — not crawlable
• A javascript link — not crawlable
• Another test site — same host, different tree

External links

• example.com — answers 200
• httpbin.org/status/404 — answers 404
• an unresolvable host — fails to be reached
• an external PDF — a file rather than a page