What a crawler actually does
It fetches a page, reads what the page says about itself, takes every link out of it, throws away the ones it has already seen and puts the rest in a queue. That is the whole loop, and it fits in a paragraph. What does not fit in a paragraph is everything the loop runs into on the open web.
Fetching
A page is not a file. It is whatever a server decides to send to the particular client asking for it, which is why a crawler that announces itself is served differently from a browser, and why a report is only worth reading if it describes the page a person would have been shown. Redirects, retries, timeouts and the occasional server that answers a crawler with a login form are all part of the fetch rather than exceptions to it.
Parsing
Markup on the open web is broken, and a parser that rejects broken markup will reject most of the web. The parser has to do what a browser does: repair, guess, and carry on. Once it has a tree, the interesting parts are the title, the description, the canonical link, the metadata the sharing platforms read, the heading outline and the links.
Queueing
The queue is where a crawl either stays polite or turns into a denial of service. It needs a rate, a budget and a memory of what it has already seen, and the memory has to work on URLs that mean the same page while looking different: a trailing slash, a reordered query, a tracking parameter, a fragment.
Stopping
The last part is knowing when to stop, which is the part every naive crawler gets wrong. A calendar with a next-month link is infinite. A search page with facets is combinatorial. A sitemap may be fifty megabytes. A single page may be ten. The budget is not a nicety, it is the only thing standing between a crawl and a machine that never comes back.