The algorithm for a web crawler fits on a napkin:

1. Take a URL off a queue.
2. Download the page.
3. Extract its links.
4. Put the new ones back on the queue. Repeat.

Write that and you have a crawler. Point it at the open web and within about ten minutes you will have been rate-limited, IP-banned, trapped in an infinitely deep calendar page, and served the same article eleven times under eleven different URLs.