A simple python based crawler to crawl and find all the URLs on your website to see what is broken.
Created completely using GPT 4 and Cursor. Tested on a very basic site and it works.
On larger sites it tends to I guess overcrawl and I'm not sure why it does that.