Website Crawl Audit (Mini Site Crawler)
This free website crawl audit starts at one URL, follows internal links and checks up to 25 pages inside a 20 second budget. The report lists every page with its HTTP status, title, description and H1 status, redirects and noindex tags, and shows internal links that lead to broken pages. It is a quick health check, not a full crawl.
Website Crawl Audit
Technical SEO
About this tool
A site audit usually begins with a crawl, a robot that moves from link to link the way a search engine does and records what it finds. Broken pages, redirect chains, pages with no title and pages blocked by a stray noindex tag are all easy to spot once the data is in one table, and very hard to spot by browsing.
This mini crawler starts at the address you enter and follows internal links on the same host, www and non-www treated as one. It stops at 25 pages or 20 seconds, whichever comes first, and obeys the robots.txt rules for all crawlers. For each page you see the status code, redirect hops, title, whether a description and an H1 exist, and whether the page is set to noindex. A second table lists internal links pointing to pages that returned an error.
It is a sample, not an inventory. Large sites have thousands of pages, so the report shows the pages closest to the start URL, which are usually the most important ones. Links to pages beyond the 25 crawled are not tested. Each crawl is cached for one hour, and the tool shares a small hourly fetch allowance per visitor, so run it on the start page that matters most.
Frequently asked questions
How does the mini crawler choose which pages to check?
It reads the start page, collects its internal links, and visits them in the order they appear, then follows the links found on those pages. Pages in the main navigation and near the top are visited first. It stops at 25 pages or when the 20 second budget runs out.
Why did the crawl stop before 25 pages?
Either the 20 second time budget ran out, the site has fewer pages reachable through links from the start page, robots.txt blocked some of them, or the hourly fetch allowance for your connection was used up. The report says which reason applied.
What is a redirect chain and why does it matter?
A redirect chain is a URL that redirects to another URL that redirects again. Each hop adds delay for visitors and crawlers, and long chains may be abandoned. Update internal links to point straight at the final address, and keep redirects to one hop.
Is this a replacement for Screaming Frog or Search Console?
No. A desktop crawler checks every page, renders JavaScript and exports full data, and Search Console shows what Google actually crawled and indexed. Use this tool for a fast first look at a site, then use those for depth. It is free and needs no install.