Guide · chapter 02

First crawl: whole site or URL list

Enter an address, choose Whole site or URL list mode and send the bee out. What happens while the crawl runs and when it finishes.

2 min readUpdated 21 October 2026

Whole site or URL list

On the start screen you choose one of two modes:

ModeWhat it doesWhen to use it
Whole siteStarts at the address you enter and follows every internal link until it has visited every page.Site audits, finding broken links, preparing a sitemap.
URL listChecks only the addresses you paste in or load from a file, without following links.Checking redirects after a migration, checking a list from Search Console.

Starting

  1. Enter the address with https://, e.g. https://example.com. If the site redirects to www or the other way round, the crawler sees it and reports it.
  2. Check the settings if the defaults don't suit you: depth limit, speed, JavaScript. For a first crawl the defaults are enough. More in configuration.
  3. Click Start.

While the crawl runs

The table fills in real time, and at the top you can see how many URLs have been processed and how many are waiting. You can pause a crawl and resume it later, and with the saved file you can even resume it after closing the app.

Be polite to the server. On small hosting, lower the number of concurrent requests. If the server starts returning 429 or 503, the bee slows down by itself, but it's better not to get that far.

When the crawl finishes

The bee goes once more through everything it has gathered and works out what you can't see from a single page: inlinks, orphan pages, duplicates, redirect chains and hreflang return links. Only then are the numbers in the tabs final.

On the right is the Overview, with the number of URLs per tab and filter, and a list of issues sorted by importance, each with a short explanation of what it means and how to fix it.

Next: how to read tabs and filters.