Crawl settings

The default settings work for most sites. This chapter is for when they don't: a large site, a slow server, content built by JavaScript, or part of the site behind a login.

Limits and speed

• Depth: how many clicks from the start page the crawler goes. 0 means no limit.
• Max URLs: useful when you want a quick sample of a large site.
• Speed: the number of concurrent requests and the maximum URLs per second. Keep it low for someone else's server or a small one.
• Response timeout: after how many seconds a request counts as No response.

robots.txt and sitemaps

The crawler respects robots.txt the way Googlebot does, and reports blocked addresses in the Response codes tab, under the Internal: blocked by robots.txt filter. Turn on Read XML sitemaps so the crawler reads the sitemaps listed in robots.txt or at /sitemap.xml. Without it there are no orphan page filters. With a licence you also get a tool to test your own robots.txt before you put it on the server.

JavaScript rendering

By default, the crawler reads the HTML exactly as the server sends it. That is the fastest option and enough for sites whose content isn't built by JavaScript. For React, Vue and similar sites, turn on JavaScript rendering: each page opens in the built-in Chromium, and the crawler reads it only after the scripts have run.

• Wait for late content: how many seconds the crawler waits for content that loads afterwards.
• Browsers at once: more is faster, but uses more memory.

The JavaScript tab then shows the difference in word count before and after JS, plus errors from the console.

Include and exclude URLs

Regex rules, one per line. Crawl only matching URLs limits the crawl to, for example, ^/(blog|vijesti)/. Exclude rules leave out filters, tracking parameters or whole folders. You test a rule on a single address before you start.

URL rewriting removes parameters or changes part of an address before the crawl, e.g. /stari-shop/ => /shop/.

Logging in to a site

For staging or content behind a login, add a login in Settings, under Site logins: Basic auth or a login form. Use test accounts only. The crawler clicks links, so with a real admin account it could also open links that delete things.

Settings presets

Save your settings as a preset, e.g. "shop with JS" or "quick sample of 1,000 URLs", and use it for new crawls, schedules and the command line.

Next: tabs and filters.