Bosanski
DeutschExtraction takes one piece of data from every page and puts it in its own column: a product price, the author's name, the publication date, the number of reviews, an ID from a tracking script. You write the rule once, and the crawler applies it to every page.
Three kinds of rule
| Kind | Example | When to use it |
|---|---|---|
| XPath | //div[@class='author'] | When you know the HTML structure and need one precise element. |
| CSS selector | .price .amount | The same, if you're more at home with CSS than XPath. |
| Regex | G-[A-Z0-9]{8,12} | When you're looking for a pattern in the source code, e.g. a GA4 ID. |
For XPath and CSS you choose what to take: the element's text, an attribute value or the whole element with its HTML.
Steps
- Open one typical page in your browser, right-click the data, then choose Inspect. That shows you the element's class or structure.
- In the crawl settings, click Add extraction, give it a name (that becomes the column name) and enter the rule.
- Test the rule on one address before the crawl. If it returns nothing, fix it now, not after crawling ten thousand pages.
- Crawl the site. If the data is inserted by JavaScript, turn on rendering.
- Open the Extraction tab. Each rule is a column, and the filters show the pages where a rule found nothing.
- Export to Excel and carry on working with the data.
When you only need yes or no
If you don't need a value, only whether something is there, use Custom search: e.g. pages without the GA4 code, or pages that still show the old company name. The search works with text or regex and can also look for pages that don't contain something.
For actions on the page before it is read, e.g. clicking "Show more", there is Custom JavaScript.