🛠 copy & metadata
to discover & index web pages
🛠 Incremental web crawlers
🛠 Distributed crawlers
🛠 Parallel crawlers
🛠 Deep web crawlers
🛠 Screen scrapers
Used by the search engine to:
🎯 Crawl websites
🎯 View images
🎯 View links
🎯 Index them on the internet
Commercial Bots
Used by some SEO websites to provide users with SEO reports of a selected website so as to solve any SEO issues on the site
Examples:
🕸️ Ahrefsbot by ahref.com
🕸️ SemrushBot by Semrush.com
🕸️ Barkrowler by Babbar.tech
Feed Fetchers Bots
Used to collect thumbnails & titles of the contents to display on their website
Examples:
Facebook external hit – used by the Facebook website
Twitter bot – used by Twitter
Monitoring Bots
Used to check the performance of the websites performances 🎯 uptime
🎯 pinback
📍 Unlike crawlers, which collect information about websites
scrapers are focused on the CONTENTS of the site, EXTRACTING:
📝 texts
🖼 images
📹 videos
🗣 audio
💲 prices
🎯 any other specific elements