Crawler — A crawler is an automated program that visits webpages and follows links to collect information. Search engines use crawlers to discover and index content, while other organizations use them for research and monitoring. Their reach depends on configuration, access restrictions, and the material they can retrieve.

Added November 6, 2026