Crawl a website or sitemap
Define a website source’s scope, start an authorized crawl, and review what was indexed.
2 min read
- Project
- Knowledge
- Add source
- Website
Use the Knowledge library for website sources. Choose Website crawl or Sitemap in Add source; once the source appears, use Start Scraping on its card.
Start with the smallest web scope that can answer the agent’s job, then verify what was actually retained.
- Finish with
- A narrow, authorized website source you have inspected
Choose the smallest useful scopeOne page, linked pages, or a maintained sitemap.
In Project → Knowledge, select Add source, then choose Website. Use Single page for one canonical URL; Website crawl for linked pages (depth 0–5, maximum 1–10,000 pages); or Sitemap for a maintained URL list. For
https://help.example.com, start at depth 1 and a modest maximum. Include/exclude patterns focus scope but never replace inspection.Confirm authority before startingOnly index content you control or are authorized to use.
Complete the URL authority attestation in the Add source flow before saving. Respect robots.txt, terms, access controls, and applicable restrictions; never submit private, authenticated, or third-party material without authority. Avoid signed or session-specific URLs.
Start, inspect, and assignReady means indexed—not automatically useful.
After saving, return to the source card in Knowledge and select Start Scraping. Follow progress there. When the card is Ready, open it to inspect pages and folders; then go to Assistant → Knowledge to assign the source and ask representative questions. A crawl can contain fewer pages than its maximum when fewer are discoverable or relevant.
Last updated on