Inspect indexed pages, folders, and discarded content

What you’ll accomplish

Use the crawl result to identify missing material before changing prompts or retrying the source.

Finish with

Evidence of what an agent can and cannot retrieve

1. Sample the pages that matterCompare indexed count with intended scope, then read key previews.

   Pages show URL, title, content preview, word count, and folder. A shortened or absent preview is not proof the full page is absent; representative-document fallbacks are marked truncated and are diagnostic, not exports.

2. Read the organization signalFolders show taxonomy, chunks, and approximate words.

   /_unclassified means no matching taxonomy folder—not automatically taxonomy-optimized content. Inspect it before calling a result a retrieval failure.

3. Explain discarded material before retryingDiscarded content is not retrievable.

   Review /_discarded_content, totals, discard rate, and reasons. A high discard rate usually calls for narrower scope, less duplicate/navigation-heavy content, or a cleaner authoritative source. Re-sync only after correcting the responsible source or scope.

Remove a page from this source

In Project → Knowledge, open the website, Shopify or WordPress source and select the page or record. Choose Remove from source, review the confirmation, then confirm Remove page. For a product or one of its variants, the action is Remove product and variants. The confirmation names the product family and language scope being removed.

Removal excludes that item from this source immediately and saves it under Excluded pages. Future normal and recurring syncs skip it. The change applies to every assistant assigned this source; it does not edit the website or store. The same content may still be available through another source.

Removing a Shopify product excludes its current and future variants across localized displays. Removing a WooCommerce product excludes its current and future variants in the selected language; other translated product families are separate. Excluded products stop contributing Knowledge, recommendations and new cart additions or quantity increases. Existing carts and orders stay in the store; shoppers can still inspect their cart, reduce quantities or remove existing items when the store connection permits it.

Allow an excluded page again

Open Excluded pages, search for the page or product, and choose Allow again. The item becomes eligible for a later normal sync. This does not immediately restore its content, start a sync, spend indexing credits or resume paused updates. After the next permitted sync, check the active content again. If the source is paused, disconnected or at capacity, resolve that existing blocker first.

Deleting the whole source also deletes its saved exclusions. Adding the source again starts with an empty exclusion list. Reconnecting the same existing source preserves its exclusions.

For a website crawl, Chat4U skips an excluded page and does not follow links from it. Pages discoverable only through that page may therefore be absent from a later crawl. Other pages already indexed are not removed as a group. Allowing the page again does not guarantee discovery; the next crawl must still reach it within the configured scope and limits.

Last updated on