Knowledge sources

Hosted kinds are website root, sitemap, single public URL, PDF, DOCX, TXT, Markdown, and custom Q&A. Every Hosted source is visitor-readable. Do not upload private runbooks or secrets. To scope knowledge to specific pages, add those pages as single public URL sources or a sitemap of just those pages, instead of a website root, which crawls the host.

add_answers_url_source accepts kind website, sitemap, or url, plus a public http or https URL. It does not accept file bytes, a question, or a provider key. Files and custom Q&A stay on the dashboard.

Index a dedicated public page whose body already states the fact. A homepage of product blurbs is a weak first source. Keep each price or number next to what it describes. For facts you want stated exactly, add a custom Q&A.

A website or sitemap crawl also fetches /llms.txt and /llms-full.txt on that host when robots.txt allows them. Text and Markdown are indexed. A missing file is ignored. BYO refresh() does the same for each origin in its URL list or sitemap. A single public URL that is the file itself is indexed as text.

Custom Q&A

Custom Q&A is always considered, even when page passages outrank it, and a matching entry is answered first. Use it for prices, policies, your support contact, and the honest “we do not state that” answers you want given.

Hosted: add entries on the dashboard. Each entry is indexed on save, usually within about a minute, and the knowledge list shows its status (indexing, then indexed).

BYO: change entries at runtime with no refresh. Call await answers.upsertQa({ id, question, answer }), answers.removeQa(id), or answers.setQa([...]) on the backend from createAnswersBackend. Only the changed entry is embedded, and it answers on the next question.

Crawl canonicalizes the URL, drops tracking parameters such as utm_, follows redirects only to public hosts, and honors robots.txt. Private hosts are rejected. Navigation chrome is stripped before chunking. A page that is only a nav shell is rejected and does not become a citation. A failed crawl leaves the last good index in place.

Related: Hosted index · Limits