Train Your AI Chatbot with Knowledge Sources

Knowledge Sources

Knowledge sources provide the information your chatbot uses for grounded answers. Open Dashboard → your assistant → Knowledge to add website URLs, upload files, enter text directly, or connect external platforms such as Confluence, Notion, SharePoint, Google Drive, Nextcloud, or SFTP/FTPS.

The Knowledge tab also shows account-wide character usage. Its source table can be searched and filtered by type or category and shows each source's indexing status and last update. Row actions let you re-import, edit, inspect, download, stop, or delete a source where supported. Select multiple rows for bulk re-indexing or deletion.

You can also connect a folder — SharePoint, Google Drive, Nextcloud or an SFTP server. Everything in it is indexed automatically, and every change is picked up on the next sync without anyone re-uploading a file. See Connected folders.

Website URLs

Enter URLs to crawl and index. WebChatAgent automatically crawls the pages and extracts content for your chatbot's knowledge base.

Adding a Website

1. Click Add Data Source in the Knowledge tab and choose Website
2. Enter the website URL (e.g. example.com — https:// is added automatically)
3. Configure optional advanced settings (see below)
4. Click Add to start crawling

Advanced Settings

When to Use CSS Selectors

CSS selectors help you extract only the relevant content from a page. This is useful when:

• Your pages have navigation menus, footers, or sidebars you want to exclude
• You want to focus on the main article content only
• The page contains ads or unrelated widgets

Example: To extract only the main content area and skip navigation:

article, .main-content, #post-body

When to Use Exclude URL Patterns

Exclude patterns prevent specific pages from being indexed (added to your chatbot's knowledge base). Common use cases:

• Login/admin pages: /login, /admin, /wp-admin
• Legal pages: /impressum, /privacy-policy, /terms
• Duplicate content: */amp/*, */print/*
• Category/tag archives: /category/*, /tag/* (often duplicate content)
• Query parameters: ?hitsPerPage — excludes all URLs containing this parameter (e.g. search result pages with pagination)

Important details:

• Patterns are case-sensitive. /About and /about are treated as different patterns. Make sure your patterns match the exact casing used in your URLs.
• Excluded pages are not indexed, but their links are still followed. The crawler will still discover and crawl pages linked from excluded URLs — only the excluded page itself is skipped for indexing. This means excluding a category page like /blog/category/news will prevent that listing page from being indexed, but the individual blog posts linked from it will still be crawled and indexed (unless they also match an exclude pattern).

Language Considerations

If your website has multilingual content (e.g. /en/about and /de/about), it is strongly recommended to index only one language. Indexing the same content in multiple languages leads to:

• Duplicate information consuming your content quota (measured in characters) unnecessarily
• Lower retrieval quality as the AI may pull answers from the wrong language
• Wasted token budget on redundant content

The chatbot replies in the visitor's language regardless of which language you index, so there's no benefit to indexing the same page twice. Use the Exclude URL Patterns field to skip the other language folders (e.g. /de/*, /fr/*).

Choose the language your customers most frequently use, or the language that matches your chatbot's primary audience.

Text Data Sources

You can manually enter text content as a data source. This is ideal for FAQs, internal knowledge, or content that doesn't exist on a website.

Adding a Text Document

1. Click Add Data Source in the Knowledge tab and choose Text Input
2. Fill in the fields:

File Upload

Upload documents directly to your chatbot's knowledge base.

Supported Formats

File Size Limits

AI Auto-Filename

On paid plans, if you upload a file with a generic name (e.g. document.txt or untitled.pdf), the system automatically generates a descriptive filename based on the file content.

Confluence Cloud

Connect your Confluence Cloud instance to import wiki pages directly into your chatbot's knowledge base.

Adding a Confluence Source

1. Click Add Data Source and select Confluence Cloud
2. Enter your Confluence URL (e.g. https://yourcompany.atlassian.net)
3. Enter the email associated with your Atlassian account
4. Enter an API token from Atlassian API Tokens
5. Click Test Connection — available spaces will be loaded
6. Select the spaces to index (leave empty to import all accessible spaces)
7. Configure the sync interval and click Create & Index

Editing a Confluence Source

After adding a Confluence source, click the edit icon to change:

• Space selection — Add or remove spaces to index
• Include child pages — Whether to include nested pages
• Re-indexing interval — How often to automatically sync

Changes take effect on the next re-index.

Incremental Sync

Confluence sources support incremental sync. Only pages that have changed since the last sync are re-indexed, making updates fast and efficient.

Notion

Connect Notion to import pages and databases into your chatbot's knowledge base.

Adding a Notion Source

1. Click Add Data Source and select Notion
2. Create an Internal Integration at Notion Integrations
3. Share the desired pages or databases with your integration in Notion
4. Enter the integration token (starts with ntn_ or secret_)
5. Click Test Connection — available databases will be loaded
6. Select the databases to index (leave empty to import all accessible pages)
7. Configure the sync interval and click Create & Index

Editing a Notion Source

After adding a Notion source, click the edit icon to change:

• Database selection — Add or remove databases to index
• Include child pages — Whether to include nested pages
• Re-indexing interval — How often to automatically sync

Incremental Sync

Notion sources support incremental sync. Only pages modified since the last sync are re-processed.

Connected folders

A connected folder is a folder that stays connected. You point the chatbot at a folder in SharePoint, Google Drive, Nextcloud or on an SFTP server once; from then on it reads every indexable file in it and checks back on a schedule you choose. Change a PDF in that folder and the chatbot knows the new version after the next sync. Delete a file and it disappears from the knowledge base. Nobody has to re-upload anything.

What gets indexed

Skipped files are listed in the Indexed files dialog with the reason, so you can see at a glance why something is missing.

How it stays in sync

Each file carries a version marker from the storage system — SharePoint and Google Drive supply a content fingerprint, SFTP and Nextcloud use size plus modification time. On every sync the chatbot compares markers and only downloads what actually changed. An unchanged file costs nothing: no download, no processing, no quota.

Choose the sync interval when you connect the folder: daily, every 3 days, weekly, or every 30 days. You can also trigger a sync by hand at any time with the refresh icon.

Telling users where a file lives

Off by default, a switch per folder: Name the storage location to users. With it on, the chatbot may mention the folder path in an answer, and each answer shows a card per file it used — file name, path, storage system, and a link straight into SharePoint, Drive or Nextcloud where one exists.

Leave it off for a public website widget. Otherwise every visitor can read your internal folder structure. With the switch off, neither the file name nor the path is ever sent to the AI model, so it cannot leak them into prose either.

Excluding files

Add one rule per line under Skip certain files:

* stands for any characters within one path segment, ** for any number of subfolders. Excluded files are removed from the knowledge base on the next sync.

You can also exclude a single file after the fact: open Indexed files, find the row, and click Remove from index. It stays excluded on later syncs.

SharePoint / OneDrive

Reads a document library from Microsoft 365. Your Microsoft administrator registers an app once and gives you three values.

Getting the credentials

1. Sign in to the Microsoft Entra admin center with an account that may register applications
2. Go to Identity → Applications → App registrations and click New registration
3. Give it a name (e.g. WebChatAgent Knowledge), leave the defaults, click Register
4. On the overview page, copy two values:
   • Directory (tenant) ID
   • Application (client) ID
5. Go to Certificates & secrets → New client secret, choose an expiry, click Add, and copy the Value column immediately — Microsoft shows it exactly once
6. Go to API permissions → Add a permission → Microsoft Graph → Application permissions and add:
   • Sites.Selected — recommended: access to nothing until an administrator explicitly grants a site, or
   • Sites.Read.All — read access to all SharePoint sites; simpler, considerably broader
7. Click Grant admin consent — without this the app has no access at all
8. For Sites.Selected, an administrator additionally has to grant the app read access to the specific site (via PowerShell or Graph Explorer, Sites.Selected permission on that site)

Connecting it

1. Add data source → SharePoint / OneDrive
2. Enter tenant ID, client ID and client secret
3. Optionally enter the site URL (e.g. https://contoso.sharepoint.com/sites/marketing) to jump straight to one site
4. Test connection — the document libraries you have access to are loaded
5. Pick the library, drill down to the folder you want
6. Choose the sync interval and click Connect and read in

Google Drive

Reads a Drive folder using a Google service account — a technical account that belongs to your organization, not to a person, so nothing breaks when an employee leaves.

Getting the credentials

1. Open the Google Cloud Console and select a project (or create one — it is free)
2. Go to APIs & Services → Library, search for Google Drive API, and click Enable
3. Go to APIs & Services → Credentials → Create credentials → Service account
4. Give it a name, click Create and continue, skip the optional role steps, click Done
5. Click the new service account, open the Keys tab, choose Add key → Create new key → JSON. A .json file downloads — that file is the credential
6. Copy the service account's e-mail address, it looks like name@project-id.iam.gserviceaccount.com

Now give that address access to your files, in one of two ways:

• Share the folder (simplest): in Google Drive, right-click the folder → Share → paste the service account address → role Viewer → Share
• Domain-wide delegation (Google Workspace only): a Workspace administrator authorizes the service account's client ID for the scope https://www.googleapis.com/auth/drive.readonly under Admin console → Security → API controls → Domain-wide delegation. Then enter the e-mail of the user whose files should be read in the Act as user field. Use this route when your Workspace blocks external sharing — a service account counts as external.

Connecting it

1. Add data source → Google Drive
2. Paste the contents of the JSON key file
3. Optionally fill Act as user (domain-wide delegation only)
4. Paste the folder link or its ID — open the folder in Drive and copy the URL; the ID is the part after /folders/
5. Test connection, choose the interval, click Connect and read in

Google Docs, Sheets and Slides carry no file bytes, so they are converted on the fly: Docs to Markdown, Sheets to Excel (first worksheet only), Slides to plain text. Google caps this conversion at 10 MB per file; anything larger is listed as skipped.

Nextcloud / WebDAV

Works with Nextcloud, ownCloud and any other WebDAV server — including the WebDAV interface many NAS boxes expose.

Getting the credentials

For Nextcloud and ownCloud, use an app password, never your account password:

1. Sign in to your Nextcloud
2. Click your avatar → Settings → Security
3. Scroll to Devices & sessions, enter a name (e.g. WebChatAgent), click Create new app password
4. Copy the generated password — it is shown once

An app password can be revoked individually and does not work for the web login, so a leak stays contained. For other WebDAV servers, use the username and password your administrator gave you.

Connecting it

1. Add data source → Nextcloud / WebDAV
2. Server URL — just the address, no path: https://cloud.example.com
3. Username and app password
4. WebDAV path — leave empty for Nextcloud and ownCloud, they use the standard path. Only fill this for other WebDAV servers (your administrator will know it, often something like /dav or /webdav)
5. Test connection, drill down to the folder, choose the interval, click Connect and read in

SFTP / FTPS

For your own server, a hosting package, or a NAS with SSH access.

What you need

Ask whoever runs the server for:

• Server address and port (SFTP usually 22, FTPS usually 21 or 990)
• Username
• Password or, better, an SSH key

An SSH key beats a password: it cannot be guessed, and you can revoke it without changing anyone else's login. If you have to create one yourself, on Mac or Linux run:

ssh-keygen -t ed25519 -f ~/wca-key -C "webchatagent"

This produces two files. Paste the contents of wca-key (the private key) into the dialog, and hand wca-key.pub (the public key) to your administrator, who appends it to ~/.ssh/authorized_keys for the account. If you set a passphrase, enter it in the passphrase field.

Connecting it

1. Add data source → SFTP / FTPS
2. Choose the protocol, enter server address, port, username
3. Either enter the password, or switch on Use SSH key instead of password and paste the private key
4. Test connection — with SFTP a server fingerprint appears
5. Compare that fingerprint with the one your administrator gave you and confirm it. It is stored and checked on every later sync; if it ever changes, the sync stops instead of quietly connecting to a different machine
6. Drill down to the folder, choose the interval, click Connect and read in

When "always read in again" helps

SFTP has no content fingerprint; the chatbot compares size and modification time. Some sync tools (rsync -t, certain backup jobs) preserve both when the content changed, and such a change would go unnoticed. For those cases there is a switch Always read files in again in the edit dialog. It re-reads every file on every sync — slower, but nothing gets missed. Leave it off unless you actually have this problem.

When something goes wrong

Categories

Categories help you organize your data sources. You can:

• Assign a category when adding or editing any data source
• Filter the data source list by category
• Use categories to group related content (e.g. "Products", "Support", "Legal")

Categories are stored in lowercase and are shared across all data source types.

Data Source List

The data source list shows each source with its current status:

You can filter the list by:

• Search — Find sources by name or URL. Connected folders match on their display name and on the folder path, so typing sharepoint or part of the path finds them
• Type — Filter by Website, File, Confluence, Notion, SharePoint, Google Drive, Nextcloud / WebDAV, or SFTP / FTPS
• Category — Filter by assigned category

Search content (the button above the list) searches the indexed text rather than the source names, and covers connected folders as well — useful for finding which file a particular sentence came from.

Editing Data Sources

• Website URLs: Click the edit icon to change URL, crawl depth, CSS selectors, exclude patterns, and re-index interval
• Text documents: Click the edit icon to change the name, category, and content
• File documents: Click the edit icon to change the name and category (content cannot be changed — re-upload the file instead)
• Confluence sources: Click the edit icon to change space selection, child page inclusion, and sync interval
• Notion sources: Click the edit icon to change database selection, child page inclusion, and sync interval
• Connected folders: Click the edit icon to change the display name, sync interval, subfolder inclusion, the storage-location switch, exclusion rules, search priority, and to re-enter credentials. The file contents themselves cannot be edited here — the storage system stays the single source of truth, and any edit would be overwritten on the next sync

Re-indexing

You can manually re-index individual data sources to update their content. This is useful when:

• Website content has changed
• You want to refresh after fixing CSS selectors or exclude patterns
• A previously errored source has been corrected

Re-indexing required (orange badge)

When a data source shows an orange "Re-indexing required" badge, its indexed content can no longer be used by the chatbot and has to be rebuilt.

This happens automatically when you change the chatbot's AI provider (for example from OpenAI to Google Gemini) in the chatbot settings. Each provider stores your content in its own embedding format, and these formats are not interchangeable. So when the provider changes, the old indexed data is cleared and every data source is flagged Re-indexing required.

When at least one source needs re-indexing, Re-import all becomes available at the top of the Knowledge tab. Click it to rebuild every affected source in one go, or use the refresh action on a single source to re-index just that one.

You don't need to re-add anything — re-indexing reuses your existing URLs, files, text, Confluence, Notion and connected folders. Websites are re-crawled and external sources re-synced, so a large knowledge base can take a few minutes. Once it finishes, the badge clears and the chatbot answers from your content again.

Content Limits

Training content is measured in characters of indexed text across all your data sources (web pages, files, Q&A entries) — this replaced the former per-page limit, so a few long pages and many short ones count the same way. Your character allowance grows with your plan; for the current per-plan limits, see the pricing page.

When the limit is reached, indexing pauses until you remove content or upgrade. After an upgrade, the interrupted sources resume indexing by themselves; there is nothing to click.

Best Practices

• Keep content clear and well-structured — The AI performs better with organized, readable content
• Index only relevant pages — Exclude login pages, admin areas, and duplicate content
• Use CSS selectors to focus on main content and exclude navigation, footers, and ads
• Monitor the Questions dashboard to identify knowledge gaps and add missing content
• Update data sources regularly to ensure accurate, current answers
• Use categories to organize large numbers of data sources
• Limit to one language when your site has identical content in multiple languages
• Point connected folders at a curated folder, not at the entire drive — a "Chatbot knowledge" folder someone maintains beats indexing every draft and old contract lying around
• Keep the storage-location switch off for public widgets, on for internal assistants and team wikis