URL Search Tool
The URL Search tool enables real-time website crawling during conversations. The chatbot can fetch and search specific URLs on-the-fly, which is useful for dynamic or frequently updated content that changes too often for standard indexing.
URL Search is available on paid plans only.
Configuration
| Field | Description | Default |
|---|---|---|
| Enable URL Search | Activates real-time URL searching for this chatbot. | Off |
URL Groups
URLs are organized into groups. Each group defines a set of URLs and a prompt that tells the chatbot when to search them.
You can create up to 10 URL groups per chatbot.
Group Fields
| Field | Description |
|---|---|
| Group Name | A descriptive name for this URL group (e.g. "Weather", "Stock Prices", "Support Articles"). |
| Prompt | Instructions for when the chatbot should search these URLs (e.g. "Search these URLs when the user asks about current weather conditions"). |
| URLs | One URL per line. Only http:// and https:// URLs are accepted. Up to 10 URLs per group. |
Example Configuration
Group: Product Prices
- Name: Product Prices
- Prompt: Search these URLs when the visitor asks about current prices, pricing, or product costs.
- URLs:
https://shop.example.com/pricing https://shop.example.com/products
Group: Support Articles
- Name: Support Articles
- Prompt: Search these URLs when the visitor has a technical issue or asks about troubleshooting.
- URLs:
https://help.example.com/faq https://help.example.com/troubleshooting
How It Works
- The visitor asks a question
- The chatbot evaluates the question against each group's prompt
- If a group matches, the chatbot fetches the URLs in real-time
- The fetched content is used to generate the answer
- The visitor gets an answer based on the latest content
URL Search vs. Standard Data Sources
| Feature | URL Search | Standard Data Sources |
|---|---|---|
| Content freshness | Always current (fetched in real-time) | Updated on re-index |
| Response speed | Slightly slower (needs to fetch) | Fast (pre-indexed) |
| Best for | Dynamic, frequently changing content | Static or rarely changing content |
| Cost | Uses more tokens per query | Pre-processed, lower per-query cost |
Best Practices
- Use for dynamic content only — Static content is better served by standard data sources
- Write clear prompts — The prompt determines when the chatbot uses each URL group
- Limit URLs per group — Fewer URLs means faster responses
- Combine with data sources — Use URL Search for dynamic data and standard sources for static content