Knowledge Base

​

Overview

​

How It Works

​

Vector Search Technology

1. Content Ingestion
   • You add a URL or document
   • Content is fetched and cleaned
   • Text is split into chunks (~500 tokens each)
2. Embedding Generation
   • Each chunk is converted to a vector embedding
   • Embeddings capture semantic meaning
   • Stored in PostgreSQL with pgvector
3. Runtime Retrieval
   • When a caller asks a question
   • Question is converted to an embedding
   • Similar chunks are found via cosine similarity
   • Top 3-5 most relevant chunks are retrieved
4. Context Injection
   • Retrieved chunks are added to the agent’s context
   • Agent uses this information to answer accurately
   • Responses cite actual content from your knowledge base

​

Why RAG?

“What are your business hours?” “I don’t have that information.” ❌

“What are your business hours?” “We’re open Monday through Friday, 9 AM to 5 PM, and Saturday from 10 AM to 2 PM.” ✓

​

Adding Knowledge Sources

​

From URL

​

Crawl Modes

​

Single Page Mode

• Fetches and processes one URL immediately
• Fastest option - results in seconds
• Good for FAQs, about pages, individual articles

​

Sitemap Mode

• Automatically discovers sitemap.xml
• Crawls all pages listed in the sitemap
• Handles sitemap indexes (nested sitemaps)
• Respects robots.txt rules
• Progress tracked in real-time

​

Recursive Mode

• Starts from your URL and follows internal links
• Configure crawl depth (1-5 levels)
• Configure max pages (up to 500)
• Only follows same-domain links
• Skips login pages, file downloads, and non-content URLs

​

Paste Content

1. Click + Add Source
2. Select the Paste Content tab
3. Enter a title for the content
4. Paste your text (minimum 50 characters)
5. Click Add Source

​

Upload File

1. Click + Add Source
2. Select the Upload File tab
3. Enter a title for the file
4. Click Choose File or drag and drop
5. Click Upload

​

Protected Pages

1. Check “Requires authentication”
2. Select “Bearer Token”
3. Enter your token: your-api-key-here

1. Check “Requires authentication”
2. Select “Basic Auth”
3. Enter username and password

​

Supported Content Types

​

Managing Sources

​

Source Dashboard

​

Source Statuses

• Syncing - Content is being fetched and processed
• Completed - Successfully processed and indexed
• Failed - Error occurred (click to see details)

​

Editing Sources

• Change the sync schedule
• Update authentication credentials
• Force an immediate re-sync
• View processing logs

​

Deleting Sources

1. Click the source to expand
2. Click Delete Source
3. Confirm deletion

​

Crawl Progress Tracking

• Progress bar shows percentage complete
• Pages crawled / Pages discovered counters
• Current URL being processed
• Failed pages count (with details)

​

Viewing Parsed URLs

1. Click the source to expand
2. Click View Parsed URLs
3. See all crawled pages with:
   • Page title
   • URL
   • Chunk count per page

​

Content Best Practices

​

What to Add

​

What NOT to Add

• Entire websites - Too much noise, dilutes relevance
• Login-protected portals - Customer-specific data
• Frequently changing content - Will become stale
• Competitor information - Could confuse the agent
• Internal documents - Security risk

​

Content Quality Tips

1. Keep it factual - Agents repeat what’s in the knowledge base
2. Use clear language - Avoid jargon unless your customers use it
3. Structure with headings - Helps chunking algorithm
4. Include common variations - “hours” and “business hours” both work
5. Update regularly - Re-sync when your content changes

​

Automatic Sync

​

How Sync Works

• At the scheduled interval, Magpipe re-fetches your URL
• Content is compared to existing version
• If changed, new chunks are generated
• Old chunks are replaced atomically
• Agent immediately uses new content

​

Sync Schedules

​

Manual Re-sync

1. Click the knowledge source
2. Click Sync Now
3. Wait for processing to complete

​

Agent Integration

​

Per-Agent Knowledge

• All knowledge sources you’ve added
• Relevant chunks are retrieved per-question
• No configuration needed - automatic

​

Prompt Integration

[System Prompt - your configuration]

[Knowledge Context]
The following information was retrieved from the knowledge base:

--- From: FAQ Page ---
Q: What are your business hours?
A: We're open Monday-Friday 9 AM to 5 PM, Saturday 10 AM to 2 PM.

--- From: Services Page ---
Our basic plan starts at $29/month and includes...
[End Knowledge Context]

[User Question]
What time do you open on Saturdays?

​

Knowledge Limitations

• 3-5 chunks retrieved per question (most relevant)
• ~2000 tokens max knowledge context per response
• Chunks prioritized by relevance score
• Less relevant chunks truncated if limit exceeded

​

Monitoring & Analytics

​

Knowledge Usage

• Which sources are accessed most
• Common questions by topic
• Retrieval success rate
• Gaps in knowledge (unanswered questions)

​

Retrieval Logs

1. Open a conversation in Inbox
2. View the transcript
3. See which knowledge chunks were retrieved

​

Limits

​

Troubleshooting

​

Source Shows “Failed”

• URL is not accessible
• Page requires JavaScript to render
• Authentication credentials incorrect
• Content exceeds size limit

​

JavaScript-Rendered Pages

1. Direct fetch (fastest)
2. Firecrawl API (JavaScript rendering)
3. Jina Reader API (fallback)
4. Microlink API (final fallback)

• Behind a login wall
• Loaded dynamically after page load
• Protected by bot detection

​

Agent Not Using Knowledge

1. Source status is “completed”
2. Content was successfully chunked (count > 0)
3. Question is related to the content
4. Try asking a direct question from the content

​

Outdated Information

1. Check when source last synced
2. Force a manual re-sync
3. Verify the source URL shows current content