Overview
Overview of Data Sources, RAG, and source connectivity
Data Sources (RAG)
Data sources are the foundation for retrieval-augmented generation (RAG) in meinGPT. Content from connected sources is indexed and made available to your assistants as knowledge. You can also attach a data source directly to a chat when you only want to look something up once.
How search works
Customer knowledge bases are often large. Hundreds or thousands of gigabytes of Word, PDF, and other files are not unusual. Sending every search query through every file would be far too slow. That is why a search index is built up front. It works similar to Google, just for your internal documents.
The initial indexing can take hours to days, depending on the data volume. Word, PDF, and similar formats are binary, so the text has to be extracted first. This one-time effort pays off in fast search results afterwards.
You do not have to wait for all of it. The index is built in two stages: keyword search first, semantic search second. A file is therefore searchable as soon as its keyword index is ready — semantic search follows in the background, file by file. You can see how far each stage has got under Indexing progress.
Search characteristics:
- Keyword search: Finds exactly the terms you type — case numbers, article numbers, proper names. It is built first and is always available.
- Semantic: Documents are converted into mathematical representations (called embeddings) that capture meaning, not just individual words. Can be turned off per data source.
- Sorted by relevance: Hits are ranked by content fit, not by frequency of a search term. A document that thematically matches the question can rank higher than one with the exact keyword.
- Number of results: By default, the ten most relevant sources are returned. The number is configurable in the settings.
- Filename search: Besides content, you can also search specifically by filename, for example "Show me file XY".
Attention
No filtering by custom metadata fields. Data sources search file contents (semantically) and file names. You can additionally filter by built-in file properties: path, extension, MIME type, and created/modified dates. Custom metadata fields from source systems are not carried over as separate, filterable fields — at best they exist as plain text inside the document content, so they can only be found via semantic search, not filtered on reliably.
Search settings per data pool
Each data pool has its own Search settings panel (open the data pool → Search settings) with four controls:
- Semantic search — additionally finds what matches in meaning but is worded differently. On by default. Turning it off is worth it when you search for exact terms anyway and the corpus is large: indexing then finishes far sooner and costs less.
- Number of Search Results — how many chunks are passed to the model per search (default 10).
- Enable Re-Ranking — adds a second relevance-ranking pass on top of the initial semantic search.
- Search agent (experimental) — routes searches for this pool through a dedicated retrieval subagent that runs several internal queries, broadening or narrowing them as needed, instead of a single direct search call.
These settings apply per data pool, not per assistant — they take effect the same way whether the pool is used directly in a chat or through an assistant.
Note
Turning semantic search off later deletes nothing. It only stops new files from being embedded — whatever is already embedded stays searchable. Turning it back on builds semantic search up gradually from the next sync run.
Indexing progress
At the top of a data source's detail page, Indexing shows one bar per stage:
- Keyword search — how many files are already searchable. Once the first files appear here, the data source is usable.
- Semantic search — how far the embeddings have caught up. If semantic search is off for this source, that is stated here instead of a bar.
Below that you can see how many files did not make it into the index: those that failed, and those skipped because the file type is not supported.
If the platform cannot query the progress right now — for example because an Outpost is offline — the section is hidden rather than showing zeroes.
Cloud data source at a glance
For most teams, the cloud-based data source is the right choice.
| Property | Value |
|---|---|
| Hosting | meinGPT Cloud (Hetzner, Germany) |
| Sync interval | Every 15 minutes |
| Search results per query | Default 10, configurable |
| Search method | Keyword search and semantic search (embeddings, sorted by relevance); semantic search can be turned off per data source |
| Availability | Included in the standard package |
Connection status
Each data source shows a status dot for its current connection state:
| Status | Meaning |
|---|---|
| 🟢 Ready | Connected and syncing regularly. |
| 🟠 Not ready | The connection exists, but the last sync failed or was incomplete. |
| 🔴 Unreachable | The source could not be reached during the last check. |
| ⚪ Unknown | Status has not been determined yet (e.g. right after creation). |
Common causes of Unreachable for a single data source:
- The source's access token (e.g. SharePoint, Google Drive, Confluence) has expired or was revoked
- Permissions were revoked in the source system
- Network or firewall changes on the source system's side
Attention
If all of your data sources show "Unreachable" at the same time — including sources with completely different underlying systems (e.g. SharePoint, Google Drive, and Confluence simultaneously) — this usually does not point to a problem with the individual sources, but to a temporary platform-side issue. In that case, contact meinGPT support instead of reconnecting individual sources.
Monitor sync status and fix failures
Every data pool syncs automatically every 15 minutes. If a sync fails, you can see it and fix it directly in meinGPT.
Automatic alert email
Admins automatically get an email when there's a problem with a data pool:
- Sync failing – a data pool has failed to sync several times in a row. New or changed documents stop being indexed until this is fixed.
- Data source connection lost – an OAuth connection (e.g. SharePoint) repeatedly fails to refresh its access token.
- Spike in sync errors – an unusually large share of files errored out during a sync.
Each email links directly to the affected data source in your settings. You can toggle these notifications individually under Settings → Notifications → Data vault alerts.
Check sync status
- Open Settings → Data sources
- Click the affected data pool
- The Sync Logs tab shows recent sync runs, including per-file error messages
Manually trigger a resync
- Open the affected data pool under Settings → Data sources
- Click Re-Synchronize to start a new run immediately
Every sync run – whether automatic or manually triggered – also removes entries for files that have been deleted from the linked source. A manual resync is therefore the right way to clear out stale index entries without having to recreate the data source. In rare cases, a single deletion may be missed during the sync and will still be present after a resync – if that happens, reach out to meinGPT support so we can look at the affected data source directly.
Re-authenticate the connection (OAuth sources)
- Open the affected data pool under Settings → Data sources
- Click Reconnect and authenticate again
- Your existing source settings stay unchanged — only the authorization is refreshed
If that doesn't help, contact support@meingpt.com.
File formats
Well supported are all formats that primarily consist of text:
- Office documents: DOCX, PPTX, XLSX (with caveats, see below)
- Email files: EML and MSG (see "Emails and attachments" below)
- TXT, Markdown, HTML
- Code files
Attention
Excel tables are a special case. When splitting documents into searchable chunks (a process called chunking), the table context gets lost. A single data row without its column headers often no longer makes sense. For calculations, analyses, and visualizations from Excel files, use the Code Sandbox instead. It processes the original file directly.
OneNote workaround. OneNote files are currently not indexed directly because the format is proprietary. Workaround: export OneNote content regularly via Make or n8n, as PDF or text. The exported files can then be connected like any other source.
Emails and attachments
You can place email files in EML (open standard) and MSG (Outlook) format directly into a data pool. The subject, sender, recipients, and message body are extracted and made searchable. Both formats are supported equally - you do not need to convert MSG to EML beforehand.
Attention
Attachments inside email files are a special case. For smaller emails, the content of attachments (e.g. a PDF) is read and indexed. For larger emails, attachments are reduced to their filename for performance reasons - their content is then not searchable. If you want a PDF attachment indexed as reliably as a PDF that sits directly in the data pool, also add the attachment as its own file to the data pool.
Access control
Data sources can be restricted to specific teams. You create teams in the admin interface and assign them to specific data sources. This lets you control which user groups see which data.
Details on creating and managing teams: Team management.
How meinGPT works with files (3-stage model)
Not every request needs the same processing depth. meinGPT decides per request how deep it has to go. There are three stages:
| Stage | What happens | Sufficient for |
|---|---|---|
| 1. Search | The platform searches all configured sources and returns snippets and filenames | Simple questions like "Is there a document about topic X?" |
| 2. Full-text retrieval | The model loads the complete content of individual files that look relevant after stage 1 | Content questions about individual, not overly large documents |
| 3. Code Sandbox | The original file is opened in an isolated environment (the sandbox) and processed with Python | Calculations, analyses, charts from large or structured files (e.g. Excel with many rows) |
You do not have to configure anything manually. More about the sandbox: Code Sandbox.
Attention
Full-text retrieval can use a lot of context. Stage 2 loads the complete content of a file or Confluence page into the chat at once – not just the relevant excerpt. When several large sources are connected to the same assistant or chat (for example a data source and a Confluence space), a single question can already consume a significant share of the model's context window. Keep an eye on the context window indicator in the chat input, and prefer narrow, targeted questions for very large or heavily nested pages (e.g. Confluence pages with many embedded macros). More on this: Chat too large for the model.
Note
Note for on-premise setups: Stage 3 (sandbox) temporarily uploads original files into the meinGPT Cloud, because the sandbox environments run there. The files are deleted immediately after processing. For privacy-sensitive setups, communicate this transparently to your stakeholders.
SharePoint Connector vs. data source: which one when?
If you want to use SharePoint data in meinGPT, you have two options: the native Microsoft 365 Connector, or a data source with SharePoint as a source. Both have their strengths.
| Criterion | Microsoft 365 Connector | Data source with SharePoint source |
|---|---|---|
| Search method | Direct access in real time | Pre-built index, sync every 15 minutes |
| Permissions | Respects SharePoint permissions automatically (at user level via OAuth) | Admin configures manually. SharePoint permissions do not apply automatically |
| Authentication | Each user authenticates individually | Centrally configured |
| Scaling | Good for targeted research in single sites or folders | Scales to large data volumes, multiple sources combinable |
| Combining sources | Only SharePoint and OneDrive | Multiple sources in one data source (SharePoint, local files, Drive, …) |
Rule of thumb:
- Microsoft 365 Connector for most SharePoint use cases. Especially when each user should only see what they are allowed to see in SharePoint itself.
- Data source when you want to index large stocks centrally, mix multiple sources, or build a central knowledge base without individual permissions.
Configuration & recommendations
Narrow the data scope
There is no hard data limit. But the more data a data source covers, the more irrelevant hits compete with the relevant ones. Recommendation:
- Connect 500 to 1,000 relevant files per data source in a targeted way, not the entire SharePoint
- Prefer multiple specialized data sources over one huge one, for example "HR policies", "Product specifications", "Sales material"
- The more focused a data source, the better the results
Write the short description carefully
The short description of a data source is not just documentation. The model uses it to decide whether a data source is relevant for a given request. A poor description leads to data sources not being searched even when they should be.
Good: "Contains all internal HR policies, process descriptions, and onboarding documents."
Less good: "HR documents."
Reference the tool in the system prompt
In assistant instructions, it pays to reference data sources explicitly. For example: "Start every conversation by retrieving relevant information from the connected data source." This makes data source usage more reliable.
Symptom: the assistant ignores a linked data pool. If an assistant answers from general knowledge instead of the data pool even though the pool is correctly synced and linked, a missing reference in the system prompt is the most common cause — not a misconfiguration of the data pool itself. Check first whether the instructions explicitly mention the data pool before investigating sync status or permissions.
Advanced: Customer-Managed Outpost (On-Premise)
If you want to run your own on-premise knowledge infrastructure, for example for regulated industries or special security requirements, you can deploy your own Outpost. Data does not leave your network in this setup. The exception is the temporary sandbox processing, see above.
- Choose network model: On-Premise Connections
- Outpost operations and configuration: Manual will follow
Sources
All supported sources are listed here:
Typical sources:
- SharePoint and OneDrive
- Google Drive
- Confluence
- Amazon S3
- WebDAV
- SMB, IMAP mailboxes, and local filesystems (Enterprise Outpost only — configured via the Outpost, not the meinGPT UI)
Custom Data Preparation Pipelines
For the dedicated pattern with S3 handover for third-party systems, see:
Related pages
- Code Sandbox: stage 3 of file processing
- Code Sandbox: structured table analysis
- Microsoft 365 Connector: direct SharePoint, Outlook, and Teams access
- Team management: access control per data source
- Data Sources: full list of source types