How search works in data pools, release levels, citations, and the 3-stage model
How MeinGPT searches a data pool, what shows up in an answer, and how deep the AI goes into an individual file when needed. Setup and configuration of a data source: Data Sources & RAG.
Customer knowledge bases are often large. Hundreds or thousands of gigabytes of Word, PDF, and other files are not unusual. Sending every search query through every file would be far too slow. That is why a search index is built up front. It works similar to Google, just for your internal documents.
The initial indexing can take hours to days, depending on the data volume. Word, PDF, and similar formats are binary, so the text has to be extracted first. This one-time effort pays off in fast search results afterwards.
You do not have to wait for all of it. The index is built in two stages: keyword search first, semantic search second. A file is therefore searchable as soon as its keyword index is ready — semantic search follows in the background, file by file. You can see how far each stage has got under Indexing progress.
Search characteristics:
Keyword search: Finds exactly the terms you type — case numbers, article numbers, proper names. It is built first and is always available.
Semantic: Documents are converted into mathematical representations (called embeddings) that capture meaning, not just individual words. Can be turned off per data source.
Sorted by relevance: Hits are ranked by content fit, not by frequency of a search term. A document that thematically matches the question can rank higher than one with the exact keyword.
Number of results: By default, the ten most relevant sources are returned. The number is configurable in the settings.
Filename search: Besides content, you can also search specifically by filename, for example "Show me file XY".
Attention
No filtering by custom metadata fields. Data sources search file contents (semantically) and file names. You can additionally filter by built-in file properties: path, extension, MIME type, and created/modified dates. Custom metadata fields from source systems are not carried over as separate, filterable fields — at best they exist as plain text inside the document content, so they can only be found via semantic search, not filtered on reliably.
Each data pool has four settings that decide how it is searched (open the data pool → Overview tab):
Semantic search — additionally finds what matches in meaning but is worded differently. On by default. Turning it off is worth it when you search for exact terms anyway and the corpus is large: indexing then finishes far sooner and costs less.
Passages per answer — how many passages may go into a single answer (default 10, range 1 to 100).
Deep search(experimental) — hands retrieval to a dedicated search agent that runs several queries, broadening or narrowing them as needed, instead of a single direct search call. Cloud data pools only, and additionally requires deep search to be enabled for your organization.
Improve result order — currently has no effect. The switch stores a value that is not evaluated in any search path. Leave it off.
These settings apply per data pool, not per assistant — they take effect the same way whether the pool is used directly in a chat or through an assistant.
Turning semantic search off later deletes nothing. It only stops new files from being embedded — whatever is already embedded stays searchable. Turning it back on builds semantic search up gradually from the next sync run.
At the top of a data source's detail page, Indexing shows one bar per stage:
Keyword search — how many files are already searchable. Once the first files appear here, the data source is usable.
Semantic search — how far the embeddings have caught up. If semantic search is off for this source, that is stated here instead of a bar.
Below that you can see how many files did not make it into the index: failed and skipped ones, each with its reason — such as "no text in the file", "password protected" or "too large for the configured limits". Files whose extension is not on the list of file formats are not counted here: the sync never picks them up in the first place.
For a cloud data source, this area also shows four figures: File upload (how many files the CloudVault has taken in), Searchable (how many files are currently searchable), Chunks (how many indexed text passages exist), and Last sync (the time and status of the most recent sync run). These four numbers come from two status checks against the CloudVault, not four independent ones: File upload and Chunks come from one check, Searchable and Last sync from the other. If a check fails, both numbers it covers show "Unknown" together. If all four show "Unknown" at the same time, the same underlying cause as the "Indexing status unavailable" message below has usually affected the whole area.
If the platform cannot query the progress right now, the section shows the message "Indexing status unavailable" instead — for a cloud data source (CloudVault) because the CloudVault did not answer this status check, for an Outpost data source because the Outpost is offline or unreachable.
Right after creating a source, this is normal
This message can appear before you have uploaded or synced a single file. It only means this one status check failed — not that anything is blocked. Uploading, connecting, and any indexing run already in progress are unaffected and keep working. If it does not clear on its own within a few minutes, contact MeinGPT support.
Answers from a data source include numbered citations as clickable badges next to the text. Clicking one opens a preview with the title, location, and an excerpt of the original document - and, depending on the data source's release level, an option to open or download the original file.
Attention
Citation isn't clickable? That is not expected behavior. Contact support@meingpt.com with a link to the affected chat so we can investigate.
How much is available behind a citation is decided by a setting on the data source — not on the assistant. You find it under Settings → Data sources: click the source, open the Admin tab, setting "What employees get from this source".
Level
What leaves the source
Matching passages only
The matching passages and the file name
Text only
Plus the full text of the documents found
Original files
Plus the file itself
Only Original files unlocks opening and downloading the original file from a citation — and lets the assistant load an original file into the chat or the Code Sandbox when needed.
A cloud data source inherits the Cloud Vault's level by default; the selector shows which one that currently is (at present Original files). You only need a level of its own when one source should release less than the rest. For an Outpost data source the level is set exclusively in the Outpost app, where Text only is the preset; the platform only displays the reported value.
Note
A citation opens the copy stored in MeinGPT, not a deep link into the source system. Jumping straight to the document in SharePoint or Google Drive is not part of a citation from a data pool.
Every cloud data source has its own Knowledge Map tab (open the data source → Knowledge Map). It shows how the documents in this pool are actually used - not manually curated, but computed from real searches over the last 30 days:
Folder activity: The folders the assistant returns to most often.
Knowledge map graph: A network of the documents most frequently retrieved together. Clicking a document shows its retrieval count and connected documents.
Note
The tab needs prior search activity. Until this pool has been searched, the tab shows a message that knowledge is being indexed - this is not an error. The first snapshot usually appears within a few minutes of a search happening against this pool.
Not every request needs the same processing depth. MeinGPT decides per request how deep it has to go. There are three stages:
Stage
What happens
Sufficient for
1. Search
The platform searches all configured sources and returns snippets and filenames
Simple questions like "Is there a document about topic X?"
2. Full-text retrieval
The model loads the complete content of individual files that look relevant after stage 1
Content questions about individual, not overly large documents
3. Code Sandbox
The original file is opened in an isolated environment (the sandbox) and processed with Python
Calculations, analyses, charts from large or structured files (e.g. Excel with many rows)
You do not have to configure anything manually. More about the sandbox: Code Sandbox.
Attention
Full-text retrieval can use a lot of context. Stage 2 loads the complete content of a file or Confluence page into the chat at once – not just the relevant excerpt. When several large sources are connected to the same assistant or chat (for example a data source and a Confluence space), a single question can already consume a significant share of the model's context window. Keep an eye on the context window indicator in the chat input, and prefer narrow, targeted questions for very large or heavily nested pages (e.g. Confluence pages with many embedded macros). More on this: Chat too large for the model.
On-premise data sources carry one extra restriction at stage 3:
Note
Note for on-premise setups: Stage 3 (sandbox) temporarily uploads original files into the MeinGPT Cloud, because the sandbox environments run there — but only when that Outpost folder is set to Text and original files (ORIGINAL_FILES). With Text only (FULL_TEXT) or the legacy PASSAGES level, the sandbox request is rejected before an original file is uploaded. Permitted files are deleted immediately after processing. For privacy-sensitive setups, communicate this transparently to your stakeholders.
Separately from who may use a data source at all (Access control), the release level determines which form of the content that source provides - for Cloud data sources as well as Outpost ones. Open the data source under Settings → Data sources → What employees get from this source:
Text only - only text leaves this source: the content of the relevant documents, not the files themselves. Citations in chat answers still show the title, location and excerpt - but no option to open or download the original file appears.
Original files - the file itself, in addition to the text. Users can open or download it from a citation.
A Cloud data source inherits its level from the associated Cloud Vault by default; each data source can override that. For an Outpost data source this field is read-only - the level is set only in the Outpost app, on the folder itself (Outpost).
Attention
The release level applies equally to everyone with access to this source. There is currently no way to differentiate it by role or user group - for example, releasing original files to one group while limiting another to text only. Anyone with access to the data source gets the same release level as everyone else with access.