Web Crawler
Crawl public or internal websites directly in MeinGPT and make them searchable
Set up in MeinGPT
Open Settings → Data sources, click Add data source, and choose Web Crawler. Enter the Start URL and click Connect — every other field is optional and has a sensible default.
The web crawler runs as a Cloud data source; it needs no Outpost of your own, and this source type is not currently supported for that.
Basic settings
| Field | Default | Description |
|---|---|---|
| Start URL | — | The URL the crawler starts from |
| Scraping Method | Basic (HTTP) | Basic uses HTTP requests, Browser enables JavaScript rendering — needed for pages that load content via JS |
| Max Depth | 3 | How many clicks deep the crawler follows from the start URL |
| Max Pages | 1000 | Upper bound on the number of pages crawled |
Login (optional)
For password-protected intranets: the crawler signs in once and stays signed in for the whole crawl.
| Field | Description |
|---|---|
| Login URL | Target URL of the login form — must use HTTPS |
| Username / Password | Credentials for the login form |
Advanced settings
| Field | Default | Description |
|---|---|---|
| Include / Exclude Paths | — | Path or full URL wildcard patterns, one per line |
| Wait for Selector | — | Browser mode only: CSS selector to wait for before extraction |
| Page Timeout | 30s | Timeout per page |
| Request Delay | 1.0s | Wait time between requests |
| Concurrent Requests | 5 | Parallel crawl requests |
| Retry Attempts | 2 | Retries on failure |
| Output Format | Markdown | Markdown, HTML, or Text |
| Main Content Only | On | Extracts only the main content area, without navigation and footer |
| Max Age | 24h | Pages older than this threshold get re-crawled |
| Proxy Server / Username / Password | — | For crawls routed through a proxy |
| User Agent | — | Custom User-Agent string |
Path filters
Use /docs/* to include pages below /docs/, or /docs/private/* to exclude a subsection. * matches any text, including slashes; ? matches one character. Exclusions take precedence.
Patterns beginning with / match only the URL path, without query parameters. Full URL patterns such as https://example.com/docs/* are also supported. Regular expressions are not supported; the simple legacy spelling /docs/.* remains compatible. Invalid rules are rejected instead of being ignored.
The start page may be fetched to discover matching links, but its content is indexed only if it matches the filters. Filters apply to subsequent crawls; they do not automatically remove content that was already indexed outside the configured scope.
Synchronization
- MeinGPT crawls from the start URL and stores extracted content per discovered page.
- Every data pool is checked by the sync scheduler every 15 minutes (see Sync interval); a page is only actually re-crawled once the configured max age (default 24 hours) has passed since it was last crawled.
- Use path filters to keep crawl scope tight.