Open Settings → Data sources, click Add data source, and choose Web Crawler. Enter the Start URL and click Connect — every other field is optional and has a sensible default.
The web crawler runs as a Cloud data source; it needs no Outpost of your own, and this source type is not currently supported for that.
| Field | Default | Description |
|---|
| Start URL | — | The URL the crawler starts from |
| Scraping Method | Basic (HTTP) | Basic uses HTTP requests, Browser enables JavaScript rendering — needed for pages that load content via JS |
| Max Depth | 3 | How many clicks deep the crawler follows from the start URL |
| Max Pages | 1000 | Upper bound on the number of pages crawled |
For password-protected intranets: the crawler signs in once and stays signed in for the whole crawl.
| Field | Description |
|---|
| Login URL | Target URL of the login form — must use HTTPS |
| Username / Password | Credentials for the login form |
| Field | Default | Description |
|---|
| Include / Exclude Paths | — | URL path regex patterns, one per line |
| Wait for Selector | — | Browser mode only: CSS selector to wait for before extraction |
| Page Timeout | 30s | Timeout per page |
| Request Delay | 1.0s | Wait time between requests |
| Concurrent Requests | 5 | Parallel crawl requests |
| Retry Attempts | 2 | Retries on failure |
| Output Format | Markdown | Markdown, HTML, or Text |
| Main Content Only | On | Extracts only the main content area, without navigation and footer |
| Max Age | 24h | Pages older than this threshold get re-crawled |
| Proxy Server / Username / Password | — | For crawls routed through a proxy |
| User Agent | — | Custom User-Agent string |
- meinGPT crawls from the start URL and stores extracted content per discovered page.
- Every data pool is checked by the sync scheduler every 15 minutes (see Sync interval); a page is only actually re-crawled once the configured max age (default 24 hours) has passed since it was last crawled.
- Use path filters to keep crawl scope tight.