---
title: "Web Crawler"
description: "Crawl public or internal websites directly in meinGPT and make them searchable"
canonical_url: "https://meingpt.com/en/docs/integrations/sources/webcrawler"
language: en
---

# Web Crawler

## Set up in meinGPT

Open **Settings → Data sources**, click **Add data source**, and choose Web Crawler. Enter the **Start URL** and click **Connect** — every other field is optional and has a sensible default.

The web crawler runs as a Cloud data source; it needs no Outpost of your own, and this source type is not currently supported for that.

## Basic settings

| Field | Default | Description |
| --- | --- | --- |
| **Start URL** | — | The URL the crawler starts from |
| **Scraping Method** | Basic (HTTP) | Basic uses HTTP requests, Browser enables JavaScript rendering — needed for pages that load content via JS |
| **Max Depth** | 3 | How many clicks deep the crawler follows from the start URL |
| **Max Pages** | 1000 | Upper bound on the number of pages crawled |

## Login (optional)

For password-protected intranets: the crawler signs in once and stays signed in for the whole crawl.

| Field | Description |
| --- | --- |
| **Login URL** | Target URL of the login form — must use HTTPS |
| **Username** / **Password** | Credentials for the login form |

## Advanced settings

| Field | Default | Description |
| --- | --- | --- |
| **Include / Exclude Paths** | — | URL path regex patterns, one per line |
| **Wait for Selector** | — | Browser mode only: CSS selector to wait for before extraction |
| **Page Timeout** | 30s | Timeout per page |
| **Request Delay** | 1.0s | Wait time between requests |
| **Concurrent Requests** | 5 | Parallel crawl requests |
| **Retry Attempts** | 2 | Retries on failure |
| **Output Format** | Markdown | Markdown, HTML, or Text |
| **Main Content Only** | On | Extracts only the main content area, without navigation and footer |
| **Max Age** | 24h | Pages older than this threshold get re-crawled |
| **Proxy Server / Username / Password** | — | For crawls routed through a proxy |
| **User Agent** | — | Custom User-Agent string |

## Synchronization

- meinGPT crawls from the start URL and stores extracted content per discovered page.
- Every data pool is checked by the sync scheduler every 15 minutes (see [Sync interval](/en/docs/integrations/data-pools-rag#cloud-data-source-at-a-glance)); a page is only actually re-crawled once the configured max age (default 24 hours) has passed since it was last crawled.
- Use path filters to keep crawl scope tight.