Stop feeding raw HTML to your RAG pipeline

nesting fourteen levels deep. Embed that and your retrieval quality is garbage before you've written a single prompt.

I kept rebuilding the same preprocessing stack for every project: fetch, strip boilerplate, convert to Markdown, chunk on heading boundaries, attach source metadata. So I turned it into an Apify actor and stopped rebuilding it.

What "RAG-ready" actually means

A record you can push into a vector store with zero further processing needs:

• Clean text — main content only. Readability-style extraction, not regex hacks.
• Markdown structure — headings survive, so chunk boundaries mean something.
• Sane chunks — split on heading/paragraph boundaries near a target token size, not mid-sentence at a character offset.
• Metadata on every chunk — source URL, page title, heading path, chunk index, crawl timestamp. Without this, citations are impossible and re-crawls are destructive.

The actor

Website to RAG Chunks does the whole path in one run: give it start URLs, it crawls same-domain pages (configurable depth and limits), extracts main content, converts to Markdown, chunks it, and emits one dataset record per chunk:

{
  "url": "https://docs.example.com/guide/auth",
  "title": "Authentication Guide",
  "headingPath": ["Guide", "Authentication", "API Keys"],
  "chunkIndex": 3,
  "totalChunks": 7,
  "markdown": "## API Keys\n\nTo generate a key...",
  "tokenEstimate": 412,
  "crawledAt": "2026-07-27T12:00:00Z"
}

From there it's one fetch from the dataset API to your embedding job. The dataset is also directly consumable by Apify's MCP integration, so an AI agent can trigger a crawl and query the result inside one conversation.

Design choices worth stealing

Chunk on structure, not character count. Fixed-size chunking splits tables and code blocks in half. Splitting on heading boundaries first, then paragraph boundaries when a section is oversized, keeps semantic units intact — retrieval quality goes up measurably.

Keep the heading path. "API Keys" means nothing alone. Guide → Authentication → API Keys gives your LLM the context to disambiguate. It costs one array field.

Estimate tokens cheaply. chars/4 is within ~10% for English prose. Good enough for chunk budgeting; skip the tokenizer dependency in the ingestion path.