Power LLMs with Web Context
Give your AI models real-time access to the live web
LLMs hallucinate when they lack current information. Give them exactly what they need: live web content, on demand, in the format models understand best.
How It Works
Scrape web context
Pass any URL or domain to retrieve HTML, Markdown, images, or a full sitemap in one API call
Inject into your LLM prompt
Include the scraped content in your system prompt or context window alongside your user query
Get grounded, accurate responses
Your model answers with real web context—no hallucinations about outdated or missing information
Scrape web context
Pass any URL or domain to retrieve HTML, Markdown, images, or a full sitemap in one API call
Inject into your LLM prompt
Include the scraped content in your system prompt or context window alongside your user query
Get grounded, accurate responses
Your model answers with real web context—no hallucinations about outdated or missing information
The Endpoints
Four endpoints, each optimized for a different stage of the LLM context pipeline. Use them together or independently depending on what your model needs.
Scrape Raw HTML
HTMLGET /v1/web/scrape/htmlRetrieve the full HTML source of any URL. Use this when you need structured DOM data, need to extract specific elements, or want to feed raw page content into a custom parser before handing off to an LLM.
DOM parsing, structured extraction, custom pipelines
Scrape to Markdown
MarkdownGET /v1/web/scrape/markdownConvert any webpage to clean GitHub Flavored Markdown. The best format for LLM context windows—strips HTML noise while preserving semantic structure. Reduces token usage dramatically versus raw HTML.
LLM context, RAG pipelines, knowledge bases
Scrape Images
ImagesGET /v1/web/scrape/imagesExtract every image from a page—img tags, inline SVGs, base64 data URIs, picture/source elements, and video posters. Returns src, element type, format, and alt text for each. Use for visual AI pipelines and multimodal models.
Multimodal AI, visual search, image indexing
Crawl Sitemap
SitemapGET /v1/web/scrape/sitemapDiscover up to 500 page URLs on any domain by crawling its sitemap. Supports sitemap index files recursively. Use this to build a URL list before batch-scraping an entire site's content into your vector store.
Batch indexing, content discovery, RAG ingestion
Who This Is For
RAG Pipeline Engineers
Building retrieval-augmented generation systems
Keeping LLM knowledge current requires constant re-indexing. Models hallucinate when their training data is outdated relative to live web content.
Use the Markdown endpoint to fetch and embed fresh web pages on demand. Combine with the Sitemap endpoint to bulk-ingest entire domains into your vector store.
Always-current RAG context without manual scraping infrastructure. Your models answer with facts, not guesses.
AI Agent Developers
Building autonomous agents that browse the web
Agents need to read web pages, but running a browser per-agent is expensive, slow, and hard to scale. Managing proxies and bot detection is a full-time job.
Replace headless browser calls with single API requests. Get HTML or Markdown with automatic proxy escalation built in—no infrastructure to maintain.
Agents that browse the web reliably at scale. Reduce per-agent infrastructure cost and eliminate browser management complexity.
AI Search Products
Building AI-powered search and answer engines
Answering questions about current events or specific URLs requires live web access. Training data cutoffs leave models unable to answer about recent content.
Scrape the relevant URL to Markdown and inject it as context before generating an answer. Give your model real-time access to any page on the web.
Search results grounded in real, current content. Users trust answers backed by live sources.
Competitive Intelligence Platforms
Monitoring competitors and market changes
Tracking competitor websites for pricing, product, and messaging changes requires scrapers that constantly break against site updates and bot detection.
Use the HTML or Markdown endpoint to snapshot competitor pages on a schedule. Diff the output and feed changes to an LLM for automated analysis and alerts.
Always-current competitive intelligence. Know when a competitor changes pricing or messaging within hours, not weeks.
Document & Knowledge Base Tools
Ingesting web content into knowledge systems
Users want to import web pages and articles into their knowledge bases, but HTML is noisy and hard to chunk correctly for embedding.
Use the Markdown endpoint for clean, well-structured content ready for chunking. Use the Sitemap endpoint to discover all pages in a documentation site for bulk import.
Faster, cleaner knowledge base ingestion. Better embeddings and retrieval from content that is actually readable.
LLM Evaluation & Testing Teams
Testing and benchmarking language models
Evaluating how well a model handles real-world web content requires a reliable way to fetch diverse, current web pages at scale.
Programmatically scrape a wide range of pages using the HTML or Markdown API to build evaluation datasets grounded in real, current web content.
Richer, more realistic model evaluations. Test against the actual web your users interact with—not stale snapshots.
Content Automation Platforms
Generating content from web sources at scale
Content teams want AI that can research topics from the live web before writing, but connecting models to live web data requires custom infrastructure.
Fetch relevant pages as Markdown and inject into the generation prompt. Your content AI researches, summarizes, and cites live sources automatically.
AI-written content grounded in current sources. Higher quality output with citations users can verify.
Multimodal AI Applications
Building apps that reason over images and text together
Extracting images from web pages to feed into vision models requires parsing complex HTML, handling SVGs, base64 URIs, and multiple image element types.
Use the Images endpoint to extract every image from any URL in a structured array. Each image is classified by type and element source—ready for multimodal model input.
Reliable image extraction without brittle scrapers. Feed vision models web images without building custom extraction logic.
Personalize at scale
Join 4,000+ businesses using Brand.dev to personalize their products.