Loading pattern preview
BlocksTools & Utilities
Cheerio Web Scraper
Cheerio Web Scraper
Extract content from websites using Cheerio. Includes parsing, link discovery, and rate limiting.
Use this when you need a URL form that scrapes a page with Cheerio in a rate-limited server action, with no chat or model involved.
Loading pattern preview
Problems solved
Technical challenges this implementation handles out of the box.
- Scrape HTML pages with Cheerio in a tool-calling agent
- Extract readable text and structure from arbitrary URLs
- Feed scraped page content into a streaming chat analysis
Use cases
Products and workflows this pattern is designed to support.
- Competitive page teardowns
- Content extraction assistants
- SEO research that starts from live HTML
- Internal link and heading audits
Compare with
Similar patterns, and when each one is the better fit.
Scrape - Cheerio (lightweight)FreeUse this when a chat model should scrape one page or crawl a site's internal links itself with Cheerio tools, without a scraping service.
Jina AI Web ScraperUse this when you need a URL form that fetches a page through Jina Reader in a rate-limited server action, with no chat or model involved.
Firecrawl Scrape Tool ChatCrawl and extract content using Firecrawl's API. Includes URL processing, markdown conversion, and structured data extraction.
Scrape - Markdown.new (free)FreeUse this when a chat model should read a URL as markdown through a markdown.new tool, with no scraping API key to set up.
Setup
Requirements, wiring steps, and what this pattern adds to your project.
Getting started
Pick how you want to pull this pattern in. Then wire env vars and routes the same way.
- 1
Download → Next.js app
Use Download → Next.js app in the preview toolbar. You get a zip scaffold (Pro downloads rewrite preview `/view/.../api/*` paths to `/api/*` automatically).
- 2
Extract the zip and install dependencies
Unzip the archive, then run `pnpm install` (or npm / yarn) in the project folder.
- 3
Customize the agent and tool files
Adapt prompts, tools, and stop conditions in the generated project for your product.
- 4
Run the app and open the pattern route
Run `pnpm dev`, open the setup/checklist page if present, then navigate to the generated pattern route and verify responses.
Files added
3 files
app/page.tsxapp/layout.tsxREADME.md
Dependencies
15 total
npm packages
7Upstashcheeriolucide-reactNext.jsReactsonnerZod
Registry components
alertbadgebuttoncardinputscroll-areaskeletontabs
Critical files
2 files
lib/scraper.tsCore scraping engine: implements fetchWithRetry with exponential backoff, uses cheerio to parse HTML and extract title, meta description, main content, links, and images. Handles rate limiting, error codes, and URL normalization.
lib/actions.tsServer action entry point: validates URLs with Zod schema, integrates rate limiting, and calls the scraper.
Requirements
- Complexity
- Beginner
- Setup time
- ~5 minutes
