Cheerio Web Scraper

Extract content from websites using Cheerio. Includes parsing, link discovery, and rate limiting.

Loading preview…

Problems solved

Technical challenges this implementation handles out of the box.

Products and workflows this pattern is designed to support.

Requirements, wiring steps, and what this pattern adds to your project.

Pick how you want to pull this pattern in. Then wire env vars and routes the same way.

1
Download → Next.js app
Use Download → Next.js app in the preview toolbar. You get a zip scaffold (Pro downloads rewrite preview `/view/.../api/*` paths to `/api/*` automatically).
2
Extract the zip and install dependencies
Unzip the archive, then run `pnpm install` (or npm / yarn) in the project folder.
3
Customize the agent and tool files
Adapt prompts, tools, and stop conditions in the generated project for your product.
4
Run the app and open the pattern route
Run `pnpm dev`, open the setup/checklist page if present, then navigate to the generated pattern route and verify responses.

3 files

14 total

npm packages

Upstashcheeriolucide-reactNext.jsReactsonnerZod

Registry components

alertbadgebuttoncardinputscroll-areaskeleton

lib/scraper.ts
Core scraping engine: implements fetchWithRetry with exponential backoff, uses cheerio to parse HTML and extract title, meta description, main content, links, and images. Handles rate limiting, error codes, and URL normalization.
lib/actions.ts
Server action entry point: validates URLs with Zod schema, integrates rate limiting, and calls the scraper.

Loading preview…