PatternsTools & Utilities

Cheerio Web Scraper

Extract content from websites using Cheerio. Includes parsing, link discovery, and rate limiting.

Loading preview…

Problems solved

Technical challenges this implementation handles out of the box.

Products and workflows this pattern is designed to support.

Requirements, wiring steps, and what this pattern adds to your project.

Pick how you want to pull this pattern in — then wire env vars and routes the same way.

1
Install from the preview toolbar
Copy the install command above and run it in your project — Pro patterns include a short-lived token.
2
Customize the agent and tool files
Adapt prompts, tools, and stop conditions for your product — Copy for AI in the toolbar helps seed that work.
3
Run your dev server and open the pattern route
Install dependencies if needed, then start the app and verify responses.

3 files

14 total

npm packages

Upstashcheeriolucide-reactNext.jsReactsonnerZod

Registry components

alertbadgebuttoncardinputscroll-areaskeleton

lib/scraper.ts
Core scraping engine: implements fetchWithRetry with exponential backoff, uses cheerio to parse HTML and extract title, meta description, main content, links, and images. Handles rate limiting, error codes, and URL normalization.
lib/actions.ts
Server action entry point: validates URLs with Zod schema, integrates rate limiting, and calls the scraper.