Loading pattern preview
TemplatesTools & Utilities
Scrape - Cheerio (lightweight)
Scrape - Cheerio (lightweight)
Scrape websites using Cheerio's fast HTML parser. Includes URL scraping, website crawling, and content extraction without external API costs.
Loading pattern preview
Problems solved
Technical challenges this implementation handles out of the box.
- Scrape page HTML with Cheerio inside a tool call
- Extract readable text from a URL for the model
- Demonstrate lightweight Cheerio scraping with the AI SDK
Use cases
Products and workflows this pattern is designed to support.
- Cheerio scrape onboarding demos
- URL content extraction tools
- Competitive page text grabs
- Lightweight HTML parsing samples
Setup
Requirements, wiring steps, and what this pattern adds to your project.
Getting started
Pick how you want to pull this pattern in. Then wire env vars and routes the same way.
- 1
Download → Next.js app
Use Download → Next.js app in the preview toolbar. You get a zip scaffold (Pro downloads rewrite preview `/view/.../api/*` paths to `/api/*` automatically).
- 2
Extract the zip and install dependencies
Unzip the archive, then run `pnpm install` (or npm / yarn) in the project folder.
- 3
Fill .env.local
The download includes .env.local placeholders when needed. Add real values for: OPENAI_API_KEY.
- 4
Customize the agent and tool files
Adapt prompts, tools, and stop conditions in the generated project for your product.
- 5
Run the app and open the pattern route
Run `pnpm dev`, open the setup/checklist page if present, then navigate to the generated pattern route and verify responses.
Environment variables
OPENAI_API_KEYGet key
Capabilities
AI SDK APIs
streamTextconvertToCoreMessagestooluseChatProviders
OpenAI
Files added
6 filesapp/page.tsxapp/layout.tsxapp/api/tool-scrape-cheerio/route.tscomponents/cheerio-scrape-demo.tsxtools/cheerio.tstools/cheerio-ui.tsx
Dependencies
10 totalnpm packages
6AISDKcheeriolucide-reactNext.jsReactZod
Registry components
alertemptyinput-groupseparator
Critical files
tool-scrape-cheerio/route.tsAPI route implementing a web scraping assistant using streamText with Cheerio tools. Contains the AI pipeline with system prompt, tool binding, and step count limits.
tools/cheerio.tsCore scraping engine and tool definitions using Cheerio. Implements scrapeUrl (HTML parsing, content extraction, retry with exponential backoff) and crawlWebsite (BFS crawler with depth limits).
Requirements
- Complexity
- Beginner
