Loading pattern preview
BlocksTools & Utilities
Jina AI Web Scraper
Jina AI Web Scraper
Scrape websites using Jina AI with JavaScript rendering and proxy rotation. Handles rate limiting and bot detection.
Use this when you need a URL form that fetches a page through Jina Reader in a rate-limited server action, with no chat or model involved.
Loading pattern preview
Problems solved
Technical challenges this implementation handles out of the box.
- Fetch clean page content through the Jina Reader API
- Convert URLs into LLM-friendly markdown for analysis
- Use scraped content inside a streaming tool-calling chat
Use cases
Products and workflows this pattern is designed to support.
- Research agents that need clean page text
- Documentation ingestion from public URLs
- Competitive content analysis
- Citation gathering from web pages
Compare with
Similar patterns, and when each one is the better fit.
Scrape - Jina AI (advanced)FreeUse this when you are prototyping a research or docs ingestion chat that turns links users paste into markdown via Jina Reader.
Cheerio Web ScraperUse this when you need a URL form that scrapes a page with Cheerio in a rate-limited server action, with no chat or model involved.
Scrape - Cheerio (lightweight)FreeUse this when a chat model should scrape one page or crawl a site's internal links itself with Cheerio tools, without a scraping service.
Workflow - URL AnalysisUse this when you need to run a URL through fixed scrape, analyze, and summarize steps in sequence and stream each step's status to the UI.
Setup
Requirements, wiring steps, and what this pattern adds to your project.
Getting started
Pick how you want to pull this pattern in. Then wire env vars and routes the same way.
- 1
Download → Next.js app
Use Download → Next.js app in the preview toolbar. You get a zip scaffold (Pro downloads rewrite preview `/view/.../api/*` paths to `/api/*` automatically).
- 2
Extract the zip and install dependencies
Unzip the archive, then run `pnpm install` (or npm / yarn) in the project folder.
- 3
Customize the agent and tool files
Adapt prompts, tools, and stop conditions in the generated project for your product.
- 4
Run the app and open the pattern route
Run `pnpm dev`, open the setup/checklist page if present, then navigate to the generated pattern route and verify responses.
Capabilities
External services
Jina
Files added
3 files
app/page.tsxapp/layout.tsxREADME.md
Dependencies
14 total
npm packages
6Upstashlucide-reactNext.jsReactsonnerZod
Registry components
alertbadgebuttoncardinputscroll-areaskeletontabs
Critical files
2 files
lib/scraper.tsCore scraping engine using Jina AI service (r.jina.ai): fetches via Jina's proxy endpoint, parses returned HTML using regex-based extractors for title, meta, content, links, and images.
lib/actions.tsServer action entry point: URL validation with Zod, rate limiting, and orchestration of the Jina scraping pipeline.
Requirements
- Complexity
- Beginner
- Setup time
- ~5 minutes
