Scrape - Jina AI (advanced)

Scrape websites using Jina AI's reader service with JavaScript rendering. Includes URL scraping, website crawling, and content extraction with AI-powered parsing.

Loading preview…

Problems solved

Technical challenges this implementation handles out of the box.

Fetch clean page markdown through Jina Reader
Feed URL content into a tool-calling model flow
Demonstrate Jina scrape wiring with the AI SDK

Use cases

Products and workflows this pattern is designed to support.

Jina Reader onboarding demos
URL-to-markdown research tools
Documentation ingestion samples
Clean web content extraction prototypes

Setup

Requirements, wiring steps, and what this pattern adds to your project.

Getting started

Pick how you want to pull this pattern in — then wire env vars and routes the same way.

1
Install from the preview toolbar
Copy the install command above and run it in your project — Pro patterns include a short-lived token.
2
Add environment variables to .env.local
OPENAI_API_KEY
3
Point client fetches at your API routes
Replace preview `/view/.../api/...` paths with `/api/...` in the installed files.
4
Customize the agent and tool files
Adapt prompts, tools, and stop conditions for your product — Copy for AI in the toolbar helps seed that work.
5
Run your dev server and open the pattern route
Install dependencies if needed, then start the app and verify responses.

Environment variables

OPENAI_API_KEYGet key

Capabilities

AI SDK APIs

streamTextconvertToCoreMessagestooluseChat

Providers

OpenAI

External services

Jina

Files added

6 files

app/page.tsx
app/layout.tsx
app/api/tool-scrape-jina/route.ts
components/jina-scrape-demo.tsx
tools/jina.ts
tools/jina-ui.tsx

Dependencies

9 total

npm packages

AISDKlucide-reactNext.jsReactZod

Registry components

alertemptyinput-groupseparator

Critical files

tool-scrape-jina/route.ts
API route implementing a web scraping assistant using streamText with Jina AI tools. Contains the AI pipeline with system prompt, tool binding, and step count limits.
tools/jina.ts
Core scraping engine and tool definitions using Jina AI Reader. Implements scrapeUrl (proxied through Jina service with regex-based parsing) and crawlWebsite (BFS crawler with depth limits).