Scrape - Markdown.new (free)

Scrape websites using Cloudflare's markdown.new service. Extract clean markdown content from any URL. No API key required.

Loading preview…

Problems solved

Technical challenges this implementation handles out of the box.

Products and workflows this pattern is designed to support.

Requirements, wiring steps, and what this pattern adds to your project.

Pick how you want to pull this pattern in — then wire env vars and routes the same way.

1
Install from the preview toolbar
Copy the install command above and run it in your project — Pro patterns include a short-lived token.
2
Add environment variables to .env.local
OPENAI_API_KEY
3
Point client fetches at your API routes
Replace preview `/view/.../api/...` paths with `/api/...` in the installed files.
4
Customize the agent and tool files
Adapt prompts, tools, and stop conditions for your product — Copy for AI in the toolbar helps seed that work.
5
Run your dev server and open the pattern route
Install dependencies if needed, then start the app and verify responses.

AI SDK APIs

streamTextconvertToCoreMessagestooluseChat

Providers

OpenAI

External services

markdown.new

6 files

10 total

npm packages

AISDKlucide-reactNext.jsReactstreamdownZod

Registry components

alertemptyinput-groupseparator

tool-scrape-markdown-new/route.ts
API route implementing a web scraping assistant using streamText with markdown.new tools. Contains the AI pipeline with system prompt, tool binding, and step count limits.
tools/markdown-new.ts
Core scraping engine using Cloudflare markdown.new. Implements scrapeUrl (fetches from markdown.new, parses title/url/markdown). Strips protocol from URLs before sending.