Loading pattern preview
BlocksTools & Utilities
Cloudflare Markdown.new Web Scraper
Cloudflare Markdown.new Web Scraper
Convert any webpage to markdown using Cloudflare's markdown.new. Clean, structured output for AI pipelines and content processing.
Use this when you need a URL form that converts a page to markdown via markdown.new in a rate-limited server action, with no chat or model.
Loading pattern preview
Problems solved
Technical challenges this implementation handles out of the box.
- Convert URLs to markdown via Cloudflare markdown.new
- Extract title and markdown content without a custom scraper stack
- Handle rate limits and forbidden responses from the markdown service
Use cases
Products and workflows this pattern is designed to support.
- Documentation ingestion from public pages
- Competitive content research
- Knowledge-base drafting from live URLs
- Lightweight URL-to-markdown utilities
Compare with
Similar patterns, and when each one is the better fit.
Scrape - Markdown.new (free)FreeUse this when a chat model should read a URL as markdown through a markdown.new tool, with no scraping API key to set up.
Firecrawl Scrape Tool ChatCrawl and extract content using Firecrawl's API. Includes URL processing, markdown conversion, and structured data extraction.
Markdown multi-format file exportUse this when you are migrating or exporting docs by section and want markdown parsed with unified and remark entirely client side.
Cheerio Web ScraperUse this when you need a URL form that scrapes a page with Cheerio in a rate-limited server action, with no chat or model involved.
Setup
Requirements, wiring steps, and what this pattern adds to your project.
Getting started
Pick how you want to pull this pattern in. Then wire env vars and routes the same way.
- 1
Download → Next.js app
Use Download → Next.js app in the preview toolbar. You get a zip scaffold (Pro downloads rewrite preview `/view/.../api/*` paths to `/api/*` automatically).
- 2
Extract the zip and install dependencies
Unzip the archive, then run `pnpm install` (or npm / yarn) in the project folder.
- 3
Customize the agent and tool files
Adapt prompts, tools, and stop conditions in the generated project for your product.
- 4
Run the app and open the pattern route
Run `pnpm dev`, open the setup/checklist page if present, then navigate to the generated pattern route and verify responses.
Capabilities
External services
cloudflare
Files added
3 files
app/page.tsxapp/layout.tsxREADME.md
Dependencies
16 total
npm packages
7Upstashlucide-reactNext.jsReactsonnerstreamdownZod
Registry components
alertbadgebuttoncardinputscroll-areaskeletontabstooltip
Critical files
2 files
lib/scraper.tsCore scraping engine using Cloudflare markdown.new: fetches via https://markdown.new/<url>, parses response to extract title, URL source, and markdown content.
lib/actions.tsServer action entry point: URL validation with Zod, rate limiting, and orchestration of the markdown.new scraping pipeline.
Requirements
- Complexity
- Beginner
- Setup time
- ~5 minutes
