Loading pattern preview
BlocksTools & Utilities
Chunk strategy compare
Chunk strategy compare
Compare small, large, and paragraph-window chunking on one document with cosine similarity — no vector database.
Use this when you are tuning chunking as a retrieval setting and want to compare three strategies side by side with cosine similarity scores.
Loading pattern preview
Problems solved
Technical challenges this implementation handles out of the box.
- See how chunk size and packing change retrieval on the same document
- Rank three chunking strategies with AI SDK Core cosineSimilarity
- Teach chunking as a retrieval hyperparameter without a vector database
Use cases
Products and workflows this pattern is designed to support.
- Choosing a chunking strategy for RAG
- Workshops on embeddings without a vector database
- Comparing retrieval quality across chunk windows
Compare with
Similar patterns, and when each one is the better fit.
Local RAG IngestUse this when a small docs or knowledge-base prototype needs its text turned into embeddings held in React state and saved as a portable index.
Local RAG ChatUse this when you already have an exported embedding index of text chunks and want users to chat over it with cited, streamed answers.
Agent DOCX ArtifactUse this when your agent drafts documents that users must download as editable Word files, built client-side from Markdown streamed via a tool call.
Cloudflare Markdown.new Web ScraperUse this when you need a URL form that converts a page to markdown via markdown.new in a rate-limited server action, with no chat or model.
Setup
Requirements, wiring steps, and what this pattern adds to your project.
Getting started
Pick how you want to pull this pattern in. Then wire env vars and routes the same way.
- 1
Download → Next.js app
Use Download → Next.js app in the preview toolbar. You get a zip scaffold (Pro downloads rewrite preview `/view/.../api/*` paths to `/api/*` automatically).
- 2
Extract the zip and install dependencies
Unzip the archive, then run `pnpm install` (or npm / yarn) in the project folder.
- 3
Fill .env.local
The download includes .env.local placeholders when needed. Add real values for: AI_GATEWAY_API_KEY.
- 4
Customize the agent and tool files
Adapt prompts, tools, and stop conditions in the generated project for your product.
- 5
Run the app and open the pattern route
Run `pnpm dev`, open the setup/checklist page if present, then navigate to the generated pattern route and verify responses.
Environment variables
AI_GATEWAY_API_KEYGet key
Capabilities
AI SDK APIs
embedcosineSimilarityExternal services
gatewayvercel-ai-gatewayUpstash
Files added
16 files
app/page.tsxapp/layout.tsxapp/api/query-embedding/route.tscomponents/local-rag-chunk-compare.tsxhooks/use-local-rag-chunk-compare.tslib/hash.tslib/embedding-config.tslib/chunk-text.tslib/chunk-compare-index.tslib/rate-limit.tslib/request-utils.tslib/retrieval.tslib/sample-index.tsfixtures/source.mdfixtures/sample-compare-index.jsonREADME.md
Dependencies
7 total
npm packages
6UpstashAISDKlucide-reactNext.jsReactZod
Registry components
button
Critical files
4 files
lib/chunk-text.tsParameterized chunker: fixed-small, fixed-large, and paragraph-window-v1. The packParagraphs path must stay parity-tested against local-rag-ingest chunkNormalizedText.
lib/chunk-compare-index.tsV1 educational compare-index contract. Not importable into local-rag-chat. Reject incompatible schema/version/model/dimensions/strategy identity before cosineSimilarity.
lib/retrieval.tsRank all three strategy lanes in the browser with cosineSimilarity. Winner is highest top-1 score; dim below-threshold snippets instead of dropping them.
query-embedding/route.tsStateless Core embed boundary for one query string. Locked model openai/text-embedding-3-small with providerOptions.openai.dimensions=512. Never add @ai-sdk/openai solely for embeddings.
Requirements
- Complexity
- Intermediate
- Setup time
- ~10 minutes
- Node
- 24+
- Memory
- 256MB
Upstash Redis for deployed demo rate limiting
