RAG Corpus Deduplicator
$1.99OfficialUse when a retrieval corpus has exact or near-duplicate docs/chunks inflating embedding cost or polluting top-k results, and you must detect and prune them.
dataragdeduplicationembeddingsvector-searchminhashdata-qualitycost-optimization· by SkillingMain
What you get
- ✓7-step procedure
- ✓Runnable Python included
- ✓7-point quality checklist
- ✓8 pitfalls to avoid
- ✓Installs into 6 tools
- Version
- v1 →
- Last updated
- today
- Length
- 6 min read
- Requires
- Works with any modern AI assistant
Works in: Claude Code, Codex, Cline, opencode, OpenClaw, Hermes · Handles multi-file projects
Preview
When to use
- Invoke before indexing/re-indexing a corpus, or when retrieval returns several near-identical chunks inside the top-k.
- Trigger when embedding/storage bills grow faster than unique content, or when answer quality drops because redundant context crowds out diverse evidence.
- Use for both document-level dedup (raw files before chunking) and chunk-level dedup (records already in a vector store).
- Do NOT use to merge distinct versions you must keep separately (e.g., API v1 vs v2 docs) — that is version control, not deduplication.
Inputs to gather
- Corpus handle: path to JSONL/Parquet, or a vector-store connection (pgvector, Qdrant, Weaviate, Pinecone, Chroma) plus coll
…
🔒 Buy once ($1.99) to unlock the full playbook, download it, and install it in every tool you use.