Indexing
FlowRAG uses a batch indexing pipeline: scan files, chunk them, extract entities via LLM, generate embeddings, and store everything.
Basic Usage
Section titled “Basic Usage”await rag.index('./content');This processes all files in the directory recursively.
Incremental Indexing
Section titled “Incremental Indexing”By default, FlowRAG skips unchanged documents. Each document’s content is hashed (SHA-256) after processing and stored in KV storage. On re-index, only modified or new files are processed.
// Only processes changed/new documentsawait rag.index('./content');
// Force re-process everythingawait rag.index('./content', { force: true });Pipeline Stages
Section titled “Pipeline Stages”Files → Scanner → Chunker → Extractor (LLM) → Embedder → Storage → Entity Embedding1. Scanner
Section titled “1. Scanner”Reads files from the input path. Supports text files (.txt, .md, .json, etc.).
2. Chunker
Section titled “2. Chunker”Splits documents into chunks using token-based splitting:
- Chunk size: ~1200 tokens (default)
- Overlap: 100 tokens between chunks
- Uses tiktoken for accurate tokenization
3. Extractor
Section titled “3. Extractor”The LLM reads each chunk and extracts:
- Entities: named things (services, databases, protocols…)
- Relations: connections between entities (uses, produces, depends on…)
- Custom fields: if defined in the schema
Extraction results are cached in KV storage to avoid re-processing identical chunks.
4. Embedder
Section titled “4. Embedder”Generates vector embeddings for each chunk, enabling semantic search.
5. Storage
Section titled “5. Storage”Saves everything to three stores:
- KV: documents, chunks, extraction cache, document hashes
- Vector: chunk embeddings for semantic search — document metadata fields (from
DocumentMetadata.fields) are automatically included in vector records, so they’re available in search results without additional lookups - Graph: entities and relations for knowledge graph traversal
6. Entity Embedding
Section titled “6. Entity Embedding”After all chunks are processed, every entity in the knowledge graph is embedded and stored in vector storage. This enables semantic entity search via searchEntities() — find entities by meaning rather than exact name.
Concurrency Control
Section titled “Concurrency Control”Configure parallel processing via options:
const rag = createFlowRAG({ schema, ...createLocalStorage('./data'), options: { indexing: { chunkSize: 1200, // tokens per chunk chunkOverlap: 100, // overlap between chunks maxParallelInsert: 2, // concurrent documents llmMaxAsync: 4, // concurrent LLM calls embeddingMaxAsync: 16, // concurrent embedding calls }, },});Human-in-the-Loop
Section titled “Human-in-the-Loop”When using the CLI with --interactive, you can review extracted entities before they’re stored:
flowrag index ./content --interactiveThis shows each extracted entity and relation, letting you accept, reject, or edit them. See CLI Reference for details.
Document Deletion
Section titled “Document Deletion”Delete a document and automatically clean up its entities and relations from the knowledge graph:
await rag.deleteDocument('doc:readme');Entities shared with other documents are preserved (only their sourceChunkIds are updated). Orphaned entities and relations are removed automatically.
During folder re-indexing, stale documents (files that no longer exist on disk) are detected and deleted automatically.
Document Parsers
Section titled “Document Parsers”By default, FlowRAG reads text files (.txt, .md, .json, .yaml, etc.). For non-text documents, register custom parsers:
const rag = createFlowRAG({ schema, ...createLocalStorage('./data'), parsers: [new PDFParser(), new DocxParser()],});Parsers implement the DocumentParser interface — see Interfaces for details. Files with matching extensions are parsed instead of read as plain text.
Extraction Gleaning
Section titled “Extraction Gleaning”Run the LLM multiple times on the same chunk for higher extraction accuracy. Each additional pass receives previously found entities as context:
const rag = createFlowRAG({ schema, ...createLocalStorage('./data'), options: { indexing: { extractionGleanings: 2, // 2 additional passes per chunk }, },});Results are deduplicated automatically. More passes improve recall at the cost of additional LLM calls.
Entity Merging
Section titled “Entity Merging”Merge duplicate entities extracted by the LLM:
await rag.mergeEntities({ sources: ['Auth Service', 'AuthService', 'auth-service'], target: 'Auth Service',});All relations are redirected to the target entity, duplicates are removed, and source entities are deleted. Self-relations created by the merge are automatically skipped.
Pipeline Hooks
Section titled “Pipeline Hooks”For programmatic control, use the onEntitiesExtracted hook:
const rag = createFlowRAG({ schema, ...createLocalStorage('./data'), hooks: { onEntitiesExtracted: async (extraction, context) => { // Filter, modify, or log extracted entities console.log(`Chunk ${context.chunkId}: ${extraction.entities.length} entities`); return extraction; }, },});