Stage 5: Embedding DB & RAG-based QA

Problem

Traditional QA over full texts is slow, expensive, and context-limited. LLMs can't process 15K+ papers in a single query.

Solution

Build a semantic embedding database (ChromaDB) and apply advanced RAG (Retrieval-Augmented Generation) for scalable, precise QA.

Key Novelty

• Optimal chunking: NLTK sentence-based, 1500-char chunks, 300-char overlap, section-aware
• Scientific embeddings: SPECTER2/all-mpnet-base-v2, GPU-accelerated indexing
• Multi-query retrieval: Multiple query variants per question, deduplication, abstract inclusion
• LLM voting system: Combines RAG and full-text answers, leveraging strengths of both approaches