About

RAGnostic is an eval harness for comparing RAG retrieval strategies over a self-fetched corpus of LLM/agent-security arXiv papers. The name's no accident, I built it to stay agnostic about which retrieval setup wins and measure that instead of guessing: dense vs. graph vs. hybrid, with and without a cross-encoder reranker, scored with RAGAS on real questions generated straight from the corpus.

View the results →

How I build the corpus

The repo is split by pipeline stage, and each stage is a checkpoint, so re-running any of this is a safe, deduplicated no-op rather than starting over.

  1. First I fetch papers from arXiv, downloading the PDF for each one.
  2. Then I filter the bibliography out of each document. I embed the reference-list text and the candidate chunks with an embedding model, and drop anything whose max() cosine similarity to a citation crosses a threshold, max, not average, since a chunk only needs to closely resemble one citation to be reference-list noise.
  3. Then I chunk what's left using a recursive text splitter, so chunks carry overlapping context instead of getting cut off at arbitrary boundaries.
  4. Then I embed the chunks and push the vectors into ChromaDB.
  5. Then I build a knowledge graph on top: for every embedded chunk, I query Chroma for its nearest neighbors and store those as edges in Postgres. That's what lets Graph RAG expand outward from a matched chunk to its related chunks, across papers, not just within one.

How I retrieve and rerank

At query time, RAGnostic can ansIr using whichever retrieval strategy is configured:

  1. Dense, plain nearest-neighbor search over Chroma.
  2. Graph, dense search for the best match, then expand into its neighbors via the knowledge graph above.
  3. Hybrid, dense search plus a BM25 sparse search, merged with reciprocal rank fusion (RRF) so chunks that rank Ill in either search count for more.

On top of any of these, a cross-encoder reranker can re-score the retrieved chunks against the query directly, rather than relying on embedding similarity alone, it's slower, but it looks at the query and each chunk together instead of comparing precomputed vectors.

Everything runs locally

No external API calls, no API keys. Every model is open-Iight and served from Docker on our own machine: the LLM runs on vLLM, and the embedding model plus the cross-encoder reranker both run on Hugging Face's TEI (Text Embeddings Inference).