We tried to make an open-source RAG engine cite a document that doesn't exist. It refused 60 times out of 60.

Testing RAGFlow's central claim — grounded citations, no fabrication — against our own document archive, on our own local hardware. Includes a grading bug we caught in our own harness before trusting the result. Measured on a DGX Spark, September 2026.

Every retrieval-augmented-generation tool claims it doesn't make things up. That claim is cheap to state and expensive to verify, so we built the actual test instead of reading the README: 321 real documents already sitting in our own research-fleet database, 471 real claims with a known correct source, and 95 questions engineered to be impossible to answer from that corpus — about Prince Andrew, Tesla's valuation, ambient lighting, none of which appear anywhere in a corpus about AI agent systems. If the tool is honest, it refuses all 95. If it's a typical language model with a retrieval step bolted on, it answers most of them confidently, citing whatever came back from the search.

It refused all 60 we ran to completion, including nine where the retrieval step handed it evidence-shaped chunks anyway.

The setup

RAGFlow (infiniflow/ragflow, Apache 2.0, ~90k GitHub stars) had no working ARM64 image, so we built it from source for the Spark's aarch64/CUDA 13 architecture — about 14 minutes. Everything ran local: bge-m3 for embeddings, qwen2.5:32b for generation, both served by the same Ollama daemon the rest of this fleet uses. No cloud API, no tokens spent, nothing left the machine.

The corpus was deliberately not easy: 3,522 chunks across nine research runs, all about the same general topic (AI agent systems), so the retrieval test can't cheat by matching on subject alone. 471 gold claims were built by taking real evidence quotes our own fleet had already extracted and verifying the quote still exists, verbatim, in its source document — a second, independent check on a property we'd only measured one other way before (more on that below).

What held up

Zero fabricated citations is the number that matters most for anything downstream that trusts this system's output unread. It's also the number most retrieval demos never actually test — most "look, it cites sources" showcases use questions the corpus can obviously answer, which tells you nothing about what happens on a question it can't.

The bug we caught in ourselves

The first grading pass scored 15% of the 58 answerable questions as refusals. That's a strange number to accept at face value from an automated classifier, so we read the actual text instead of trusting the count. Seven of those nine "refusals" were real answers: "not found in the provided documents... however, the dataset does indicate..." followed by a fully-cited answer. The classifier keyed on "not found" and stopped reading. True refusal rate: 3.3%, not 15%.

We're reporting this because a piece about citation integrity that hides its own grading mistake would undercut the point it's trying to make. The instrument was wrong, not the model — the same shape of error as the six-item benchmark that couldn't tell a 7B from a 32B, elsewhere in this collection: measure something, get a number, and check whether the number is telling you about the subject or about your own test before you act on it.

The side finding, about us

Building the gold set meant re-checking 659 of our own fleet's stored evidence quotes against their original source pages — a real-world reuse of a corpus built for an entirely different project. 74.8% matched byte-for-byte, 91.7% matched once punctuation and unicode are normalized, and the remaining 8.3% were reformatted lists and bullets, not invented text. We'd already measured our own fabrication rate at effectively zero one other way; this is the same conclusion arrived at from a completely different angle, which is worth more than either measurement alone. It also confirms something practical for anyone building this kind of check: a naive verbatim-quote gate fails roughly one real quote in four on formatting alone. Normalize before you gate, or you'll be flagging honest quotes as fabricated.

Should we run it in production

No, not as a wholesale replacement, and the reasoning is worth stating plainly rather than leaving as an implied "obviously." Our own search stack already does hybrid retrieval over Elasticsearch with a citation-forced answering layer — the same core substrate RAGFlow runs on by default. RAGFlow doesn't offer a different way to retrieve; it offers a packaged pipeline, a UI, and one genuinely distinct capability in front of that: DeepDoc, its document-parsing stage, which does OCR, table-structure recognition, and layout recognition together rather than flattening a scanned or table-heavy document to text first. Independent reports (not ours) note it catching table structure that even paid OCR services miss. That's the actual reason to consider it — a purpose-built answer to messy source documents, worth bolting on as an ingestion front-end if the archive ever needs to swallow scanned filings or dense financial tables at scale. Running the whole stack (its own MySQL, Redis, MinIO, and a second Elasticsearch cluster) just to get that one stage, before that need is real, is buying a warehouse to store one filing cabinet.

One more honest data point, unrelated to our own test but relevant to anyone evaluating this project from the outside: RAGFlow's own maintainers opened a GitHub discussion this year titled "Call for Enterprise Stories: How Is Your Organization Using RAGFlow?" — a project with 90,000 stars still short enough on independently-documented deployments that it's asking its own users to supply them. Adoption and verification are different things. We measured the second one directly, on our own hardware, against our own documents, and it held.