Every retrieval-augmented-generation tool claims its answers are grounded — traceable to a real document, never invented. That claim costs nothing to state in a README and a great deal to actually verify, so rather than take one at its word, we built the test ourselves against an open-source tool called RAGFlow, using our own document archive, on our own DGX Spark, with nothing sent to any cloud service.
What RAGFlow actually is
RAGFlow (infiniflow/ragflow, Apache 2.0) is not a vector database and not a prompt library. It's a packaged pipeline that takes a raw file and produces a cited answer. The stage it stakes its reputation on is called DeepDoc — parsing that runs OCR, table-structure recognition, and layout recognition together, so a scanned contract or a spreadsheet with merged cells doesn't get flattened into an unreadable wall of text before anyone asks it a question. Rather than slicing every document into identical blind chunks, an uploaded file gets matched against one of roughly a dozen named templates — Q&A, resume, academic paper, technical manual — each tuned to how that document type is actually structured, and a person can see and correct the resulting chunks before they're indexed.
Underneath, retrieval is hybrid: vector similarity plus keyword matching plus a reranker, running over Elasticsearch — the same class of engine a lot of production search infrastructure, including our own, already runs on. On top of that sits an optional knowledge-graph layer that can be updated incrementally as new documents arrive, rather than requiring a full rebuild. The project has grown fast: it's sitting at roughly 90,000 GitHub stars, up from about 10,000 two years ago.
The test: can you make it cite something that doesn't exist
We built RAGFlow from source and pointed it at 321 real documents already in our own archive — 3,522 chunks, all on one general subject, so retrieval couldn't cheat by matching on topic alone. We built 471 "gold" claims by taking evidence quotes our own systems had already extracted and confirming each one still exists, word for word, in its source document. Then we wrote 95 questions engineered to be unanswerable from that corpus — about subjects like Prince Andrew and Tesla's valuation, which appear nowhere in it. A grounded system should refuse every one of them. A language model with a retrieval step bolted on carelessly will often answer anyway, citing whatever the search happened to return.
On the retrieval side, the numbers were solid and in line with published benchmarks for this class of system: the correct source document ranked first 82.6% of the time, and appeared somewhere in the top 10 results 93.4% of the time, with a mean reciprocal rank of 0.864.
On citation integrity, the result was the one that actually matters: of 136 citation markers the system produced answering the answerable questions, zero pointed to a document that doesn't exist. It named the genuinely correct source 93.1% of the time, and the exact gold quote was present in what it cited 81.0% of the time. On the 60 unanswerable questions we ran to completion, it refused all 60 — including 9 cases where the retrieval step handed it evidence-shaped chunks anyway, and it still declined to build an answer from them. Zero smuggled or unsupported content on that set.
The mistake we caught in our own test before trusting it
Our first automated grading pass scored 15% of the answerable questions as refusals — a number worth doubting rather than accepting. Reading the actual text showed most of those weren't refusals at all: the system had written something like "not found in the provided documents... however, the data does show..." followed by a properly cited answer, and an automated classifier had keyed on the word "not found" and stopped reading. The corrected true refusal rate was 3.3%, not 15%. We're reporting the miss because a piece about citation integrity that hid its own grading error would be a strange way to make the point — the lesson generalizes past this one test: when a measurement produces a number, check whether the number is describing the thing you're testing or a flaw in your own test, before you act on it.
Building the gold set had a side benefit unrelated to RAGFlow itself: it meant re-checking 659 of our own stored evidence quotes against their original source pages, using a method completely different from how we'd checked that before. 74.8% matched byte-for-byte, 91.7% matched once punctuation and formatting differences are normalized, and the remaining 8.3% turned out to be reformatted lists, not invented text — a naive exact-match check would have wrongly flagged real quotes as fabricated roughly one time in four, purely on formatting.
What independent reports say breaks in practice
A tool's own demo and an outside user's experience are different things, so we checked what RAGFlow's public issue tracker actually shows. Several rough edges are documented there directly. One closed bug, filed after a version update, reports memory climbing toward exhaustion with no request traffic at all, needing repeated restarts as the only workaround. A separate report describes document parsing that doesn't parallelize and runs for hours on a single large file. A third documents an out-of-memory crash parsing a 400-plus-page PDF. And the project's own FAQ acknowledges that getting good results requires manually tuning retrieval parameters — it isn't a system that simply works well by default.
There's also a real gap worth naming for anyone considering this for anything regulated: a discussion opened on the project's own issue tracker argues that RAGFlow deployments handling sensitive documents are plausible candidates for high-risk classification under the EU's AI Act, and that the project currently ships no compliance checklist, no retrieval audit trail, and no built-in way to handle a right-to-erasure request.
And on adoption: despite the star count, RAGFlow's own maintainers opened a GitHub discussion this year titled "Call for Enterprise Stories: How Is Your Organization Using RAGFlow?" — a project this large still short enough on independently documented deployments that it's asking its own users to supply the case studies. One Hacker News commenter's criticism of the broader category of tools RAGFlow belongs to is worth including because it names the actual tradeoff plainly: these engines tend to "reimplement each part (PDF parser, etc etc) in a way where they are pretty much useless except in their specific engine" — meaning the parsing quality doesn't necessarily travel if you wanted to use just that one piece somewhere else.
Should you run the whole thing
Not as a wholesale replacement for a search stack that already does grounded, citation-forced retrieval — and the reasoning is worth stating rather than leaving implied. RAGFlow's default retrieval layer runs on the same class of engine, Elasticsearch, that a lot of existing infrastructure already uses. It isn't offering a different way to retrieve information; it's offering a packaged pipeline, a management UI, and one genuinely distinct capability sitting in front of that: DeepDoc's document parsing, done properly, for messy source material.
That's the actual reason to consider it at all — not as a platform swap, but as a purpose-built ingestion front end if you regularly need to swallow scanned filings or dense, table-heavy documents that a plain text pipeline mangles. Running its full stack — its own database, cache, object storage, and a second search cluster — just to get that one parsing stage, before that specific need is real, is a lot of infrastructure to stand up for one job.
The claim that actually matters here held up under a real test: on our own documents, on our own hardware, with nothing sent to any outside service, it did not fabricate a single citation, and it refused every genuinely unanswerable question we gave it, even when the retrieval step handed it material that could have been misused to answer anyway.
