GraphRAG and Knowledge Graphs: Beyond Vector-Only Retrieval

GraphRAG earns its complexity on exactly one class of workload: global, multi-hop questions over a bounded corpus — “what are the themes, who is connected to what, summarize across everything.” For local factoid Q&A, which is most enterprise retrieval traffic, vector search plus reranking remains the right default, because the graph index is expensive to build and to keep current. This guide covers what the pattern actually adds, when to skip it, and the three realistic implementation paths.

1. By the numbers

2. The question vector search cannot see

Vector retrieval answers local questions: the answer lives in a handful of chunks that are semantically near the query, so nearest-neighbor search finds them. It fails a different shape of question entirely — one whose answer is a property of the whole corpus, not of any chunk. “Which suppliers appear across our incident reports, and how are they connected?” has no chunk to retrieve; it requires reading everything and synthesizing. The Microsoft Research paper that anchors this pattern — “From Local to Global: A Graph RAG Approach to Query-Focused Summarization” (arXiv 2404.16130) — names the failure precisely^[1].

The paper’s abstract states the mechanism in full: “Our approach uses an LLM to build a graph index in two stages: first, to derive an entity knowledge graph from the source documents, then to pregenerate community summaries for all groups of closely related entities. Given a question, each community summary is used to generate a partial response, before all partial responses are again summarized in a final response to the user.”^[1] Three ideas matter for a platform lead. First, the knowledge graph is LLM-extracted, not hand-curated — entities and relationships are pulled from your documents by a model, which is where the cost lives. Second, community detection groups related entities into clusters at multiple levels, and each cluster gets a pregenerated summary — the corpus is pre-digested at index time so global questions become cheap-ish at query time. Third, answering a global question is a map-reduce over community summaries, not a top-k chunk lookup: partial answers per community, then a final synthesis.

3. Vector-only vs. GraphRAG: the real tradeoff

4. When GraphRAG is not worth it

The strongest warning comes from the source. The microsoft/graphrag repository — the reference implementation, MIT-licensed — tells users that “GraphRAG indexing can be an expensive operation, please read all of the documentation to understand the process and costs involved, and start small.”^[3] Every document passes through LLM extraction, and every entity community gets an LLM-written summary, at multiple hierarchy levels. That is an LLM bill proportional to corpus size at index time, before a single user query — and it recurs whenever the corpus changes enough to invalidate communities.

So the disqualifiers are concrete. A high-churn corpus (tickets, chat, news feeds) turns the index into a treadmill: extraction and community summaries chase the data forever. A local-question workload never exercises the graph — the paper claims wins for “global sensemaking questions,” not for factoid lookup^[1], and for local questions well-tuned hybrid search plus reranking (see the sibling pattern guide at /guides/advanced-rag-patterns-guide) closes most quality gaps at a fraction of the cost. An unbounded corpus breaks the economics outright; note that even AWS’s managed offering caps a data source at 1,000 files by default, 10,000 by exception^[2]. And extraction quality is a new failure mode: the graph is only as good as what the extraction model pulled out, so entity resolution errors (two names, one supplier) silently corrupt every downstream traversal.

5. Three implementation paths

1. The reference library: microsoft/graphrag

The first-party open-source implementation is “a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using the power of LLMs.”^[3] It is the fastest way to run the actual paper pattern — LLM extraction, community detection, pregenerated summaries — against your own documents and measure whether global questions improve. Given its maintenance-mode status, position it as a pilot vehicle and a design document, not a production commitment.

2. Managed: Amazon Bedrock Knowledge Bases with Neptune Analytics

For teams already on Bedrock, AWS sells the pattern as a checkbox: “Amazon Bedrock Knowledge Bases offers a fully managed GraphRAG feature with Amazon Neptune.” The feature “automatically identifies and uses relationships between entities and structural elements within documents ingested into Knowledge Bases,” and at query time it runs vector search first, then “expands on these related chunks by traversing the graph and retrieving their details from the graph database.”^[2] You choose a foundation model for graph construction when creating the knowledge base — the managed path does not remove the LLM extraction cost, it just meters it through Bedrock.

The constraints are documented and worth reading before committing: graph-build configuration is not customizable, autoscaling is not supported for the Neptune Analytics graph, Amazon S3 is the only supported data source, and with hierarchical chunking only child chunks are retrieved^[2]. Managed GraphRAG is a genuine de-risking of the operational surface, at the price of AWS’s opinions and a deeper tie into one cloud’s retrieval stack — a lock-in tradeoff to weigh deliberately, not discover later.

3. Build-your-own on a property graph database

The DIY path stores extracted entities and relationships in a general-purpose property graph database and queries it with a graph query language, layering LLM extraction and retrieval orchestration around it. It offers the most control — custom schemas, your own entity-resolution logic, traversal tuned to your domain — and the most ways to fail: you own extraction quality, graph refresh, query design, and the joinery between the vector store and the graph store. Choose it when the graph itself is a durable enterprise asset (a supplier network, a product ontology, a research corpus) that will outlive any single RAG application, and staff it as a data-engineering product, not a prompt-engineering side quest.

6. The middle grounds: structure without the full pattern

Full GraphRAG — LLM-extracted graph plus community summaries plus map-reduce querying — is the maximal version of a broader idea: give retrieval access to relationships, not just similarity. Several cheaper points on that spectrum capture much of the value, and they compose with the ingestion discipline covered at /guides/rag-ingestion-and-chunking rather than replacing it.

7. What the evidence actually shows — and what it doesn't

Be precise about the claim you are buying. The paper’s abstract states: “For a class of global sensemaking questions over datasets in the 1 million token range, we show that GraphRAG leads to substantial improvements over a conventional RAG baseline for both the comprehensiveness and diversity of generated answers.”^[1] That is a scoped result: a specific question class (global sensemaking), a specific corpus scale (~1M tokens), and specific qualities (comprehensiveness and diversity of answers). It is not a claim that GraphRAG improves factoid accuracy, reduces hallucination on local questions, or wins at every corpus size — so your evaluation plan should test the claim on your data rather than assume a general uplift. AWS frames its managed feature’s benefit the same conditional way: better responses “particularly when the information needs to be connected through multiple logical steps.”^[2]

Practically, that means a two-track eval before any production commitment: a local track (your existing retrieval eval set — GraphRAG must not regress it) and a global track (sensemaking and multi-hop questions your analysts actually ask, judged on comprehensiveness and coverage). If the global track is empty because nobody asks those questions, that is your answer.

8. Honest objections

“Long-context models make this obsolete — just stuff the corpus in the window.” Partially fair at small scale: a corpus that fits in one context window can be summarized directly. But context windows are metered per query, so paying to re-read the corpus on every global question inverts GraphRAG’s economics, which pay extraction once and amortize it across queries. The honest synthesis is that long context raises the corpus size at which graph indexing starts to pay — it does not remove the crossover point.

“The graph is hallucination-prone — you’ve built your retrieval on LLM output.” This objection deserves respect. LLM extraction does err, and errors in the index are stickier than errors in a single answer: they persist and propagate into every traversal. Mitigations exist — extraction spot-audits, entity-resolution review, grounding answers in source chunks rather than the graph alone — but they are real operating costs that belong in the TCO, not footnotes. This is also the strongest argument for the deterministic middle grounds above, which cannot hallucinate.

“We already have a knowledge graph program; isn’t this the same thing?” No — and conflating them burns budgets. A curated enterprise knowledge graph is a governed data asset with schema and stewardship. GraphRAG’s index is a disposable, machine-generated artifact optimized for retrieval, rebuilt when the corpus shifts. The best implementations connect the two; the worst try to make one serve as the other.

9. The read

GraphRAG is a scoped instrument, not a successor to vector retrieval. The decision rule: default to vector-plus-reranking for local Q&A; add deterministic graph structure (metadata, citations, entity tags) as cheap insurance; reserve full GraphRAG for bounded, high-value corpora where global and multi-hop questions are demonstrably the workload — investigations, research synthesis, supplier and risk networks, program-wide analysis. Pilot with the reference library or a managed offering to validate lift on your own global-question eval set before building anything durable, and price the index refresh, not just the build. Teams that follow this sequence get the pattern’s genuine capability — answers to questions their stack previously could not see — without funding a graph their users never query.