Why NLP Before the Graph

The knowledge graph doesn't build itself. Before a single node lands in Neo4j, every paper has to go through two NLP stages: named entity recognition to identify what exists in the text, and relation extraction to identify how those things connect.

Get this wrong and the graph is noise. Get it right and you have a structured knowledge base that no keyword search can replicate.

Here's exactly how both stages work.

The Pipeline Structure

The entire NLP workflow is a reproducible DVC pipeline with four tracked stages:

fetch_arxiv → ner → relations → graph_loader

Every stage is version-controlled. If a parameter changes — confidence threshold, batch size, entity type mapping — DVC detects it, marks downstream stages stale, and re-runs only what's affected. Every run logs metrics to MLflow under the insight-engine-nlp experiment.

This means every entity count, relation count, and domain breakdown is reproducible and comparable across runs. If I tighten the confidence threshold tomorrow, I can immediately see the effect on graph density without touching the codebase.

Stage 1 — Data Ingestion

Script: src/ingestion/arxiv_fetcher.py

The pipeline starts with the arXiv API. The fetcher queries 12 domain-specific search terms, pulls paper metadata — title, abstract, authors, categories, published date — and writes to JSONL files.

Three behaviours worth noting:

Deduplication is on arxiv_id — re-running the fetcher never downloads the same paper twice. New papers land in new batch files, which DVC detects as a change and marks the NER stage stale automatically. The fetcher is rate-limited to 3 seconds between requests per arXiv's API requirements.

Current result: 229,498 papers across 12 domains — Aerospace, Medical Devices, Materials, Energy, Biotechnology, Robotics, Quantum, Nanotechnology, Environment, Semiconductors, Pharma, and Neuroscience.

Stage 2 — Named Entity Recognition

Script: src/nlp/ner_pipeline.py Model: en_core_sci_lg (SciSpacy large biomedical model)

The NER stage reads every paper's title and abstract and extracts named entities, mapping each to one of 12 canonical entity types:

Why en_core_sci_lg over the general English model?

Scientific text is a different language. "Transformer" in a general corpus means a piece of electrical equipment. In a machine learning paper it's an architecture. SciSpacy's biomedical model has pre-trained representations for scientific vocabulary that the general model completely misses.

Results: 10,779,699 entity mentions extracted across 229,498 documents — an average of 47 entities per document. Every run logs these counts to MLflow per domain, so I can immediately see if a new data batch skews the distribution.

Stage 3 — Relation Extraction (v1 Rule-Based)

Script: src/nlp/relation_extractor.py

This is where the architecture gets interesting — and where I made a deliberate, opinionated tradeoff.

Relations in Insight Engine are not extracted by reading sentences. They are assigned by entity type pairs. If a Material and a Device co-occur in the same document, the rule table says: emit a COMPOSED_OF relationship with confidence 0.80.

Here's the full rule table:

Why rules instead of sentence-level extraction?

Sentence-level relation extraction — reading the actual text between entities — requires models that are significantly slower and much harder to run at scale. At 229,498 documents on a single RTX 4060, a sentence-level model would make this stage prohibitively slow.

More importantly, the relation types were designed around the intelligence modules the product needs to power:

Relations are the product's knowledge model, not a neutral linguistic extraction. This makes the v1 rule-based approach a reasonable first pass — fast, predictable, and sufficient to validate the core product thesis.

The known limitation: a paper stating "compound X does NOT treat disease Y" still emits a TREATS relation. v2 will use sentence-level extraction via SciSpacy relation models or Mistral 7B to read actual sentence text. That's a future phase.

All thresholds are tunable via params.yaml:

relations:
  confidence_threshold: 0.7
  max_relations_per_doc: 10

confidence_threshold drops low-confidence rules before they hit the graph. max_relations_per_doc prevents combinatorial explosion on entity-dense papers.

Results: 2,294,895 relations extracted across all 12 domains.

What Each Stage Stores

One important detail: after Stage 4 (graph loading), the JSONL files are no longer the source of truth. Neo4j deduplicates 10.7M raw entity mentions into 1,529,916 unique nodes via MERGE. The files are only needed if you want to rebuild the graph from scratch via dvc repro.

What's Next

Stage 4 takes the JSONL output and loads it into Neo4j. But loading 10.7M entity mentions into a graph database naively takes hours — and produces a graph full of duplicates.

Part 3 covers the graph loader: the Neo4j schema, the MERGE strategy that deduplicates 10.7M mentions into 1.5M unique nodes, the batch Cypher patterns that make it tractable at scale, and the embedding pipeline that links every entity node to its Qdrant vector via an embedding_id property.

Follow the series on Hashnode to get Part 3 when it drops.

Questions about the NLP pipeline, the relation type design, or DVC setup? Drop them in the comments — I read everything.

The Problem Nobody Has Solved Well

Innovation rarely happens inside a single domain. The surgical robot borrowed from automotive manufacturing. mRNA vaccines borrowed from cancer research. Lithium-ion battery chemistry borrowed from aerospace.

Yet almost every research discovery tool available today — Google Scholar, Semantic Scholar, internal enterprise search — is fundamentally siloed. They help you go deeper into a domain. They cannot help you go across them.

This is the problem Insight Engine is built to fix.

What Insight Engine Actually Is

Insight Engine is a B2B SaaS platform for cross-domain innovation discovery. Given a research question, it doesn't just retrieve similar papers — it traverses a knowledge graph to find structural connections between research fields that no single paper ever put together.

The core thesis:

A battery-longevity breakthrough in aerospace may directly solve an unsolved problem in cardiology. But nobody ever makes the connection — because they're reading different journals, attending different conferences, and searching different databases.

Insight Engine is the matchmaker.

The Data

The platform currently operates on:

• 229,498 arXiv papers across 12 research domains: Aerospace, Medical Devices, Materials, Energy, Biotechnology, Robotics, Quantum, Nanotechnology, Environment, Semiconductors, Pharma, and Neuroscience
• 10,779,699 entity mentions extracted via NER (47 per document on average)
• 2,294,895 relationships between entities
• 1,529,916 unique entity nodes in the knowledge graph after deduplication
• 1,583,613 RELATES_TO edges connecting them
• 1,837,582 cross-domain edges — connections between entities from different domains

This is a live, growing dataset. New domains and data sources are being added as the project progresses.

The Tech Stack

Every tool was chosen for a specific reason, and all of it runs locally on a single RTX 4060 8GB GPU — zero cloud cost during development.

Data & NLP

• spaCy 3.7 + SciSpacy en_core_sci_lg — biomedical NER for entity extraction
• DVC 3.x — Git-like versioning for the full data pipeline
• MLflow 2.x — experiment tracking across every pipeline run
• Apache Airflow 2.9 — DAG-based pipeline scheduling

Graph & Vector

• Neo4j Community 5.x — knowledge graph with full Cypher query support
• Qdrant 1.9 — vector store for semantic entity search
• nomic-embed-text v1.5 — 768-dim embeddings, runs via Ollama
• LlamaIndex 0.10 — GraphRAG query layer connecting Neo4j and Qdrant

LLM

• Mistral 7B (Q4_K_M) via Ollama — fits in 4.1GB VRAM, primary synthesis model
• Llama 3.1 8B — fallback model for more complex reasoning

Backend

• FastAPI 0.111 + SQLAlchemy 2.0 async — API layer with JWT auth
• PostgreSQL 16 — user, auth, and session storage
• Celery + Redis — background task processing

Frontend

• Next.js 14 + TypeScript + Tailwind + shadcn/ui — dashboard
• Sigma.js 3.x — WebGL graph visualisation handling 100K+ nodes in-browser
• TanStack Query 5.x — server state management

Infrastructure

• Docker Compose — all 13 services orchestrated in a single file
• GitHub Actions — CI/CD with automated testing and container publishing to ghcr.io
• Prometheus + Grafana — metrics and monitoring

Where the Project Stands

Phase 3 is active. Here's what's complete:

✅ Full data ingestion pipeline — 229,498 papers fetched across 12 domains
✅ NER pipeline — 10.7M entities extracted, all 4 DVC pipeline stages reproducible
✅ Knowledge graph loaded — 1.5M entity nodes, 1.6M edges in Neo4j
✅ Embedding pipeline — 1,529,916 entities embedded and stored in Qdrant
✅ GraphRAG query engine — first cross-domain query working end-to-end
✅ FastAPI backend — JWT auth, refresh token rotation, async endpoints
✅ Next.js dashboard — 5 tabs, Sigma.js graph visualisation
✅ CI/CD — GitHub Actions running tests on every PR

GraphRAG query latency currently sits at ~32 seconds (p95): 1s embedding, 0.2s Qdrant, 12s Neo4j traversal, 14s Mistral synthesis. Optimisation is ongoing.

What Makes This Different from Vanilla RAG

Standard vector RAG — what most tools do — finds things that are already known to be related because they co-occur in text and are therefore close in embedding space.

Graph traversal finds things that are structurally connected through chains of evidence that no single document ever assembled.

When you ask "aerospace materials for cardiac implants", a vector search returns aerospace papers and cardiac papers ranked by similarity. The connection between a titanium alloy fatigue resistance paper and an aortic stenosis treatment paper — a connection that spans a 3-hop graph path — is invisible to pure vector search.

Part 3 will cover how Neo4j and Qdrant work together to solve this. For now: Qdrant gets you to the right door, Neo4j takes you through it.

Why Build in Public?

Two reasons.

First — writing forces clarity that coding alone doesn't. If I can't explain a decision, I probably don't understand it well enough.

Second — most build-in-public content shows polished results. I want to show the process: the architectural decisions, the dead ends, the production bugs, the tradeoffs. There have been plenty already — DVC broken by an fsspec version conflict, an embedding pipeline pagination bug that silently skipped entities, a Neo4j MERGE key design that had to be rethought mid-load. All of it will be documented here.

What's Coming in Part 2

Next post covers the NLP pipeline in detail — how spaCy and SciSpacy extract 10.7 million entities from scientific text, the 12 entity types and why they were chosen, and how 2.3 million relationships are assigned using a rule-based system designed around the intelligence modules the product needs to power.

Follow the series on Hashnode to get Part 2 when it drops.

*Questions about the architecture, stack choices, or anything else? Drop them in the comments