Serverless Data Lineage &
Enterprise Knowledge Graph
LineagIQ unifies fragmented data pipelines into an active, queryable semantic knowledge graph on Delta Lake. Detect point-in-time schema drift, prevent breaking pipeline changes before deployment, and deliver context-rich copilots for your data team—with zero dedicated graph database infrastructure.
Eliminate Silent Data Drift With Time-Travel Diffing
Data pipelines fail silently when upstream tables alter column types, rename keys, or drop fields unannounced. LineagIQ captures every structural revision and dependency shift across time through Delta Lake's ACID transactional commit logs.
-
Continuous State Reconstruction: Scrub through intuitive timeline controls to recreate and inspect the precise topological state of your data assets at any past timestamp or commit version.
-
Automated Topology Diffing (T₁ → T₂): Instantly isolate additions, removals, column mutations, and rerouted lineage dependencies between any two arbitrary points in time.
-
Regulatory Audit Readiness: Fulfill stringent financial, governance, and privacy mandates (BCBS 239, SOX, GDPR) with immutable historical proof of upstream provenance and data lineage.
Instant Blast Radius & Root-Cause Attribution
Before deploying schema migrations or refactoring dbt models, LineagIQ computes downstream blast radius in milliseconds—preventing downstream data incidents and broken executive dashboards.
-
Recursive Downstream Traversal: Discover every indirect dependency across data warehouses, feature stores, ML models, reverse-ETL jobs, and BI dashboards across arbitrary graph depths.
-
Upstream Root-Cause Isolation: When an executive KPI or dashboard metric diverges, trace backward through marts, transformations, and ingestion pipelines to uncover the failure source in seconds.
-
Targeted Stakeholder Notification: Map asset ownership, domain boundaries, and downstream business consumers directly onto impacted subgraphs to orchestrate targeted incident response.
Autonomous AI Assistant for Architectural Governance
LineagIQ bridges graph intelligence with generative AI through Graph-Augmented Generation (GraphRAG). Ground your LLMs with deep structural lineage context to eliminate hallucinations during architectural audits and incident investigations.
-
Private Local & Cloud LLM Flexibility: Run completely offline with local Ollama instances (Llama 3, Mistral, Qwen) for zero network egress, or connect directly to cloud APIs (OpenAI GPT-4o, Google Gemini).
-
Automated Impact Synthesis: Feed time-travel schema diffs directly into the AI Assistant to synthesize executive incident summaries, risk scores, and remediation playbooks on demand.
-
Standardized Agentic Tooling: Out-of-the-box Python retriever functions for LangChain, AutoGen, and LlamaIndex to query blast radius and historical lineage programmatically.
Inject Any Enterprise Data Source Directly Into the Collection Agent
The LineagIQ Collection Agent operates as a lightweight, stateless utility inside your secure VPC, CI/CD pipeline, or Kubernetes cluster. It ingests schema definitions, warehouse catalogs, audit logs, and runtime orchestrator events—transforming raw operational signals into unified graph models and semantic vector embeddings without ever touching raw customer record data.
dbt Core & dbt Cloud
Transformation DAGs & Schemas
Directly ingests compilation artifacts (manifest.json and catalog.json). Extracts data models, seeds, snapshots, source declarations, column-level documentation, tags, and tests, compiling the complete transformation DAG into unified DERIVED_FROM dependency edges.
python -m collection_agent.src.cli --dbt-manifest <path> --dbt-catalog <path>
SQL Catalogs & Schemas
Warehouse Metadata & Relational DDL
Parses standard INFORMATION_SCHEMA.TABLES, COLUMNS, and foreign key constraint definitions. Extracts relational schemas, column data types, ordinal positions, and physical JOINS_WITH relationships across relational databases and cloud warehouses.
python -m collection_agent.src.cli --sql-schema <path_to_schema.json>
SQL Query Logs & Audit History
Access Auditing & Runtime Join Mining
Analyzes warehouse audit histories (Snowflake QUERY_HISTORY, BigQuery INFORMATION_SCHEMA.JOBS, Databricks Query History). Mines actual user and service dataset consumption patterns (CONSUMED_BY), query execution frequencies, and implicit multi-table join predicates.
python -m collection_agent.src.cli --query-logs <path_to_logs.json>
OpenLineage Standard Events
Orchestration & Compute Lineage
Ingests standardized OpenLineage JSON run events emitted by workflow orchestrators and distributed compute engines. Seamlessly maps job run IDs, execution timestamps, input dataset consumption edges (CONSUMED_BY), and output production edges (PRODUCED_BY).
python -m collection_agent.src.cli --openlineage <path_to_events.json>
The "Slim Stack" Revolution: Why No Dedicated Graph DB?
Legacy enterprise data catalogs mandate dedicated Neo4j, TigerGraph, or Amazon Neptune clusters—burdening teams with persistent infrastructure bills, JVM memory management, and specialized graph query skills. LineagIQ re-architects graph intelligence from first principles.
Collibra / Alation / Neo4j Stack
- Heavyweight Infrastructure: 24/7 provisioned JVM graph nodes (Neo4j Enterprise, Amazon Neptune) requiring massive RAM and continuous tuning.
- Network & Perimeter Risk: Often requires granting third-party SaaS vendors persistent inbound network tunnels or broad warehouse read permissions.
- Destructive Overwrite Semantics: Traditional graph databases overwrite properties in-place, making historical state recreation cumbersome, slow, or impossible.
- Proprietary Graph Lock-in: Lineage topologies trapped in niche graph formats and query dialects (Cypher, Gremlin) isolated from standard lakehouse tools.
Serverless S3 Delta + Embedded In-Memory Engine
- Serverless Storage Economics: Topologies persist directly on commodity object storage (Amazon S3, Google Cloud Storage, or local disk) in open Delta Lake Parquet format.
- Ephemeral Zero-Egress Collection: Collection agents run as lightweight batch jobs inside your VPC or CI/CD pipelines. Operational customer data never leaves your trust boundary.
- Native ACID Time-Travel Replay: Delta Lake's transactional commit logs capture every lineage revision automatically with zero storage bloat and full historical reproducibility.
- High-Performance Embedded Graph Engine: Evaluates multi-hop lineage and blast radius in microseconds (< 25 µs) with an 18.6x reduction in memory footprint—running enterprise data topologies seamlessly on standard serverless containers.
Hardware Sizing vs. Graph Scale (Nodes & Edges)
Unlike traditional graph databases that require massive dedicated memory and multi-node clusters, LineagIQ processes large enterprise graph topologies with virtually zero infrastructure footprint. Run comprehensive data lineage with sub-millisecond query responses on basic, low-cost serverless containers.
| Scale Tier | Graph Topology (Nodes & Edges) | In-Memory Cache Footprint | Recommended Hardware | Recursive Traversal Latency | Legacy JVM Graph Equivalent |
|---|---|---|---|---|---|
| Mid-Market | 50,000 nodes • 150,000 edges | ~1.5 MB |
0.25 vCPU, 512 MB RAM (e.g., ~$5/mo) | < 0.05 ms (20 µs) | 16 GB RAM Neo4j (~$300/mo) |
| Enterprise | 250,000 nodes • 1,200,000 edges | ~12 MB |
0.5 vCPU, 1 GB RAM (e.g., ~$10/mo) | < 0.15 ms (150 µs) | 32 GB RAM Neptune (~$850/mo) |
| Large Enterprise | 1,000,000 nodes • 5,000,000 edges | ~50 MB |
1 vCPU, 2 GB RAM (e.g., ~$20/mo) | < 0.50 ms | 64 GB RAM HA Cluster (~$2,200/mo) |
| Hyperscale Global | 5,000,000+ nodes • 25,000,000+ edges | ~250 MB |
2 vCPU, 4 GB RAM (e.g., ~$45/mo) | < 2.50 ms | 128 GB+ Distributed Cluster ($5,000+/mo) |
The Three-Tier Decoupled Architecture
Engineered for total data sovereignty, ironclad perimeter security, and sub-millisecond operational performance. LineagIQ cleanly separates edge metadata collection, open lakehouse storage, and serverless intelligence—eliminating proprietary database lock-in and persistent cluster costs.
Stateless Collection Agent
Runs as an ephemeral batch task inside your VPC, Kubernetes cluster, or CI/CD runner. Harvests schema definitions, manifests, and query logs with zero outbound data egress—customer PII and raw records never leave your network.
Open Delta Lake Storage
Stored directly in your own cloud object storage (Amazon S3, Google Cloud Storage, or Azure Blob). Full ACID transaction guarantees, snappy columnar Parquet tables, and native point-in-time time travel replay—perpetual ownership without vendor lock-in.
Serverless Intelligence Plane
High-performance query service executing multi-hop lineage reachability and blast-radius attribution in sub-milliseconds over Delta Lake snapshots. Powers interactive timeline scrubbing, CI/CD deployment gates, and context-grounded AI copilots.
Sovereign Edge Ingestion: Private, Ephemeral & Frictionless
Traditional data governance vendors demand intrusive VPN tunnels, static database superuser passwords, or direct inbound network access into production warehouses. LineagIQ reverses this operational paradigm:
- Runs Within Your Perimeter: The collection agent executes as a stateless CLI task, Docker container, or Kubernetes CronJob inside your private network.
-
Pure Structural Metadata Extraction: Inspects compilation artifacts (dbt
manifest.json), data catalog catalogs, and audit logs. Sensitive customer rows, PII, and financial records are never inspected or stored. - Universal Ecosystem Compatibility: Seamlessly hooks into existing pipelines—orchestrated via Airflow DAGs, dbt Cloud webhooks, or automated GitHub Actions on pull requests.
Open Lakehouse Storage: 100% Data Ownership & Audit Replay
Instead of locking your critical dependency graphs inside opaque, proprietary graph databases that cost thousands per month, LineagIQ stores all metadata in open Delta Lake tables on your own cloud bucket:
- Zero Vendor Lock-In: Metadata is written in open Apache Parquet format. Query it anytime with standard data engineering engines—DuckDB, Apache Spark, Trino, or Python pandas.
- Immutable ACID Audit Log: Every metadata update is an atomic commit. Never worry about corrupt state or unrecorded schema drift—every table revision is preserved for regulatory audits (SOX, BCBS 239).
- Point-in-Time Time Travel: Replay the historical topology of your entire data warehouse at any past timestamp to verify data provenance, investigate past incidents, or satisfy compliance checks.
_delta_log
Snappy Parquet
Zero Licensing Fees
Serverless Intelligence Plane: Rapid Triage, CI/CD Gates & AI Copilots
The lightweight control plane transforms static lineage snapshots into active operational guardrails for your engineering team:
- Microsecond Reachability Traversals: Evaluate multi-hop upstream root cause and downstream blast radius across 50,000+ nodes in sub-milliseconds with an 18.6x memory reduction.
- Automated CI/CD Breaking Change Gates: Block risky PRs before merging by checking if a modified model or deleted column impacts downstream executive dashboards or ML models.
- Hallucination-Free AI Assistant: Feed authoritative graph topology and time-travel diffs directly to internal LLMs (OpenAI, Gemini, or private local Ollama) for instant natural-language incident triage.
Embed Lineage Intelligence Directly Into Your CI/CD & AI Workflows
Evaluate blast radius, detect breaking schema drift, and gate PRs automatically using clean REST API calls from GitHub Actions, GitLab CI, Airflow, or AI agents:
import requests, sys
# 1. Query downstream blast radius for a modified data asset
res = requests.post("https://plane.lineagiq.com/api/v1/blast-radius", json={
"node_id": "postgres.raw.raw_customers",
"max_depth": 5
}).json()
# 2. Block PR merge if critical downstream assets are affected
critical = [n["name"] for n in res["impacted_nodes"] if n.get("properties", {}).get("tier") == "executive_gold"]
if critical:
sys.exit(f"❌ CI/CD Gate Blocked: {len(critical)} critical downstream assets affected: {critical}")
print(f"✅ Lineage Gate Passed: {len(res['impacted_nodes'])} downstream dependencies verified safe.")Quick Start: Run LineagIQ in 60 Seconds
Pull pre-built, production-ready multi-architecture containers directly from GitHub Container Registry. No local Python build or cloud setup required.
ghcr.io/timor-dataworks/lineagiq-control-plane:latest
ghcr.io/timor-dataworks/lineagiq-collection-agent:latest
Generate or Ingest Metadata
Run the ephemeral agent to generate a 3-version historical lineage evolution dataset (or point to your own dbt manifests / SQL catalogs).
docker run --rm \
-v $(pwd)/data:/data \
ghcr.io/timor-dataworks/lineagiq-collection-agent:latest \
--multiversion-demo./data (Delta Lake tables: nodes, edges, vectors, commit logs).
Launch Serverless Control Plane
Start the query runtime. LineagIQ mounts your Delta Lake directory and immediately serves the interactive visualizer and REST APIs.
docker run -d --name lineagiq-control-plane \
-p 8000:8000 \
-e DATA_PATH=/data \
-v $(pwd)/data:/data \
ghcr.io/timor-dataworks/lineagiq-control-plane:latesthttp://localhost:8000 with sub-second response times.
Launch with Docker Compose
Spin up the complete pipeline (demo dataset generation + serverless control plane) with a single command:
# Download docker-compose.yml and start stack
curl -sSL https://raw.githubusercontent.com/timor-dataworks/LineagIQ/main/docker-compose.yml \
-o docker-compose.yml
docker compose run --rm demo_generator
docker compose up -d control_planeEverything You Need to Know About LineagIQ
Got questions about our serverless architecture, time-travel lineage diffing, CI/CD governance gates, or AI copilots? Find answers below.
What is LineagIQ and how does it provide data lineage without a graph database?
LineagIQ is a serverless data lineage and enterprise semantic knowledge graph platform. Unlike traditional data catalogs (Collibra, Alation) that mandate provisioned Neo4j or Amazon Neptune clusters, LineagIQ persists topology datasets directly on open Delta Lake Parquet tables in cloud object storage (Amazon S3, Google Cloud Storage, or local disk).
At query runtime, LineagIQ loads these open tables into an ultra-fast, zero-overhead in-memory graph index, delivering sub-millisecond lineage traversals without persistent cluster overhead, JVM memory tuning, or 6-figure enterprise license fees.
How fast are lineage queries and impact analyses with LineagIQ?
By avoiding heavyweight graph database servers, network hops, and object overhead, LineagIQ resolves multi-hop upstream and downstream dependency queries in under 25 microseconds (< 25 µs).
In extensive benchmarks, this architecture delivers an 18.6x reduction in memory footprint compared to legacy catalog solutions, allowing 50,000+ data assets to be analyzed effortlessly within lightweight serverless containers or standard CI/CD runners.
What data sources and pipeline integrations does LineagIQ support?
LineagIQ features native, turnkey ingestion adapters for:
• dbt Core & dbt Cloud: Direct ingestion of compiled manifest.json execution artifacts.
• SQL Dialect Catalogs: Snowflake, BigQuery, Databricks, PostgreSQL, and DuckDB INFORMATION_SCHEMA.
• Query History & Audit Logs: AST-based SQL query log parsing for dynamic runtime lineage inference.
• OpenLineage Standard: Ingestion of JSON run events from Apache Airflow, Apache Spark, Dagster, and Apache Flink.
How does Delta Lake Time-Travel enable historical lineage diffing?
Delta Lake maintains an append-only JSON commit log (_delta_log) that records transactional snapshots over time with full ACID guarantees. LineagIQ leverages this protocol for zero-copy time travel.
Data teams can compare graph topologies at commit T₁ against commit T₂ to instantly detect silent schema drift, removed columns, added dependencies, or altered transformation logic—providing immutable audit trails for BCBS 239, SOX, and GDPR compliance.
How does LineagIQ calculate downstream blast radius?
When any dataset or column is modified or reviewed in a pull request, LineagIQ traverses the full dependency chain downstream in real time.
It isolates every affected data mart, downstream machine learning model, reverse-ETL connector, and executive BI dashboard—giving engineering teams clear visibility into the impact before changes reach production.
How does LineagIQ empower AI agents and teams with lineage context?
LineagIQ provides structured semantic context and graph relationships directly to your engineering team and AI assistants.
When investigating incidents (e.g., "What broke the executive churn dashboard?"), LineagIQ pinpoints the exact upstream pipeline failure and delivers authoritative context to OpenAI, Claude, Gemini, or private local models (Ollama) without hallucination.
How is LineagIQ deployed in production or local environments?
LineagIQ provides pre-built, production-ready multi-architecture container images from GitHub Container Registry (GHCR):
• Control Plane: ghcr.io/timor-dataworks/lineagiq-control-plane:latest
• Collection Agent: ghcr.io/timor-dataworks/lineagiq-collection-agent:latest
You can launch the entire stack in under 60 seconds with docker compose up -d, or run the ephemeral agent as a Kubernetes CronJob inside your private VPC.