Next-generation system provenance data collection

ProvCollector

A distributed platform for collecting, storing and querying whole-system provenance. Agents on monitored hosts turn OS telemetry into a uniform event stream; a backend resolves that stream into stable object identities and writes it into a partitioned PostgreSQL provenance graph; analysis tools query, visualise and report on the result; a management plane installs, configures and updates the agent fleet.

Get started Read the architecture Source

Introduction

System provenance is a record of the causal dependencies between the objects an operating system manages. Processes, files, sockets, and other system objects form the vertices of a directed graph; each operation observed between two of them (i.e. a process creating another process, a process writing to a file, a process receiving a network packet) forms a directed edge representing the flow of data. This graph-structured log data enables tracing causal flows forward in time, to uncover the downstream effects of a system event, or backward, to find the root cause of an event.

ProvCollector implements the collection and storage layer of a system provenance pipeline. The ProvCollector agent translates diverse operating system telemetry into a uniform event representation. A streaming backend resolves transient descriptors (e.g. PID, file path) to persistent object identifiers, and materialises the resulting graph into a database. Alongside the data collection pipeline, we include analysis tools to conduct interactive graph exploration, extract subgraphs via backtracking and forwardtracking, and monitor deployment health. ProvCollector also provides a management plane to simplify the deployment, configuration, and maintenance of the agent across a large fleet of hosts.

Provenance-based intrusion detection systems (PIDSs) can consume the ProvCollector graph by utilizing the database for historical analysis, or subscribing to the Kafka stream for live detection.

Architecture at a glance

%% ProvCollector — the data plane, agent to graph. Editable Mermaid source.
%% Preview at https://mermaid.live, in VS Code, or render with the Mermaid CLI:
%%   npx -p @mermaid-js/mermaid-cli mmdc -i architecture.mmd -o architecture.svg
%% Keep this file free of colours. The palette is applied at render time from the
%% active colour scheme — see _includes/components/mermaid.html. Node appearance
%% is chosen with the semantic classes at the bottom of the file.
%% Two layout notes, both learned the hard way:
%%   - never leave a bare "%%" line; Mermaid's comment stripper needs content
%%     after the marker, so an empty one survives and breaks the parse;
%%   - a subgraph's "direction" is ignored once it has edges crossing its border,
%%     so keep subgraphs to things that are genuinely self-contained.

flowchart TB

  agent["<b>1 · Agent</b><br/>Windows: ETW + optional driver · Linux: <i>TODO</i><br/>"]

  raw(["<code>RawEvents</code> Kafka topic"])

  subgraph ep ["EventProcessor"]
    order["<b>2 · Reordering</b><br/>Events buffered per agent to improve temporal ordering"]
	resolve["<b>3 · Identity resolution</b><br/>Transient descriptors (PID, path, etc.) → stable UUIDs"]
	changelog[("<b>Identity store</b><br/>Minimal descriptor to UUID mapping<br/>Identities expire after 7d (configurable)")]
    datareduction["<b>4 · Data reduction modules</b><br/>In-flight reduction of redundant events"]
	resolve <--> changelog
    order --> resolve
    resolve --> datareduction
  end


  processed(["<code>ProcessedEvents</code> Kafka topic"])

  subgraph streamconsumers ["Stream consumers"]
    throughput["<b>ThroughputMonitor</b><br/>Live rates, read from the Kafka<br/>topics rather than the database"]
    writer["<b>5 · Database.Postgres.Writer</b><br/>Bulk insert nodes & edges"]
  end


  postgres[("<b>PostgreSQL 18</b><br/>nodes & edges tables")]

  subgraph dbconsumers ["Database consumers"]
	tracker["<b>6 · Tracker</b><br/>Batch CLI<br/>NetworkX JSON"]
	visualizer["<b>6 · Visualizer</b><br/>Blazor Server<br/>Sigma.js graph"]
	reporter["<b>6 · BotReporter</b><br/>DB statistics<br/>Slack reports"]
  end
  
  agent --> raw
  raw --> order
  datareduction --> processed
  processed --> writer
  processed -.-> throughput
  writer --> postgres
  postgres -- "SQL · time-respecting traversal" --> tracker
  postgres --> visualizer
  postgres --> reporter

  class raw,processed topic
  class changelog,postgres store

• Service
• Kafka topic
• Durable store

Figure 1. The data plane as a set of independently replaceable stages, joined by Kafka topics rather than direct calls. The agent only produces events; reordering and identity resolution run inside the EventProcessor middleware as separate stages with distinct responsibilities; the database writer only simply materializes the resolved graph into PostgreSQL. The final graph can be consumed from either the ProcessedEvents topic or the database.

The Architecture section describes each stage in more detail.

Where to go next

• Getting started

  Clone the superproject, build the solution, bring up the Compose stack, and get an agent reporting into it.

• Architecture

  The pipeline stage by stage, the wire contract and identity model, and the security boundary between the two planes.

• Components

  What lives in each of the five repositories, and which project does what inside them.

• Operations

  The Compose topology, Kafka topics and ACLs, configuration precedence, secrets, CI and installer builds.

• Reference

  Tracker CLI options, event and object type registries, and the settings that matter in production.

• FAQ

  The questions that come up most: coverage gaps, retention, why PIDs are not identities.