Architecture
How an OS event becomes a row in a provenance graph, and why each stage between the two exists.
%% ProvCollector — the data plane, agent to graph. Editable Mermaid source.
%% Preview at https://mermaid.live, in VS Code, or render with the Mermaid CLI:
%% npx -p @mermaid-js/mermaid-cli mmdc -i architecture.mmd -o architecture.svg
%% Keep this file free of colours. The palette is applied at render time from the
%% active colour scheme — see _includes/components/mermaid.html. Node appearance
%% is chosen with the semantic classes at the bottom of the file.
%% Two layout notes, both learned the hard way:
%% - never leave a bare "%%" line; Mermaid's comment stripper needs content
%% after the marker, so an empty one survives and breaks the parse;
%% - a subgraph's "direction" is ignored once it has edges crossing its border,
%% so keep subgraphs to things that are genuinely self-contained.
flowchart TB
agent["<b>1 · Agent</b><br/>Windows: ETW + optional driver · Linux: <i>TODO</i><br/>"]
raw(["<code>RawEvents</code> Kafka topic"])
subgraph ep ["EventProcessor"]
order["<b>2 · Reordering</b><br/>Events buffered per agent to improve temporal ordering"]
resolve["<b>3 · Identity resolution</b><br/>Transient descriptors (PID, path, etc.) → stable UUIDs"]
changelog[("<b>Identity store</b><br/>Minimal descriptor to UUID mapping<br/>Identities expire after 7d (configurable)")]
datareduction["<b>4 · Data reduction modules</b><br/>In-flight reduction of redundant events"]
resolve <--> changelog
order --> resolve
resolve --> datareduction
end
processed(["<code>ProcessedEvents</code> Kafka topic"])
subgraph streamconsumers ["Stream consumers"]
throughput["<b>ThroughputMonitor</b><br/>Live rates, read from the Kafka<br/>topics rather than the database"]
writer["<b>5 · Database.Postgres.Writer</b><br/>Bulk insert nodes & edges"]
end
postgres[("<b>PostgreSQL 18</b><br/>nodes & edges tables")]
subgraph dbconsumers ["Database consumers"]
tracker["<b>6 · Tracker</b><br/>Batch CLI<br/>NetworkX JSON"]
visualizer["<b>6 · Visualizer</b><br/>Blazor Server<br/>Sigma.js graph"]
reporter["<b>6 · BotReporter</b><br/>DB statistics<br/>Slack reports"]
end
agent --> raw
raw --> order
datareduction --> processed
processed --> writer
processed -.-> throughput
writer --> postgres
postgres -- "SQL · time-respecting traversal" --> tracker
postgres --> visualizer
postgres --> reporter
class raw,processed topic
class changelog,postgres store
- Service
- Kafka topic
- Durable store
EventProcessor middleware as separate stages with distinct
responsibilities; the database writer only simply materializes the resolved
graph into PostgreSQL. The final graph can be consumed from either the ProcessedEvents topic or the database.
Two planes
Two independent planes share the monitored host and nothing else.
The data plane carries provenance. It is one-way and asynchronous: agent to Kafka, to the processor, back to Kafka, to the writer, into PostgreSQL.
The management plane carries fleet control. The Updater runs as a second
service beside the agent and holds a SignalR connection to FleetServer;
operators drive it through AgentDashboard. No functionality in the management
plane is required to collect or analyze provenance data.
%% ProvCollector — the management plane. Editable Mermaid source; see the notes
%% at the top of architecture.mmd. Colours come from the active colour scheme.
flowchart LR
subgraph host ["Monitored host"]
agent["<b>ProvCollector.Agent</b><br/>Collects provenance.<br/>Knows nothing about the fleet."]
updater["<b>ProvCollector.Updater</b><br/>Heartbeat, config delivery,<br/>verified package install."]
end
raw(["RawEvents → the data plane"])
fleet["<b>FleetServer</b><br/>Agent hub + operator API<br/><i>X-Api-Key checked before registration</i>"]
dashboard["<b>AgentDashboard</b><br/>Blazor operator UI<br/><i>no state of its own</i>"]
packages[("<b>Packages</b><br/>SHA-256 digest per package,<br/>verified before install")]
agent --> raw
updater <-- "SignalR · heartbeat, commands, logs" --> fleet
fleet <-- "HTTP · operator API" --> dashboard
fleet --- packages
class raw topic
class packages store
class agent dataplane
class updater,fleet,dashboard management
- Data plane
- Durable store
- Management plane
RawEvents, the processed stream or the database, so an operator
with dashboard access cannot read collected provenance — and the collection
path, which holds a write-only Kafka credential, cannot issue installer
instructions.
Keeping them apart means an operator with dashboard access cannot read collected provenance, and a compromised collection path cannot issue installer instructions. Security model covers the boundary in detail.
The data pipeline
| # | Stage | Component | What it does |
|---|---|---|---|
| 1 | Collection | ProvCollector.Agent |
OS telemetry → ProvenanceEvent → RawEvents |
| 2 | Ordering | ProvCollector.EventProcessor |
Per-agent event reordering |
| 3 | Identity resolution | ProvCollector.EventProcessor |
Transient descriptors → stable UUIDs |
| 4 | Data reduction | ProvCollector.EventProcessor |
Drop/merge redundant events |
| 5 | Storage | Database.Postgres.Writer |
ProcessedEvents → nodes / edges tables |
| 6 | Analysis | Tracker, Visualizer, BotReporter |
Graph traversal, visualization, etc. |
Each stage is modular: collection sources are pluggable, data reduction strategies can be customized, and the database writer can be replaced with alternative graph stores.
The pipeline explains each stage in further detail.
In this section
-
Each of the six stages, what it guarantees, and the failure it is defending against.
-
The protobuf wire contract, the identity resolution rules, the changelog, and the storage schema.
-
The boundary between the planes, the per-role Kafka ACLs, and credential handling at rest and in flight.