Private Enrichment
PacketSnitch normally enriches a capture by asking a public service: GeoIP from the bundled MaxMind database, WHOIS from RDAP, IP reputation from IPSum, exit nodes from the Tor Project, open ports from Shodan InternetDB, ASN from HackerTarget, and file/IP reputation from VirusTotal.
Many operators cannot send an indicator to any of those. A government network, a regulated enterprise, or an air-gapped SOC usually owns a better database than the public one — and is forbidden from using the public one at all.
Private enrichment closes that gap. You load a JSON bundle of the metadata you already hold, and the backend serves it ahead of any public upstream. In local-only mode it never makes an outbound request at all.
Everything is reachable through one deliberately small driver: scripts/ps-enrich.py. It is a single file with no dependencies — standard library only — because it has to run on a jump host inside your network with nothing but the system Python.
Table of Contents
- Quick Start
- The Three Modes
- The Bundle Format
- Supported Services
- Target Types and CIDR Blocks
- Partial Records
- Field Aliases
- Policy: Authoritative vs Supplementary
- Expiry
- The Driver
- HTTP API
- Capture-Time Enrichment
- Where the Data Lives
- Worked Example: A Private GeoIP and Tor Feed
- Worked Example: Feeding an Internal VirusTotal Replacement
- Operational Notes
- Troubleshooting
Quick Start
# 1. Check the bundle parses and every record is admissible (writes nothing)
python3 ps-enrich.py validate --bundle acme-ti.json
# 2. Load it (idempotent — re-running with the same content changes nothing)
python3 ps-enrich.py ingest --bundle acme-ti.json
# 3. Prove it is being served
python3 ps-enrich.py lookup --service geoip --target 203.0.113.9
# 4. Lock the install down so no public service is ever contacted
python3 ps-enrich.py mode --mode local-only
# 5. Export exactly what the backend holds (a backup, or a diff basis)
python3 ps-enrich.py export --out acme-ti.export.json
The driver talks to the backend’s HTTP bridge, which PacketSnitch already runs. The default address is http://127.0.0.1:9020; override it with --base-url or the PACKETSNITCH_BACKEND_URL environment variable.
The Three Modes
The mode decides what happens when a lookup arrives. It is a global setting with optional per-service overrides.
| Mode | A matching record exists | No record matches |
|---|---|---|
local-first (default) | served from your database, no network call | the public service is queried, exactly as before |
local-only | served from your database | supported: false + skippedReason: "not-in-local-dataset"; no network call |
remote-only | your database is ignored entirely | the public service is queried |
remote-only is the escape hatch for an install that wants the store present but inactive — every payload is byte-identical to a build without this feature.
Two consequences are worth stating plainly:
local-firstis not a cache. An authoritative record means the answer, not a hint. If your database says an address is in Reston, no public service is asked and no public service can disagree.local-onlyis a guarantee, not a preference. The check happens before the network request is constructed, so a miss cannot fall through to the internet. It also skips the public-response cache, so an install that was online before cannot leak a stale public answer afterwards.
Mode changes take effect on the next lookup. No restart is needed.
# Global
python3 ps-enrich.py mode --mode local-only
# Per service: keep VirusTotal public, everything else private
python3 ps-enrich.py mode --mode local-only --service-mode virustotal=remote-only
# Remove an override so a service inherits the global mode again
python3 ps-enrich.py mode --service-mode virustotal=inherit
# Read the current configuration
python3 ps-enrich.py mode
The Bundle Format
A bundle is one JSON document:
{
"schema": "packetsnitch.enrichment/v1",
"provider": {
"id": "acme-ti",
"name": "ACME Threat Intelligence",
"url": "https://ti.acme.internal",
"license": "internal-only"
},
"records": [
{
"service": "geoip",
"target": { "type": "cidr", "value": "203.0.113.0/24" },
"data": { "country": "United States", "city": "Reston",
"latitude": 38.9586, "longitude": -77.3570 }
}
]
}
| Key | Required | Meaning |
|---|---|---|
schema | recommended | packetsnitch.enrichment/v1. Any other value is rejected, so a future format cannot be silently misread. |
provider.id | recommended | Filesystem-safe bundle name. This is how a re-ingest finds the bundle it should merge into. Defaults to default. |
provider.name / .url / .license | optional | Shown in the payload’s enrichment.providers list, so an analyst can see whose data an answer came from. |
records | yes | Array of records (see below). |
replace | optional | true replaces the provider’s records instead of merging into them. |
updatedAt | optional | Free-form timestamp carried through to the status report. |
A record:
| Key | Required | Meaning |
|---|---|---|
service | yes | geoip, whois, ipsum, tor, shodan, asn, virustotal — or any for the multi-service form. |
target | yes | A string, or { "type": ..., "value": ... }. The type may be omitted and is inferred from the value. |
data | yes | The payload. Every field inside it is optional. |
policy | optional | authoritative (default) or supplementary. See Policy. |
provider | optional | Per-record provider override; useful when one export mixes sources. |
expiresAt / ttlSeconds | optional | When this record stops being served. |
observedAt | optional | When the fact was true — the anchor ttlSeconds counts from. |
confidence | optional | 0.0–1.0; reported in the annotation, and the minimum across merged records wins. |
notes | optional | Free text carried through to the export. |
Accepted shorthand keys for data are fields, payload and values, so an export script written against a different vocabulary usually works unchanged.
A bundle may also be a bare array of records, a single record ({"service": ..., "target": ..., "data": ...}), or NDJSON (one record per line, blank lines and # comments ignored). NDJSON is what you want for a large export: a streaming job can write it out without building one giant JSON array.
Rejected records do not discard the bundle
A 200,000-row export from a private database will eventually contain one typo. That typo rejects that record, not the run:
{
"success": true,
"accepted": 199999,
"rejected": [{ "index": 91204, "reason": "record 91204: invalid IP address '10.0.0.300'" }]
}
Use --strict in a pipeline to turn any rejection into a non-zero exit code.
Supported Services
Each service accepts the subset of fields it can render. Everything else is preserved verbatim and comes back out in ps-enrich export, so a round trip is lossless.
GeoIP
Replaces or supplements the bundled MaxMind database, both for on-demand lookups and — importantly — for every packet in the capture (see Capture-Time Enrichment).
{ "service": "geoip", "target": "203.0.113.9",
"data": { "country": "United States", "countryCode": "US", "region": "Virginia",
"city": "Reston", "postalCode": "20190",
"latitude": 38.9586, "longitude": -77.3570,
"timezone": "America/New_York", "isp": "ACME", "asn": "AS64500" } }
Aliases: lat, lon/lng, zip, tz, country_code, organisation.
WHOIS / RDAP
{ "service": "whois", "target": "203.0.113.9",
"data": { "isp": "ACME Corporation", "netName": "ACME-NET",
"netType": "DIRECT ALLOCATION", "parent": "NET-203-0-0-0-0",
"registrationDate": "2011-04-05T00:00:00Z",
"updatedDate": "2025-01-02T00:00:00Z",
"rangeStart": "203.0.113.0", "rangeEnd": "203.0.113.255",
"cidr": "203.0.113.0/24", "rirHost": "rdap.arin.net" } }
IPSum (IP reputation / block listing)
{ "service": "ipsum", "target": "203.0.113.9", "data": { "hits": 7 } }
Only hits is needed. The A–F grade badge is this build’s banding of that count (the same bands the public feed uses), not something you have to compute.
Tor exit nodes
{ "service": "tor", "target": { "type": "cidr", "value": "203.0.113.0/24" },
"data": { "isExitNode": true, "nickname": "acmeexit", "platform": "Tor 0.4.8.10" } }
isExitNode: false is meaningful: it marks an address as not an exit node, so a local-only install can answer the question rather than reporting a miss. Exit-node records reach the capture pipeline, so tor.exit.node is set on packets during analysis without any network access.
Shodan InternetDB
{ "service": "shodan", "target": "203.0.113.9",
"data": { "ports": [22, 443, 8443], "hostnames": ["vpn.acme.example"],
"cpes": ["cpe:/a:openbsd:openssh:9.6"], "tags": ["vpn"],
"vulns": ["CVE-2024-0001"] } }
ASN
{ "service": "asn", "target": "203.0.113.9",
"data": { "asn": "AS64500", "asnName": "ACME-AS, US",
"organization": "ACME Corporation", "domain": "acme.example",
"asnRange": "203.0.113.0 - 203.0.113.255" } }
The ISO country code is derived from the trailing , XX in asnName when you do not supply countryCode — the same rule the public path uses.
VirusTotal
Keyed by IP, CIDR, domain, URL or hash. Only the counters are needed.
{ "service": "virustotal", "target": { "type": "hash", "value": "44d88612fea8a8f36de82e1278abb02f" },
"data": { "malicious": 58, "suspicious": 0, "harmless": 0, "undetected": 3,
"reputation": -90, "meaningfulName": "eicar.com",
"typeDescription": "EICAR Test File", "tags": ["test"] } }
Shorthand counters (malicious, suspicious, harmless, undetected, timeout, failure, confirmed_timeout, type_unsupported) are collected into the analysis block the UI renders.
Multi-service records
When one indicator carries several kinds of metadata — the normal case — write one record with "service": "any":
{ "service": "any", "target": "203.0.113.9",
"data": {
"shodan": { "ports": [443, 8443], "tags": ["vpn"] },
"asn": { "asn": "AS64500", "organization": "ACME" },
"whois": { "netName": "ACME-NET", "isp": "ACME Corporation" },
"ipsum": { "hits": 7 },
"tor": { "isExitNode": false },
"virustotal": { "malicious": 4, "reputation": -11 },
"geoip": { "country": "United States", "city": "Reston" }
} }
This expands into one record per service at ingest, so a single target entry feeds every surface at once.
Target Types and CIDR Blocks
| Type | Example | Matched by |
|---|---|---|
ip | 203.0.113.9, 2001:db8::1 | exact address |
cidr | 203.0.113.0/24, 2001:db8::/32 | any address inside the block |
domain | vpn.acme.example | exact name (VirusTotal only) |
url | https://acme.example/x | exact URL (VirusTotal only) |
hash | MD5 / SHA-1 / SHA-256 / SHA-512 | exact digest (VirusTotal only) |
The type is inferred when you omit it. CIDR blocks are the important part: a single /8 describes sixteen million addresses in one record, and the matcher always prefers the most specific block. That is how a private database can say “everything in this range is ours” without emitting a row per address.
When several records match one address, the more specific block wins, and an exact host record beats any block. Records from different providers at the same specificity are merged, so two teams can each contribute part of the picture.
Partial Records
Every field is optional. A private database rarely knows everything a public one does, and this feature is designed around that reality rather than against it.
A record that supplies only what it knows is accepted, and the response says so:
"enrichment": {
"servedFrom": "local",
"providers": [{ "id": "acme-ti", "name": "ACME Threat Intelligence" }],
"partial": true,
"missingFields": ["analysis.malicious"]
}
partial means “an essential field for this surface is absent” — for VirusTotal that is the malicious count, for GeoIP the country or city, and so on. The field still appears in the payload with its neutral default (0, [], "Unknown"), so the UI renders normally; partial is what tells a reader the difference between “we checked and it is zero” and “we do not know”.
Field Aliases
The bundle accepts the short names an operator is likely to write as well as the canonical ones. The authoritative list is served by the backend — never guess:
python3 ps-enrich.py schema
That prints every service, its target types, its accepted aliases, its essential fields and the current limits, generated from the same table the ingester uses. A field this build does not recognise is preserved, not dropped, so a newer schema can add fields and an older build will still round-trip them.
Policy: Authoritative vs Supplementary
| Policy | local-first behaviour | Use when |
|---|---|---|
authoritative (default) | served without a network call | your database is the truth for this indicator |
supplementary | the public service is queried and your fields are merged into its answer | you have an extra field, or a local annotation, to add |
A merge never replaces a scalar the public answer already has unless the local record sets it explicitly, and lists are replaced rather than unioned — an authoritative record that removes a false positive has to be able to remove it.
If any matching record is authoritative, the whole answer is served locally.
Expiry
Indicators go stale. A record can say when it stops being true:
{ "service": "virustotal", "target": "203.0.113.9",
"data": { "malicious": 4 }, "ttlSeconds": 86400 }
ttlSeconds counts from observedAt (or from now, when observedAt is absent) and is converted to an absolute expiresAt. An expired record is not served — it is skipped exactly as if it had never been ingested, and the count of dropped records appears in ps-enrich status as expiredDropped.
The Driver
ps-enrich [--base-url URL] [--timeout SECONDS] COMMAND
| Command | What it does |
|---|---|
status | What the backend holds, and which mode it is in. |
schema | The machine-readable bundle contract, straight from the backend. |
validate | Check a bundle without writing. Falls back to an offline structural preview when the backend is unreachable. |
ingest | Validate and store. Idempotent. |
lookup | Ask for one enriched target — the fastest way to confirm the store is being served. |
mode | Read or set the global/per-service mode. |
export | Everything the backend holds, as a bundle you can re-ingest. |
clear | Delete one provider, or everything with --all. |
Useful flags:
--provider ID— set the bundle id when the bundle does not name one.--replace— replace the provider’s records rather than merging.--strict— exit3if any record was rejected.--require-local(onlookup) — exit3unless the answer came from your store. This is the assertion to put in a smoke test.--json— print the raw report.
Exit codes: 0 success, 1 usage/input error, 2 the backend was unreachable or answered non-2xx, 3 records were rejected (with --strict) or a --require-local lookup was served from somewhere else.
HTTP API
The driver is a thin client over endpoints you can also call directly from your own tooling.
| Method | Path | Purpose |
|---|---|---|
GET | /enrichment/status | counts, providers, modes, limits |
GET | /enrichment/schema | the bundle contract |
GET | /enrichment/export?service=&provider= | everything held, as a bundle |
POST | /enrichment/validate | { "bundle": ... } — check without writing |
POST | /enrichment/ingest | { "bundle": ..., "providerId"?, "replace"? } |
POST | /enrichment/config | { "mode": ..., "services": {...} } |
POST | /enrichment/clear | { "providerId": ... } or { "all": true } |
POST | /control | { "action": "set-runtime-config", "enrichment": {...} } |
Every enrichment lookup payload carries an enrichment block naming the mode, the serving path (local, merged, bundled or none), the providers and the partiality. A remote-only install emits no such block at all.
Capture-Time Enrichment
Two surfaces reach the packets, not just the on-demand lookups — which is what makes the private data visible in the Stats map, the filters and the AI summary:
GeoIP. A matching record supplies every
loc.src.*/loc.dst.*field the capture pipeline reads, so a private GeoIP database drives the heatmap andloc.*filters exactly like MaxMind does. The record is markedEnrichment: "local"(andloc.<side>.enrichment), with the provider name inEnrichment Provider.In
local-onlymode a miss falls back to the bundled MaxMind database, because that file ships with the application and is not a third-party query. The payload is annotatedservedFrom: "bundled"so the distinction stays visible.Tor exit nodes. Exit-node records are merged into the capture-time lookup, keyed by address and by CIDR block, so
tor.exit.nodeis set during analysis with no Onionoo request. Inlocal-onlymode the public list is never fetched.
Where the Data Lives
<userData>/enrichment/
├── config.json # mode + per-service overrides
└── bundles/
├── acme-ti.json
└── internal-geoip.json
<userData>is Electron’s per-user application directory.- Override the whole location with
--enrichment-dirorPACKETSNITCH_ENRICHMENT_DIR; set the mode with--enrichment-modeorPACKETSNITCH_ENRICHMENT_MODE. - Bundles are ordinary JSON files. Hand-editing one is supported: the store fingerprints the directory and hot-reloads changed files, so dropping a new bundle in and re-running a lookup picks it up without restarting PacketSnitch.
- Writes are atomic (temp file + rename), so a crash mid-ingest cannot leave a half-written bundle. A corrupt bundle is skipped, never fatal.
Worked Example: A Private GeoIP and Tor Feed
Suppose you own two CSVs: an address-to-office map and your own exit-node list.
#!/usr/bin/env python3
"""Convert our internal feeds into a ps-enrich bundle."""
import csv, ipaddress, json, sys
records = []
with open("office-ranges.csv", newline="") as handle:
for row in csv.DictReader(handle): # cidr,city,country,lat,lon
records.append({
"service": "geoip",
"target": {"type": "cidr", "value": row["cidr"]},
"data": {
"city": row["city"],
"country": row["country"],
"latitude": float(row["lat"]),
"longitude": float(row["lon"]),
},
})
with open("exit-nodes.csv", newline="") as handle:
for row in csv.DictReader(handle): # address,nickname
records.append({
"service": "tor",
"target": {"type": "ip", "value": row["address"]},
"data": {"isExitNode": True, "nickname": row["nickname"]},
})
json.dump(
{
"schema": "packetsnitch.enrichment/v1",
"provider": {"id": "internal-feeds", "name": "Internal Feeds", "license": "internal"},
"records": records,
},
sys.stdout,
)
python3 to-bundle.py > internal-feeds.json
python3 ps-enrich.py validate --bundle internal-feeds.json --strict
python3 ps-enrich.py ingest --bundle internal-feeds.json --replace
python3 ps-enrich.py lookup --service geoip --target 198.18.0.5 --require-local
The --replace matters here: the export is regenerated wholesale, so replacing prevents a decommissioned range from lingering in the store.
Worked Example: Feeding an Internal VirusTotal Replacement
# One trusted analyst's lookup becomes a record for the whole team.
curl -s -X POST http://127.0.0.1:9020/enrichment/ingest \
-H 'Content-Type: application/json' \
-d '{"bundle": {"provider": {"id": "ir-team"},
"records": [{"service": "virustotal",
"target": {"type": "hash", "value": "44d88612fea8a8f36de82e1278abb02f"},
"data": {"malicious": 58, "meaningfulName": "eicar.com"},
"ttlSeconds": 604800}]}}'
Now every analyst on that install sees the verdict, the store answers in milliseconds, and nothing was sent to VirusTotal. A week later the record expires on its own.
Operational Notes
- Scale. A bundle holds up to 500,000 records; the store reads up to 64 bundles. Lookups are a dict hit for an exact address; a CIDR lookup walks the blocks, so thousands of blocks are fine and hundreds of thousands are not.
- Memory. The index is in memory. A 500k-record bundle of small records is tens of megabytes — comfortable, but not free on a 2 GB VM.
- No network. Ingest and lookup are local. The only outbound traffic is the one you asked for in
local-firstmode on a miss. - Idempotent. Re-running an identical ingest reports
added: 0and writes nothing new. This is safe to put in a cron job. - Auditability. Every served payload names its providers, so a report can say where an answer came from.
ps-enrich exportis the full record. - Privacy. Nothing in this feature transmits your metadata anywhere. The bundles are files on your machine, read by your local backend.
Troubleshooting
“cannot reach the PacketSnitch backend” — PacketSnitch must be running, and the bridge must be enabled. Check the address with --base-url, or set PACKETSNITCH_BACKEND_URL.
“No local enrichment record matches this target” — either the record was not ingested (ps-enrich status) or the target is outside your blocks. Remember that a CIDR record only matches addresses the block contains; a 10.0.0.0/8 record does not cover 192.168.1.5.
A record was rejected — the reason names the record index and the problem. The rest of the bundle was stored regardless. Run validate --strict in your pipeline so a database export that starts failing cannot go unnoticed.
The answer still came from a public service — the record’s policy is supplementary, or the mode is local-first and no authoritative record matched. ps-enrich lookup --require-local turns that into a failure so it is visible.
A record stopped working — check expiredDropped in ps-enrich status. An expired record behaves exactly like an absent one.
