Last updated:

Private Enrichment

PacketSnitch normally enriches a capture by asking a public service: GeoIP from the bundled MaxMind database, WHOIS from RDAP, IP reputation from IPSum, exit nodes from the Tor Project, open ports from Shodan InternetDB, ASN from HackerTarget, and file/IP reputation from VirusTotal.

Many operators cannot send an indicator to any of those. A government network, a regulated enterprise, or an air-gapped SOC usually owns a better database than the public one — and is forbidden from using the public one at all.

Private enrichment closes that gap. You load a JSON bundle of the metadata you already hold, and the backend serves it ahead of any public upstream. In local-only mode it never makes an outbound request at all.

Everything is reachable through one deliberately small driver: scripts/ps-enrich.py. It is a single file with no dependencies — standard library only — because it has to run on a jump host inside your network with nothing but the system Python.

Table of Contents

Quick Start

# 1. Check the bundle parses and every record is admissible (writes nothing)
python3 ps-enrich.py validate --bundle acme-ti.json

# 2. Load it (idempotent — re-running with the same content changes nothing)
python3 ps-enrich.py ingest --bundle acme-ti.json

# 3. Prove it is being served
python3 ps-enrich.py lookup --service geoip --target 203.0.113.9

# 4. Lock the install down so no public service is ever contacted
python3 ps-enrich.py mode --mode local-only

# 5. Export exactly what the backend holds (a backup, or a diff basis)
python3 ps-enrich.py export --out acme-ti.export.json

The driver talks to the backend’s HTTP bridge, which PacketSnitch already runs. The default address is http://127.0.0.1:9020; override it with --base-url or the PACKETSNITCH_BACKEND_URL environment variable.

The Three Modes

The mode decides what happens when a lookup arrives. It is a global setting with optional per-service overrides.

ModeA matching record existsNo record matches
local-first (default)served from your database, no network callthe public service is queried, exactly as before
local-onlyserved from your databasesupported: false + skippedReason: "not-in-local-dataset"; no network call
remote-onlyyour database is ignored entirelythe public service is queried

remote-only is the escape hatch for an install that wants the store present but inactive — every payload is byte-identical to a build without this feature.

Two consequences are worth stating plainly:

  • local-first is not a cache. An authoritative record means the answer, not a hint. If your database says an address is in Reston, no public service is asked and no public service can disagree.
  • local-only is a guarantee, not a preference. The check happens before the network request is constructed, so a miss cannot fall through to the internet. It also skips the public-response cache, so an install that was online before cannot leak a stale public answer afterwards.

Mode changes take effect on the next lookup. No restart is needed.

# Global
python3 ps-enrich.py mode --mode local-only

# Per service: keep VirusTotal public, everything else private
python3 ps-enrich.py mode --mode local-only --service-mode virustotal=remote-only

# Remove an override so a service inherits the global mode again
python3 ps-enrich.py mode --service-mode virustotal=inherit

# Read the current configuration
python3 ps-enrich.py mode

The Bundle Format

A bundle is one JSON document:

{
  "schema": "packetsnitch.enrichment/v1",
  "provider": {
    "id": "acme-ti",
    "name": "ACME Threat Intelligence",
    "url": "https://ti.acme.internal",
    "license": "internal-only"
  },
  "records": [
    {
      "service": "geoip",
      "target": { "type": "cidr", "value": "203.0.113.0/24" },
      "data": { "country": "United States", "city": "Reston",
                "latitude": 38.9586, "longitude": -77.3570 }
    }
  ]
}
KeyRequiredMeaning
schemarecommendedpacketsnitch.enrichment/v1. Any other value is rejected, so a future format cannot be silently misread.
provider.idrecommendedFilesystem-safe bundle name. This is how a re-ingest finds the bundle it should merge into. Defaults to default.
provider.name / .url / .licenseoptionalShown in the payload’s enrichment.providers list, so an analyst can see whose data an answer came from.
recordsyesArray of records (see below).
replaceoptionaltrue replaces the provider’s records instead of merging into them.
updatedAtoptionalFree-form timestamp carried through to the status report.

A record:

KeyRequiredMeaning
serviceyesgeoip, whois, ipsum, tor, shodan, asn, virustotal — or any for the multi-service form.
targetyesA string, or { "type": ..., "value": ... }. The type may be omitted and is inferred from the value.
datayesThe payload. Every field inside it is optional.
policyoptionalauthoritative (default) or supplementary. See Policy.
provideroptionalPer-record provider override; useful when one export mixes sources.
expiresAt / ttlSecondsoptionalWhen this record stops being served.
observedAtoptionalWhen the fact was true — the anchor ttlSeconds counts from.
confidenceoptional0.0–1.0; reported in the annotation, and the minimum across merged records wins.
notesoptionalFree text carried through to the export.

Accepted shorthand keys for data are fields, payload and values, so an export script written against a different vocabulary usually works unchanged.

A bundle may also be a bare array of records, a single record ({"service": ..., "target": ..., "data": ...}), or NDJSON (one record per line, blank lines and # comments ignored). NDJSON is what you want for a large export: a streaming job can write it out without building one giant JSON array.

Rejected records do not discard the bundle

A 200,000-row export from a private database will eventually contain one typo. That typo rejects that record, not the run:

{
  "success": true,
  "accepted": 199999,
  "rejected": [{ "index": 91204, "reason": "record 91204: invalid IP address '10.0.0.300'" }]
}

Use --strict in a pipeline to turn any rejection into a non-zero exit code.

Supported Services

Each service accepts the subset of fields it can render. Everything else is preserved verbatim and comes back out in ps-enrich export, so a round trip is lossless.

GeoIP

Replaces or supplements the bundled MaxMind database, both for on-demand lookups and — importantly — for every packet in the capture (see Capture-Time Enrichment).

{ "service": "geoip", "target": "203.0.113.9",
  "data": { "country": "United States", "countryCode": "US", "region": "Virginia",
            "city": "Reston", "postalCode": "20190",
            "latitude": 38.9586, "longitude": -77.3570,
            "timezone": "America/New_York", "isp": "ACME", "asn": "AS64500" } }

Aliases: lat, lon/lng, zip, tz, country_code, organisation.

WHOIS / RDAP

{ "service": "whois", "target": "203.0.113.9",
  "data": { "isp": "ACME Corporation", "netName": "ACME-NET",
            "netType": "DIRECT ALLOCATION", "parent": "NET-203-0-0-0-0",
            "registrationDate": "2011-04-05T00:00:00Z",
            "updatedDate": "2025-01-02T00:00:00Z",
            "rangeStart": "203.0.113.0", "rangeEnd": "203.0.113.255",
            "cidr": "203.0.113.0/24", "rirHost": "rdap.arin.net" } }

IPSum (IP reputation / block listing)

{ "service": "ipsum", "target": "203.0.113.9", "data": { "hits": 7 } }

Only hits is needed. The A–F grade badge is this build’s banding of that count (the same bands the public feed uses), not something you have to compute.

Tor exit nodes

{ "service": "tor", "target": { "type": "cidr", "value": "203.0.113.0/24" },
  "data": { "isExitNode": true, "nickname": "acmeexit", "platform": "Tor 0.4.8.10" } }

isExitNode: false is meaningful: it marks an address as not an exit node, so a local-only install can answer the question rather than reporting a miss. Exit-node records reach the capture pipeline, so tor.exit.node is set on packets during analysis without any network access.

Shodan InternetDB

{ "service": "shodan", "target": "203.0.113.9",
  "data": { "ports": [22, 443, 8443], "hostnames": ["vpn.acme.example"],
            "cpes": ["cpe:/a:openbsd:openssh:9.6"], "tags": ["vpn"],
            "vulns": ["CVE-2024-0001"] } }

ASN

{ "service": "asn", "target": "203.0.113.9",
  "data": { "asn": "AS64500", "asnName": "ACME-AS, US",
            "organization": "ACME Corporation", "domain": "acme.example",
            "asnRange": "203.0.113.0 - 203.0.113.255" } }

The ISO country code is derived from the trailing , XX in asnName when you do not supply countryCode — the same rule the public path uses.

VirusTotal

Keyed by IP, CIDR, domain, URL or hash. Only the counters are needed.

{ "service": "virustotal", "target": { "type": "hash", "value": "44d88612fea8a8f36de82e1278abb02f" },
  "data": { "malicious": 58, "suspicious": 0, "harmless": 0, "undetected": 3,
            "reputation": -90, "meaningfulName": "eicar.com",
            "typeDescription": "EICAR Test File", "tags": ["test"] } }

Shorthand counters (malicious, suspicious, harmless, undetected, timeout, failure, confirmed_timeout, type_unsupported) are collected into the analysis block the UI renders.

Multi-service records

When one indicator carries several kinds of metadata — the normal case — write one record with "service": "any":

{ "service": "any", "target": "203.0.113.9",
  "data": {
    "shodan":     { "ports": [443, 8443], "tags": ["vpn"] },
    "asn":        { "asn": "AS64500", "organization": "ACME" },
    "whois":      { "netName": "ACME-NET", "isp": "ACME Corporation" },
    "ipsum":      { "hits": 7 },
    "tor":        { "isExitNode": false },
    "virustotal": { "malicious": 4, "reputation": -11 },
    "geoip":      { "country": "United States", "city": "Reston" }
  } }

This expands into one record per service at ingest, so a single target entry feeds every surface at once.

Target Types and CIDR Blocks

TypeExampleMatched by
ip203.0.113.9, 2001:db8::1exact address
cidr203.0.113.0/24, 2001:db8::/32any address inside the block
domainvpn.acme.exampleexact name (VirusTotal only)
urlhttps://acme.example/xexact URL (VirusTotal only)
hashMD5 / SHA-1 / SHA-256 / SHA-512exact digest (VirusTotal only)

The type is inferred when you omit it. CIDR blocks are the important part: a single /8 describes sixteen million addresses in one record, and the matcher always prefers the most specific block. That is how a private database can say “everything in this range is ours” without emitting a row per address.

When several records match one address, the more specific block wins, and an exact host record beats any block. Records from different providers at the same specificity are merged, so two teams can each contribute part of the picture.

Partial Records

Every field is optional. A private database rarely knows everything a public one does, and this feature is designed around that reality rather than against it.

A record that supplies only what it knows is accepted, and the response says so:

"enrichment": {
  "servedFrom": "local",
  "providers": [{ "id": "acme-ti", "name": "ACME Threat Intelligence" }],
  "partial": true,
  "missingFields": ["analysis.malicious"]
}

partial means “an essential field for this surface is absent” — for VirusTotal that is the malicious count, for GeoIP the country or city, and so on. The field still appears in the payload with its neutral default (0, [], "Unknown"), so the UI renders normally; partial is what tells a reader the difference between “we checked and it is zero” and “we do not know”.

Field Aliases

The bundle accepts the short names an operator is likely to write as well as the canonical ones. The authoritative list is served by the backend — never guess:

python3 ps-enrich.py schema

That prints every service, its target types, its accepted aliases, its essential fields and the current limits, generated from the same table the ingester uses. A field this build does not recognise is preserved, not dropped, so a newer schema can add fields and an older build will still round-trip them.

Policy: Authoritative vs Supplementary

Policylocal-first behaviourUse when
authoritative (default)served without a network callyour database is the truth for this indicator
supplementarythe public service is queried and your fields are merged into its answeryou have an extra field, or a local annotation, to add

A merge never replaces a scalar the public answer already has unless the local record sets it explicitly, and lists are replaced rather than unioned — an authoritative record that removes a false positive has to be able to remove it.

If any matching record is authoritative, the whole answer is served locally.

Expiry

Indicators go stale. A record can say when it stops being true:

{ "service": "virustotal", "target": "203.0.113.9",
  "data": { "malicious": 4 }, "ttlSeconds": 86400 }

ttlSeconds counts from observedAt (or from now, when observedAt is absent) and is converted to an absolute expiresAt. An expired record is not served — it is skipped exactly as if it had never been ingested, and the count of dropped records appears in ps-enrich status as expiredDropped.

The Driver

ps-enrich [--base-url URL] [--timeout SECONDS] COMMAND
CommandWhat it does
statusWhat the backend holds, and which mode it is in.
schemaThe machine-readable bundle contract, straight from the backend.
validateCheck a bundle without writing. Falls back to an offline structural preview when the backend is unreachable.
ingestValidate and store. Idempotent.
lookupAsk for one enriched target — the fastest way to confirm the store is being served.
modeRead or set the global/per-service mode.
exportEverything the backend holds, as a bundle you can re-ingest.
clearDelete one provider, or everything with --all.

Useful flags:

  • --provider ID — set the bundle id when the bundle does not name one.
  • --replace — replace the provider’s records rather than merging.
  • --strict — exit 3 if any record was rejected.
  • --require-local (on lookup) — exit 3 unless the answer came from your store. This is the assertion to put in a smoke test.
  • --json — print the raw report.

Exit codes: 0 success, 1 usage/input error, 2 the backend was unreachable or answered non-2xx, 3 records were rejected (with --strict) or a --require-local lookup was served from somewhere else.

HTTP API

The driver is a thin client over endpoints you can also call directly from your own tooling.

MethodPathPurpose
GET/enrichment/statuscounts, providers, modes, limits
GET/enrichment/schemathe bundle contract
GET/enrichment/export?service=&provider=everything held, as a bundle
POST/enrichment/validate{ "bundle": ... } — check without writing
POST/enrichment/ingest{ "bundle": ..., "providerId"?, "replace"? }
POST/enrichment/config{ "mode": ..., "services": {...} }
POST/enrichment/clear{ "providerId": ... } or { "all": true }
POST/control{ "action": "set-runtime-config", "enrichment": {...} }

Every enrichment lookup payload carries an enrichment block naming the mode, the serving path (local, merged, bundled or none), the providers and the partiality. A remote-only install emits no such block at all.

Capture-Time Enrichment

Two surfaces reach the packets, not just the on-demand lookups — which is what makes the private data visible in the Stats map, the filters and the AI summary:

  1. GeoIP. A matching record supplies every loc.src.* / loc.dst.* field the capture pipeline reads, so a private GeoIP database drives the heatmap and loc.* filters exactly like MaxMind does. The record is marked Enrichment: "local" (and loc.<side>.enrichment), with the provider name in Enrichment Provider.

    In local-only mode a miss falls back to the bundled MaxMind database, because that file ships with the application and is not a third-party query. The payload is annotated servedFrom: "bundled" so the distinction stays visible.

  2. Tor exit nodes. Exit-node records are merged into the capture-time lookup, keyed by address and by CIDR block, so tor.exit.node is set during analysis with no Onionoo request. In local-only mode the public list is never fetched.

Where the Data Lives

<userData>/enrichment/
├── config.json          # mode + per-service overrides
└── bundles/
    ├── acme-ti.json
    └── internal-geoip.json
  • <userData> is Electron’s per-user application directory.
  • Override the whole location with --enrichment-dir or PACKETSNITCH_ENRICHMENT_DIR; set the mode with --enrichment-mode or PACKETSNITCH_ENRICHMENT_MODE.
  • Bundles are ordinary JSON files. Hand-editing one is supported: the store fingerprints the directory and hot-reloads changed files, so dropping a new bundle in and re-running a lookup picks it up without restarting PacketSnitch.
  • Writes are atomic (temp file + rename), so a crash mid-ingest cannot leave a half-written bundle. A corrupt bundle is skipped, never fatal.

Worked Example: A Private GeoIP and Tor Feed

Suppose you own two CSVs: an address-to-office map and your own exit-node list.

#!/usr/bin/env python3
"""Convert our internal feeds into a ps-enrich bundle."""
import csv, ipaddress, json, sys

records = []

with open("office-ranges.csv", newline="") as handle:
    for row in csv.DictReader(handle):          # cidr,city,country,lat,lon
        records.append({
            "service": "geoip",
            "target": {"type": "cidr", "value": row["cidr"]},
            "data": {
                "city": row["city"],
                "country": row["country"],
                "latitude": float(row["lat"]),
                "longitude": float(row["lon"]),
            },
        })

with open("exit-nodes.csv", newline="") as handle:
    for row in csv.DictReader(handle):          # address,nickname
        records.append({
            "service": "tor",
            "target": {"type": "ip", "value": row["address"]},
            "data": {"isExitNode": True, "nickname": row["nickname"]},
        })

json.dump(
    {
        "schema": "packetsnitch.enrichment/v1",
        "provider": {"id": "internal-feeds", "name": "Internal Feeds", "license": "internal"},
        "records": records,
    },
    sys.stdout,
)
python3 to-bundle.py > internal-feeds.json
python3 ps-enrich.py validate --bundle internal-feeds.json --strict
python3 ps-enrich.py ingest   --bundle internal-feeds.json --replace
python3 ps-enrich.py lookup --service geoip --target 198.18.0.5 --require-local

The --replace matters here: the export is regenerated wholesale, so replacing prevents a decommissioned range from lingering in the store.

Worked Example: Feeding an Internal VirusTotal Replacement

# One trusted analyst's lookup becomes a record for the whole team.
curl -s -X POST http://127.0.0.1:9020/enrichment/ingest \
  -H 'Content-Type: application/json' \
  -d '{"bundle": {"provider": {"id": "ir-team"},
        "records": [{"service": "virustotal",
                     "target": {"type": "hash", "value": "44d88612fea8a8f36de82e1278abb02f"},
                     "data": {"malicious": 58, "meaningfulName": "eicar.com"},
                     "ttlSeconds": 604800}]}}'

Now every analyst on that install sees the verdict, the store answers in milliseconds, and nothing was sent to VirusTotal. A week later the record expires on its own.

Operational Notes

  • Scale. A bundle holds up to 500,000 records; the store reads up to 64 bundles. Lookups are a dict hit for an exact address; a CIDR lookup walks the blocks, so thousands of blocks are fine and hundreds of thousands are not.
  • Memory. The index is in memory. A 500k-record bundle of small records is tens of megabytes — comfortable, but not free on a 2 GB VM.
  • No network. Ingest and lookup are local. The only outbound traffic is the one you asked for in local-first mode on a miss.
  • Idempotent. Re-running an identical ingest reports added: 0 and writes nothing new. This is safe to put in a cron job.
  • Auditability. Every served payload names its providers, so a report can say where an answer came from. ps-enrich export is the full record.
  • Privacy. Nothing in this feature transmits your metadata anywhere. The bundles are files on your machine, read by your local backend.

Troubleshooting

“cannot reach the PacketSnitch backend” — PacketSnitch must be running, and the bridge must be enabled. Check the address with --base-url, or set PACKETSNITCH_BACKEND_URL.

“No local enrichment record matches this target” — either the record was not ingested (ps-enrich status) or the target is outside your blocks. Remember that a CIDR record only matches addresses the block contains; a 10.0.0.0/8 record does not cover 192.168.1.5.

A record was rejected — the reason names the record index and the problem. The rest of the bundle was stored regardless. Run validate --strict in your pipeline so a database export that starts failing cannot go unnoticed.

The answer still came from a public service — the record’s policy is supplementary, or the mode is local-first and no authoritative record matched. ps-enrich lookup --require-local turns that into a failure so it is visible.

A record stopped working — check expiredDropped in ps-enrich status. An expired record behaves exactly like an absent one.