Log Redaction Pipeline

Log Redaction Pipeline

You've joined the observability team. Every service sends its log records through this redaction pipeline before they reach the central log store, so personal data (emails, phone numbers, card numbers, IP addresses) never leaves the host. The security team has filed one bug and two features.

Codebase

How your code is run

run(operations: list[list]) -> list processes the operations in order with a fresh pipeline and returns one result per operation. Errors are returned as "error: <code>" strings.

• A record is a JSON object whose keys are non-empty strings (dots are just characters: "auth.token" is one field) and whose values are strings, numbers, booleans or null. Anything else is invalid record. The input record is never modified.
• Scanning a string runs the detectors in the order EMAIL, CARD, IPV4, PHONE; each replaces its matches with [EMAIL], [CARD], [IPV4] or [PHONE] and adds one to its counter per match. Non-string values are never scanned. Today every field is scanned.
• logfmt (preview): field=value pairs sorted by field name, joined by single spaces; null, true/false and numbers are written as JSON; a string that is empty or contains a space, " or = is wrapped in double quotes (with " escaped as \").
• Unknown operation → "error: unknown operation"; wrong number of arguments → "error: invalid arguments".

Worked example

[["redact", {"msg": "login ok for ana@example.com from 10.0.0.12", "status": 200, "user": null}],
 ["preview", {"msg": "card 4111 1111 1111 1111 declined", "retry": false}],
 ["stats"]]

returns

[{"msg": "login ok for [EMAIL] from [IPV4]", "status": 200, "user": null},
 "msg=\"card [CARD] declined\" retry=false",
 {"EMAIL": 1, "CARD": 1, "IPV4": 1, "PHONE": 0}]

Part 1 — Phone and email detection is wrong (SEC-412)

Goal: make the EMAIL and PHONE detectors match exactly the definitions below.

Reports from the security review: "call +1 415-555-0134 is logged as call +1 [PHONE], and +14155550134 is not redacted at all", and "user=bob@example.com; comes out as [EMAIL]; — we lose the user= key — and alice@a.io,bob@b.io is counted as one email".

Rules

1. Email: a local part of one or more letters, digits or . _ % + -, then @, then a domain of one or more labels (letters, digits, -) separated by dots, ending in a final label of 2 or more letters. Nothing else is part of the match, so surrounding characters such as =, :, <, >, (, ), quotes, ,, ; and a sentence-ending . stay in the text.
2. Phone: an optional country code +1 (optionally followed by one space, - or .), then a 3-digit area code that may be wrapped in parentheses, then 3 digits, then 4 digits. Between the area code, the 3-digit group and the 4-digit group there may be one space, - or ., or nothing. The whole match, including +1, becomes [PHONE].
3. A phone number is not matched when it is directly preceded or followed by a letter, digit or _. Other country codes (e.g. +44 ...) are out of scope and stay as they are.
4. Each email or phone number found counts once in stats.

tests/test_patterns.py::test_phone_with_country_code_is_fully_redacted currently fails. The CARD and IPV4 detectors are known-good.

Part 2 — Per-field policies (LOG-77)

Goal: let operators choose what happens to each field instead of scanning everything.

Rules

1. Actions: scan (current behaviour), mask (value becomes "[REDACTED]"), drop (field removed from the output), keep (value passed through untouched). Action names are case-sensitive.
2. A selector is either an exact field name ("password") or a wildcard "<prefix>.*", which matches every field whose name starts with "<prefix>." ("auth.*" matches auth.token and auth.session.id, but not auth or authz.role). The prefix must be non-empty and no other * may appear anywhere; anything else is invalid selector.
3. Resolution for a field: an exact selector wins; otherwise the matching wildcard with the longest prefix; otherwise scan.
4. set_policy on a selector that already has a rule replaces it — including the defaults.
5. The defaults in config.DEFAULT_POLICIES are in force from the start: password → drop, auth.* → mask.
6. mask turns any non-null value (string, number or boolean) into "[REDACTED]"; null stays null.
7. Only scan fields are scanned, so only they affect stats.

Example

[["redact", {"user": "ana@example.com", "password": "hunter2", "auth.token": "abc123", "authz": "admin"}],
 ["set_policy", "user", "keep"],
 ["set_policy", "debug.*", "drop"],
 ["redact", {"user": "ana@example.com", "debug.sql": "select 1", "msg": "ok"}],
 ["policy_for", "auth.session.id"]]

returns

[{"user": "[EMAIL]", "auth.token": "[REDACTED]", "authz": "admin"}, true, true,
 {"user": "ana@example.com", "msg": "ok"}, "mask"]

Part 3 — Partial masking and deterministic tokens (LOG-81)

Goal: support two more actions so analysts can still join and debug on sensitive fields.

Rules

1. last4: replace every digit (0-9) in the value with * except the last four digits; all other characters are kept. If the value has 4 or fewer digits, all of them are replaced. "4111-1111-1111-1234" → "****-****-****-1234", "pin 12" → "pin **".
2. token: replace the value with "tok_" + the first 16 hex characters of sha256(f"{TOKEN_SALT}:{normalized}") (UTF-8), where normalized is the value with leading and trailing whitespace removed, lower-cased, and TOKEN_SALT comes from config.py. Equal inputs give equal tokens in every field, record and call.
3. Both actions accept strings and integers (an integer is first converted with str(), so 42 and "42" give the same result). Any other non-null value (float, boolean) becomes "[REDACTED]"; null stays null.
4. set_policy accepts last4 and token with all the Part 2 selector and precedence rules. Fields using them are not scanned and do not affect stats.

Example

[["set_policy", "card_number", "last4"],
 ["set_policy", "customer.*", "token"],
 ["redact", {"card_number": "4111-1111-1111-1234", "customer.email": " Ana@Example.com", "customer.id": 1001}],
 ["redact", {"customer.email": "ana@example.com"}]]

returns

[true, true,
 {"card_number": "****-****-****-1234", "customer.email": "tok_1c617de3bd79dcd0", "customer.id": "tok_116398aef4a1de6e"},
 {"customer.email": "tok_1c617de3bd79dcd0"}]

Notes

• scanner.py, records.py, logfmt.py and stats.py are known-good, as are the CARD and IPV4 detectors.
• You may use the AI assistant freely; check its regexes against the rules above with real examples.
• Later parts must not break earlier ones; hidden tests re-check earlier behaviour.