Lantern Log Search: Query Semantics, Pipelines, Alerts and Scale

Lantern Log Search: Query Semantics, Pipelines, Alerts and Scale

You've joined the observability team that owns Lantern, the log search backend behind the company's incident tooling. Structured log events are ingested from collectors; engineers search them with a small query language such as

service:checkout level:error "payment declined" duration_ms>500 -env:staging

Searches in the field are returning wrong answers, on-call wants aggregation in the search bar, SRE wants saved searches that page them, and the search box has started to crawl on busy days. The work is split into four Parts. Everything is deterministic and in memory.

Codebase

How your code is run

run(operations: list[list]) -> list executes the operations in order against a fresh service and returns one result per operation. Every operation is [name, *args]. Errors are returned as "error: <code>" strings, never raised.

• An unknown or non-string operation name returns "error: unknown operation". A wrong number of arguments, an empty operation and an operation that is not a list return "error: invalid arguments".
• Accepted events are numbered ev-1, ev-2, ... in the order they are accepted. A rejected event does not use up an id.
• ingest_batch generates count events with generator.generate_events(seed, count, start_ts, jitter) and offers each one, in order, to exactly the same path as ingest. seed, count, start_ts, jitter must be integers >= 0, with count <= 100000 and jitter <= 1000, otherwise invalid arguments. Events rejected as too late are counted in rejected; nothing else can be rejected.

Events

{"ts": 1700000123, "service": "checkout", "level": "error",
 "message": "Payment declined for order 7", "fields": {"duration_ms": 612, "env": "prod"}}

ts is an integer number of seconds >= 0 (not a boolean), service a non-empty string, level one of debug info warn error, message a string (possibly empty). fields is optional (default {}); its keys match [A-Za-z_][A-Za-z0-9_.]* and are not ts, service, level or message, and its values are strings or integers (not booleans, not floats). Anything else is invalid event.

An event is rejected as too late when ts < newest - ALLOWED_LATENESS_S, where newest is the largest ts accepted so far (no check while nothing has been accepted). Today config.ALLOWED_LATENESS_S is 0: collectors must deliver events in timestamp order (equal timestamps are fine). Validation happens before the lateness check.

The query language

A query is a filter, optionally followed by a pipeline stage (Part 2). The filter is built from:

Tokenization (one definition for messages, words and phrases). tokenizer.message_tokens lower-cases and keeps maximal runs of the ASCII characters A-Z a-z 0-9; every other character (spaces, punctuation, _, -, =, non-ASCII letters) separates tokens. So "Payment_declined: card=4242" has the tokens payment declined card 4242. A bare word or a quoted phrase is tokenized the same way and matches when its tokens appear consecutively and in order among the message's tokens. A word is never a substring match: pay does not match payment. A bare word that yields several tokens (payment-declined) is the phrase of those tokens.

Fields. Besides custom fields, a query can use ts, service, level and message (the whole message text, compared exactly). The *canonical text* of a value is its decimal digits for an integer and the string itself for a string (events.field_text), so status:500 matches both 500 and "500". A field the event does not have never satisfies field:value.

Numeric comparisons. The event's value counts as a number if it is an integer, or a string that is entirely an optional - followed by digits ("90", "-5", "007"). Then it is compared numerically. Any other value ("n/a", "1.5", " 7") and a missing field never satisfy a comparison. Strings are never compared alphabetically.

Negation is plain boolean NOT. -duration_ms>500 matches every event that does *not* satisfy duration_ms>500, including events whose duration_ms is missing or non-numeric. Likewise -env:staging matches events with no env at all.

Precedence, loosest to tightest: OR, then implicit AND, then -. So a b OR c d is (a AND b) OR (c AND d), -a b is (NOT a) AND b, and -(a OR b) is NOT (a OR b). - may be stacked (--a is a).

Lexical details.

• Whitespace (anything Python's str.isspace accepts) separates terms. (, ) and | are always their own tokens outside quotes.
• A *word* is a maximal run of characters other than whitespace, (, ), " and |.
• OR in exactly those capitals is the operator; or, Or and AND are ordinary words.
• A - that starts a term and is immediately followed by a non-space character negates what follows. A - followed by whitespace or the end of the text is a syntax error. A - in the middle of a word is just part of the word.
• A word that starts with a field name ([A-Za-z_][A-Za-z0-9_.]*) directly followed by :, >, >=, < or <= is a field clause, so https://example.com is the clause https equals //example.com. After : the value must be non-empty unless it is the quoted form field:"...". After a comparison operator it must be an integer matching -?[0-9]+, otherwise the query is invalid.
• Every other word is a message word. If it has no tokens at all (!!!), the query is invalid. A quoted phrase has no escapes, must be terminated, and must contain at least one token.
• An empty or all-whitespace filter matches every event. An empty group (), a missing operand (a OR, OR a, a OR OR b), unbalanced parentheses and every case marked above are invalid query.

Searching

["search", query] or ["search", query, options], where options is an object with optional keys from, to, limit (anything else is invalid arguments; so is a non-string query). from and to are integers (or null, meaning open) selecting events with from <= ts < to; from > to selects nothing. limit is an integer from 1 to 100 (default 10). Options are validated before the query is parsed.

Result: {"total": <number of matching events in range>, "hits": [<ids>]}; hits lists the ids of the first limit matches newest first: larger ts first, and for equal ts the event that arrived later first.

Worked example. After ingesting these six events (timestamps 100 to 105, in this order):

ev-1 ts 100 checkout error  "Payment declined for order 7"                    duration_ms 612    env prod
ev-2 ts 101 checkout warn   "payment_declined: card expired"                  duration_ms "90"   env prod
ev-3 ts 102 payments error  "Upstream timeout after 3 retries"                duration_ms 2500   env staging
ev-4 ts 103 checkout info   "Order 7 created"                                                    env prod
ev-5 ts 104 auth     error  "login failed for user u7"                        duration_ms "n/a"  env prod
ev-6 ts 105 checkout error  "Payment pending, awaiting declined-card review"  duration_ms 700    env dev

Part 1: Search is returning wrong results (LAN-4121 .. LAN-4152)

Goal: make search follow the language reference above. Support has collected these reports from the field. Each one is a symptom; there may be more than one defect behind a report, and one defect may explain several reports.

• LAN-4121. "service:checkout level:error OR level:warn only returns checkout events. Warnings from other services never show up, and a b OR c d seems to behave like a AND (b OR c) AND d."
• LAN-4130. "-(level:debug OR level:info) still returns debug and info events."
• LAN-4137. "Searching "declin" finds messages containing declined, and "ment dec" finds payment declined."
• LAN-4144. "duration_ms>500 returns events whose duration is the string "90", and duration_ms<100 returns events that have no duration at all."
• LAN-4149. "env:Prod returns the events from env:prod; values are supposed to be exact."
• LAN-4152. "{"to": 105} includes the event at ts 105 even though to is exclusive."

tests/ already contains tests for some of these; several currently fail. Everything else in the reference must keep working: the worked example above is the contract.

Part 2: Aggregations in the search bar (LAN-4188)

Goal: a query may end with one pipeline stage, introduced by the first | that is outside quotes:

<filter> | stats count by <field>
<filter> | top <n> <field>

The filter may be empty (| stats count by service aggregates every event). The stage text is split on whitespace and must match one of the two forms exactly: <field> is a field name as in events (builtin fields work too) and <n> is one or more ASCII digits with value >= 1. Anything else, including a second | or a missing part, is invalid query. A filter that fails to parse is invalid query as before.

The stage sees all events that match the filter within from/to; limit is validated but has no effect on it. It groups them by the field's canonical text (so integer 500 and string "500" are one group; an event without the field belongs to no group) and returns

{"total": <matching events>, "missing": <matching events without the field>, "rows": [...]}

• Groups are ordered by count, largest first; equal counts are ordered by the group's text in plain (code point) string order, so "10" comes before "9".
• stats count by: every group, as [text, count].
• top n: the first n groups, each as [text, count, percent]. percent is the group's share of the matching events that have the field (not of total), as a string with one decimal, rounded half up: 1 of 16 is "6.3", 1 of 3 is "33.3", all of them is "100.0". With no group at all, rows is [].

Example (the six events above)

Part 1 behaviour is unchanged: a query with no stage returns the same results as before.

Part 3: Out-of-order events and alerts (LAN-4203)

SRE wants saved searches that page them. Collectors batch and retry, so events now arrive late and out of order, and the alert has to be right about *event time*, not arrival order.

Rule changes.

1. config.ALLOWED_LATENESS_S becomes 30. An event is accepted iff ts >= newest - 30 (so newest - 30 is accepted and newest - 31 is too late). A rejected event is not stored, does not use an id and is invisible to everything else. This replaces the Part 1 rule that collectors must deliver in order.
2. Accepted events are searchable by their own ts, wherever they arrived. Result order, from/to and every Part 1-2 rule are unchanged. (Equal ts: later arrival is newer.)

Alerts.

• invalid arguments: name not a non-empty string, query not a string, window or threshold not an integer (booleans are not integers). invalid alert: window outside 1..86400 or threshold < 1. invalid query: the query does not parse or contains a pipeline stage. The query is evaluated with exactly the semantics of search (an empty query matches every event).
• Ticks are the multiples of config.ALERT_TICK_S (10) seconds of event time: 0, 10, 20, ...
• The watermark is the largest ts accepted so far. A tick T is closed as soon as the watermark is greater than T + ALLOWED_LATENESS_S: from then on no accepted event can have ts <= T.
• A closed tick T fires if the number of matching events with T - window < ts <= T is at least threshold; the firing is recorded as [T, count] with that final count. Ticks are decided only when they close, in increasing order, and a late event that was still accepted counts.
• An alert only considers ticks greater than the watermark at the moment it was created: the first tick is (W // 10 + 1) * 10 for watermark W, or 0 if no event had been accepted yet. Events already stored at creation time still count towards the windows of those ticks.
• alert_firings returns the firings recorded so far, in tick order.
• Every accepted event takes part, however it arrived (ingest or ingest_batch).
• Timestamps go up to 10^9 and one event can move the watermark by hundreds of millions of seconds.

Example

[["create_alert", "burst", "level:error", 20, 2],
 ["ingest", {"ts": 100, "service": "api", "level": "error", "message": "boom"}],
 ["ingest", {"ts": 105, "service": "api", "level": "error", "message": "boom"}],
 ["ingest", {"ts": 140, "service": "api", "level": "info", "message": "ok"}],
 ["ingest", {"ts": 112, "service": "api", "level": "error", "message": "late boom"}],
 ["ingest", {"ts": 109, "service": "api", "level": "error", "message": "too old"}],
 ["alert_firings", "burst"],
 ["ingest", {"ts": 141, "service": "api", "level": "info", "message": "ok"}],
 ["alert_firings", "burst"],
 ["ingest", {"ts": 200, "service": "api", "level": "info", "message": "ok"}],
 ["alert_firings", "burst"]]

returns [true, "ev-1", "ev-2", "ev-3", "ev-4", "error: too late", [], "ev-5", [[110, 2]], "ev-6", [[110, 2], [120, 2]]].

At ts 140 the watermark is 140, so only ticks up to 100 are closed (nothing fires). Event ts 112 is 28 seconds late and still counts. At 141 tick 110 closes: errors at 100 and 105 are in (90, 110]. Tick 120 closes only once the watermark passes 150: (100, 120] holds the errors at 105 and 112.

Part 4: Search at scale (LAN-4260)

On busy days the box times out. A single run can now ingest 20,000+ events (use ingest_batch) and then issue hundreds of searches, including ones with negations, phrases, comparisons, OR, time ranges and pipeline stages. The whole hidden suite must finish within a few seconds, so scanning every event for every query is too slow.

Goal: make searches fast without changing any answer: every result from Parts 1 to 3 (including total, ordering, missing and alert firings) must be identical. Late events (Part 3) are part of the data being searched.

Notes

• tokenizer.py, generator.py and events.py are known-good; matcher.py, parser.py and store.py are where the Part 1 defects live (not necessarily one per report).
• You may use the AI assistant freely; you are expected to verify what it produces.
• Later Parts must not break earlier ones; hidden tests re-check earlier behaviour.
• Parts 1 to 3 is a strong result. Part 4 is a stretch.