Experiment Platform: Ramps, Layers and Trustworthy Results

Experiment Platform: Ramps, Layers and Trustworthy Results

You've joined the product-analytics team that runs the in-house A/B testing service. The service decides which users see which experiment variant, logs who was actually exposed, and reports conversion and revenue per exposed user. Last week a ramp went badly wrong, finance has stopped trusting the dashboards, and three follow-ups are queued behind the fix.

Codebase

How your code is run

run(operations: list[list]) -> list executes the operations in order against a fresh service and returns one result per operation. Every operation is [name, ts, ...args]; ts is an integer number of seconds. ts values are not required to be increasing. Errors are returned as {"error": "<code>"}, never raised.

• A malformed operation (not a list, fewer than two items, a non-string name) returns invalid_arguments. An unknown name returns unknown_operation (this is checked before the arguments). A non-integer ts or arguments of the wrong number or type return invalid_arguments. true/false are never integers. Optional attrs is a dict of string keys to string values.
• Ids (experiment, layer, user, event, variant names) are case-sensitive non-empty strings.

Operations available from the start

users is a list of user ids, or {"prefix": "u", "count": 60} meaning u0 ... u59.

An experiment spec is a dict with these keys; a spec that is not a dict, any other key, or any value of the wrong type or range is invalid_spec:

A rejected create_experiment or set_allocation changes nothing (including not creating the layer).

Part 1: The ramp incident (EXP-412, EXP-418, EXP-421)

Goal: make assignment behave the way the rules below say, so that ramping an experiment up never changes what anybody has already seen, and so that the hash matches the notebooks.

What happened: the checkout team ramped onepage from 10% to 30%. Afterwards a large share of the users who had been in the original 10% were suddenly in the *other* variant (EXP-412), and badge, a neighbouring experiment in the same layer, could no longer be ramped even though most of the layer was empty (EXP-421). Separately, data science reports that assignments for users with non-ASCII ids (josé) differ from what their notebooks compute (EXP-418).

Rules

1. Hash. The hash of (salt, user_id) is 32-bit FNV-1a (offset basis 2166136261, prime 16777619, arithmetic modulo 2^32) over the UTF-8 bytes of the text "<salt>:<user_id>". The bucket is that hash modulo 10,000. Reference vectors: fnv1a32("") = 2166136261, fnv1a32("a") = 0xE40C292C, fnv1a32("foobar") = 0xBF9CF968; bucket("layer-checkout", "ana") = 493.
2. Layers. A layer assigns each user a bucket using the salt "layer-" + layer_id. Each bucket is owned by at most one experiment of the layer, so a user is in at most one experiment per layer. An experiment with allocation p owns p * 100 buckets.
3. Claiming. create_experiment claims the p * 100 lowest-numbered free buckets. set_allocation to a larger percent claims the lowest-numbered free buckets *in addition to* those it already owns. To a smaller percent it gives back its most recently claimed buckets. The buckets an experiment keeps never change owner or number. If the layer does not have enough free buckets, the call fails with layer_full and nothing changes. Setting the same percent is a no-op.
4. Variants. Which variant an in-experiment user gets uses the experiment's own hash, not the layer's: position = bucket(spec.salt, user_id) // 100 (0 to 99); the user gets the first variant, in declaration order, whose running weight total is greater than position. It does not depend on the allocation.
5. variant / variants return, for a given user (and optional attrs): null if the experiment's targeting does not match (every targeting key must be present in attrs with one of its allowed values), else null if the user's layer bucket is not owned by this experiment, else the variant. They never change any state.
6. layer_map lists the layer as runs of consecutive buckets with the same owner (null = free), in bucket order, covering 0 to 10000 exactly.
7. Check order for set_allocation: unknown experiment, then a percent that is not an integer in 0 to 100 (invalid_allocation), then capacity.

Example

[["create_experiment", 1, "onepage", {"layer": "checkout", "salt": "exp-onepage", "variants": [["control", 50], ["treatment", 50]], "allocation": 10, "goal": "signup"}],
 ["create_experiment", 2, "badge", {"layer": "checkout", "salt": "exp-badge", "variants": [["off", 80], ["on", 20]], "allocation": 20, "goal": "signup"}],
 ["variants", 3, "onepage", ["ana", "josé", "bob", "carol"]],
 ["set_allocation", 4, "onepage", 30],
 ["layer_map", 5, "checkout"],
 ["variants", 6, "onepage", ["ana", "josé", "bob", "carol"]]]

returns

[{"allocation": 10, "buckets": 1000}, {"allocation": 20, "buckets": 2000},
 ["treatment", "treatment", null, null],
 {"allocation": 30, "buckets": 3000},
 [["onepage", 0, 1000], ["badge", 1000, 3000], ["onepage", 3000, 5000], [null, 5000, 10000]],
 ["treatment", "treatment", null, "control"]]

ana and josé keep their variant; carol (layer bucket 3754) is a new arrival.

Part 2: Exposure logging and results (PLAT-77)

Goal: a user counts toward an experiment only if they were actually shown it. Build the exposure log and the results report, with exact arithmetic and the display rules below.

Rules

1. expose decides exactly like variant (Part 1). If the result is null nothing is recorded: a user excluded by targeting, or outside the experiment's buckets, is never counted anywhere. Otherwise the user is exposed: the first expose call for a (experiment, user) that returns a variant records the exposure with that call's ts as first_ts. Later calls return the variant but never change first_ts, even when their ts is smaller. variant and variants never expose anyone.
2. track records an event for any user, exposed or not (events are kept in case the user is exposed later). amount is an integer number of cents >= 0 (default 0); a negative amount is invalid_amount and records nothing.
3. Which events count. An event counts for an exposure iff first_ts <= event_ts < first_ts + window_s, judged on the timestamps alone: events before the exposure never count; one at exactly first_ts does, one at exactly first_ts + window_s does not; the order in which the operations arrived does not matter (an event can be tracked before the expose call and still count). Each experiment judges its own exposures, and the experiment's creation time is irrelevant.
4. A user is converted if at least one counted event is named goal. A user's revenue is the sum of the amounts of their counted events named purchase (REVENUE_EVENT in config.py). The goal may itself be purchase.
5. Report. report returns {"experiment": exp_id, "variants": [row, ...], "srm": {"chi_square": "<text>", "flagged": bool}} with one row per variant in declaration order (including variants nobody was exposed to): {"variant", "exposed", "converted", "conversion", "revenue_cents", "revenue_per_user"}, where exposed, converted, revenue_cents are integers and the other two are strings:

• conversion = converted / exposed as a percentage with two decimals, e.g. "12.34%".
• revenue_per_user = revenue_cents / exposed rounded to a whole cent, written "$12.34".
• Both round exact ties up (half-up), computed on exact numbers; exposed = 0 gives "0.00%" and "$0.00".

1. Sample-ratio check. With k variants, total exposed users and weight_i the variant weights, the expected count of variant i is total * weight_i / 100, and chi_square = sum((observed_i - expected_i)^2 / expected_i) computed exactly (use Fraction). flagged is chi_square > SRM_CRITICAL[k - 1] (strictly greater; table in config.py). chi_square is shown with three decimals, half-up, e.g. "3.141". With total = 0 it is "0.000" and not flagged.

Example (experiment onepage: allocation 100, 50/50, targeting {"country": ["US", "CA"]}, goal "signup", window_s 3600)

[["track", 5, "u1", "signup"],
 ["expose", 10, "onepage", "u1", {"country": "US"}],
 ["expose", 20, "onepage", "u3", {"country": "US"}],
 ["expose", 30, "onepage", "u2", {"country": "FR"}],
 ["track", 40, "u1", "signup"],
 ["track", 41, "u1", "purchase", 1000],
 ["track", 45, "u3", "purchase", 2500],
 ["expose", 50, "onepage", "u1", {"country": "US"}],
 ["track", 3609, "u3", "purchase", 500],
 ["track", 3610, "u1", "purchase", 700],
 ["track", 3620, "u3", "purchase", 900],
 ["report", 4000, "onepage"]]

returns [true, "control", "treatment", null, true, true, true, "control", true, true, true, REPORT] where REPORT is

{"experiment": "onepage",
 "variants": [
   {"variant": "control", "exposed": 1, "converted": 1, "conversion": "100.00%", "revenue_cents": 1000, "revenue_per_user": "$10.00"},
   {"variant": "treatment", "exposed": 1, "converted": 0, "conversion": "0.00%", "revenue_cents": 3000, "revenue_per_user": "$30.00"}],
 "srm": {"chi_square": "0.000", "flagged": false}}

u1's first signup (ts 5) is before the exposure, u2 was excluded by targeting, and u1's window is [10, 3610), so the purchase at 3610 is outside it.

Part 3: Requirements change, sticky assignments and a holdout (PLAT-91)

Goal: product has changed its mind about ramps. Once a user has been *exposed*, nothing about the experiment's configuration may move them again, even ramping an experiment down to 0%, and a global holdout of users must be kept out of every experiment so their baseline can be compared.

This supersedes Part 1 for exposed users only. For users who have never been exposed, Part 1 behaves exactly as before (ramping down still hands their buckets back, variant still reflects the current buckets).

Rules

1. Resolution. variant, variants and expose all use the same steps, after unknown_experiment:
2. *Pinned.* If the user has a pinned assignment in this experiment's layer: for this experiment, return the pinned variant (targeting, buckets and allocation are not re-checked); for any other experiment of the layer, return null.
3. Targeting does not match: null.
4. The user's layer bucket is not owned by this experiment: null.
5. *Holdout.* The user's holdout group is held out if bucket("global-holdout", user_id) < holdout_percent * 100, otherwise eligible (HOLDOUT_SALT in config.py). The default percent is 0. If the user has a frozen holdout group (below), that group is used instead of recomputing it. Held out: null.
6. Otherwise the variant from Part 1 rule 4.
7. Pinning. When expose returns a variant, the user is pinned in that layer to (experiment, variant) for good. variant and variants never pin or freeze anything.
8. Freezing. An expose call that reaches step 4 freezes the user's holdout group for every experiment from then on (the group computed under the then-current percent). Later set_holdout calls only affect users with no frozen group.
9. Holdout exposures. When expose finds the user held out, it returns null and, the first time for that (experiment, user), records a *holdout exposure* with that ts. Holdout exposures count events exactly like Part 2 (same window, goal and revenue rules). report gains a key "holdout" holding a row ("variant": "holdout") only if the experiment has at least one holdout exposure; the other keys of the report are unchanged. Holdout users are not part of the sample-ratio check.

Example (experiment onepage, allocation 100, 50/50)

[["set_holdout", 1, 5],
 ["expose", 2, "onepage", "u5"],
 ["expose", 3, "onepage", "u1"],
 ["set_holdout", 4, 0],
 ["expose", 5, "onepage", "u5"],
 ["set_allocation", 7, "onepage", 0],
 ["variant", 8, "onepage", "u1"],
 ["variant", 8, "onepage", "u2"]]

returns [5, null, "control", 0, null, {"allocation": 0, "buckets": 0}, "control", null]. u5's holdout bucket is 228 (held out at 5%) and stays held out after the holdout goes to 0; u1 keeps control with 0% allocation, while the never-exposed u2 is out.

Part 4: Live dashboards at scale (PLAT-103)

Goal: experimenters watch their experiments while traffic is still arriving, and the dashboard refreshes constantly. Add the "who is winning" banner and make every query cheap no matter how much traffic has been logged.

Rules

1. leading compares converted / exposed of the variants exactly (as fractions, not the displayed strings). A variant nobody was exposed to has rate 0. Ties go to the variant declared first. The holdout is never a candidate.
2. simulate already exists (read the docstring in traffic.py). It drives the real expose and track with generated users and, when poll_every > 0, calls leading after every poll_every-th user to mimic a refreshing dashboard. count and poll_every are non-negative integers, prefix a non-empty string, attrs a dict of strings. Do not change what it generates: the hidden tests depend on it.
3. The hidden tests use simulate to create tens of thousands of users and tens of thousands of events from a handful of operations, with hundreds of leading polls between users, so leading and report must not rescan the exposure and event logs on every call. Everything earlier must stay exactly as is, including events that arrive before the exposure they belong to, events outside the window, and users who are exposed to several experiments.

Example (experiment onepage: allocation 100, 50/50, goal "signup", window_s 600)

[["simulate", 100, "onepage", "sim-", 40, {}, 10],
 ["leading", 200, "onepage"]]

returns [{"given": 40, "leading": {"control": 3, "treatment": 1}}, "control"]: the dashboard was polled 4 times during the run (after users 10, 20, 30 and 40), control led three times.

Notes

• validate.py, targeting.py, errors.py and config.py are known-good.
• You may use the AI assistant freely; you are expected to verify what it produces.
• Later parts must not break earlier ones; hidden tests re-check earlier behaviour after later changes.
• Parts 1 and 2 are the core. Parts 3 and 4 are the stretch.