Request Gateway: Schema Validation, References and Scale

Request Gateway: Schema Validation for API Bodies

You've joined the API-platform team that owns gate, the service in front of every public endpoint at a payments-and-logistics company. Customers register a schema for each endpoint, and gate checks every request body against it before the body reaches the backend. Schemas are written in a documented subset of JSON Schema (below). The first release shipped with a handful of bugs that support keeps escalating, and three requests are queued behind them: references and composition, a rule change plus anyOf/oneOf and defaults, and a performance floor for the import team.

Everything about the dialect is defined in this file or in the code. Part 1 fixes gate as it exists; each later Part adds to it, and Part 3 replaces two rules you implement in Part 1.

Codebase

How your code is run

run(operations: list[list]) -> list executes the operations in order against a fresh gateway and returns one result per operation. Errors are returned as "error: <code>" strings, never raised.

• An unknown operation name returns "error: unknown operation". Every operation takes exactly two arguments after its name, and the first must be a non-empty string; otherwise "error: invalid arguments". (Unknown operation is checked first.)
• Registering a name again replaces its schema; later requests use the new one.
• An error is a three-element list [path, keyword, message] (strings). path is a JSON Pointer into the instance: "" is the body itself, "/items/2/name" is instance["items"][2]["name"]. Array indices are 0-based. A property name is escaped: ~ is written ~0 and / is written ~1 (a property a/b is /a~1b).
• validate and accept never modify the instance they are given.

The schema dialect

A schema is a JSON object. Keywords it does not know (title, $comment, ...) are ignored. Malformed keyword values other than those listed under *invalid schema* are out of scope.

Rules

1. Every keyword in a schema is evaluated independently; a failing type does not hide other errors. A keyword only looks at instances of the kind in the "applies to" column and ignores the rest (minimum ignores a string; required ignores an array).
2. Types: integer is a whole number (3 and 3.0 both qualify, 3.5 does not); number is any number. Booleans are never numbers or integers — not for type, and not for the numeric keywords.
3. required is checked against the object itself, whether or not properties mentions the name.
4. properties and patternProperties both apply: a property that is listed and also matches a pattern must satisfy both schemas.
5. A property name may match several patterns; each matching schema applies.
6. additionalProperties (default true) governs the properties that are neither listed in properties nor matched by any patternProperties pattern. false: each such property is an error at its own path. A schema object: each such property's value must satisfy it, at the property's path.
7. JSON equality for enum/const: a boolean equals only the same boolean; 1 equals 1.0; arrays are equal element by element in order; objects are equal when they have the same keys with equal values (key order is irrelevant). Strings never equal numbers.
8. Glob dialect (pattern, patternProperties): * matches any run of characters including none; ? matches exactly one character; \ makes the next character literal (\*, \?, \\), and a \ that ends the pattern is a literal backslash; everything else matches itself, case-sensitively. The pattern must match the whole string.
9. Ordering (Part 1 rule). A result list is sorted by (path, keyword, message), comparing each as a plain string (code-point order). Duplicate errors are kept.

Worked example (Part 1)

[["register", "order", {
   "type": "object",
   "required": ["id", "items"],
   "properties": {
     "id": {"type": "integer", "minimum": 1},
     "items": {"type": "array", "minItems": 1, "items": {
       "type": "object", "required": ["sku"],
       "properties": {"sku": {"type": "string", "pattern": "SKU-*"},
                      "qty": {"type": "integer", "minimum": 1}}}}},
   "additionalProperties": false}],
 ["validate", "order", {"id": true,
   "items": [{"sku": "SKU-1", "qty": 2}, {"sku": "X9", "qty": 0}, {"qty": 1}],
   "note": "hi"}]]

returns

[true,
 [["/id", "type", "expected type integer"],
  ["/items/1/qty", "minimum", "expected a value >= 1"],
  ["/items/1/sku", "pattern", "string does not match pattern \"SKU-*\""],
  ["/items/2", "required", "missing required property \"sku\""],
  ["/note", "additionalProperties", "property not allowed"]]]

Part 1 — Fix the validator (GW-1412, GW-1419, GW-1423, GW-1430, GW-1436)

Goal: make validate follow rules 1–9 above for every keyword in the table (except $ref and composition, which come later).

Customer reports:

1. GW-1412 — "{"age": true} passes "type": "integer", and a minimum rule seems to be applied to true/false as if they were 1/0."
2. GW-1419 — "I list email in required but don't define it under properties: bodies with no email are accepted."
3. GW-1423 — "Errors inside arrays point at the wrong element, and a property called a/b shows up as /a/b instead of /a~1b."
4. GW-1430 — "additionalProperties: false rejects properties that I allowed with patternProperties."
5. GW-1436 — "const: 1 and enum: [0, 1] accept true and false."

Existing tests in tests/ show the expected shape; some fail today. Keep every Part 1 rule working as you add the later Parts.

Part 2 — $ref, $defs, recursion and allOf (GW-1451)

Goal: let schemas reuse and nest definitions, including recursive ones such as a tree of nodes, and reject schemas that could never terminate.

1. $ref is "#" (the whole registered document) or "#" followed by a JSON Pointer into the same document, e.g. #/$defs/node or #/$defs/a~1b/items. Pointer tokens are unescaped with ~1 → / and ~0 → ~ (decode ~1 first: ~01 is the two characters ~1). A token addresses an object key, or an array element when the target is a list and the token is a canonical index (0, 12).
2. {"$ref": R} means "the instance must also satisfy the schema at R". Errors of that schema are reported at the instance's own path (no wrapper error). Other keywords next to $ref are evaluated normally, as usual.
3. allOf is a non-empty list of schemas; the instance must satisfy every branch. Errors of all branches are reported at the instance's own path. In Part 2 an error produced twice is listed twice.
4. A property *named* $ref (inside properties) is just a property; $ref inside const, enum or default is data. Only the schema positions count: the root, values of $defs, properties and patternProperties, items, additionalProperties (when an object), and elements of allOf/anyOf/oneOf.
5. register returns "error: invalid schema" when: the schema (or any schema position) is not an object; a $ref is not a string of the form above, or does not resolve, or resolves to something that is not an object; allOf is not a non-empty list of objects; or there is an unguarded cycle: starting from some schema and following only $ref and the branches of allOf/anyOf/oneOf (steps that stay on the *same* instance value) you can reach the same schema object again. Recursion through properties, patternProperties, items or additionalProperties is fine: it consumes the instance.
6. Registration is atomic: a rejected schema does not replace one registered earlier under that name.

Worked example (Part 2)

[["register", "tree", {
   "$defs": {"node": {"type": "object", "required": ["name"],
     "properties": {"name": {"type": "string"},
                    "children": {"type": "array", "items": {"$ref": "#/$defs/node"}}}}},
   "$ref": "#/$defs/node"}],
 ["validate", "tree", {"name": "root", "children": [{"name": "a"}, {"name": 5, "children": [{}]}]}],
 ["register", "loop", {"$defs": {"a": {"$ref": "#/$defs/b"}, "b": {"allOf": [{"$ref": "#/$defs/a"}]}}}]]

returns

[true,
 [["/children/1/children/0", "required", "missing required property \"name\""],
  ["/children/1/name", "type", "expected type string"]],
 "error: invalid schema"]

Part 3 — Requirements change, anyOf/oneOf and defaults (GW-1480)

The console team reviewed real error output and changed two rules. Then product asked for alternatives and defaults.

3a. Two rules change (they replace Part 1 rule 9 and the Part 2 duplicate rule).

1. Duplicates: an error list never contains the same [path, keyword, message] twice; keep one.
2. Document order: errors are ordered by path in *document order*, then keyword, then message. Compare paths segment by segment (split on /, keeping the escaped form of each segment):

• a segment that is a *canonical array index* (0, or digits without a leading zero) sorts numerically, and before every segment that is not;
• other segments compare as plain strings (code-point order);
• a path sorts before every path it is a prefix of.

So /items/2 < /items/10, /a/b < /a-b (segments a < a-b, even though - sorts before / in the raw string), and /01 (not canonical) sorts after /9. Keyword and message are still compared as plain strings.

3b. anyOf and oneOf (non-empty lists of schemas, checked at registration like allOf). Every branch is evaluated against the same instance, independently. A branch *matches* when it yields no errors. The errors of a branch are its distinct errors (rule 3a.1 applied inside the branch) and include whatever nested anyOf/oneOf produced.

1. anyOf passes if at least one branch matches. Otherwise its errors are exactly the errors of the best branch: the one with the fewest errors; ties go to the lowest index. They are reported as they are (their own paths); there is no extra anyOf error.
2. oneOf passes if exactly one branch matches. If none matches, it reports the best branch's errors exactly like anyOf. If N >= 2 branches match, it reports one error at the instance path: keyword oneOf, message matched N branches, expected exactly 1.

3c. accept and default. ["accept", name, instance] validates; if there are errors it returns {"ok": false, "errors": [...]} (same list validate gives) and nothing is filled. Otherwise it returns {"ok": true, "value": V} where V is a copy of the instance with defaults filled in. Defaults are filled only after a successful validation, and the default values themselves are never validated.

1. The *effective default* of a schema is its own default value if it has the key (even null); otherwise the effective default of its $ref target, recursively; otherwise none. (allOf, anyOf and oneOf are not searched for defaults.)
2. At an object location with schema S, contributions happen in this order, and an earlier contribution wins for a property name: (1) for each S.properties entry in schema order, if the object does not have that property and the entry has an effective default, append the property with a deep copy of the default; (2) S's $ref target, processed as a schema for the same instance; (3) each allOf branch in order; (4) the first anyOf branch that matches; (5) the one oneOf branch that matches; (6) the walk continues into children.
3. The walk into children covers only properties that were in the original instance (never a value inserted from a default; defaults are used verbatim, nested defaults inside them are not applied). A child is processed with every schema that applies to it: its properties schema, each matching patternProperties schema, or additionalProperties (when it is an object and no property or pattern covered the name), in that order. Array elements are processed with items.
4. Which anyOf/oneOf branch matches is decided against the original instance, not the partially filled copy.
5. Appended properties go at the end of the object, in the order they were filled. Every inserted default is its own copy: filling one payload must never change the schema or a later payload.

Worked examples (Part 3)

[["register", "pay", {"oneOf": [
   {"type": "object", "required": ["card"], "properties": {"card": {"type": "string", "pattern": "????-????"}}, "additionalProperties": false},
   {"type": "object", "required": ["iban"], "properties": {"iban": {"type": "string", "pattern": "DE*"}}, "additionalProperties": false}]}],
 ["validate", "pay", {"card": "1234-56"}],
 ["validate", "pay", {"card": "1234-5678", "iban": "DE00"}],
 ["register", "num", {"oneOf": [{"type": "integer"}, {"type": "number", "minimum": 0}]}],
 ["validate", "num", 5],
 ["validate", "num", "x"],
 ["register", "rows", {"type": "array", "items": {"type": "integer"}}],
 ["validate", "rows", [0, 0, "a", 0, 0, 0, 0, 0, 0, 0, "b"]]]

returns

[true,
 [["/card", "pattern", "string does not match pattern \"????-????\""]],
 [["/iban", "additionalProperties", "property not allowed"]],
 true,
 [["", "oneOf", "matched 2 branches, expected exactly 1"]],
 [["", "type", "expected type integer"]],
 true,
 [["/2", "type", "expected type integer"], ["/10", "type", "expected type integer"]]]

(In the second result both branches have exactly one error, so the first branch wins the tie.)

[["register", "profile", {"type": "object", "required": ["name"], "properties": {
   "name": {"type": "string"},
   "role": {"type": "string", "default": "viewer"},
   "prefs": {"type": "object", "default": {"theme": "light"},
     "properties": {"theme": {"type": "string"}, "beta": {"type": "boolean", "default": false}}}}}],
 ["accept", "profile", {"name": "Ada"}],
 ["accept", "profile", {"name": "Bo", "prefs": {}}],
 ["accept", "profile", {"role": 5}]]

returns

[true,
 {"ok": true, "value": {"name": "Ada", "role": "viewer", "prefs": {"theme": "light"}}},
 {"ok": true, "value": {"name": "Bo", "prefs": {"beta": false}, "role": "viewer"}},
 {"ok": false, "errors": [["", "required", "missing required property \"name\""], ["/role", "type", "expected type string"]]}]

Part 4 — Scale (GW-1502)

The import team pushes customer CSVs through gate, and the load-test harness replays production shapes. Everything above must keep returning exactly the same results, but fast. The bulk operations let the harness describe big inputs compactly (see gate/fixtures.py):

• A template is JSON in which {"$repeat": n, "$each": T} expands to a list of n copies of T; inside it "$i" becomes the integer index of the innermost enclosing repeat and {i} inside any other string becomes that index in decimal. Object keys are never rewritten. A malformed $repeat (not exactly the keys $repeat and $each, or n not a non-negative integer) is invalid template.
• ["validate_fixture", name, template]: expand the template to one instance, validate it once. Returns {"count": <number of errors>, "first": <first 3 errors of the final list>}.
• ["validate_rows", name, template]: the template must expand to a list (else invalid template); each element is validated as its own request, exactly like a validate call. Returns {"rows": n, "invalid_rows": k, "errors": <sum of every row's error count>, "first": [[row, error], ...]} where first has up to 3 entries, one per invalid row in row order, holding that row's first error.

Workloads you should expect (all must give the same answers as the plain rules, in a few seconds of browser Python):

• one body with tens of thousands of values;
• thousands of small bodies validated one by one against a schema with dozens of $defs;
• a recursive tree schema about 20 levels deep whose oneOf branches both recurse into the same children;
• long strings (about 50 characters) tested against a glob with many *s that does not match;
• a schema re-registered under the same name between bulk calls.

Worked example (Part 4)

[["register", "rowschema", {"type": "object", "required": ["id", "n"],
   "properties": {"id": {"type": "string", "pattern": "row-*"}, "n": {"type": "integer", "maximum": 4}}}],
 ["validate_rows", "rowschema", {"$repeat": 6, "$each": {"id": "row-{i}", "n": "$i"}}],
 ["validate_fixture", "rowschema", {"id": "row-{i}"}]]

returns

[true,
 {"rows": 6, "invalid_rows": 1, "errors": 1, "first": [[5, ["/n", "maximum", "expected a value <= 4"]]]},
 {"count": 1, "first": [["", "required", "missing required property \"n\""]]}]

(Outside a $repeat, {i} is left as written, and a string without {i} is unchanged.)