Reduce label bias

When a model chooses between candidates labelled A and B, the label assignment can influence its answer. Label permutations compare the same candidates under different assignments and combine their probabilities before choosing the winning original ID.

Use this for comparisons where the IDs are interchangeable, such as selecting a response or proposal. It adds inference work and can improve consistency, but does not guarantee a more accurate answer. Ordinary requests keep their usual single-pass behavior.

Define the candidates and the output

This example compares two answers to a word problem. Save both files in the directory where you run the client:

label-choice.input.json

Download label-choice.input.json

{
  "question": "A box contains 6 packets with 4 pencils in each packet. How many pencils are in the box?",
  "responses": {
    "A": "There are 10 pencils, because 6 + 4 = 10.",
    "B": "There are 24 pencils, because 6 × 4 = 24."
  }
}

label-choice.schema.json

Download label-choice.schema.json

{
  "type": "object",
  "properties": {
    "winner": {
      "type": "string",
      "enum": ["A", "B"],
      "description": "Choose the response that correctly answers input.question. Return its key from input.responses.",
      "x-runtime": {
        "label_permutations": {"scope": "/input/responses"},
        "probabilities": "all"
      }
    }
  },
  "required": ["winner"],
  "additionalProperties": false
}

The annotation belongs on winner, the field making the choice. Its scope is a JSON Pointer to the candidate object: /input/responses. You choose the property name; responses is not a reserved word. The object's keys must exactly match the field's string enum.

Each candidate's content stays intact. The returned winner always identifies the candidate in your original input. The question, descriptions and input outside the scope remain unchanged.

Run the request

You need a Blatent API key and the dependencies listed beside your chosen client:

Before you run this example

Call https://api.blatent.ai from your server. Set your API key in the same terminal where you run the example:

export BLATENT_API_KEY='YOUR_SERVICE_KEY'

Python requires Python 3.10+ and HTTPX:

python -m pip install "httpx>=0.28,<1"

TypeScript requires Node.js 24+ and uses its built-in fetch. The cURL examples require curl and jq.

Save the displayed files and the selected client in one directory. Run the client there with python example.py, node example.ts or bash example.sh. Downloads use the example's name in place of example.

Download Python

Python

import json
import os
from pathlib import Path
import httpx

request = {
    "input": json.loads(Path("label-choice.input.json").read_text(encoding="utf-8")),
    "schema": json.loads(Path("label-choice.schema.json").read_text(encoding="utf-8")),
}
with httpx.Client(
    base_url="https://api.blatent.ai",
    headers={"Authorization": f"Bearer {os.environ['BLATENT_API_KEY']}"},
    timeout=195,
) as client:
    response = client.post("/v1/resolve", json=request)
    response.raise_for_status()
    print(json.dumps(response.json()["output"]))

TypeScript

import { readFileSync } from "node:fs";

const request = {
    "input": JSON.parse(readFileSync("label-choice.input.json", "utf8")),
    "schema": JSON.parse(readFileSync("label-choice.schema.json", "utf8")),
};
const response = await fetch(
  "https://api.blatent.ai" + "/v1/resolve",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.BLATENT_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(request),
    signal: AbortSignal.timeout(195_000),
  },
);
if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
console.log(JSON.stringify((await response.json()).output));

cURL

set -eu

body=$(jq -n --argjson body '{}' \
  --slurpfile f0 'label-choice.input.json' \
  --slurpfile f1 'label-choice.schema.json' \
  '$body | setpath(["input"]; $f0[0]) | setpath(["schema"]; $f1[0])')
curl --fail-with-body --silent --show-error \
  "https://api.blatent.ai/v1/resolve" \
  -H "Authorization: Bearer $BLATENT_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary "$body"

The expected answer is {"winner":"B"}. The same schema can be registered and reused; its scope is checked against each request's input.

Read the optional probabilities

Set "probabilities": true on winner to include probabilities["/winner"], with value, probability and log_probability for each original candidate ID. The example's "all" spelling is a compatible alias. Set false or omit the flag to hide this metadata; label permutations still run and use the same selection rule.

You can also request probabilities without label permutations. These are relative preferences among supplied candidates, not calibrated confidence that an answer is correct.

Choose an appropriate scope

Keep candidate-specific facts inside the scoped values. Avoid references elsewhere such as “candidate A is preferred”: reassigning the display labels would make those references ambiguous. Descriptions should explain the comparison rather than attach meanings to A or B.

Do not apply this directly to meaningful category names such as approved and rejected. For a taxonomy, use interchangeable IDs whose values contain each category's name and description, then map the selected original ID back in your application.

Cost and supported combinations

Candidate count	Label assignments evaluated
2	2
3	6
4 or more	3 distinct, reproducible assignments

This release supports one annotated finite string field. Other output fields must be fixed by the schema once that choice is known. The scope must select an object; arrays, missing keys and mismatched labels are rejected.

You can supply optional input context, state and parameters. Combining label permutations with OPA policies, multiple independently generated fields, or ordering-only permutations is not supported. Unsupported annotations and combinations return an error; they are never silently ignored. See the approved annotations and response fields.