The Awards Bench

Points or Cash · Pointonomics study · Data version 2026-11-04 · Run 2026-11-10 · General questions (JSON) · Personal questions (JSON)

1. Introduction

People increasingly ask language models how to use their points: which partner to transfer to, whether a fare is worth paying in miles, what status a card gives. The answers depend on rules that change several times a year and differ by country, and a confident wrong answer costs real money: points moved at the wrong ratio cannot be moved back. Existing travel benchmarks test itinerary planning and airline customer service [1–4]; none tests loyalty-program knowledge, where most questions have one right answer that can be checked.

We make three contributions: (i) a benchmark of 171 items with exact reference answers, each traceable to a dated source; (ii) a construction method that only keeps items whose answer is stable under the uncertainty in the data; (iii) results for 5 current models without tools and one with the dataset as tools.

2. Related work

τ-bench [1] evaluates agents that serve simulated airline customers under written policies and grades the final booking database, with repeated trials (pass^k) to measure consistency. TravelPlanner [2] tests multi-day itinerary planning under constraints in a fixed sandbox. TravelBench [3] adds multi-turn dialogue, implied preferences and requests that cannot be done, using real user questions in Chinese travel scenarios. TREK [4] emphasises executable plans with a deterministic evaluator. The Awards Bench is narrower and closer to a knowledge test: it asks about the rules of loyalty programs and the arithmetic of using them, and, like τ-bench and TREK, grades without a judge model.

3. The benchmark

3.1 Construction

No item is written by hand. A generator reads the research dataset (transfer partner lists per bank and per issuing country, award charts, hotel program terms, card benefit pages, each with its source and the date it was checked) and emits the items with their reference answers. Items are kept only when the answer does not depend on something the data cannot pin down. We exclude programs that price awards dynamically, prices whose notes give distance bands (a published price is for one gateway; a named route may sit in another band), a program that changed its pricing recently, card tiers named only by third-party blogs or worded inconsistently by the issuer, and award prices from Europe, which the data does not yet record as single numbers.

Decisions place the cash fare far from the decision line used by the assistant (1.5¢ per point to use points, 1¢ to pay cash): at 2.4¢ a point for a use-points item and 0.6¢ for a pay-cash item, so an award price that is a third off in the data does not change the reference answer. A pay-cash item is kept only when even the cheapest award recorded anywhere in the data, including price floors of dynamic programs, still loses to cash.

The personal part uses invented travellers on neutral routes. Its reference answers come from the functions a deployed assistant uses to track trips (nights without a hotel, legs without a flight, the US 24-hour refund rule, shared tickets, what a cancellation leaves open) and to reason about status (tier thresholds, whether buying hotel nights pays under stated perk values, wallets after a transfer at a country's ratio). Only status-run items where the nights cost at least twice the perks, or the perks are worth at least three times the nights, are kept.

4. Evaluation protocol

Each item is asked once, in a fresh session, with the system prompt below and a reply format (a JSON object). Closed-book models get no tools and are told not to search; we checked the event log of the OpenAI runs for search or command calls and found none. Claude models run through the Claude Agent SDK and the OpenAI model through the Codex CLI, each with its default reasoning settings. The open-book contestant is Claude Opus 5.5 with the dataset exposed as tools (transfer options, award charts, status rules, card benefits, points value, hotel status runs).

You are taking a quiz on airline and hotel loyalty programs, credit card points and award travel, as the rules stood in <month year>. Answer every question with your best answer, even when unsure. Reply with the JSON object asked for and nothing else.

Scoring is deterministic. Numbers must match (award prices within 5%, rounding within 0.5%); tiers must name the tier; dates, routes and sets of flights must match exactly; a refund deadline within one minute. A use-points decision is correct with the right choice and a points cost between 15% and 120% of the cheapest reference option (160% where the price comes from a distance band), so a cheaper option the model finds counts. Because most decisions are use-points, the decisions score is balanced accuracy, the mean of the two classes: always answering "points" scores 50%. Intervals are 95% Wilson score intervals [5] over items.

5. Results

5.1 General knowledge and decisions

5.2 A traveller's own trip and status

6. Analysis

Errors cluster where the rules differ by country: of the wrong answers on European Amex ratios (for example 3:2 to Avios on a Belgian card, 5:4 to Flying Blue on a French one), 44% give the US 1:1. Award prices are answered correctly 46% of the time across closed-book models; several of these charts changed in 2025–2026. With the dataset as tools, the same items are answered 99% correctly, so these are mostly gaps in knowledge rather than in arithmetic.

7. Limitations

Reference answers come from our dataset. It is sourced and dated, but it can be wrong; a first run of this benchmark surfaced five construction errors (§9), which is why every item links to its source and corrections are invited. One sample per item. Scores carry sampling noise beyond the item-level intervals shown; repeated runs would tighten them. Different harnesses. Claude and OpenAI models run through their vendors' own agent tools with default settings, not identical API calls. Contamination. Published items may end up in future models' training data. About 20% of the items (34 of 171), chosen by a fixed hash of their id, are held back and never published; scores cover all items, and a later version can compare a model's accuracy on the published and held-back items to tell knowing the rules from having seen the answers. New versions also regenerate items from newer data. Ceiling effects. Keeping only clear-cut decisions makes them easy: most models score close to 100% on decisions and on trip tracking, so those parts separate models less than the facts and personal-status items do. Harder decisions, near the line and with the data's uncertainty stated, are planned for the next version. Scope. Trips from the US for decisions; European award prices are not yet included; Gemini models were not available to us for this run.

8. Data and ethics

The personal part uses invented travellers; no user data is used. The items, reference answers and sources (all but the held-back 34) are published at /awards-bench/questions.json and /awards-bench/questions-personal.json under the Creative Commons Attribution 4.0 (CC BY 4.0) licence: reuse them, with attribution to Points or Cash and a link to this page. Corrections: hello@pointsorcash.app.

9. Version history

2026-11-04, November 2026. First release. A preliminary run was discarded after it showed five construction errors: a price attached to a route in another distance band, a program whose prices changed too recently to publish, transfer items that did not say "standard ratio" while a temporary bonus ran, two pay-cash items a cheaper award could overturn, and an unbalanced decisions score. All five were fixed before the run reported here. In the personal part, one item (the flight to rebook after a cancelled LaGuardia–Montreal flight) was re-scored to accept either New York airport, as the item already specified; answers were not re-run.

How to cite

@misc{awardsbench2026,
  title  = {The Awards Bench: What Language Models Know About Loyalty Programs},
  author = {{Points or Cash}},
  year   = {2026},
  note   = {Pointonomics study, data version 2026-11-04},
  url    = {https://pointsorcash.app/awards-bench}
}

References

1. Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. arxiv.org/abs/2406.12045
2. Xie, J. et al. (2024). TravelPlanner: A Benchmark for Real-World Planning with Language Agents. arXiv:2402.01622. arxiv.org/abs/2402.01622
3. TravelBench: A Broader Real-World Benchmark for Multi-Turn and Tool-Using Travel Planning (2026). ACL 2026. arXiv:2512.22673. arxiv.org/abs/2512.22673
4. TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning (2026). arXiv:2607.26977. www.alphaxiv.org/abs/2607.26977
5. Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212.