How to Choose an LLM for a Business Application in 2026: Evals, Data Residency and Model Retirement

Most model comparisons start with a leaderboard. The data behind those leaderboards now argues against using them as the deciding factor. Stanford HAI's 2026 AI Index reports that top model performance is converging, with four companies now clustered within 25 Elo points of each other. It also finds that the gap between the best closed and the best open model was 3.3% in March 2026, up from 0.5% in August 2024 but still small.

The same report warns that benchmarks meant to stay difficult for years are saturated within months, and cites a review that found invalid-question rates ranging from 2% on MMLU Math to 42% on GSM8K. A public score tells you a model is capable in general. It does not tell you whether it reads your invoices, follows your refund policy or writes in your brand's Japanese. That has to be measured on your own work.

Both major vendors put this step first. OpenAI's model selection guide says to choose acceptance criteria before comparing models, to repeat the comparison across representative tasks, and to measure completion time alongside quality, including the time spent reviewing or repairing the result. Anthropic's guide says that having a good evaluation set is the most important step when deciding whether to upgrade or change models.

In practice, a usable test set is a spreadsheet of real inputs from the workflow, the answer a competent employee would give, and a rule for marking each output pass or fail.

Anthropic describes two starting points. Efficiency-first means beginning with a fast, low-cost model, testing thoroughly and upgrading only for specific capability gaps; it suits prototypes, tight latency requirements and high-volume, straightforward tasks. Capability-first means beginning with the strongest model for the task and moving to more efficient options once the workflow is proven; it suits complex reasoning and work where accuracy outweighs cost.

OpenAI's guide frames the same trade-off as four questions: how often the workflow runs, how quickly you need the result, how the output will be used, and how important quality is. Its advice is to compare options on the same inputs and keep the lightest setting that meets your quality bar. Both vendors also offer a reasoning or effort setting within a single model, and Anthropic notes that tuning effort is often a better lever than switching models.

Price per token is a poor guide on its own. A cheaper model that needs more retries, longer prompts or more human correction can cost more per finished task than a stronger one, which is why OpenAI's guide counts review and repair time as part of the comparison. Compare candidates on the cost of a correct, usable result.

For Canadian and Japanese companies, this step often narrows the shortlist more than any benchmark. OpenAI's data controls documentation states that data sent to its API has not been used to train its models since March 1, 2023 unless the customer opts in, and that abuse monitoring logs, which can contain prompts and responses, are kept for up to 30 days by default.

Data residency is where the detail matters. OpenAI offers a Canada region and a Japan region for API projects, but its own table lists both as regional storage only, with no regional processing. The documentation explains that in regions without regional processing, customer content may be processed and temporarily stored outside the region. Using any region other than the United States also requires approval for abuse monitoring controls and a Modified Retention amendment, and data residency endpoints carry a 10% price uplift for eligible models released on or after March 5, 2026.

The lesson applies to every vendor, not only OpenAI: ask where data is stored at rest, where inference runs, what is retained and for how long, and what the regional option costs. Get the answers in writing before the privacy review, not after the build.

The same model can often be reached three ways: directly from the model vendor, through a cloud provider's AI platform, or, for open-weight models, on infrastructure you control. The choice changes who you contract with, which regions are available and how the model's lifecycle is managed. Anthropic's deprecation documentation, for example, notes that partner-operated platforms such as Amazon Bedrock set their own retirement schedules, so a model's status and dates can differ from the vendor's own API.

Open-weight models deserve a fair look. With a 3.3% gap to the best closed model in Stanford's March 2026 measurement, they can be good enough for many workflows, and self-hosting gives full control over where inference runs. The trade-off is operational: you take on hosting, scaling, security patching and model updates that a vendor API handles for you. Run them through the same test set as everything else.

Whichever model you choose, that exact version will not be available forever. Anthropic's documentation says it regularly retires older models, gives at least 60 days' notice before retiring publicly released models, and that requests to a model past its retirement date fail. Its own table shows the claude-sonnet-4-5-20250929 version deprecated on September 30, 2026, with a tentative retirement date of November 30, 2026. Anthropic recommends testing replacement models well before the retirement date.

This is why the test set from step 1 is an asset, not a one-off exercise. A migration becomes a rerun and a comparison instead of a guess.

Use this list to structure a model decision with the people who will sign off on it: the product owner, whoever handles privacy, and the team that will maintain the system.