Which model actually finishes the job?

Our engineering perspective. Examples are illustrative, not client results or claims of measured model performance. Primary resources are linked in the text.

“Which model is best?” is an incomplete engineering question. Best at extracting product facts, planning a stock transfer or preparing a brand-sensitive customer reply? Under what latency, budget and review constraints? With which tools and which version of the business context?

A model can win a public benchmark and still be the wrong choice for a narrow production task. Public results are useful for discovery. Deployment selection needs evidence from the task you actually intend to run.

Evaluate the whole unit of work

Define the task boundary first. For a landing-page workflow, a successful result might be a factually correct draft that preserves the product format, includes supporting sources and passes a human review. For fulfillment, completion may require verifying a partner state and preparing a policy-compliant response.

Do not collapse these tasks into one score. Separate hard failures from subjective quality. A beautiful customer reply with a fabricated delivery date fails a different requirement from a correct reply that feels slightly formal. A weighting scheme should never let polished language cancel an unauthorized action.

Build a dataset with the difficult cases left in

Start with representative, permissioned examples. Include missing identifiers, contradictory stock records, multiple currencies, ambiguous product variants and incomplete policies. Keep a separate test set that is not used to tune the prompt. Track the origin and expected disposition of each case.

Use both common traffic and a deliberately difficult challenge set. Report their results separately. A model that handles routine cases well may still require strict routing away from rare consequential cases. Averaging them into a single attractive percentage can hide that distinction.

Repeat enough runs to expose variability. Retain model version, settings, tool definitions and context snapshot. If you change the prompt and the model together, the improvement cannot be attributed to the model alone.

Reasoning depth is a budget, not a virtue

A routine classification may need a short, constrained pass. A cross-functional plan may benefit from a larger reasoning budget. Giving every task maximum effort increases latency and cost without demonstrating that the final work improved.

Set escalation rules using observable conditions. Missing evidence, a failed validation, conflicting records or an action outside scope can trigger a stronger model or a human review. Do not rely solely on a model announcing its own confidence. A fluent answer can still be wrong.

Model-supported settings differ. Map your internal categories, such as focused, balanced and deep, onto each provider’s supported controls. Record that mapping with the deployment version. Identical labels across providers do not imply identical effort or comparable cost.

Count accepted results, not cheap calls

For a fixed evaluation set, add model cost, tool cost, infrastructure allocation and human correction time. Divide by the number of results meeting the acceptance criteria. Keep the rejected work in the numerator. Otherwise a model can look inexpensive by producing answers that someone else has to repair.

cost_per_accepted_result = (
    inference + tools + infrastructure + human_correction
) / accepted_results

Report alongside:
  hard_failure_count
  acceptance_rate
  median_and_tail_latency
  human_minutes_per_result
  unresolved_execution_count

This is a proposed accounting method, not a published Looqus performance table. There are no invented pass rates here. The purpose is to make an eventual comparison reproducible and useful to the operating team.

Promote a candidate gradually

First replay historical cases without external actions. Then use shadow mode against current inputs while the existing workflow remains responsible for delivery. Compare disagreement cases, not only averages. A limited release should have an owner, stop conditions and a known rollback version.

Keep the context and harness constant where possible. Model evaluation becomes much harder when the source data, permissions and downstream tools are changing at the same time. After promotion, continue sampling accepted results because the business and its inputs will drift.

Useful starting resources include Anthropic’s workflow patterns for deciding where routing helps, and the vLLM structured-output guide for machine-readable contracts. Neither replaces task-specific evaluation.

The commercial payoff is discipline. Model choice becomes a reversible engineering decision supported by evidence, while the business keeps the same context, ownership and completion standard.