Devansh Kamlani
When I am asked to choose an LLM for an enterprise system, the question I always push back on is “which model is best?” The honest answer is “best for what?” A model that crushes a generic benchmark might be expensive, slow and a poor fit for a regulated workflow. A model that scores poorly on multilingual reasoning might be perfect for a narrow English-language summarisation use case. The real work of model selection is not picking a winner from a leaderboard; it is building an evaluation that answers the questions your business actually cares about.
LLM Evaluation Frameworks: How Architects Pick Models
In this article I want to share how I run evaluation as a solution architect, from the canonical benchmarks I use as a sanity check to the bespoke eval suites I design for specific applications. I will cover the categories of evaluation that matter (capability, task-specific, regression, safety, cost-performance), the frameworks I reach for (lm-evaluation-harness, BIG-bench, Ragas, OpenAI Evals), and the discipline of running evaluations in CI so quality regressions are caught before users notice.
This is written for architects, ML leads and platform engineers who need a defensible answer when a leader asks “why did you pick this model?”. If you have ever lost a model selection argument to vibes, this article is for you.
Why evaluation is the architect’s job
In most organisations, evaluation falls into a gap between data science and platform engineering. Data scientists prefer to iterate on models; engineers prefer to deploy what is given. Nobody owns the question of whether the chosen model is fit for purpose, and the consequence is that model selection ends up being driven by leaderboards, vendor pitches or whichever model is in the news that week.
I argue that evaluation is fundamentally an architect’s responsibility. The architect is the person who has to defend the model choice to leadership, integrate it with downstream systems and own the consequences when it fails. That makes evaluation a design artefact, not a one-off science project.
If you cannot point to a measurable test that says “this model is the right choice for this use case”, you have not actually made a choice. You have made an assumption.
A practical implication: I keep an evaluation portfolio for every system I architect. It lives in the same repository as the system design, and it evolves with the system over time.
The five evaluation categories
I think about evaluation in five categories. A serious model selection process touches all five, in some proportion.
| Category | Question it answers | When you need it |
|---|---|---|
| Capability benchmarks | Is this model smart enough generally? | Initial filtering |
| Task-specific | Is it good at the specific job? | Final selection |
| Regression | Did the system get worse this week? | Production monitoring |
| Safety | Does it behave under attack? | Pre-launch and continuous |
| Cost-performance | Is the quality worth the cost? | Final selection and ongoing |
The cost of getting any one of these wrong is high. A model that fails the capability bar wastes engineering effort. A model that passes capability but fails task-specific evals ships broken features. A model with no regression suite drifts silently. A model with no safety suite becomes the next news story.
The discipline of separating the categories also helps you choose the right tool. Lm-evaluation-harness is great for capability, useless for safety. Ragas is great for RAG, irrelevant for general agentic tasks.
Capability benchmarks and their limits
Capability benchmarks are the famous leaderboards: MMLU, GPQA, HumanEval, GSM8K, BBH, ARC, MATH. They measure broad reasoning and knowledge across standardised tasks. They are useful as a first filter.
I use capability benchmarks to answer three questions:
- Is the model at the intelligence tier I need?
- Has the model been trained on the languages and domains I care about?
- Are there obvious gaps (maths, code, reasoning) that disqualify it?
The limit of capability benchmarks is that they correlate poorly with task-specific performance. A model with a higher MMLU score might generate worse customer-service responses than a smaller, better-fine-tuned competitor. I have seen organisations pick a model based purely on a benchmark and then discover, six months in, that the benchmark told them nothing about the actual workload.
Treat capability benchmarks like a credit score for the model: necessary, not sufficient.
I usually filter to the top three or four candidate models using benchmarks, then run task-specific evaluations to make the final call.
Task-specific evaluation suites
Task-specific evaluation is where the real architectural work happens. The goal is to build a representative dataset of inputs your system will see, with expected outputs or scoring rubrics, and to run candidate models against it.
A good task-specific suite has:
- At least a few hundred examples, ideally a thousand or more, covering the diversity of real traffic.
- A mix of easy, medium and hard cases drawn from production logs (sanitised) or expert curation.
- Multiple scoring methods: exact match for deterministic tasks, rubric-based scoring for open-ended tasks, LLM judges where neither applies.
- Documentation of edge cases the system must handle.
I draw test data from three sources: real production traffic (when permitted), expert-written prompts that exercise hard cases, and synthetic data generated by a separate model. The combination prevents the suite from becoming brittle.
One client I worked with built a 1,200-example evaluation suite for their claims-summarisation system. When we ran four candidate models, the cheapest open-weight model actually outperformed the premium closed-source option on their specific task. We saved roughly seventy per cent of their projected inference budget.
Regression evaluation in production
Regression evaluation is the continuous version of task-specific evaluation. The same suite runs against the production system on a schedule, and the results are tracked over time.
The patterns I deploy:
- An hourly smoke-test eval on a 20-example subset to catch immediate problems.
- A nightly full eval on the complete suite to detect slower drift.
- A weekly extended eval against an expanded suite for thorough review.
- A pre-deployment eval as a CI gate before any prompt or model change.
Regression evaluation is what catches silent model updates from providers, prompt regressions from well-meaning engineers and retrieval-index degradation from data pipeline issues. I have seen it pay for itself in a single incident.
The regression suite has to be stable; if the suite changes, the metric becomes meaningless. I version the suite as carefully as I version the application code.
Safety and red-team evaluation
Safety evaluation answers questions like: does the model leak system prompts, does it refuse harmful requests, does it produce hate speech, does it allow jailbreaks. These questions are mandatory in regulated industries and increasingly mandatory under the EU AI Act and similar regimes.
I build safety suites around three sources:
- Public adversarial datasets such as HarmBench, AdvBench and the JailbreakBench corpus.
- Internal red-team prompts written by security engineers who understand the application context.
- Domain-specific safety cases (financial advice, medical advice, legal advice) where regulation applies.
The output of safety evaluation is two metrics: a refusal rate on adversarial prompts (should be high) and a compliance rate on benign prompts (should be high). Both matter. A model that refuses everything is as broken as a model that refuses nothing.
Safety evaluation must run continuously. Models change; prompts change; the threat landscape changes. A safety baseline from six months ago is a polite fiction.
Cost-performance evaluation
Cost-performance evaluation is the architect’s economic check. It pairs every quality measurement with a cost measurement, so the conversation about model selection becomes “is the extra quality worth the extra cost?”
I track:
- Cost per request, including prompt tokens, completion tokens and any tool calls.
- Quality score on the task-specific suite.
- Latency at p50 and p95.
- Cost per “good” answer, defined as a quality score above a threshold.
The last metric is the one that surprises people. A cheap model that produces good answers seventy per cent of the time can be more expensive per good answer than a premium model that produces good answers ninety-five per cent of the time, because of retry logic and downstream remediation cost.
A simple table I produce for every model selection:
| Model | Quality score | Cost per request | Latency p95 | Cost per good answer |
|---|---|---|---|---|
| Model A | 0.92 | 0.012 GBP | 1,200 ms | 0.013 GBP |
| Model B | 0.78 | 0.003 GBP | 800 ms | 0.004 GBP |
| Model C | 0.85 | 0.006 GBP | 1,000 ms | 0.007 GBP |
The right answer depends on the use case, but the table makes the trade-off explicit.
Evaluation frameworks worth knowing
The framework landscape has consolidated nicely. The frameworks I reach for in 2026:
- lm-evaluation-harness by EleutherAI. The de facto standard for capability benchmarks.
- BIG-bench by Google. Broader and more diverse than MMLU, good for stress testing.
- OpenAI Evals. Open framework with strong support for custom evals and a large community library.
- Ragas. Specialised for RAG systems, scores retrieval and generation separately.
- TruLens. Strong observability-evaluation crossover, good LLM-judge templates.
- DeepEval. Pytest-style framework for custom evals, integrates well with CI.
- Promptfoo. CLI-driven eval framework, popular for prompt comparison.
The choice depends on what you are evaluating. Capability is lm-evaluation-harness territory. Custom application evals are usually DeepEval or Promptfoo. RAG-specific is Ragas. The frameworks complement rather than compete.
lm-evaluation-harness and BIG-bench
Lm-evaluation-harness is the framework I run first when evaluating a new candidate model. It supports hundreds of tasks out of the box, runs against any model with a chat or completion interface, and produces results comparable to public leaderboards.
A typical run for a candidate model:
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-70B \
--tasks mmlu,gsm8k,hellaswag,arc_challenge,truthfulqa \
--batch_size 8 \
--output_path results/llama-3.1-70b
BIG-bench complements lm-evaluation-harness with a broader and more diverse set of tasks, including some that target reasoning, social understanding and edge cases. I use BIG-bench when I want to stress-test a model beyond the standard benchmarks.
Neither framework tells you whether a model is good for your specific application. They tell you whether the model is in the right tier of capability to be worth further evaluation.
Ragas for RAG systems
Ragas is the framework I default to for RAG evaluation. It separates retrieval quality from generation quality, which is the right architectural distinction.
The metrics Ragas exposes:
- Context precision. Of the retrieved documents, what fraction are relevant.
- Context recall. Of the relevant documents, what fraction were retrieved.
- Faithfulness. Is the generated answer grounded in the retrieved context.
- Answer relevance. Is the answer responsive to the question.
For a RAG architect, these four metrics tell the story. A high faithfulness with low context recall means the model is honest but missing information. Low faithfulness means the model is hallucinating despite having the right context. The diagnosis maps cleanly to architectural fixes.
I run Ragas continuously against production RAG systems with a sampled subset of real traffic. The trend lines on the four metrics are the most useful single signal for RAG health.
OpenAI Evals and custom evals
OpenAI Evals is the framework I reach for when I need to write something specific to the application. It supports rule-based scoring, LLM-judge scoring and custom Python scoring, and it integrates well with CI.
A simple custom eval looks like:
id: claim-summary-faithfulness
samples_jsonl: data/claims_eval.jsonl
eval: ClassifierEval
class: evals.classifier.Classifier
modelgraded_spec: closedqa
For domain-specific evals (legal, medical, financial), I usually write a custom Python scorer that combines deterministic checks (does the output mention the required disclaimers?) with an LLM-judge component (is the explanation correct and helpful?).
The discipline of writing custom evals forces clarity about what “good” means for the application. That clarity is often the most valuable artefact of the evaluation process, regardless of which framework you use.
LLM-as-judge patterns
LLM-as-judge is the pattern that makes large-scale evaluation feasible. A large model scores the output of a candidate model against a rubric, producing a quantitative metric that would otherwise require human labelling.
The patterns that work:
- Single judge with clear rubric. Best for binary or three-point scores.
- Pairwise comparison. Best for ranking two candidate models, robust to scaling biases.
- Multi-judge ensembling. Average over three or more judges for higher-stakes decisions.
- Calibration against human labels. Run the judge on a sample also scored by humans, measure agreement, retune the prompt or rubric.
The pitfalls are real. Judge models have biases (they often prefer their own family’s outputs), they can be lazy on subtle distinctions, and their behaviour drifts over time as providers update them. Calibration against human labels is the only way to keep the judge trustworthy.
I always specify the judge model version in the eval pipeline. Using “GPT-4” is not enough; I want “gpt-4o-2024-08-06” or its successor, pinned and tracked.
Designing a custom evaluation suite
When I design a custom suite for a client, I follow a sequence.
- Define the user goals the system must achieve.
- Translate each goal into one or more measurable behaviours.
- Collect or write 500 to 2,000 examples covering the behaviours.
- Split into curated (gold-label) and sampled (real traffic) sets.
- Define scoring methods (exact match, rubric, judge) per behaviour.
- Calibrate judges against human labels on a 50 to 100 example sample.
- Establish a baseline score with the current model.
- Wire the suite into CI as a release gate.
- Run the suite continuously in production.
- Review the suite quarterly and refresh examples.
Steps six and ten are the ones most teams skip and the ones that determine whether the suite ages well. A well-maintained suite gets more valuable over time; a neglected suite becomes a liability.
The biggest design choice is the ratio of curated to sampled data. I usually aim for roughly half and half. Curated data ensures coverage of hard cases; sampled data ensures the suite reflects reality.
Evaluation in CI and production
Eval in CI closes the loop between development and production. Every prompt change, model swap or retrieval-index update should trigger an automatic eval run, with a defined pass threshold that gates promotion.
A typical pipeline:
- PR opened with a prompt or config change.
- CI loads the change, runs the regression eval suite against staging.
- Results are posted to the PR with pass/fail and per-metric breakdowns.
- PR is blocked if any guarded metric regresses beyond a tolerance.
- After merge, the change is rolled out behind a feature flag.
- Production eval runs every hour and alerts on regressions.
The most underrated piece is the per-PR results posting. Engineers learn to think about evals when they see scores on every change. Without that feedback loop, evals become invisible and ignored.
I treat the eval suite as a product owned by the AI platform team. It has a roadmap, an owner and a review cadence. The teams that take this seriously ship faster and break less.
