Evaluation & Benchmarking

LLM Benchmark can be divided into three levels:

LLM Evaluation
├── Model Capability Benchmark
│   ├── Knowledge / Reasoning
│   ├── Mathematics / Coding
│   ├── Instruction Following
│   └── Long Context / Tool Calling
│
├── Inference Performance Benchmark
│   ├── TTFT / TPOT / ITL
│   ├── Latency / Throughput
│   ├── Concurrency / Goodput
│   └── GPU Memory
│
└── End-to-End Benchmark
    ├── Agent Task Success
    ├── RAG Quality
    ├── Tool Call Accuracy
    └── Latency / Cost / Reliability

1. Model Capability Benchmark

Used to measure the capabilities of the model itself, rather than the speed of the inference engine.

Common evaluation dimensions:

• Knowledge and factual accuracy.
• Mathematical and logical reasoning.
• Code generation.
• Instruction following.
• Multilingual capabilities.
• Long context understanding.
• Tool Calling and Agent capabilities.

Common indicators

Pass@k

For example, for a code question:

Generate 5 Solutions

Solution 1 → Fail
Solution 2 → Fail
Solution 3 → Pass
Solution 4 → Fail
Solution 5 → Fail

At least one of the five candidates sampled this time is correct, so the corresponding Pass@5 event is successful.

In formal evaluation, Pass@k is usually counted for multiple questions and multiple samplings; if n candidates are used and c pass the test, standard unbiased estimation can be used:

$$\operatorname{pass@k}=1-\frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Note: Pass@k is not simply multiplying the single answer Accuracy by k.

Review FAQ

• Data Contamination: Test questions or answers enter the training data.
• Benchmark Leakage: Test set, answers or review information leaked.
• Prompt Sensitivity: Different Prompt / Chat Template affects the results.
• Sampling Differences: Temperature and Top-p have different generation times.
• Judge Bias: Length preference, position preference, or model preference for LLM Judge.

When comparing models, try to unify the evaluation configuration and record the model version, Prompt, generation parameters and evaluation program version.

2. Common model capabilities Benchmark

SWE-bench

SWE-bench lets the model or Agent handle software problems in the real GitHub Repository.

GitHub Issue
      ↓
Agent Reads Repository
      ↓
Modify Code
      ↓
Run Tests
      ↓
Resolved / Failed

It is closer to real software engineering than HumanEval.

However, such results are highly dependent on Agent Scaffold, tool permissions, test environment, and execution budget and cannot be entirely attributed to the underlying model.

In addition, in 2026, OpenAI publicly pointed out that SWE-bench Verified had been significantly contaminated by data, and recommended that newer and more reliable evaluation benchmarks be used instead. Verified leaderboard scores should therefore be used with caution to judge cutting-edge coding capabilities.

3. Inference Performance Benchmark

Inference Performance Benchmark Study the running efficiency of the same model under different hardware, inference engines, quantization schemes and loads.

core indicators

TTFT and TPOT

For requests with an output Token number greater than 1:

$$TPOT=\frac{Latency-TTFT}{OutputTokens-1}$$

ITL counts the delay between adjacent outputs, while TPOT is generally the request-level average.

In Streaming, the two are related but not identical.

The measurement positions and statistical calibers of different Benchmark tools may be different, and the formula needs to be confirmed before comparison.

Goodput

A high Throughput does not necessarily mean a good user experience.

Suppose the server completes 100 requests per second, but many of these requests are delayed longer than required.

Goodput only counts requests that satisfy Service Level Objective (SLO).

For example:

SLO:
TTFT < 1 s
TPOT < 50 ms

100 requests/s total

80 requests/s meet SLO
20 requests/s violate SLO

Goodput = 80 requests/s

The actual SLO threshold should be determined by product requirements.

Goodput is especially important for producing Serving because it combines system processing power with user experience constraints.

4. Inference performance test scenario

load model

Two important concepts:

Closed-loop Benchmark

Fixed a certain number of concurrent clients. After one request is completed, the next one will be initiated.

N Concurrent Clients
       ↓
Request → Response → Next Request

Open-loop Benchmark

Send requests according to the set request arrival rate without waiting for the previous request to complete.

5 Requests/s
10 Requests/s
20 Requests/s
...

Open-loop is more suitable for studying queuing, latency, and system capacity for a given request arrival rate.

Testing considerations

Should be fixed:

• Model and numerical precision.
• GPU model and quantity.
• Inference engine version.
• Input/output Token length distribution.
• Request arrival rate or number of concurrencies.
• Prefix Cache settings.
• Sampling parameters.

Warming-up, test duration and multiple repeat measurements should also be considered.

It is recommended to report at least P50, P95, P99 for latency rather than just reporting the average.

5. End-to-End Benchmark

End-to-end evaluation is concerned with whether the entire application actually accomplishes user goals.

User Request
      ↓
Application / Agent
      ↓
Retrieval / Tool Calling
      ↓
LLM
      ↓
Final Result
      ↓
Task Evaluation

Common indicators:

Just because the model answers well does not mean that the entire Agent or RAG application will be able to complete the task correctly.

6. Agent Evaluation

Agent reviews focus on tool selection, parameters, execution paths, and final results.

User Task
    ↓
Plan / Reason
    ↓
Select Tool
    ↓
Generate Arguments
    ↓
Execute Tool
    ↓
Observe Result
    ↓
Continue / Finish

core indicators

most important principle

Task Success Rate is usually more important than Tool-call Accuracy alone.

For example, if a Agent tool selection is completely correct, but the user task is not completed in the end, the task should still be considered a failure.

At the same time, fewer Steps does not necessarily mean better; reducing steps cannot be at the expense of reducing the success rate.

For Agent systems, the evaluation environment should also be as repeatable as possible, including tool return values, external status, and test data.

7. RAG Evaluation

RAG is usually divided into two parts for evaluation: Retrieval and Generation.

User Query
     ↓
Retriever
     ↓
Retrieved Documents
     ↓
LLM
     ↓
Generated Answer

Retrieval indicator

Recall@k

For example, a certain Query has 4 related documents.

Top-5 found 3 of them:

[ Recall@5=\frac34=0.75 ]

If you only care about finding at least one relevant document, you can use Hit Rate@k, but it's not exactly the same as Recall@k.

Precision@k

There are 3 related articles in Top-5:

[ Precision@5=\frac35=0.6 ]

MRR

Mean Reciprocal Rank focuses on the position of the first relevant result.

Rank 1 → 1
Rank 2 → 1/2
Rank 5 → 1/5
No relevant result → 0

Average the reciprocal ranks of multiple Query.

NDCG@k

Normalized Discounted Cumulative Gain Also consider:

• Document relevance level.
• Document sort position.

Highly relevant documents ranked higher usually receive higher scores.

Common value range is 0~1.

Generation indicator

Faithfulness and Correctness

These two concepts are easily confused.

Faithfulness measures whether the answer is based on the search context.

Correctness measures whether the answer is factually correct.

For example, the retrieved document contains the error message, which LLM faithfully repeats:

Faithfulness → may be high
Correctness  → probably low

Therefore, RAG cannot only look at Faithfulness, but also needs to evaluate the quality of the search source and the correctness of the final answer.

Ragas uses Faithfulness, Context Precision, Context Recall, Response Relevancy, etc. as different evaluation indicators.

8. Commonly used evaluation tools

vLLM Benchmark example

vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --model your-model-id \
  --dataset-name random \
  --num-prompts 100 \
  --request-rate 5

This example uses random synthetic requests and is suitable for preliminary performance testing; you need to start a compatible model service and ensure that the model names are consistent before running.

The actual stress test should try to use the Prompt length and request distribution that are similar to those in the production environment.

9. Reliability of evaluation results

The credibility of Benchmark depends not only on the evaluation set but also on the experimental design.

Key points to check:

Especially in Agent/RAG applications, a small internal evaluation set corresponding to actual business tasks should be established.

For example, 100 real business requests are usually more valuable for decision-making than relying only on a general model ranking list.

Summary

The three most important principles:

1. capabilities, speed and system success rate need to be evaluated separately for, and you cannot just look at one Benchmark score.
2. A fair comparison requires fixed test conditions, otherwise the results may not be comparable.
3. The production system should focus on the comprehensive performance of quality, latency, throughput and cost, rather than just pursuing the highest Tokens/s.

---

Reference documentation

• lm-evaluation-harness
• vLLM Benchmarking
• GuideLLM
• Ragas Metrics
• DeepEval
• SWE-bench