servebench
Load-tests OpenAI-compatible LLM endpoints and reports TTFT, TPOT, throughput and capacity.
Tested on Linux
Use it
Linux x86_64
You need first
- Runtimes:
python 3.11+ - Needs the network
install servebench with mock extra via uv tool
uv tool install "servebench[mock]"
Where the README and reality differ
- README: uv tool install servebench / pip install servebench
What worked: uv tool install "servebench[mock]" to provide servebench mock-server
Check that it works
servebench --help 2>&1The output should contain “mock-server”.
First thing to try
servebench mock-server --port 8000mkdir -p runs && servebench run --url http://127.0.0.1:8000/v1 --model mock-model --concurrency 1,2 --num-requests 4 --input-len 16 --output-len 8 --out runs/run-a.json 2>&1Run servebench run --url <server-url>/v1 --model <model> --out <file> against a running OpenAI-compatible server to start a benchmark.
What it does
- Load-test OpenAI-compatible LLM endpointsTested
How to use it
- Start your OpenAI-compatible server, or try a local one with
servebench mock-server --port <port>. - Run
servebench run --url <server-url>/v1 --model <model> --concurrency <levels> --num-requests <n> --input-len <tokens> --output-len <tokens> --out <file>. - Results are saved to the file you gave with
--out.
- Start your OpenAI-compatible server, or try a local one with
- Find how many concurrent users meet latency targetsTested
How to use it
- Run
servebench find-capacity --url <server-url>/v1 --model <model> --max <n> --num-requests <n> --out <file>. - Read the reported capacity and the TTFT and TPOT targets it was measured against.
- Run
- Compare benchmark runs side by side with chartsTested
How to use it
- Save two or more runs with
servebench run ... --out <file>. - Run
servebench compare-models <run-a-file> <run-b-file>to see the Model Serving Comparison.
- Save two or more runs with
- Find serving bottlenecks with a roofline modelTested
How to use it
- Save a run with
servebench run ... --out <file>. - Run
servebench roofline <file> --gpu-name "<your GPU>". - Read the Verdict line to see which limit the server is bound by.
- Save a run with
How far pickfit got
Linux x86_64
- Installs
- Starts
- 4 of 4 features work
- Works offline
Windows: the project says it runs there; pickfit hasn’t tried it.
Good for
- sweep a running LLM server across concurrency levels
- find how many concurrent users fit latency targets
- compare multiple benchmark runs side by side
- analyze hardware roofline bottleneck and utilization
- generate engine launch flags from auto-tuning
An alternative to vLLM's vllm bench serve, NVIDIA GenAI-Perf, guidellm and LLMPerf.
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
Linux x86_64
Its service answers requests
- Replayed from scratch
- Works offline
- Checked on 30 Oct 2026
- Version 1.1.3
Differences the test machine hit
- README: generic --version check
In the test: servebench --help (shows commands); --version exits 2 with No such option - README: servebench quick answers in ~1 minute
In the test: servebench quick auto-detected mock at 127.0.0.1:8000 and kept doubling load past c=16, still running after 90s