pickfit

BrowseDeveloper CLIs

servebench

Load-tests OpenAI-compatible LLM endpoints and reports TTFT, TPOT, throughput and capacity.

Tested on Linux

Use it

Linux x86_64

You need first

  • Runtimes: python 3.11+
  • Needs the network
  1. install servebench with mock extra via uv tool

    uv tool install "servebench[mock]"
Where the README and reality differ
  • README: uv tool install servebench / pip install servebench
    What worked: uv tool install "servebench[mock]" to provide servebench mock-server

Check that it works

servebench --help 2>&1

The output should contain “mock-server”.

First thing to try

servebench mock-server --port 8000
mkdir -p runs && servebench run --url http://127.0.0.1:8000/v1 --model mock-model --concurrency 1,2 --num-requests 4 --input-len 16 --output-len 8 --out runs/run-a.json 2>&1

Run servebench run --url <server-url>/v1 --model <model> --out <file> against a running OpenAI-compatible server to start a benchmark.

What it does

  • Load-test OpenAI-compatible LLM endpoints
    How to use it
    1. Start your OpenAI-compatible server, or try a local one with servebench mock-server --port <port>.
    2. Run servebench run --url <server-url>/v1 --model <model> --concurrency <levels> --num-requests <n> --input-len <tokens> --output-len <tokens> --out <file>.
    3. Results are saved to the file you gave with --out.
    Tested
  • Find how many concurrent users meet latency targets
    How to use it
    1. Run servebench find-capacity --url <server-url>/v1 --model <model> --max <n> --num-requests <n> --out <file>.
    2. Read the reported capacity and the TTFT and TPOT targets it was measured against.
    Tested
  • Compare benchmark runs side by side with charts
    How to use it
    1. Save two or more runs with servebench run ... --out <file>.
    2. Run servebench compare-models <run-a-file> <run-b-file> to see the Model Serving Comparison.
    Tested
  • Find serving bottlenecks with a roofline model
    How to use it
    1. Save a run with servebench run ... --out <file>.
    2. Run servebench roofline <file> --gpu-name "<your GPU>".
    3. Read the Verdict line to see which limit the server is bound by.
    Tested

How far pickfit got

Linux x86_64

  1. Installs
  2. Starts
  3. 4 of 4 features work
  4. Works offline

Windows: the project says it runs there; pickfit hasn’t tried it.

Good for

  • sweep a running LLM server across concurrency levels
  • find how many concurrent users fit latency targets
  • compare multiple benchmark runs side by side
  • analyze hardware roofline bottleneck and utilization
  • generate engine launch flags from auto-tuning

An alternative to vLLM's vllm bench serve, NVIDIA GenAI-Perf, guidellm and LLMPerf.

How pickfit tested it

Each run starts from a clean machine and follows the steps above. How pickfit verifies

Linux x86_64

Its service answers requests

  • Replayed from scratch
  • Works offline
  • Checked on 30 Oct 2026
  • Version 1.1.3

Differences the test machine hit

  • README: generic --version check
    In the test: servebench --help (shows commands); --version exits 2 with No such option
  • README: servebench quick answers in ~1 minute
    In the test: servebench quick auto-detected mock at 127.0.0.1:8000 and kept doubling load past c=16, still running after 90s

How to read a tool card

Stitching
pickfit installed this tool itself. Everything under the stitches comes from that run. A tool pickfit has not run never gets stitches.
Verified on your OS
The install steps ran on the system you chose. “Ran on Linux only” means it has not been run on yours yet.
macOS, Linux and WindowsWhere it ran
Green OS marks: pickfit ran the install steps there. Grey ones: the README says it runs there, and pickfit has not tried.
How far the run got
Installs and starts, its main function works, its service answers, it produces model output. A window means a desktop app opened; a plug, an MCP client used it. The mark sits beside the run’s result, under “Why it fits and how we know” and in “How pickfit tested it”.
Visual check
pickfit looked at screenshots of the app. Green: it looks right. Amber with a number: things that look wrong; open it to see them. Grey and dashed: not checked yet.
Known vulnerabilities
Known high or critical vulnerabilities (OSV.dev) in what it installs. Green: none known. Red with a number: how many; open it for the list, and read it before you install. Grey and dashed: not checked, which does not mean safe.
About the run
The day it ran, and the version that ran.
Source of a reason
Every reason a pick fits rests on one profile field. Open the reason to see the field and the value it relies on.
Hidden gem
High quality, little known.
Quality and popularity
Two scores out of 100. Quality: how well it installed and worked when pickfit tried it, and how good its install docs and upkeep are. Popularity: from stars, downloads and Hacker News.
Features checked
Of the main features the README advertises, how many the run actually exercised (green), and how many failed (red). Grey ones were not checked, and the card says why.
Not verified
A well-known tool Fitz mentions from general knowledge. No run record, no steps, no score.