pickfit

BrowseLocal AI tools

mini-infer

Minimal educational LLM inference engine with continuous batching and paged KV-cache for Qwen2.5.

Tested on Linux

Use it

Linux x86_64

You need first

  • Runtimes: python 3.11+
  • A GPU is optional
  • Needs the network
  1. Install mini-infer via uv tool

    uv tool install mini-infer
Where the README and reality differ
  • README: InferenceEngine accepts num_blocks/block_size implying any block size works
    What worked: Sequence BlockTable is hardcoded to block_size 16, so InferenceEngine must be built with block_size=16 (default); custom block_size=4 crashes in write_sequence_kv

Check that it works

mini-infer --help 2>&1

The output should contain “serve”.

First thing to try

mini-infer info 2>&1

Run mini-infer info to see the tool's details and available commands.

What it does

  • Run LLM inference with continuous batching
    How to use it
    1. In Python, build an InferenceEngine from mini_infer.engine with a Qwen2 model and its config.
    2. Queue prompts with engine.add_request(prompt_token_ids=<tokens>, max_tokens=<n>).
    3. Call engine.step() repeatedly until each request has its output tokens.
    Tested
  • Manage the KV cache in paged blocks
    How to use it
    1. Create an InferenceEngine with num_blocks=<n> and block_size=<size>.
    2. Check free blocks with engine.block_pool.num_free and a request's blocks with seq.block_table.physical_block_ids.
    Tested
  • Reuse shared prompt prefixes with a cache
    How to use it
    1. Create a PrefixCache and a BlockPool from mini_infer.cache.
    2. Store a prefix with insert_prefix_block(<tokens>, <block>, <pool>).
    3. Look up a new prompt with match_prefix(<tokens>, <pool>) to get the cached blocks.
    Tested
  • Generate tokens with speculative decoding
    How to use it
    1. Load a target model and a smaller draft model.
    2. Call speculative_step(target_model=<model>, draft_model=<draft>, prefix_ids=<tokens>, k_draft=<k>) from mini_infer.speculative.verifier.
    3. Read the new tokens and the number of accepted draft tokens it returns.
    Tested

How far pickfit got

Linux x86_64

  1. Installs
  2. Starts
  3. 4 of 4 features work
  4. Works offline

macOS and Windows: the project says it runs there; pickfit hasn’t tried it.

Good for

  • serve Qwen2.5 models via OpenAI-compatible streaming API
  • chat with codebase in terminal with persistent memory
  • generate token-budgeted AST skeleton for a codebase
  • serve coding model locally as editor backend

An alternative to vLLM, llama.cpp and HuggingFace Transformers.

How pickfit tested it

Each run starts from a clean machine and follows the steps above. How pickfit verifies

Linux x86_64

Its main function works

  • Replayed from scratch
  • Works offline
  • Checked on 30 Oct 2026
  • Version 0.2.7

How to read a tool card

Stitching
pickfit installed this tool itself. Everything under the stitches comes from that run. A tool pickfit has not run never gets stitches.
Verified on your OS
The install steps ran on the system you chose. “Ran on Linux only” means it has not been run on yours yet.
macOS, Linux and WindowsWhere it ran
Green OS marks: pickfit ran the install steps there. Grey ones: the README says it runs there, and pickfit has not tried.
How far the run got
Installs and starts, its main function works, its service answers, it produces model output. A window means a desktop app opened; a plug, an MCP client used it. The mark sits beside the run’s result, under “Why it fits and how we know” and in “How pickfit tested it”.
Visual check
pickfit looked at screenshots of the app. Green: it looks right. Amber with a number: things that look wrong; open it to see them. Grey and dashed: not checked yet.
Known vulnerabilities
Known high or critical vulnerabilities (OSV.dev) in what it installs. Green: none known. Red with a number: how many; open it for the list, and read it before you install. Grey and dashed: not checked, which does not mean safe.
About the run
The day it ran, and the version that ran.
Source of a reason
Every reason a pick fits rests on one profile field. Open the reason to see the field and the value it relies on.
Hidden gem
High quality, little known.
Quality and popularity
Two scores out of 100. Quality: how well it installed and worked when pickfit tried it, and how good its install docs and upkeep are. Popularity: from stars, downloads and Hacker News.
Features checked
Of the main features the README advertises, how many the run actually exercised (green), and how many failed (red). Grey ones were not checked, and the card says why.
Not verified
A well-known tool Fitz mentions from general knowledge. No run record, no steps, no score.