mini-infer
Minimal educational LLM inference engine with continuous batching and paged KV-cache for Qwen2.5.
Tested on Linux
Use it
Linux x86_64
You need first
- Runtimes:
python 3.11+ - A GPU is optional
- Needs the network
Install mini-infer via uv tool
uv tool install mini-infer
Where the README and reality differ
- README: InferenceEngine accepts num_blocks/block_size implying any block size works
What worked: Sequence BlockTable is hardcoded to block_size 16, so InferenceEngine must be built with block_size=16 (default); custom block_size=4 crashes in write_sequence_kv
Check that it works
mini-infer --help 2>&1The output should contain “serve”.
First thing to try
mini-infer info 2>&1Run mini-infer info to see the tool's details and available commands.
What it does
- Run LLM inference with continuous batchingTested
How to use it
- In Python, build an
InferenceEnginefrommini_infer.enginewith a Qwen2 model and its config. - Queue prompts with
engine.add_request(prompt_token_ids=<tokens>, max_tokens=<n>). - Call
engine.step()repeatedly until each request has its output tokens.
- In Python, build an
- Manage the KV cache in paged blocksTested
How to use it
- Create an
InferenceEnginewithnum_blocks=<n>andblock_size=<size>. - Check free blocks with
engine.block_pool.num_freeand a request's blocks withseq.block_table.physical_block_ids.
- Create an
- Reuse shared prompt prefixes with a cacheTested
How to use it
- Create a
PrefixCacheand aBlockPoolfrommini_infer.cache. - Store a prefix with
insert_prefix_block(<tokens>, <block>, <pool>). - Look up a new prompt with
match_prefix(<tokens>, <pool>)to get the cached blocks.
- Create a
- Generate tokens with speculative decodingTested
How to use it
- Load a target model and a smaller draft model.
- Call
speculative_step(target_model=<model>, draft_model=<draft>, prefix_ids=<tokens>, k_draft=<k>)frommini_infer.speculative.verifier. - Read the new tokens and the number of accepted draft tokens it returns.
How far pickfit got
Linux x86_64
- Installs
- Starts
- 4 of 4 features work
- Works offline
macOS and Windows: the project says it runs there; pickfit hasn’t tried it.
Good for
- serve Qwen2.5 models via OpenAI-compatible streaming API
- chat with codebase in terminal with persistent memory
- generate token-budgeted AST skeleton for a codebase
- serve coding model locally as editor backend
An alternative to vLLM, llama.cpp and HuggingFace Transformers.
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
Linux x86_64
Its main function works
- Replayed from scratch
- Works offline
- Checked on 30 Oct 2026
- Version 0.2.7