vllm-rlt
Inference engine for recurrent language models with loop-level continuous batching.
Tested on Linux
Use it
Linux x86_64
You need first
- Runtimes:
Python 3.10+
clone repo from GitHub
git clone --depth 1 https://github.com/InferMatrixLab/vllm-rlt.git repocreate venv and uv pip install -e ./repo
uv venv --python 3.12 /tmp/venv && source /tmp/venv/bin/activate && uv pip install -e ./repo
Check that it works
/tmp/venv/bin/python -m vllm_rlt.entrypoints.serve --helpThe output should contain “--attention-backend”.
First thing to try
/tmp/venv/bin/python -c "
from vllm_rlt import SchedulerConfig, CacheConfig, ExecutionConfig
# F1: loop-level continuous batching
cfg = SchedulerConfig(mode='refill', max_num_seqs=4, wavefront_chunked_prefill=True)
print('F1 SchedulerConfig ok', cfg.mode, cfg.wavefront_chunked_prefill)
# also mixed What it does
- Batch recurrent model requests at the loop levelNot shown working
- Manage a paged KV cache that tracks loop depthNot shown working
- Decode with adaptive early exit within loop boundsNot tested
- Serve streaming OpenAI-compatible completionsNot shown working
How far pickfit got
Linux x86_64
- Installs
- Starts
- 0 of 4 features work
- Works offline
Good for
- run fixed-depth decoding or adaptive early-exit decoding
- serve streaming OpenAI-compatible completions
- run prefill/decode disaggregation across GPUs on one host
An alternative to vLLM.
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
Linux x86_64
Its main function works
- Replayed from scratch
- Works offline
- Checked on 1 Nov 2026
- Version 0.2.0
Differences the test machine hit
- README: uv pip install -e .
In the test: uv venv first or add --system flag; plain uv pip install fails with No virtual environment found