jitllm
JIT inference engine that adapts to hardware to run LLMs faster, even when they exceed memory.
Tested on macOS and Linux
Use it
macOS arm64
You need first
- Runtimes:
Go 1.26 or newer
brew tap and install jitllm via Homebrew
brew tap jitllm/jitllm 2>&1 brew trust jitllm/jitllm 2>&1 brew install jitllm 2>&1
Where the README and reality differ
- README: jitllm convert qwen3-30b-a3b (and flags after file)
What worked: Go flag parser requires flags BEFORE file: jitllm convert -o DIR <file> and jitllm run -n 16 <file> "prompt"; placing -o or -n after file errors
Check that it works
which jitllm && which jitllmd && jitllm --help 2>&1 | head -n 100; echo ---VERSION---; jitllm --version 2>&1; jitllm version 2>&1; jitllmd --help 2>&1The output should contain “jitllm convert”.
First thing to try
mkdir -p ~/models && curl -fL -o ~/models/stories260K.gguf https://huggingface.co/ggml-org/test-model-stories260K/resolve/main/stories260K-f32.gguf && ls -lh ~/models/stories260K.ggufjitllm convert -o ~/models/stories260K.jlm ~/models/stories260K.gguf 2>&1Start by converting a model to the local format with jitllm convert -o <output.jlm> <model.gguf>.
Linux x86_64
You need first
- Runtimes:
Go 1.26 or newer
Download install script for inspection
curl -fsSL https://jitllm.org/install.sh -o /tmp/jitllm-install.shInstall jitllm and jitllmd from release script
sh /tmp/jitllm-install.sh
Where the README and reality differ
- README: curl -fsSL https://jitllm.org/install.sh | sh
What worked: Download to /tmp/jitllm-install.sh, read it, then run sh /tmp/jitllm-install.sh; installer verifies checksums.txt and installs to ~/.local/bin - README: jitllm --version or --help works
What worked: jitllm version subcommand prints 0.2.6; jitllm --version and jitllm --help exit 2 with usage on stderr - README: On Linux the install script starts jitllmd as a systemd user service
What worked: Installer prints 'no systemd user session here: start the server with jitllmd serve' and skips service install where no systemd user session exists
Check that it works
jitllm versionThe output should contain “0.2.6”.
First thing to try
jitllm versionStart by converting a model to the local format with jitllm convert -o <output.jlm> <model.gguf>.
What it does
These come from the README; pickfit didn’t check them one by one.
- Generate optimized kernels for your hardwareNot tested
- Run models larger than memory using pagingNot tested
- Serve chat and completions through compatible APIsNot tested
How far pickfit got
macOS arm64
- Installs
- Starts
- 0 of 3 features work
- Works offline
Linux x86_64
- Installs
- Starts
- Answers requests
- Works offline
Other systems
Windows: the project says it runs there; pickfit hasn’t tried it.
Good for
- convert catalog models to local format
- chat with a local model
- serve chat and embeddings via API
- call tools with constrained decoding
An alternative to llama.cpp, vLLM and mistral.rs.
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
macOS arm64
Its main function works
- Replayed from scratch
- Works offline
- Checked on 1 Nov 2026
- Version 0.2.6
Differences the test machine hit
- README: jitllm convert -o output writes file directly
In the test: -o is a directory; convert creates DIR/basename.jlm inside it, nested when -o already looks like *.jlm; server only lists *.jlm files directly under -models, so copy to ~/models/tiny.jlm for discovery - README: curl http://localhost:4242/v1/chat/completions with model qwen3-30b-a3b
In the test: base model without chat template is rejected for chat; use /v1/completions or a chat-template model for chat completions
Linux x86_64
Its service answers requests
- Replayed from scratch
- Works offline
- Checked on 28 Oct 2026
- Version 0.2.6