PocketLLM
Single-device inference engine that runs Qwen3 text, vision and speech models via a C core.
Tested on Linux
Use it
Linux x86_64
You need first
- Runtimes:
python>=3.10
clone PocketLLM source from GitHub
git clone --depth 1 https://github.com/lvyufeng/PocketLLM.git /tmp/PocketLLMpip install PocketLLM from cloned source
pip install --user /tmp/PocketLLM 2>&1 && python3 -m pocketllm --help
Where the README and reality differ
- README: pip install pocketllm then `pocketllm devices`
What worked: clone the GitHub tree and run `pip install --user /tmp/PocketLLM`, then invoke as `python3 -m pocketllm devices` (no `pocketllm` executable is shipped; pyproject.toml defines no [project.scripts]) - README: pip install pocketllm installs the documented tool
What worked: PyPI pocketllm 0.1.1 fails to build in isolation: setup.py defines `class BuildExtensions(BuildExtension)` with BuildExtension=None when torch is absent, raising `TypeError: NoneType takes no arguments`; the current GitHub tree (0.2.0.dev0, numpy-only, no ext_modules) installs cleanly
Check that it works
python3 -m pocketllm --help 2>&1The output should contain “usage: pocketllm”.
First thing to try
python3 -m pocketllm devices 2>&1Run python3 -m pocketllm --help to see the available commands.
What it does
These come from the README; pickfit didn’t check them one by one.
- Run large language models on a single acceleratorNot tested
- Serve models over an OpenAI-compatible HTTP APINot tested
- Transcribe speech audio to textNot tested
- Process images with a vision-language modelNot tested
How far pickfit got
Linux x86_64
- Installs
- Starts
- Main function works
- Works offline
Good for
- generate text from a Qwen3 GGUF checkpoint via CLI
- serve model behind OpenAI-compatible HTTP
- transcribe speech to transcript with Qwen3-ASR
An alternative to llama.cpp.
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
Linux x86_64
Its main function works
- Replayed from scratch
- Works offline
- Checked on 30 Oct 2026
- Version 0.2.0.dev0