Helios
Readable PyTorch engine that serves Qwen3-4B locally with batching and paged attention.
Tested on Linux
Use it
Linux x86_64
You need first
- Runtimes:
python>=3.11 - A GPU is required
- Needs the network
Clone Helios source
git clone https://github.com/sidmanale643/helios.git /tmp/helios-srcInstall dependencies with uv sync
cd /tmp/helios-src && uv sync
Where the README and reality differ
- README: uv sync
What worked: Clone the repo first (git clone https://github.com/sidmanale643/helios.git) then run uv sync inside the clone; the work dir ships only README.md - README: Standard --version/--help CLI flags
What worked: helios entrypoint is a server with no --version/--help flags; version confirmed via importlib.metadata (helios 0.1.0) and /health reports model and warmup_batch_sizes
Check that it works
cd /tmp/helios-src && ls -l .venv/bin/helios && .venv/bin/python -c "import importlib.metadata, torch; print('helios', importlib.metadata.version('helios')); print('torch', torch.__version__); print('cuda', torch.cuda.is_available())"The output should contain “helios 0.1.0”.
First thing to try
cd /tmp/helios-src && ls -l .venv/bin/helios && .venv/bin/python -c "import importlib.metadata, torch; print('helios', importlib.metadata.version('helios')); print('torch', torch.__version__); print('cuda', torch.cuda.is_available())"What it does
Good for
- serve OpenAI-style chat completions locally
- batch concurrent chat requests continuously
- benchmark concurrent inference with timing data
How far pickfit got
Linux x86_64
- Installs
- Starts
- Produces output
- Works offline
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
Linux x86_64
It produces model output
- Replayed from scratch
- GPU: NVIDIA A10
- Works offline
- Checked on 1 Nov 2026
- Version 0.1.0
Differences the test machine hit
- README: First start downloads tokenizer and model weights
In the test: Pre-download Qwen/Qwen3-4B snapshot (tokenizer.json + safetensors) with huggingface_hub snapshot_download before starting the server so the server reuses cache and never fetches during the check