ResolveLab
Fault-injected HTTP environments that grade agent side effects for evaluation and training.
Tested on Linux
Use it
Linux x86_64
You need first
- Runtimes:
Python 3.11+
clone ResolveLab repo for editable install
rm -rf repo && git clone https://github.com/Ta0p1/ResolveLab.git repopip install editable with dev extras
cd repo && python3 -m pip install -e ".[dev]"
Check that it works
resolvelab --helpThe output should contain “ResolveLab: run, verify, compare.”.
First thing to try
cd repo && python examples/casedesk_quickstart.pyFrom the cloned repository folder, run python examples/casedesk_quickstart.py to see a sample agent episode graded.
What it does
- Run agents against an HTTP service that injects faultsTested
How to use it
- In the repository folder, run
python examples/casedesk_quickstart.py. - Read the grader line for the result, the claim and the number of commits.
- In the repository folder, run
- Grade episodes from server state and the commit logTested
How to use it
- In the repository folder, run
python examples/casedesk_env_loop.py. - Check the mean reward and mean steps reported over the dev episodes.
- In the repository folder, run
- Generate replay-verified training data for post-trainingTested
How to use it
- In the repository folder, run
python scripts/verify_casedesk_v2.py --dagger-every 20 --replay-every 10. - Check the output for verified provenance, the DAgger label count and the episode replay count.
- In the repository folder, run
How far pickfit got
Linux x86_64
- Installs
- Starts
- 3 of 3 features work
- Works offline
Good for
- run agents against HTTP service that injects failures
- grade episodes from server state and commit log
- produce training data with replay-verified teacher
How pickfit tested it
Each run starts from a clean machine and follows the steps above. How pickfit verifies
Linux x86_64
Its main function works
- Replayed from scratch
- Works offline
- Checked on 30 Oct 2026
- Version 0.4.0