Context & Description
Running neural network models in full precision (FP32) limits batch throughput and increases VRAM usage. We need automated model quantization pipelines converting PyTorch model weights to FP16 and TensorRT / ONNX INT8 for high-throughput GPU inference.
Location: agent-engines/gpu_worker/models.py
Requirements
- Implement quantization script for evaluation and policy heads: FP32 -> FP16 and INT8 using TensorRT / ONNX Runtime quantization tools.
- Measure evaluation error: ensure < 0.5% degradation in Top-1 move accuracy compared to baseline FP32 model.
- Achieve at least 2.5x speedup in batch evaluation throughput on NVIDIA GPUs.
- Package quantized models into standard distribution artifacts.
Acceptance Criteria
What to be done
- Create
quantize_models.py in agent-engines/gpu_worker/.
- Update
models.py to support loading quantized TensorRT/ONNX engines.
- Add benchmark and accuracy validation test suite.
What NOT to be done
- Do NOT sacrifice tactical move accuracy (must maintain >= 99.5% agreement with FP32).
- Do NOT break CPU inference fallback support.
Context & Description
Running neural network models in full precision (FP32) limits batch throughput and increases VRAM usage. We need automated model quantization pipelines converting PyTorch model weights to FP16 and TensorRT / ONNX INT8 for high-throughput GPU inference.
Location:
agent-engines/gpu_worker/models.pyRequirements
Acceptance Criteria
agent-engines/tests/test_quantization.py.What to be done
quantize_models.pyinagent-engines/gpu_worker/.models.pyto support loading quantized TensorRT/ONNX engines.What NOT to be done