Parasail serves open-weight LLMs on distributed clusters of GPUs managed by Kubernetes. The typical performance optimization loop is to reproduce a problem or measure a regression, iterate on code or config, set up and run benchmarks, and decide whether the change is worth shipping. This is a slow loop when done by humans. Most of it can run without human intervention, but getting real efficiency out of the system means using GPU capacity intelligently, validating changes on production traffic in a controlled way, and learning from both mistakes and successes.

This talk goes through what we built under the agent platform to handle the real-world constraints of GPU capacity and production traffic:

• Time-boxed GPU leases and a queue, so agents can schedule experiments without holding hardware idle or pulling capacity from customer deployments
• Deployment cloning and traffic mirroring, so a candidate and a baseline see the same live traffic before anything is promoted
• A canary stage with guardrails, automatic rollback, and a human in the loop
• Which steps stay as fixed scripts and which are left to the agent to decide