Load balancing
Session
HTTP
How GitHub Copilot Serves 400 Million Completion Requests a Day
Monday Nov 18 / 03:55PM PST
GitHub Copilot is the largest LLM powered Code Completion service in the world, serving hundreds of millions of requests per day with an average response time of under 200ms. This is the story of the architecture which powers this product.
David Cheney
Lead, Copilot Proxy @GitHub, Open Source Contributor and Project Member for Go Programming Language, Previously @VMware