As AI technologies continue to evolve, the demand for processing both structured and unstructured data across diverse industries is rapidly growing. However, scaling AI batch processing across thousands of GPUs presents significant challenges in maintaining scalability, reliability, and observability. These challenges are further amplified when aiming for high-throughput batch data processing with large language models (LLMs), due to their computational demands and complexity.

In this presentation, we will demonstrate how we built a scalable and efficient batch inference stack using Ray at Anyscale. We begin by introducing Ray as a robust, scalable AI compute engine, followed by an in-depth look at RayData, a versatile and high-performance deep learning data processing pipeline. Next, we will introduce vLLM, the leading open-source framework for LLM inference, and illustrate how the combination of RayData and vLLM offers an ideal solution for scalable batch inference.