Product

News

User Story

Engineering

Culture

Jeffrey Wang

Kourosh Hakhamaneshi

Abrar Sheikh

Seiji Eicher

See how Ray Serve LLM updates in Ray 2.55 enable DP group fault tolerance for vLLM Wide Expert Parallelism (WideEP)

DP Group Fault Tolerance for vLLM

Announcing DP Group Fault Tolerance for vLLM WideEP Deployments with Ray Serve LLM

Alex Yang

Akshay Malik

Spencer Peterson (Google)

Ray Serve upgrades deliver faster online inference with HAProxy and gRPC, enabling scalable LLM inference on Ray with lower latency and higher throughput.

Major upgrades to Ray Serve: Online Inference with 88% lower latency and 11.1x higher throughput

Rui Qiao

We are excited to announce new Ray Serve LLM APIs that make it easy to deploy state of the art serving patterns with vLLM

vLLM Ray Serve Anyscale - 3

Ray Serve LLM on Anyscale: Wide-EP and Disaggregated Serving with vLLM

Ray Serve LLM on Anyscale: APIs for Wide-EP and Disaggregated Serving with vLLM

Justin Ji

Gene Su

Ray Serve is a scalable model serving library that optimizes traffic routing, scaling, and caching. For LLMs and MoE models, it leverages prefix caching and cache-aware routing to reduce latency, avoid redundant computation, and improve efficiency in multi-turn conversations and distributed inference.

Ray Serve: Custom Routing - Hero Image

Ray Serve: Reduce LLM Inference Latency by 60% with Custom Request Routing

Ray has become the engine for AI-native computing for thousands of AI- and digital-natives. Now Anyscale and Microsoft are partnering to bring the power of Ray to every enterprise.

Announcing first-party service - Anyscale on Azure

Announcing Anyscale on Azure: Build, Run and Scale AI-Native Workloads Securely on Azure Infrastructure

Ray, the open-source distributed compute framework for AI is joining the PyTorch Foundation, part of The Linux Foundation. 

Ray joins PyTorch Foundation

Ray is Joining The PyTorch Foundation

opensource stack blog thumbnail

Components of an Open Source AI Compute Tech Stack

An Open Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM

Explore a technical comparison of leading Reinforcement Learning (RL) libraries for LLMs from Ray. This guide analyzes frameworks like TRL, Verl, and RAGEN to help developers choose the best tools for RLHF, reasoning, and agentic AI.

The architecture of a Reinforcement Learning (RL) library is split into two primary components: Generation and Training. During the generation phase, an LLM Engine performs multi-turn rollouts within an environment to produce data and reward signals. This output is then fed into the training phase to update the model's parameters. This process forms a feedback loop, where the progressively improved model generates the next iteration of data for continuous refinement.

Open Source RL Libraries for LLMs Graph

Open Source RL Libraries for LLMs

This section is used to order the "Types" and "Tags" that show up for filters on the Blog Index

Types

Products / Libraries

Blog

Anyscale is the leading AI application platform. With Anyscale, developers can build, run and scale AI applications instantly.

Posts by Seiji Eicher

No posts found.