LLM on EKS: Serving with vLLM

Last year, I mentioned that I'm interested in learning how to serve LLMs in production. At first it was just curiosity, but over time I wanted to actually try building something—not just reading about it. This post is a small step in that direction: serving an LLM using vLLM , deployed on Amazon EKS , provisioned the infra using AWS CDK , and wrapped into a simple chatbot using Streamlit . TL;DR Exploring LLM serving on a Kubernetes cluster (EKS) Using vLLM as the inference engine Provisioning t