Terrill Dickey
January 24, 2025 14:36
Explore NVIDIA’s approach to horizontal autoscaling of NIM microservices on Kubernetes using custom metrics for efficient resource management.
NVIDIA has introduced a comprehensive approach to horizontally auto-scaling NIM microservices on Kubernetes, as detailed by Juana Nakfour on the NVIDIA Developer Blog. This method leverages Kubernetes Horizontal Pod Autoscaling (HPA) to dynamically scale resources and optimize compute and memory usage based on custom metrics.
Understanding NVIDIA NIM Microservices
The NVIDIA NIM microservice serves as a deployable model inference container on Kubernetes that is critical for managing large-scale machine learning models. These microservices require a clear understanding of their compute and memory profiles in production environments to ensure efficient autoscaling.
Autoscale settings
The process begins with setting up a Kubernetes cluster equipped with the necessary components: Kubernetes Metrics Server, Prometheus, Prometheus Adapter, and Grafana. These tools are essential for scraping and displaying the metrics needed for HPA services.
The Kubernetes Metrics Server collects resource metrics from Kubelets and exposes them through the Kubernetes API Server. Prometheus and Grafana are used to scrape metrics from pods and create dashboards, and the Prometheus Adapter allows HPA to leverage custom metrics for scaling strategies.
NIM Microservice Deployment
NVIDIA provides detailed guidance on deploying NIM microservices, specifically using the NIM Model for LLM. This includes setting up the necessary infrastructure and ensuring that NIM for LLM Microservices is ready to scale based on GPU cache usage metrics.
Grafana dashboards visualize these custom metrics, making it easy to monitor and adjust resource allocation based on traffic and workload demands. The deployment process involves generating traffic using tools such as genai-perf, which helps evaluate the impact of different concurrency levels on resource utilization.
Implementing Horizontal Pod Autoscaling
To implement HPA, NVIDIA demonstrates the creation of HPA resources focusing on: gpu_cache_usage_perc Metric system. HPA runs load tests at different concurrency levels to automatically adjust the number of pods to maintain optimal performance and demonstrate efficiency in handling fluctuating workloads.
future prospects
NVIDIA’s approach paves the way for further exploration, such as scaling based on multiple metrics such as request latency or GPU compute utilization. You can also enhance autoscaling capabilities by leveraging Prometheus Query Language (PromQL) to create new metrics.
Visit the NVIDIA Developer Blog to learn more.
Image source: Shutterstock