Large-scale Large Language Model (LLM) inference systems deployed in distributed cloud environments face significant challenges in maintaining low latency, efficient GPU utilization, and energy-aware scheduling across heterogeneous hardware. Traditional Kubernetes-based orchestration frameworks are not optimized for the dynamic memory and compute characteristics of transformer workloads, often resulting in resource fragmentation and increased scheduling latency. This paper proposes Adaptive Resource Orchestration (ARO), a telemetry-driven framework designed for distributed LLM inference in heterogeneous GPU clusters. ARO introduces a Rack Affinity Group (RAG) hierarchical indexing mechanism that reduces scheduling search complexity to O(log N * M) while enabling topology-aware resource placement. The framework integrates multi-objective optimization to balance inference latency, energy efficiency, and GPU utilization. Experimental evaluation on a 32-node heterogeneous GPU cluster (H100, A100, and T4) demonstrates significant improvements over baseline Kubernetes scheduling approaches, including up to 84% reduction in P99 latency and 112% improvement in inference energy efficiency (68 tokens/J). These results highlight the importance of hardware-aware orchestration for improving performance and energy efficiency in distributed AI infrastructure.
In large scale LLM infrastructure latency plays a very pivotal part as it brings more faster processing with it for faster user experience. Frequent changes in latency brought by inter-connected components is a significant factor of bottleneck in large LLM orchestration. This change in trend with the growing demand of technology has become non-trivial to improvise the infrastructure. The use of LLM and its growing traction has changed over time from the last few years. The paper emphasizes a new technique ARO (Adaptive Resource Orchestration) method which would be a major stepping stone to outcast the scheduling difficulties that are associated with the transformers sharding framework in a distributed environment. In RAG (Rack affinity group) hierarchical indexing is utilized in the framework by dividing the hardware. This brings sense on complexity in scheduling which gets reduced in terms of logarithmically with O(logN·M). From the study performed in this paper it showcases 84% reduction of latency with P99 under ARO implementation. Efficiency was 68 Tokens which is approximately 112% increase. So this result is very critical to take into studies performed with ARO where it separates GPU scaling from the power inefficiencies. These factors are very important when we are dealing with Kubernetes where it reduces the energy cost spent in the distributed infrastructure.