Abstract

Service meshes are emerging software frameworks to manage communications among microservices of distributed applications. With a service mesh, each microservice is flanked by an L7 sidecar proxy that intercepts any incoming and outgoing requests for better observability, traffic management, and security. The sidecar proxy uses an application-level load balancing policy to route outbound requests towards possible replicas of destination microservices. A widely used load balancing policy is the Least Outstanding Request (LOR), which routes requests to the microservice replica with the fewest outstanding requests. While the LOR policy significantly reduces request latency in scenarios with a single load balancer, our comprehensive investigation, spanning analytical, simulation, and experimental methodologies, reveals that its effectiveness decreases in environments with multiple load balancers, typical of service meshes serving applications with several microservice replicas. Specifically, the resulting request latency asymptotically tends to that provided by a random load balancing policy as the number of microservice replicas increases. To address this loss in efficacy, we propose a solution based on a new Kubernetes custom resource, named Proxy-Service, offering potential improvements in performance and scalability.