Ben Diringer

View Original

Using Application Load Balancers with Kubernetes

Benjamin Diringer

Source: amazon.com

Introduction

Using the AWS Application Load Balancer (ALB) offers an easy way to provide flexible, secure, and scalable load balancing for applications running in Kubernetes.

AWS Application Load Balancers provide the following features:

  • Layer 7 (HTTP/HTTPS)
  • TLS termination
  • Path-based routing
  • Integrates with AWS WAF and AWS Shield

The AWS Load Balancer Controller allows application owners to configure a load balancer for their application using the Kubernetes Ingress API. The load balancer configuration can be saved in source control alongside the application code and deployed with a Helm chart or other methods of deploying to Kubernetes.

This guide will focus on common use cases for providing load balancing functionality for applications which are deployed to Kubernetes.

What are the benefits of the AWS Load Balancer Controller?

  • Declarative configuration management
    Load balancer configuration is managed by Kubernetes Ingress API which allows application owners to define a load balancer for their application in source control without requiring permissions to AWS.
  • Integration with pod readiness gates
    Kubernetes pod readiness gates can be used to ensure pods have passed load balancer health checks prior to being considered ready in Kubernetes.
  • Multiple applications supported with a single load balancer
    Multiple applications can share a single load balancer and hostname. Path-based routing rules are used to determine which application will receive requests.
  • Slow start mode
    Slow-start mode ensures new pods are warmed up prior to receiving a full share of traffic.
  • Integration with AWS Certificate Manager
    Certificates managed by AWS Certificate Manager (ACM) can be used by the load balancer. If encryption is enabled but no certificate is specified, then a certificate which matches the hostname of the listener will be automatically selected.

Controller Installation

Installing the AWS Load Balancer Controller requires configuring IAM, security groups, and deployment of the controller via Helm or YAML. Refer to the Installation Guide for step by step details on how to complete installation and initial configuration for the controller.

Application Load Balancing

Declarative Configuration Management

Ingress Resource

A load balancer is configured by deploying an ingress resource in Kubernetes which has the appropriate ingressClassName specified. A minimal ingress resource for an AWS ALB may look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-service
  annotations:
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: example-service.example.com
      http:
        paths:
          - backend:
              service:
                name: example-service
                port:
                  number: 443
            path: /*
            pathType: ImplementationSpecific

You can use kubectl to create or update the load balancer:

kubectl apply -f ingress.yaml

Configuration Parameters

Load balancer functionality can be customized by adding annotations to the ingress resource.

The following table describes a few of the most commonly used annotations which may be required. For a complete list, see the Ingress Annotations page in the official documentation.

Configuration Annotation Example Value
Target Type alb.ingress.kubernetes.io/target-type ip
Backend Protocol alb.ingress.kubernetes.io/backend-protocol HTTPS
Listen Ports alb.ingress.kubernetes.io/listen-ports [{"HTTPS": 443}]
Healthcheck Protocol alb.ingress.kubernetes.io/healthcheck-protocol HTTPS
Healthcheck Path alb.ingress.kubernetes.io/healthcheck-path /healthcheck/ready
SSL Policy alb.ingress.kubernetes.io/ssl-policy ELBSecurityPolicy-TLS-1-2-2017-01
Scheme alb.ingress.kubernetes.io/scheme internet-facing

Pod Readiness Gate

The AWS Load Balancer controller supports integration with Kubernetes pod readiness gates. This is useful to ensure that the load balancer considers a new target healthy before the pod is considered ready by Kubernetes. Without the pod readiness gate, a rolling deployment could terminate all pods in a replica set before any pods in a new replica set are ready to receive traffic from the load balancer which can result in performance impacts or HTTP 503 (service unavailable) errors.

Pod Readiness Gate

Rolling update to a deployment with Pod Readiness Gate enabled (assuming maxSurge: 100% and maxUnavailable: 0%).

In the above example, without the pod readiness gate, the 2nd pod in the v1 ReplicaSet would be terminated before the 2nd pod in the v2 ReplicaSet is receiving traffic from the load balancer.

The Pod Readiness Gate integration is enabled on a per-namespace basis. To enable, add the following label to the namespace where the Kubernetes service is deployed:

elbv2.k8s.aws/pod-readiness-gate-inject: enabled

End-to-End Encryption

By default, an application load balancer will use unencrypted HTTP for backend connections to targets. However, end-to-end encryption may be required if the environment is in scope for PCI DSS, HIPAA, or other regulatory frameworks.

The Setting up end-to-end TLS encryption on Amazon EKS with the new AWS Load Balancer Controller article describes how to configure end-to-end encryption using an AWS Certificate Manager Private CA. However, this solution incurs additional cost and may not be necessary since the certificate on the target does not have to be trusted by the load balancer.

Configure the following attribute to enable HTTPS on back-end targets:

alb.ingress.kubernetes.io/backend-protocol: HTTPS

If you opt to use self-signed certificates, they can be generated when building application containers:

Windows:

Import-Module WebAdministration

New-SelfSignedCertificate -CertStoreLocation
cert:\LocalMachine\My -DnsName <DNS Hostname> | New-Item IIS:\SslBindings\0.0.0.0!443

$certHash = (Get-ChildItem cert:\LocalMachine\My | Where-Object {$_.Subject -match "CN=<DNS
Hostname>"}).GetCertHashString()

Get-WebBinding -Name <IIS
Website> -Protocol https |   $_.AddSslCertificate($certHash, "My")

Linux:

openssl req -new -newkey rsa -x509 -subj '/CN=' -nodes -out MyCertificate.crt -keyout MyKey.key

Once a certificate is generated, you will need to configure the web server application to use it

Ingress Groups

The IngressGroup feature allows you to deploy multiple applications which share a single load balancer and hostname. The AWS Load Balancer Controller will automatically merge path-based routing rules for each Ingress into a single Application Load Balancer.

Ingress Group Source: amazon.com

To use ingress groups, ensure that all ingress resources use the same alb.ingress.kubernetes.io/group.name value. If you need to control the order in which rules are merged, use the alb.ingress.kubernetes.io/group.order annotation

The following example demonstrates the use of a simple IngressGroup using two ingress resources:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-service-a
  alb.ingress.kubernetes.io/group.name: example-service
spec:
  rules:
    - host: example-service.example.com
      http:
        paths:
        - backend:
            service:
              name: example-service-a
              port:
                number: 443
          path: /example-service-a/*
          pathType: ImplementationSpecific
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-service-b
  alb.ingress.kubernetes.io/group.name: example-service
spec:
  rules:
    - host: example-service.example.com
      http:
        paths:
        - backend:
            service:
              name: example-service-b
              port:
                number: 443
          path: /example-service-b/*
          pathType: ImplementationSpecific

For more information, see the IngressGroup section of the official documentation.

Slow Start Mode

Often, new containers process requests more slowly for a period of time until they are “warmed up”. Slow start mode allows new targets to receive less than their full share of requests until the slow start duration period has ended. Traffic will be gradually increased over the slow start duration.

To enable slow start mode, use the alb.ingress.kubernetes.io/target-group-attributes attribute:

alb.ingress.kubernetes.io/target-group-attributes: slow_start.duration_seconds=30

Conclusion

In this article, I’ve discussed the benefits of using AWS Application Load Balancers and the AWS Load Balancer Controller in a Kubernetes environment. I also described how to configure pod readiness gates, end-to-end encryption, ingress groups, and slow start mode.

In a future article, we’ll cover load balancer monitoring and access logs.

References

https://aws.amazon.com/blogs/containers/introducing-aws-load-balancer-controller/

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html

https://www.eksworkshop.com/beginner/180_fargate/prerequisites-for-alb/

https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/how-it-works/

https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller