Using the Virtual Traffic Manager as a Kubernetes Ingress Controller

This feature is applicable only to version 22.3 and later.

The Ingress Controller is a Virtual Traffic Manager feature that implements and meets the requirements of a Kubernetes Ingress Controller.

This feature enables you to position your Virtual Traffic Manager as an ingress point for your Kubernetes infrastructure, and to control your Virtual Traffic Manager configuration through standard Kubernetes mechanisms.

This guide assumes you have an understanding of Kubernetes features and capabilities, and that you have a previously-configured Kubernetes service infrastructure with which you want to add the Virtual Traffic Manager, In particular, make sure you are familiar with the following Kubernetes concepts:

Running services on a Kubernetes cluster

The Ingress resource concept and specification

The IngressClass specification

This feature was previously made available as part of an optional external solution. The Kubernetes resource specification for ingress supported by the former Ingress Controller feature has changed in Kubernetes v1.18.

Supported Features

The Virtual Traffic Manager acting as an Ingress Controller includes support for the following functionality:

Handling of HTTP and HTTPS traffic, including HTTPS offload

Let’s Encrypt certificate generation and renewal

The ability to watch either a single, specified namespace, or all namespaces except "kube-system"

Session persistence

Health monitoring

Integrated logging with the Virtual Traffic Manager event log

High Availability

The Virtual Traffic Manager can be deployed and configured as a cluster of instances for high-availability (HA) and redundancy. With the Ingress Controller feature enabled across the cluster, each Virtual Traffic Manager instance can assume the role of primary node. However, only the Ingress Controller on the primary node will attempt to apply configuration to the cluster. The other Virtual Traffic Manager instances still see changes on the Kubernetes platform so as to keep their state up-to-date, but they will not attempt to apply any configuration to the cluster unless they then become the primary instance.

Logging

Log messages concerning Ingress Controller activity appear in the Virtual Traffic Manager Event Log, and are prefixed with "ingresscontroller:”.

With verbose logging disabled (default), only errors are reported in the logs. If additional logging information is required, enable the vtm_verbose setting on the System > Ingress Controller page. For more details, see Configuring the Virtual Traffic Manager as an Ingress Controller.

Scope and Restrictions

The Virtual Traffic Manager remains external to the Kubernetes cluster, and is configured automatically through interaction with the Kubernetes Ingress resource.

As demonstrated in this diagram, the Virtual Traffic Manager's Ingress Controller function watches the configured Kubernetes API for Ingress and IngressClass resources. Based on these, the Virtual Traffic Manager creates resources and service configuration within your Virtual Traffic Manager cluster to meet the needs of the Kubernetes-based services.

As well as monitoring the Kubernetes API for Ingress resources, the Virtual Traffic Manager also monitors Secret resources, and creates SSL server certificate entries for any Secret resource of the type kubernetes.io/tls, assuming they also match any label selector specified in the configuration.

You can operate a cluster of Virtual Traffic Manager in this role for high-availability and redundancy purposes (see High Availability). However, the Virtual Traffic Manager is not able to load-balance requests directly to the Kubernetes pods. This is because the pods have an IP address only on the Kubernetes internal network, to which an external Virtual Traffic Manager cannot access. The data path to the pods must therefore be via a NodePort-based Service (see https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types).

Session persistence and health monitoring function only if the identity of a pod can be inferred from another piece of information known to the Virtual Traffic Manager; in this case, the Kubernetes compute node on which the pod is running. Accordingly, these features require that a podAntiAffinity rule (see https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) is defined in the Deployment (or ReplicaSet) specification to avoid multiple pods from the same Deployment being scheduled to the same compute node. Furthermore, these features require all defined Service objects within the Ingress to specify the spec.externalTrafficPolicy: Local setting, so as to avoid kube-proxy re-routing requests to a different compute node (see https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport).

Defining the Virtual Traffic Manager as the Controller in Your Ingress Specification

The Virtual Traffic Manager supports the standard Kubernetes Ingress specification, as well as some custom metadata annotations (see Ingress Metadata Annotation Reference).

For the Virtual Traffic Manager to recognize Ingress and IngressClass resource as applicable to this function, such resources must use an IngressClass defined with the following field:

spec:
controller: pulsesecure.net/vtm-ingress-controller

While deprecated, the Virtual Traffic Manager will also act on Ingress objects that contain the value “vtm” in the kubernetes.io/ingress.class annotation.

IP Address Support

The Virtual Traffic Manager Ingress Controller function supports two types of user-supplied IP address through the following Ingress annotations:

vtm.pulsesecure.net/ip-address: This annotation should be used when a dedicated IP address is allocated to an Ingress. The full range of functionality provided by other Ingress annotations can be used with this option.

vtm.pulsesecure.net/shared-ip-address This annotation enables a single IP address to be shared by multiple Ingress objects. These can use either the same or different ports; when the same port is specified by multiple Ingresses, the normal Ingress routing rules are applied in the order the Ingress objects were created. The following restrictions apply to shared IP services:

ingress.kubernetes.io/ssl-passthrough cannot be used, since the HTTP payload needs to be read to perform request routing.

vtm.pulsesecure.net/ssl-redirect can only be used if the primary Ingress bind port is 443. This can also not be specified if an existing Ingress sharing the IP uses port 80 as its bind port. If this is specified, no subsequently-created Ingress object can use port 80 as its bind port.

vtm.pulsesecure.net/use-lets-encrypt cannot be specified if an existing Ingress sharing the IP address uses port 80 as its bind port. If this annotation is specified, no subsequently-created Ingress object can use port 80 as its bind port.

If neither of the ip-address annotations are specified, the Virtual Traffic Manager will attempt to use a specified default Traffic IP Group. If no default is specified, an error is reported.

To learn more about applicable Ingress resource annotations, see Ingress Metadata Annotation Reference.

Ingress Spec Field Support

An Ingress resource definition includes a spec (specification) field to define the desired state and rules to which incoming requests must match. The Virtual Traffic Manager supports the following additional definitions within the spec: field:

TLS: If specifying a TLS section in the Ingress for SSL offload, it is important that the relevant Secret resources are created beforehand (unless Let’s Encrypt support has been specified; see Let's Encrypt Support). If an attempt is made to apply an Ingress before the Secrets exist, this will cause an error on the Virtual Traffic Manager and the Ingress will not be applied successfully.

Backend: If this section is specified, the relevant Virtual Traffic Manager pool will be created and set as the default pool for the virtual server. This section can be used alone or in conjunction with a spec.Rules section.

Rules: If this section is specified, a separate vTM pool will be created for each required backend. A TrafficScript rule will be applied to the virtual server to route requests to the relevant pool. The values of path and host fields support wildcard matches.

Let's Encrypt Support

If, in the Ingress resource definition, the vtm.pulsesecure.net/use-lets-encrypt custom metadata annotation is set to “true”, the behavior of the Virtual Traffic Manager changes with regards to how the "spec.TLS" section (see Ingress Spec Field Support) is processed.

Instead of expecting a Secret with the specified name to already exist, the Virtual Traffic Manager attempts to obtain a new certificate for the specified hostnames from Let’s Encrypt (or another ACME service), and creates a new Secret containing the certificate. A separate part of the Ingress Controller function monitors all Secrets created by the Virtual Traffic Manager and renews the certificates eight days before they expire. For Let’s Encrypt to issue a certificate, it must successfully challenge every domain name in the request. It is therefore necessary to ensure DNS can resolve the domain names to the listen IP address of the Ingress before the Ingress is created.

Prerequisites

Before using this feature, make sure you meet the following prerequisites:

For Kubernetes:

The minimum supported version is v1.19. An error message is logged in the Virtual Traffic Manager event log if the Kubernetes cluster version the ingress controller has connected to is not appropriate.

A fully deployed Kubernetes cluster.

A Service Account with appropriate permissions granted for reading and listing IngressClass resources in a Kubernetes namespace of interest.

Kubernetes Service resources of type NodePort

See also Scope and Restrictions

For the Virtual Traffic Manager:

Version 22.3 or later.

A fully deployed Virtual Traffic Manager instance or HA cluster.

Kubernetes nodes should be accessible to the Virtual Traffic Manager through your network.

Preparation for Use

To use this feature, perform the following steps:

In Kubernetes, create ServiceAccount or kube config, as well as IngressClass and Ingress resources according to the Kubernetes documentation.

In the Virtual Traffic Manager, configure and enable the Ingress Controller functionality.

Configuring the Virtual Traffic Manager as an Ingress Controller

To configure the Virtual Traffic Manager as an Ingress Controller for your Kubernetes-based services, perform the following steps:

Make sure you have access to a chosen Kubernetes cluster

Define a ServiceAccount with sufficient rights

Retrieve your Kubernetes access token string from a Secret created for the ServiceAccount

The following configuration page appears:

Ingress Controller UI page

Configure the following settings:

enable_ingctrl: Enables the Ingress Controller feature on the Virtual Traffic Manager

k8s_config_path: (Optional) The path to a Kubernetes user configuration file (for example, "/etc/kubernetes/admin.conf"). If this field is not specified, the controller uses the values configured in k8s_endpoint and k8s_token to connect to the API server.

k8s_endpoint: The Kubernetes API endpoint (for example, "https://api.mykube.local:6443"). You must specify this setting, even if a Kubernetes configuration file is specified in k8s_config_path.

k8s_token: The Kubernetes API authentication token. You must specify this setting, even if a Kubernetes configuration file is specified in k8s_config_path. If a configuration file is also specified, this token must have permission to query "service" and "endpoints" resources for all namespaces specified in the configuration file. If a configuration file is not specified, this token must have permission to read Ingress resources and read/write Secret resources in all relevant namespaces.

k8s_verify_ssl: Whether to verify the SSL certificate of the Kubernetes API server. This setting is used by both the Ingress Controller and any Service Discovery pools on the Virtual Traffic Manager.

vtm_password: The password of the Virtual Traffic Manager user authorized to make REST calls to the Kubernetes API server.

vtm_verify_ssl_cert: Whether to verify the SSL certificate of other Virtual Traffic Managers in your HA cluster.

vtm_verbose: Enable verbose logging to the Virtual Traffic Manager event log.

vtm_enable_customizations: Whether to allow the automatically-generated Virtual Traffic Manager configuration to be overridden by user-specified customizations. If this option is enabled, the vtm.pulsesecure.net/pool-customization and vtm.pulsesecure.net/virtualserver-customization annotations are read. If this option is disabled, any customizations are ignored.

vtm_enable_trafficscript: Whether to permit custom TrafficScript rules to be applied to the automatically-generated virtual servers. If this option is enabled, the vtm.pulsesecure.net/request-trafficscript and vtm.pulsesecure.net/response-trafficscript annotations are read. If this option is disabled, annotations are ignored.

To apply these settings, select Update.

Ingress Metadata Annotation Reference

This section lists all metadata annotations to the Ingress resource that control behavior of the Ingress Controller functionality in the Virtual Traffic Manager:

Annotation	Purpose	Example Value
vtm.pulsesecure.net/ip-address	The listen IP address for the Ingress. This creates a Traffic IP group on the Virtual Traffic Manager.	"10.0.2.223"
vtm.pulsesecure.net/shared-ipaddress	The shared IP address for the Ingress. If a Traffic IP group with the specified address exists, it is used for the new Ingress; otherwise one is created.	"10.0.5.223"
vtm.pulsesecure.net/port	The port to which the primary virtual server is bound. If not specified, 80 is used (where no TLS section exists in the Ingress) or 443 is used (if a TLS section exists).	“666”
ingress.kubernetes.io/ssl-redirect	If a TLS section is specified in the Ingress and this annotation is set to “true”, an additional virtual server on port 80 is created, together with a rule redirecting traffic to the HTTPS virtual server.	“true”
ingress.kubernetes.io/sslpassthrough	If this is set to “true”, the virtual server is set to pass through HTTPS without decrypting. This setting is incompatible with both TLS and Rules sections in the Ingress.	“true”
vtm.pulsesecure.net/re-encrypt	If this is set to “true”, the Virtual Traffic Manager pools are configured to re-encrypt traffic to the back-ends.	“true”
vtm.pulsesecure.net/monitor	A Virtual Traffic Manager REST-compatible JSON string defining an active health monitor. If omitted, a regular TCP connect monitor is instead used.

vtm.pulsesecure.net/persistence	A Virtual Traffic Manager REST-compatible JSON string defining a session persistence class.

vtm.pulsesecure.net/poolcustomization	A Virtual Traffic Manager REST-compatible JSON string defining customizations for every pool created to implement the Ingress resource. This annotation should be used with caution as it is possible to override the computed configuration with a non-working configuration.

vtm.pulsesecure.net/requesttrafficscript	A TrafficScript request rule as a string. Where permitted, this rule is added as the first rule in the request_rules field of the primary virtual server implementing an Ingress resource. This field is not supported for shared virtual servers (in other words, when a vtm.pulsesecure.net/shared-ip-address annotation has been specified). This annotation should be used with caution as it is possible to override the computed configuration with a non-working configuration.

vtm.pulsesecure.net/responsetrafficscript	A TrafficScript response rule as a string. Where permitted, this rule is added as the first rule in the response_rules field of the primary virtual server implementing an Ingress resource. This field is not supported for shared virtual servers (in other words, when a vtm.pulsesecure.net/shared-ip-address annotation has been specified). This annotation should be used with caution as it is possible to override the computed configuration with a non-working configuration.

vtm.pulsesecure.net/use-letsencrypt	Enable Let’s Encrypt support for all Secrets specified in the TLS section.	“true”
vtm.pulsesecure.net/virtualserver-customization	A Virtual Traffic Manager REST-compatible JSON string defining customizations for a virtual server. This field is not supported for shared virtual servers (in other words, when a vtm.pulsesecure.net/shared-ip-address annotation has been specified). This annotation should be used with caution as it is possible to override the computed configuration with a non-working configuration.