News Radar RSS

Kubernetes v1.36: Deprecation and removal of Service ExternalIPs

Kubernetes Blog Cloud & Infrastructure Score 6/10

Summary

<p>The <code>.spec.externalIPs</code> field for <a href="https://kubernetes.io/docs/concepts/services-networking/service/">Service</a> was an early attempt to provide cloud-load-balancer-like functionality for non-cloud clusters. Unfortunately, the API assumes that every user in the cluster is fully trusted, and in any situation where that is not the case, it enables various security exploits, as described in <a href="https://www.cvedetails.com/cve/CVE-2020-8554/">CVE-2020-8554</a>.</p> <p>Since Kubernetes 1.21, the Kubernetes project has recommended that all users disable <code>.spec.externalIPs</code>. To make that easier, Kubernetes also added an admission controller (<code>DenyServiceExternalIPs</code>) that can be enabled to do this. At the time, SIG Network felt that blocking the functionality by default was too large a breaking change to consider.</p> <p>However, the security problems are still there, and as a project we're increasingly unhappy with the "insecure by default" state of the feature. Additionally, there are now several better alternatives for non-cloud clusters wanting load-balancer-like functionality.</p> <p>As a result, the <code>.spec.externalIPs</code> field for Service is now formally deprecated in Kubernetes 1.36. We expect that a future minor release of Kubernetes will drop implementation of the behavior from <code>kube-proxy</code>, and will update the Kubernetes <a href="https://www.cncf.io/training/certification/software-conformance/">conformance</a> criteria to require that conforming implementations <strong>do not</strong> provide support.</p> <h2 id="terminology">A note on terminology, and what hasn't been deprecated<a class="td-heading-self-link" href="#terminology" aria-label="Heading self-link"></a></h2><p>The phrase <em>external IP</em> is somewhat overloaded in Kubernetes:</p> <ul> <li> <p>The Service API has a field <code>.spec.externalIPs</code> that can be used to add additional IP addresses that a Service will respond on.</p> </li> <li> <p>The Node API's <code>.status.addresses</code> field can list addresses of several different types, one of which is called <code>ExternalIP</code>.</p> </li> <li> <p>The <code>kubectl</code> tool, when displaying information about a Service of type LoadBalancer in the default output format, will show the load balancer IP address under the column heading <code>EXTERNAL-IP</code>.</p> </li> </ul> <p>This deprecation is about the first of those. If you are not setting the field <code>externalIPs</code> in any of your Services, then it does not apply to you.</p> <p>That said, as a precaution, you may still want to enable the <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips">DenyServiceExternalIPs</a> admission controller to block any future use of the <code>externalIPs</code> field.</p> <h2 id="alternatives">Alternatives to <code>externalIPs</code><a class="td-heading-self-link" href="#alternatives" aria-label="Heading self-link"></a></h2><p>If you are using <code>.spec.externalIPs</code>, then there are several alternatives.</p> <p>Consider a Service like the following:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterIP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-app</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">externalIPs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">"192.0.2.4"</span><span class="w"> </span></span></span></code></pre></div><h3 id="alternative-LoadBalancer">Using manually-managed LoadBalancer Services instead of <code>externalIPs</code><a class="td-heading-self-link" href="#alternative-LoadBalancer" aria-label="Heading self-link"></a></h3><p>The easiest (but also worst) option is to just switch from using <code>externalIPs</code> to using a <code>type: LoadBalancer</code> service, and assigning a load balancer IP by hand. This is, essentially, exactly the same as <code>externalIPs</code>, with one important difference: the load balancer IP is part of the Service's <code>.status</code>, not its <code>.spec</code>, and in a cluster with RBAC enabled, it can't be edited by ordinary users by default. Thus, this replacement for <code>externalIPs</code> would only be available to users who were given permission by the admins (although those users would then be fully empowered to replicate CVE-2020-8554; there would still not be any further checks to ensure that one user wasn't stealing another user's IPs, etc.)</p> <p>Because of the way that <code>.status</code> works in Kubernetes, you must create the Service without a load balancer IP, and then add the IP as a second step:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> cat loadbalancer-service.yaml </span></span><span class="line"><span class="cl"><span class="go">apiVersion: v1 </span></span></span><span class="line"><span class="cl"><span class="go">kind: Service </span></span></span><span class="line"><span class="cl"><span class="go">metadata: </span></span></span><span class="line"><span class="cl"><span class="go"> name: my-example-service </span></span></span><span class="line"><span class="cl"><span class="go">spec: </span></span></span><span class="line"><span class="cl"><span class="go"> # prevent any real load balancer controllers from managing this service </span></span></span><span class="line"><span class="cl"><span class="go"> # by using a non-existent loadBalancerClass </span></span></span><span class="line"><span class="cl"><span class="go"> loadBalancerClass: non-existent-class </span></span></span><span class="line"><span class="cl"><span class="go"> type: LoadBalancer </span></span></span><span class="line"><span class="cl"><span class="go"> selector: </span></span></span><span class="line"><span class="cl"><span class="go"> app.kubernetes.io/name: my-example-app </span></span></span><span class="line"><span class="cl"><span class="go"> ports: </span></span></span><span class="line"><span class="cl"><span class="go"> - protocol: TCP </span></span></span><span class="line"><span class="cl"><span class="go"> port: 80 </span></span></span><span class="line"><span class="cl"><span class="go"> targetPort: 8080 </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="gp">$</span> kubectl apply -f loadbalancer-service.yaml </span></span><span class="line"><span class="cl"><span class="go">service/my-example-service created </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="gp">$</span> kubectl patch service my-example-service --subresource<span class="o">=</span>status --type<span class="o">=</span>merge -p <span class="s1">'{"status":{"loadBalancer":{"ingress":[{"ip":"192.0.2.4"}]}}}'</span> </span></span></code></pre></div><h3 id="alternative-load-balancer-controller">Using a non-cloud based load balancer controller<a class="td-heading-self-link" href="#alternative-load-balancer-controller" aria-label="Heading self-link"></a></h3><p>Although <code>LoadBalancer</code> services were originally designed to be backed by cloud load balancers, Kubernetes can also support them on non-cloud platforms by using a third-party load balancer controller such as <a href="https://metallb.io/">MetalLB</a>. This solves the security problems associated with <code>externalIPs</code> because the administrator can configure what ranges of IP addresses the controller will assign to services, and the controller will ensure that two services can't both use the same IP.</p> <p>So, for example, after <a href="https://metallb.io/installation/">installing</a> and <a href="https://metallb.io/configuration/">configuring</a> MetalLB, a cluster administrator could configure a pool of IP addresses for use in the cluster:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">metallb.io/v1beta1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">IPAddressPool</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">production</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">metallb-system</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">addresses</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="m">192.0.2.0</span><span class="l">/24</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">autoAssign</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">avoidBuggyIPs</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span></span></span></code></pre></div><p>After which a user can create a <code>type: LoadBalancer</code> Service and MetalLB will handle the assignment of the IP address. MetalLB even supports the deprecated <code>loadBalancerIP</code> field in Service, so the end user can request a specific IP (assuming it is available) for backward-compatibility with the <code>externalIPs</code> approach, rather than being assigned one at random:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">LoadBalancer</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-app</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">loadBalancerIP</span><span class="p">:</span><span class="w"> </span><span class="s2">"192.0.2.4"</span><span class="w"> </span></span></span></code></pre></div><p>Similar approaches would work with other load balancer controllers. This approach can allow cluster administrators to have control over which IP addresses are assigned, rather than users.</p> <h3 id="alternative-gateway-api">Using Gateway API<a class="td-heading-self-link" href="#alternative-gateway-api" aria-label="Heading self-link"></a></h3><p>Another potential solution is to use an implementation of the <a href="https://gateway-api.sigs.k8s.io/">Gateway API</a>.</p> <p>Gateway API allows cluster administrators to define a Gateway resource, which can have an IP address attached to it via the <code>.spec.addresses</code> field. Since Gateway resources are designed to be managed by <a href="https://gateway-api.sigs.k8s.io/concepts/security/">cluster administrators</a>, RBAC rules can be put in place to only allow privileged users to manage them.</p> <p>An example of how this could look is:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">gateway.networking.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Gateway</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-gateway</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gatewayClassName</span><span class="p">:</span><span class="w"> </span><span class="l">example-gateway-class</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">addresses</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">IPAddress</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="s2">"192.0.2.4"</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">gateway.networking.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">HTTPRoute</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-route</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">parentRefs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-gateway</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">rules</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">backendRefs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-svc</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-svc</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterIP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="l">example-app</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span></span></span></code></pre></div><p>The Gateway API project is the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs within Kubernetes. Gateway API was designed to fix the shortcomings of the Service and Ingress resource, making it a very reliable robust solution that is under active development.</p> <h2 id="timeline-for-externalips-deprecation">Timeline for <code>externalIPs</code> deprecation<a class="td-heading-self-link" href="#timeline-for-externalips-deprecation" aria-label="Heading self-link"></a></h2><p>The rough timeline for this deprecation is as follows:</p> <ol> <li>With the release of Kubernetes 1.36, the field was deprecated; Kubernetes now emits <a href="https://kubernetes.io/blog/2020/09/03/warnings/">warnings</a> when a user uses this field</li> <li>About a year later (v1.40 at the earliest) support for <code>.spec.externalIPs</code> will be disabled in kube-proxy, but users will have a way to opt back in should they require more time to migrate away</li> <li>About another year later - (v1.43 at the earliest) support will be disabled completely; users won't have a way to opt back in</li> </ol>

KubernetesInfrastructure

News Radar provides aggregated summaries. Full content and copyright remain with the original publisher.