prostir zvuku
A spatial nature sound mixer for Mac Discussion | Link
A spatial nature sound mixer for Mac Discussion | Link
The inbox designed for humans and agents Discussion | Link
Until this past weekend, a contractor for the Cybersecurity & Infrastructure Security Agency (CISA) maintained a public GitHub repository that exposed credentials to several highly privileged AWS GovCloud accounts and a large number of internal CISA systems. Security experts said the public archive included files detailing how CISA builds, tests and deploys software internally, and that it represents one of the most egregious government data leaks in recent history.
Just a year ago, we launched AWS Transform for .NET, Mainframe and VMware workloads, the first agentic AI service purpose-built for modernizing enterprise applications at scale. At re:Invent 2025, we introduced AWS Transform custom, which enables organizations to modernize and transform code at scale using AWS-managed and custom transformations. You can upgrade language versions, migrate […]
Reunite lost pets using noseprints Discussion | Link
This week I'm talking with Matt Carey about Code Mode and how most of us have been thinking about MCP all wrong. Matt works on the Agents SDK and MCP at Cloudflare — we discuss how server-side Code Mode lets one MCP server expose all ~2,500 Cloudflare API endpoints in about 1,000 tokens of context, the dynamic Worker loader that runs model-written code safely in a V8 isolate, Matt's own workflow with Claude, where memory fits into the future of agents, and his Zaggy git wrapper that keeps agents from force-pushing his repos.
This article was originally published with the wrong date. It was later republished, dated the 15th of May 2026. Kubernetes v1.36 introduces a new alpha counter metric route_controller_route_sync_total to the Cloud Controller Manager (CCM) route controller implementation at k8s.io/cloud-provider . This metric increments each time routes are synced with the cloud provider. A/B testing watch-based route reconciliation This metric was added to help operators validate the CloudControllerManagerWatchBasedRoutesReconciliation feature gate introduced in Kubernetes v1.35 . That feature gate switches the route controller from a fixed-interval loop to a watch-based approach that only reconciles when nodes actually change. This reduces unnecessary API calls to the infrastructure provider, lowering pressure on rate-limited APIs and allowing operators to make more efficient use of their available quota. To A/B test this, compare route_controller_route_sync_total with the feature gate disabled (default) versus enabled. In clusters where node changes are infrequent, you should see a significant drop in the sync rate with the feature gate turned on. Example: expected behavior With the feature gate disabled (the default fixed-interval loop), the counter increments steadily regardless of whether any node changes occurred: # After 10 minutes with no node changes route_controller_route_sync_total 60 # After 20 minutes, still no node changes route_controller_route_sync_total 120 With the feature gate enabled (watch-based reconciliation), the counter only increments when nodes are actually added, removed, or updated: # After 10 minutes with no node changes route_controller_route_sync_total 1 # After 20 minutes, still no node changes — counter unchanged route_controller_route_sync_total 1 # A new node joins the cluster — counter increments route_controller_route_sync_total 2 The difference is especially visible in stable clusters where nodes rarely change. Where can I give feedback? If you have feedback, feel free to reach out through any of the following channels: The #sig-cloud-provider channel on Kubernetes Slack The KEP-5237 issue on GitHub The SIG Cloud Provider community page for other communication channels How can I learn more? For more details, refer to KEP-5237 .
Back in Kubernetes 1.28, we introduced the Mixed Version Proxy (MVP) as an Alpha feature (under the feature gate UnknownVersionInteroperabilityProxy ) in a previous blog post . The goal was simple but critical: make cluster upgrades safer by ensuring that requests for resources not yet known to an older API server are correctly routed to a newer peer API server, instead of returning an incorrect 404 Not Found . We are excited to announce that the Mixed Version Proxy is moving to Beta in Kubernetes 1.36 and will be enabled by default! The feature has evolved significantly since its initial release, addressing key gaps and modernizing its architecture. Here is a look at how the feature has evolved and what you need to know to leverage it in your clusters. What problem are we solving? In a highly available control plane undergoing an upgrade, you often have API servers running different versions. These servers might serve different sets of APIs (Groups, Versions, Resources). Without MVP, if a client request lands on an API server that does not serve the requested resource (e.g., a new API version introduced in the upgrade), that server returns a 404 Not Found . This is technically incorrect because the resource is available in the cluster, just not on that specific server. This can lead to serious side effects, such as mistaken garbage collection or blocked namespace deletions. MVP solves this by proxying the request to a peer API server that can serve it. sequenceDiagram participant Client participant API_Server_A as API Server A (Older/Different) participant API_Server_B as API Server B (Newer/Capable) Client->>API_Server_A: 1. Request for Resource (e.g., v2) Note over API_Server_A: Determines it cannot serve locally API_Server_A->>API_Server_A: 2. Looks up capable peer in Discovery Cache API_Server_A->>API_Server_B: 3. Proxies request (adds x-kubernetes-peer-proxied header) API_Server_B->>API_Server_B: 4. Processes request locally API_Server_B-->>API_Server_A: 5. Returns Response API_Server_A-->>Client: 6. Forwards Response How has it evolved since 1.28 The initial Alpha implementation was a great proof of concept, but it had some limitations and relied on older mechanisms. Here is how we have modernized it for Beta: From StorageVersion API to Aggregated Discovery In the Alpha version, API servers relied on the StorageVersion API to figure out which peers served which resources. While functional, this approach had a significant limitation: the StorageVersion API is not yet supported for CRDs and aggregated APIs. For Beta, we have replaced the reliance on StorageVersion API calls with the use of Aggregated Discovery . API servers now use the aggregated discovery data to dynamically understand the capabilities of their peers. The Missing Piece: Peer-Aggregated Discovery The 1.28 blog post noted a significant gap: while we could proxy resource requests, discovery requests still only showed what the local API server knew about. In 1.36, we have added Peer-Aggregated Discovery support! Now, when a client performs discovery (e.g., listing available APIs), the API server merges its local view with the discovery data from all active peers. This provides clients with a complete, unified view of all APIs available across the entire cluster, regardless of which API server they connected to. sequenceDiagram participant Client participant API_Server_A as API Server A participant API_Server_B as API Server B Client->>API_Server_A: 1. Request Discovery Document API_Server_A->>API_Server_A: 2. Gets Local APIs API_Server_A->>API_Server_B: 3. Gets Peer APIs (Cached or Direct) API_Server_A->>API_Server_A: 4. Merges and sorts lists deterministically API_Server_A-->>Client: 5. Returns Unified Discovery Document While peer-aggregated discovery will be the default behavior (note that peer-aggregated discovery is enabled if the --peer-ca-file flag is set, otherwise the server will fallback to showing only its local APIs), there may be cases where you need to inspect only the resources served by the specific API server you are connected to. You can request this non-aggregated view by including the profile=nopeer parameter in your request's Accept header (e.g., Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList;profile=nopeer ). Required configuration While the feature gate will be enabled by default, it requires certain flags to be set to allow for secure communication between peer API servers. To function correctly, make sure your API server is configured with the following flags: --feature-gates=UnknownVersionInteroperabilityProxy=true : This will be default in 1.36, but it is good to verify --peer-ca-file= : [CRITICAL] This is a required flag. You must provide the CA bundle that the source API server will use to authenticate the serving certificates of destination peer API servers. Without this, proxying will fail due to TLS verification errors. --peer-advertise-ip and --peer-advertise-port : These flags are used to set the network address that peers should use to reach this API server. If unset, the values from --advertise-address or --bind-address are used. If you have complex network topologies where API servers communicate over a specific internal interface, setting these flags explicitly is highly recommended. Configuring with kubeadm If you manage your cluster with kubeadm , you can configure these flags in your ClusterConfiguration file: apiVersion : kubeadm.k8s.io/v1beta4 kind : ClusterConfiguration apiServer : extraArgs : peer-ca-file : "/etc/kubernetes/pki/ca.crt" # peer-advertise-ip and port if needed Call to action If you are running multi-master clusters and upgrading them regularly, the Mixed Version Proxy is a major safety improvement. With it becoming default in 1.36, we encourage you to: Review your API server flags to ensure --peer-ca-file is set properly. Test the feature in your staging environments as you prepare for the 1.36 upgrade. Provide feedback to SIG API Machinery ( Slack , mailing list , or by attending SIG API Machinery meetings ) on your experience.
Search and ask questions inside lecture videos Discussion | Link
Amazon Bedrock Advanced Prompt Optimization enables customers to optimize their prompts for their current model or migrate prompts to new models faster than before with built-in evaluation feedback loops. Optimize your prompts and compare results for up to 5 models simultaneously.
<p>The <code>.spec.externalIPs</code> field for <a href="https://kubernetes.io/docs/concepts/services-networking/service/">Service</a> was an early attempt to provide cloud-load-balancer-like functionality for non-cloud clusters. Unfortunately, the API assumes that every user in the cluster is fully trusted, and in any situation where that is not the case, it enables various security exploits, as described in <a href="https://www.cvedetails.com/cve/CVE-2020-8554/">CVE-2020-8554</a>.</p> <p>Since Kubernetes 1.21, the Kubernetes project has recommended that all users disable <code>.spec.externalIPs</code>. To make that easier, Kubernetes also added an admission controller (<code>DenyServiceExternalIPs</code>) that can be enabled to do this. At the time, SIG Network felt that blocking the functionality by default was too large a breaking change to consider.</p> <p>However, the security problems are still there, and as a project we're increasingly unhappy with the "insecure by default" state of the feature. Additionally, there are now several better alternatives for non-cloud clusters wanting load-balancer-like functionality.</p> <p>As a result, the <code>.spec.externalIPs</code> field for Service is now formally deprecated in Kubernetes 1.36. We expect that a future minor release of Kubernetes will drop implementation of the behavior from <code>kube-proxy</code>, and will update the Kubernetes <a href="https://www.cncf.io/training/certification/software-conformance/">conformance</a> criteria to require that conforming implementations <strong>do not</strong> provide support.</p> <h2 id="terminology">A note on terminology, and what hasn't been deprecated<a class="td-heading-self-link" href="#terminology" aria-label="Heading self-link"></a></h2><p>The phrase <em>external IP</em> is somewhat overloaded in Kubernetes:</p> <ul> <li> <p>The Service API has a field <code>.spec.externalIPs</code> that can be used to add additional IP addresses that a Service will respond on.</p> </li> <li> <p>The Node API's <code>.status.addresses</code> field can list addresses of several different types, one of which is called <code>ExternalIP</code>.</p> </li> <li> <p>The <code>kubectl</code> tool, when displaying information about a Service of type LoadBalancer in the default output format, will show the load balancer IP address under the column heading <code>EXTERNAL-IP</code>.</p> </li> </ul> <p>This deprecation is about the first of those. If you are not setting the field <code>externalIPs</code> in any of your Services, then it does not apply to you.</p> <p>That said, as a precaution, you may still want to enable the <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips">DenyServiceExternalIPs</a> admission controller to block any future use of the <code>externalIPs</code> field.</p> <h2 id="alternatives">Alternatives to <code>externalIPs</code><a class="td-heading-self-link" href="#alternatives" aria-label="Heading self-link"></a></h2><p>If you are using <code>.spec.externalIPs</code>, then there are several alternatives.</p> <p>Consider a Service like the following:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterIP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-app</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">externalIPs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">"192.0.2.4"</span><span class="w"> </span></span></span></code></pre></div><h3 id="alternative-LoadBalancer">Using manually-managed LoadBalancer Services instead of <code>externalIPs</code><a class="td-heading-self-link" href="#alternative-LoadBalancer" aria-label="Heading self-link"></a></h3><p>The easiest (but also worst) option is to just switch from using <code>externalIPs</code> to using a <code>type: LoadBalancer</code> service, and assigning a load balancer IP by hand. This is, essentially, exactly the same as <code>externalIPs</code>, with one important difference: the load balancer IP is part of the Service's <code>.status</code>, not its <code>.spec</code>, and in a cluster with RBAC enabled, it can't be edited by ordinary users by default. Thus, this replacement for <code>externalIPs</code> would only be available to users who were given permission by the admins (although those users would then be fully empowered to replicate CVE-2020-8554; there would still not be any further checks to ensure that one user wasn't stealing another user's IPs, etc.)</p> <p>Because of the way that <code>.status</code> works in Kubernetes, you must create the Service without a load balancer IP, and then add the IP as a second step:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> cat loadbalancer-service.yaml </span></span><span class="line"><span class="cl"><span class="go">apiVersion: v1 </span></span></span><span class="line"><span class="cl"><span class="go">kind: Service </span></span></span><span class="line"><span class="cl"><span class="go">metadata: </span></span></span><span class="line"><span class="cl"><span class="go"> name: my-example-service </span></span></span><span class="line"><span class="cl"><span class="go">spec: </span></span></span><span class="line"><span class="cl"><span class="go"> # prevent any real load balancer controllers from managing this service </span></span></span><span class="line"><span class="cl"><span class="go"> # by using a non-existent loadBalancerClass </span></span></span><span class="line"><span class="cl"><span class="go"> loadBalancerClass: non-existent-class </span></span></span><span class="line"><span class="cl"><span class="go"> type: LoadBalancer </span></span></span><span class="line"><span class="cl"><span class="go"> selector: </span></span></span><span class="line"><span class="cl"><span class="go"> app.kubernetes.io/name: my-example-app </span></span></span><span class="line"><span class="cl"><span class="go"> ports: </span></span></span><span class="line"><span class="cl"><span class="go"> - protocol: TCP </span></span></span><span class="line"><span class="cl"><span class="go"> port: 80 </span></span></span><span class="line"><span class="cl"><span class="go"> targetPort: 8080 </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="gp">$</span> kubectl apply -f loadbalancer-service.yaml </span></span><span class="line"><span class="cl"><span class="go">service/my-example-service created </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="gp">$</span> kubectl patch service my-example-service --subresource<span class="o">=</span>status --type<span class="o">=</span>merge -p <span class="s1">'{"status":{"loadBalancer":{"ingress":[{"ip":"192.0.2.4"}]}}}'</span> </span></span></code></pre></div><h3 id="alternative-load-balancer-controller">Using a non-cloud based load balancer controller<a class="td-heading-self-link" href="#alternative-load-balancer-controller" aria-label="Heading self-link"></a></h3><p>Although <code>LoadBalancer</code> services were originally designed to be backed by cloud load balancers, Kubernetes can also support them on non-cloud platforms by using a third-party load balancer controller such as <a href="https://metallb.io/">MetalLB</a>. This solves the security problems associated with <code>externalIPs</code> because the administrator can configure what ranges of IP addresses the controller will assign to services, and the controller will ensure that two services can't both use the same IP.</p> <p>So, for example, after <a href="https://metallb.io/installation/">installing</a> and <a href="https://metallb.io/configuration/">configuring</a> MetalLB, a cluster administrator could configure a pool of IP addresses for use in the cluster:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">metallb.io/v1beta1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">IPAddressPool</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">production</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">metallb-system</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">addresses</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="m">192.0.2.0</span><span class="l">/24</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">autoAssign</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">avoidBuggyIPs</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span></span></span></code></pre></div><p>After which a user can create a <code>type: LoadBalancer</code> Service and MetalLB will handle the assignment of the IP address. MetalLB even supports the deprecated <code>loadBalancerIP</code> field in Service, so the end user can request a specific IP (assuming it is available) for backward-compatibility with the <code>externalIPs</code> approach, rather than being assigned one at random:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">LoadBalancer</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="l">my-example-app</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">loadBalancerIP</span><span class="p">:</span><span class="w"> </span><span class="s2">"192.0.2.4"</span><span class="w"> </span></span></span></code></pre></div><p>Similar approaches would work with other load balancer controllers. This approach can allow cluster administrators to have control over which IP addresses are assigned, rather than users.</p> <h3 id="alternative-gateway-api">Using Gateway API<a class="td-heading-self-link" href="#alternative-gateway-api" aria-label="Heading self-link"></a></h3><p>Another potential solution is to use an implementation of the <a href="https://gateway-api.sigs.k8s.io/">Gateway API</a>.</p> <p>Gateway API allows cluster administrators to define a Gateway resource, which can have an IP address attached to it via the <code>.spec.addresses</code> field. Since Gateway resources are designed to be managed by <a href="https://gateway-api.sigs.k8s.io/concepts/security/">cluster administrators</a>, RBAC rules can be put in place to only allow privileged users to manage them.</p> <p>An example of how this could look is:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">gateway.networking.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Gateway</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-gateway</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gatewayClassName</span><span class="p">:</span><span class="w"> </span><span class="l">example-gateway-class</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">addresses</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">IPAddress</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="s2">"192.0.2.4"</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">gateway.networking.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">HTTPRoute</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-route</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">parentRefs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-gateway</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">rules</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">backendRefs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-svc</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">example-svc</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterIP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">app.kubernetes.io/name</span><span class="p">:</span><span class="w"> </span><span class="l">example-app</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span></span></span></code></pre></div><p>The Gateway API project is the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs within Kubernetes. Gateway API was designed to fix the shortcomings of the Service and Ingress resource, making it a very reliable robust solution that is under active development.</p> <h2 id="timeline-for-externalips-deprecation">Timeline for <code>externalIPs</code> deprecation<a class="td-heading-self-link" href="#timeline-for-externalips-deprecation" aria-label="Heading self-link"></a></h2><p>The rough timeline for this deprecation is as follows:</p> <ol> <li>With the release of Kubernetes 1.36, the field was deprecated; Kubernetes now emits <a href="https://kubernetes.io/blog/2020/09/03/warnings/">warnings</a> when a user uses this field</li> <li>About a year later (v1.40 at the earliest) support for <code>.spec.externalIPs</code> will be disabled in kube-proxy, but users will have a way to opt back in should they require more time to migrate away</li> <li>About another year later - (v1.43 at the earliest) support will be disabled completely; users won't have a way to opt back in</li> </ol>
This week I'm talking with Adam Jacob, founder of System Initiative and creator of Swamp, about what happens when AI agents change the entire shape of software development. We discuss how he went from an 18-person team down to five and shipped Swamp 900 times in four weeks, why he brought User Acceptance Testing (UAT) testing back from the 90s, why software architecture (and domain-driven design) suddenly matters more than knowing how to write code, the live demo where I pointed Swamp at my Proxmox box and watched it write its own automation (blew my mind!!), and why he'll never accept a pull request to Swamp, ever.
Your n8n command center, now on your phone Discussion | Link
<p>AI/ML and batch workloads introduce unique scheduling challenges that go beyond simple Pod-by-Pod scheduling. In Kubernetes v1.35, we introduced the first tranche of <em>workload-aware scheduling</em> improvements, featuring the foundational Workload API alongside basic <em>gang scheduling</em> support built on a Pod-based framework, and an <em>opportunistic batching</em> feature to efficiently process identical Pods.</p> <p>Kubernetes v1.36 introduces a significant architectural evolution by cleanly separating API concerns: the Workload API acts as a static template, while the new PodGroup API handles the runtime state. To support this, the <code>kube-scheduler</code> features a new <em>PodGroup scheduling cycle</em> that enables atomic workload processing and paves the way for future enhancements. This release also debuts the first iterations of <em>topology-aware scheduling</em> and <em>workload-aware preemption</em> to advance scheduling capabilities. Additionally, <em>ResourceClaim support for workloads</em> unlocks <em>Dynamic Resource Allocation (<a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/">DRA</a>)</em> for PodGroups. Finally, to demonstrate real-world readiness, v1.36 delivers the first phase of integration between the Job controller and the new API.</p> <h2 id="workload-and-podgroup-api-updates">Workload and PodGroup API updates<a class="td-heading-self-link" href="#workload-and-podgroup-api-updates" aria-label="Heading self-link"></a></h2><p>The Workload API now serves as a static template, while the new PodGroup API describes the runtime object. Kubernetes v1.36 introduces the Workload and PodGroup APIs as part of the <code>scheduling.k8s.io/v1alpha2</code> <a class='glossary-tooltip' title='A set of related paths in the Kubernetes API.' data-bs-toggle='tooltip' data-bs-placement='top' href='https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups-and-versioning' target='_blank' aria-label='API group'>API group</a>, completely replacing the previous <code>v1alpha1</code> API version.</p> <p>In v1.35, Pod groups and their runtime states were embedded within the Workload resource. The new model decouples these concepts: the Workload now serves as a static template object, while the PodGroup manages the runtime state. This separation also improves performance and scalability as the PodGroup API allows per-replica sharding of status updates.</p> <p>Because the Workload API acts merely as a template, the <code>kube-scheduler</code>'s logic is streamlined. The scheduler can directly read the PodGroup, which contains all the information required by the scheduler, without needing to watch or parse the Workload object itself.</p> <p>Here is what the updated configuration looks like. Workload controllers (such as the Job controller) define the Workload object, which now acts as a static template for your Pod groups:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">scheduling.k8s.io/v1alpha2</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Workload</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workload</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">some-ns</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Pod groups are now defined as templates,</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># which contains the PodGroup objects' spec fields.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">podGroupTemplates</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">workers</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingPolicy</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gang</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># The gang is schedulable only if 4 pods can run at once</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">minCount</span><span class="p">:</span><span class="w"> </span><span class="m">4</span><span class="w"> </span></span></span></code></pre></div><p>Controllers then stamp out runtime PodGroup instances based on those templates. The PodGroup runtime object holds the actual scheduling policy and references the template from which it was created. It also has a status containing conditions that mirror the states of individual Pods, reflecting the overall scheduling state of the group:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">scheduling.k8s.io/v1alpha2</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PodGroup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workers-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">some-ns</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># The PodGroup references the Workload template it originated from.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># In comparison, .metadata.ownerReferences points to the "true" workload object,</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># e.g., a Job. </span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">podGroupTemplateRef</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">workload</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">workloadName</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workload</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">podGroupTemplateName</span><span class="p">:</span><span class="w"> </span><span class="l">workers</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># The actual scheduling policy is placed inside the runtime PodGroup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingPolicy</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gang</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">minCount</span><span class="p">:</span><span class="w"> </span><span class="m">4</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">status</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># The status contains conditions mirroring individual Pod conditions.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">conditions</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">PodGroupScheduled</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">status</span><span class="p">:</span><span class="w"> </span><span class="s2">"True"</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">lastTransitionTime</span><span class="p">:</span><span class="w"> </span><span class="ld">2026-04-03T00:00:00Z</span><span class="w"> </span></span></span></code></pre></div><p>Finally, to bridge this new architecture with individual Pods, the <code>workloadRef</code> field in the Pod API has been replaced with the <code>schedulingGroup</code> field. When creating Pods, you link them directly to the runtime PodGroup:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Pod</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">worker-0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">some-ns</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># The workloadRef field has been replaced by schedulingGroup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingGroup</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">podGroupName</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workers-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> </span></span></span></code></pre></div><p>By keeping the Workload as a static template and elevating the PodGroup to a first-class, standalone API, we establish a robust foundation for building advanced workload scheduling capabilities in future Kubernetes releases.</p> <h2 id="podgroup-scheduling-cycle-and-gang-scheduling">PodGroup scheduling cycle and gang scheduling<a class="td-heading-self-link" href="#podgroup-scheduling-cycle-and-gang-scheduling" aria-label="Heading self-link"></a></h2><p>To efficiently manage these workloads, the kube-scheduler now features a dedicated <em>PodGroup scheduling cycle</em>. Instead of evaluating and reserving resources sequentially Pod-by-Pod, which risks scheduling deadlocks, the scheduler evaluates the group as a unified operation.</p> <p>When the scheduler pops a PodGroup member from the scheduling queue, regardless of the group's specific policy, it fetches the rest of the queued Pods for that group, sorts them deterministically, and executes an atomic scheduling cycle as follows:</p> <ol> <li> <p>The scheduler takes a single snapshot of the cluster state to prevent race conditions and ensure consistency while evaluating the entire group.</p> </li> <li> <p>It then attempts to find valid Node placements for all Pods in the group using a PodGroup scheduling algorithm, which leverages the standard Pod-based filtering and scoring phases.</p> </li> <li> <p>Based on the algorithm's outcome, the scheduling decision is applied atomically for the entire PodGroup.</p> <ul> <li> <p>Success: If the placement is found and group constraints are met, the schedulable member Pods are moved directly to the binding phase together. Any remaining unschedulable Pods are returned to the scheduling queue to wait for available resources so they can join the already scheduled Pods.</p> <p>(Note: If new Pods are added to a PodGroup after others are already scheduled, the cycle evaluates the new Pods while accounting for the existing ones. Crucially, Pods already assigned to Nodes remain running. The scheduler will not unassign or evict them, even if the group fails to meet its requirements in subsequent cycles.)</p> </li> <li> <p>Failure: If the group fails to meet its requirements, the entire group is considered unschedulable. None of the Pods are bound, and they are returned to the scheduling queue to retry later after a backoff period.</p> </li> </ul> </li> </ol> <p>This cycle acts as the foundation for <em>gang scheduling</em>. When your workload requires strict <em>all-or-nothing</em> placement, the <code>gang</code> policy leverages this cycle to prevent partial deployments that lead to resource wastage and potential deadlocks.</p> <p>While the scheduler still holds the Pods in the <code>PreEnqueue</code> until the <code>minCount</code> requirement is met, the actual scheduling phase now relies entirely on the new PodGroup cycle. Specifically, during the algorithm's execution, the scheduler verifies that the number of schedulable Pods satisfies the <code>minCount</code>. If the cluster cannot accommodate the required minimum, none of the pods are bound. The group fails and waits for sufficient resources to free up.</p> <h3 id="limitations">Limitations<a class="td-heading-self-link" href="#limitations" aria-label="Heading self-link"></a></h3><p>The first version of the PodGroup scheduling cycle comes with certain limitations:</p> <ul> <li> <p>For basic <em>homogeneous</em> Pod groups (i.e., those where all Pods have identical scheduling requirements and lack inter-Pod dependencies like affinity, anti-affinity, or topology spread constraints), the algorithm is expected to find a placement if one exists.</p> </li> <li> <p>For <em>heterogeneous</em> Pod groups, finding a valid placement if one exists is not guaranteed, even when the solution might seem trivial.</p> </li> <li> <p>For Pod groups with <em>inter-Pod dependencies</em>, finding a valid placement if one exists is not guaranteed.</p> </li> </ul> <p>In addition to the above, for cases involving <em>intra-group dependencies</em> (e.g., when the schedulability of one Pod depends on another group member via inter-Pod affinity), this algorithm may fail to find a placement regardless of cluster state due to its deterministic processing order.</p> <h2 id="topology-aware-scheduling">Topology-aware scheduling<a class="td-heading-self-link" href="#topology-aware-scheduling" aria-label="Heading self-link"></a></h2><p>For complex distributed workloads like AI/ML training or batch processing, placing Pods randomly across a cluster can introduce significant network latency and bottleneck overall performance.</p> <p>Topology-aware scheduling addresses this problem by allowing you to define topology constraints directly on a PodGroup, ensuring its Pods are co-located within specific physical or logical domains:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">scheduling.k8s.io/v1alpha2</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PodGroup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">topology-aware-workers-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingPolicy</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gang</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">minCount</span><span class="p">:</span><span class="w"> </span><span class="m">4</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Enforce that the pods are co-located based on the rack topology</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingConstraints</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">topology</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">topology.kubernetes.io/rack</span><span class="w"> </span></span></span></code></pre></div><p>In this example, the <code>kube-scheduler</code> attempts to schedule the Pods across various combinations of Nodes that match the <code>rack</code> topology constraint. It then selects the optimal placement based on how efficiently the PodGroup utilizes resources and how many Pods can successfully be scheduled within that domain.</p> <p>To achieve this, the scheduler extends the PodGroup scheduling cycle with a dedicated placement-based algorithm consisting of three phases:</p> <ol> <li> <p>Generate candidate placements (subsets of Nodes that are theoretically feasible for the PodGroup's assignment) based on the group's scheduling constraints. The topology-aware scheduling plugin uses the new <code>PlacementGenerate</code> extension point to create these placements.</p> </li> <li> <p>Evaluate each proposed placement to confirm whether the entire PodGroup can actually fit there.</p> </li> <li> <p>Score all feasible placements to select the best fit for the PodGroup. The topology-aware scheduling plugins use the new <code>PlacementScore</code> extension point to score these placements.</p> </li> </ol> <p>Currently, topology-aware scheduling does not trigger Pod preemption to satisfy constraints. However, we plan to integrate workload-aware preemption with topology constraints in the upcoming release.</p> <p>While Kubernetes v1.36 delivers this foundational topology-aware scheduling, the Kubernetes project is planning expand its capabilities soon. Future updates will introduce support for multiple topology levels, soft constraints (preferences), deeper integration with Dynamic Resource Allocation (DRA), and more robust behavior when paired with the <code>basic</code> scheduling policy.</p> <h2 id="workload-aware-preemption">Workload-aware preemption<a class="td-heading-self-link" href="#workload-aware-preemption" aria-label="Heading self-link"></a></h2><p>To support the new PodGroup scheduling cycle, Kubernetes v1.36 introduces a new type of preemption mechanism called <em>workload-aware preemption</em>. When a PodGroup cannot be scheduled, the scheduler utilizes this mechanism to try making a scheduling of this PodGroup possible.</p> <p>Compared to the default preemption used in the standard Pod-by-Pod scheduling cycle, this new mechanism treats the entire PodGroup as a single preemptor unit. Instead of evaluating preemption victims on each Node separately, it searches across the entire cluster. This allows the scheduler to preempt Pods from multiple Nodes simultaneously, making enough space to schedule the whole PodGroup afterwards.</p> <p>Workload-aware preemption also introduces two additional concepts directly to the PodGroup API:</p> <ul> <li> <p>PodGroup <code>priority</code> that overrides the priority of the individual Pods forming the PodGroup.</p> </li> <li> <p>PodGroup <code>disruptionMode</code> that dictates whether the Pods within a PodGroup can be preempted independently, or if they have to be preempted together in an <em>all-or-nothing</em> fashion.</p> </li> </ul> <p>In Kubernetes v1.36, these fields are only respected by the workload-aware preemption mechanism. The people working on this set of features are hoping to extend support for these fields to other disruption sources, including default preemption used in the Pod-by-Pod scheduling cycle, in future releases.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">scheduling.k8s.io/v1alpha2</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PodGroup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">victim-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">priorityClassName</span><span class="p">:</span><span class="w"> </span><span class="l">high-priority</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">priority</span><span class="p">:</span><span class="w"> </span><span class="m">1000</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">disruptionMode</span><span class="p">:</span><span class="w"> </span><span class="l">PodGroup</span><span class="w"> </span></span></span></code></pre></div><p>In this example, when the scheduler evaluates <code>victim-pg</code> as a potential preemption victim during a workload-aware preemption cycle, it will use 1000 as its priority and preempt the PodGroup in a strictly <em>all-or-nothing</em> fashion.</p> <h2 id="dra-resourceclaim-support-for-workloads">DRA ResourceClaim support for workloads<a class="td-heading-self-link" href="#dra-resourceclaim-support-for-workloads" aria-label="Heading self-link"></a></h2><p>Since its general availability in Kubernetes v1.34, <a class='glossary-tooltip' title='A Kubernetes feature for requesting and sharing resources, like hardware accelerators, among Pods.' data-bs-toggle='tooltip' data-bs-placement='top' href='https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/' target='_blank' aria-label='DRA'>DRA</a> has enabled Pods to make detailed requests for <a class='glossary-tooltip' title='Any resource that's directly or indirectly attached your cluster's nodes, like GPUs or circuit boards.' data-bs-toggle='tooltip' data-bs-placement='top' href='https://kubernetes.io/docs/reference/glossary/?all=true#term-device' target='_blank' aria-label='devices'>devices</a> like GPUs, TPUs, and NICs. Requested devices can be shared by multiple Pods requesting the same <a class='glossary-tooltip' title='Describes the resources that a workload needs, such as devices. ResourceClaims can request devices from DeviceClasses.' data-bs-toggle='tooltip' data-bs-placement='top' href='https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceclaims-templates' target='_blank' aria-label='ResourceClaim'>ResourceClaim</a> by name. Other requests can be replicated through a <a class='glossary-tooltip' title='Defines a template for Kubernetes to create ResourceClaims. Used to provide per-Pod or per-PodGroup access to separate, similar resources.' data-bs-toggle='tooltip' data-bs-placement='top' href='https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceclaims-templates' target='_blank' aria-label='ResourceClaimTemplate'>ResourceClaimTemplate</a>, in which Kubernetes generates one ResourceClaim with a non-deterministic name for each Pod referencing the template. However, large-scale workloads that require certain Pods to share certain devices are currently left to manage creating individual ResourceClaims themselves.</p> <p>Now, in addition to Pods, PodGroups can represent the replicable unit for a ResourceClaimTemplate. For ResourceClaimTemplates referenced by one of a PodGroup's <code>spec.resourceClaims</code>, Kubernetes generates one ResourceClaim for the entire PodGroup, no matter how many Pods are in the group. When one of a Pod's <code>spec.resourceClaims</code> for a ResourceClaimTemplate matches one of its PodGroup's <code>spec.resourceClaims</code>, the Pod's claim resolves to the ResourceClaim generated for the PodGroup and a ResourceClaim will not be generated for that individual Pod. A single PodGroupTemplate in a Workload object can express resource requests which are both copied for each distinct PodGroup and shareable by the Pods within each group.</p> <p>The following example shows two Pods requesting the same ResourceClaim generated from a ResourceClaimTemplate for their PodGroup:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">scheduling.k8s.io/v1alpha2</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PodGroup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workers-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resourceClaims</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">pg-claim</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resourceClaimTemplateName</span><span class="p">:</span><span class="w"> </span><span class="l">my-claim-template</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Pod</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">topology-aware-workers-pg-pod-1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingGroup</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">podGroupName</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workers-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resourceClaims</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">pg-claim</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resourceClaimTemplateName</span><span class="p">:</span><span class="w"> </span><span class="l">my-claim-template</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Pod</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">topology-aware-workers-pg-pod-2</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedulingGroup</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">podGroupName</span><span class="p">:</span><span class="w"> </span><span class="l">training-job-workers-pg</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resourceClaims</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">pg-claim</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resourceClaimTemplateName</span><span class="p">:</span><span class="w"> </span><span class="l">my-claim-template</span><span class="w"> </span></span></span></code></pre></div><p>In addition, ResourceClaims referenced by PodGroups, either through <code>resourceClaimName</code> or the claim generated from <code>resourceClaimTemplateName</code>, become reserved for the entire PodGroup. Previously, kube-scheduler could only list individual Pods in a ResourceClaim's <code>status.reservedFor</code> field which is limited to 256 items. Now, a single PodGroup reference in <code>status.reservedFor</code> can represent many more than 256 Pods, allowing high-cardinality sharing of devices.</p> <p>Together, these changes enable massive workloads with complex topologies to utilize DRA for scalable device management.</p> <h2 id="integration-with-the-job-controller">Integration with the Job controller<a class="td-heading-self-link" href="#integration-with-the-job-controller" aria-label="Heading self-link"></a></h2><p>In Kubernetes v1.36, the Job controller can create and manage Workload and PodGroup objects on your behalf, so that Jobs representing a tightly coupled parallel application, such as distributed AI training, are gang-scheduled without any additional tooling. Without this integration, you would have to create the Workload and PodGroup yourself and wire their references into the Pod template. Now, the Job controller automates this process natively.</p> <p>When the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#WorkloadWithJob"><code>WorkloadWithJob</code></a> feature gate is enabled, the Job controller automatically:</p> <ul> <li> <p>creates a Workload and a corresponding runtime PodGroup for each qualifying Job,</p> </li> <li> <p>sets <code>.spec.schedulingGroup</code> onto every Pod the Job creates so the scheduler treats them as a single gang, and</p> </li> <li> <p>sets the Job as the owner of the generated objects, so they are garbage-collected when the Job is deleted.</p> </li> </ul> <h3 id="when-does-the-integration-kick-in">When does the integration kick in?<a class="td-heading-self-link" href="#when-does-the-integration-kick-in" aria-label="Heading self-link"></a></h3><p>To keep the first feature iteration predictable, the Job controller only creates a Workload and PodGroup when the Job has a well-defined, fixed shape:</p> <ul> <li> <p><code>.spec.parallelism</code> is greater than 1</p> </li> <li> <p><a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode"><code>.spec.completionMode</code></a> is set to <code>Indexed</code></p> </li> <li> <p><code>.spec.completions</code> is equal to <code>.spec.parallelism</code></p> </li> <li> <p>The <code>schedulingGroup</code> is not already set on the Pod template.</p> </li> </ul> <p>These conditions describe the class of Jobs that gang scheduling can reason about: each Pod has a stable identity (<code>Indexed</code>), the gang size is known and fixed at admission time (<code>parallelism</code> == <code>completions</code>), and no other controller has already claimed scheduling responsibility (<code>schedulingGroup</code> field is unset). Jobs that do not meet these conditions are scheduled Pod-by-Pod, exactly as before.</p> <p>If you set <code>schedulingGroup</code> on the Pod template yourself (for example, because a higher-level controller is managing the workload), the Job controller leaves the Pod template alone and does not create its own Workload or PodGroup. This makes the feature safe to enable in clusters that already use an external batch system.</p> <p>Here is an example of a Job that qualifies for gang scheduling:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">batch/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Job</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">training-job</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">job-ns</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">completionMode</span><span class="p">:</span><span class="w"> </span><span class="l">Indexed</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">parallelism</span><span class="p">:</span><span class="w"> </span><span class="m">4</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">completions</span><span class="p">:</span><span class="w"> </span><span class="m">4</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restartPolicy</span><span class="p">:</span><span class="w"> </span><span class="l">Never</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">containers</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">worker</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">registry.example/trainer:latest</span><span class="w"> </span></span></span></code></pre></div><p>The Job controller creates a Workload and a PodGroup owned by this Job, and every Pod it creates carries a <code>.spec.schedulingGroup</code> that points at the generated PodGroup. The Pods are then scheduled together once all four can be placed at the same time using the PodGroup scheduling cycle described earlier in this post.</p> <h3 id="what-s-not-covered-yet">What's not covered yet<a class="td-heading-self-link" href="#what-s-not-covered-yet" aria-label="Heading self-link"></a></h3><p>The current constraints limit this integration to static, indexed, fully-parallel Jobs. Support for additional workload shapes, including elastic Jobs and other built-in controllers, is tracked in <a href="https://kep.k8s.io/5547">KEP-5547</a>.</p> <p>In future Kubernetes releases, this integration will expand to support additional workload controllers, and the current constraints for Jobs may be relaxed.</p> <h2 id="what-s-next">What's next?<a class="td-heading-self-link" href="#what-s-next" aria-label="Heading self-link"></a></h2><p>The journey for workload-aware scheduling doesn't stop here. For v1.37, the community is actively working on:</p> <ul> <li> <p><strong>Graduating Workload and PodGroup APIs to Beta:</strong> Our primary goal is to mature the Workload and PodGroup APIs to the Beta stage, solidifying their foundational role in the Kubernetes ecosystem. As part of this graduation process, we also plan to introduce <code>minCount</code> mutability to unlock elastic jobs and allow dynamic workloads to scale efficiently.</p> </li> <li> <p><strong>Multi-level Workload hierarchies:</strong> To support complex modern AI workloads like JobSet or Disaggregated Inference via LeaderWorkerSet (LWS), we are working on expanding the architecture to support multi-level hierarchies. We aim to introduce a new API that allows grouping multiple PodGroups into hierarchical structures, directly reflecting the organization of real-world workload controllers.</p> </li> <li> <p><strong>Graduating advanced scheduling features:</strong> We are focused on driving the maturity of the broader workload-aware scheduling ecosystem. This includes bringing existing features, such as topology-aware scheduling and workload-aware preemption, to the Beta stage.</p> </li> <li> <p><strong>Unified controller integration API:</strong> To streamline adoption, we’re working on a controller integration API. This will provide real-world workload controllers with a unified, standardized method for consuming workload-aware scheduling capabilities.</p> </li> </ul> <p>The priority and implementation order of these focus areas are subject to change. Stay tuned for further updates.</p> <h2 id="getting-started">Getting started<a class="td-heading-self-link" href="#getting-started" aria-label="Heading self-link"></a></h2><p>All below workload-aware scheduling improvements are available as Alpha features in v1.36. To try them out, you must configure the following:</p> <ul> <li>Prerequisite: Workload and PodGroup API support: Enable the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#GenericWorkload"><code>GenericWorkload</code></a> feature gate on both the <code>kube-apiserver</code> and <code>kube-scheduler</code>, and ensure the <code>scheduling.k8s.io/v1alpha2</code> <a class='glossary-tooltip' title='A set of related paths in the Kubernetes API.' data-bs-toggle='tooltip' data-bs-placement='top' href='https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups-and-versioning' target='_blank' aria-label='API group'>API group</a> is enabled.</li> </ul> <p>Once the prerequisite is met, you can enable specific features:</p> <ul> <li>Gang scheduling: Enable the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#GangScheduling"><code>GangScheduling</code></a> feature gate on the <code>kube-scheduler</code>.</li> <li>Topology-aware scheduling: Enable the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#TopologyAwareWorkloadScheduling"><code>TopologyAwareWorkloadScheduling</code></a> feature gate on the <code>kube-scheduler</code>.</li> <li>Workload-aware preemption: Enable the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#WorkloadAwarePreemption"><code>WorkloadAwarePreemption</code></a> feature gate on the <code>kube-scheduler</code> (requires <code>GangScheduling</code> to also be enabled).</li> <li>DRA ResourceClaim support for workloads: Enable the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#DRAWorkloadResourceClaims"><code>DRAWorkloadResourceClaims</code></a> feature gate on the <code>kube-apiserver</code>, <code>kube-controller-manager</code>, <code>kube-scheduler</code> and <code>kubelet</code>.</li> <li>Workload API integration with the Job controller: Enable the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#EnableWorkloadWithJob"><code>WorkloadWithJob</code></a> feature gate on the <code>kube-apiserver</code> and <code>kube-controller-manager</code>.</li> </ul> <p>We encourage you to try out workload-aware scheduling in your test clusters and share your experiences to help shape the future of Kubernetes scheduling. You can send your feedback by:</p> <ul> <li>Reaching out via <a href="https://kubernetes.slack.com/archives/C0AHLJ0EAEL">Slack (#workload-aware-scheduling)</a>.</li> <li>Joining the <a href="https://www.kubernetes.dev/community/community-groups/sigs/scheduling/#meetings">SIG Scheduling</a> meetings.</li> <li>Filing a new <a href="https://github.com/kubernetes/kubernetes/issues">issue</a> in the Kubernetes repository.</li> </ul> <h2 id="learn-more">Learn more<a class="td-heading-self-link" href="#learn-more" aria-label="Heading self-link"></a></h2><p>To dive deeper into the architecture and design of these features, read the KEPs:</p> <ul> <li><a href="https://kep.k8s.io/4671">Workload API and gang scheduling</a></li> <li><a href="https://kep.k8s.io/5732">Topology-aware scheduling</a></li> <li><a href="https://kep.k8s.io/5710">Workload-aware preemption</a></li> <li><a href="https://kep.k8s.io/5729">DRA ResourceClaim support for workloads</a></li> <li><a href="https://kep.k8s.io/5547">Workload API support in Job controller</a></li> </ul>
Krebs on Security published: Patch Tuesday, May 2026 Edition
<p>Since its original implementation in the Linux kernel in 2018, <em>Pressure Stall Information</em> (PSI) has provided users with the high-fidelity signals needed to identify resource saturation before it becomes an outage. Unlike traditional utilization metrics, PSI tells the story of tasks stalled and time lost, all in nicely-packaged percentages of time across the CPU, memory, and I/O.</p> <p>With the recent release of Kubernetes v1.36, users across the ecosystem have a stable, reliable interface to observe resource contention at the node, pod, and container levels. In this post, we will dive into the improvements and performance testing that proved its readiness for production.</p> <h2 id="beyond-utilization-why-psi">Beyond utilization: why PSI?<a class="td-heading-self-link" href="#beyond-utilization-why-psi" aria-label="Heading self-link"></a></h2><p>Monitoring CPU or memory usage alone can be misleading. A node may report XX% (below 100%) CPU utilization while certain tasks are experiencing severe latency due to scheduling delays. PSI fills this gap by providing:</p> <ul> <li><strong>Cumulative Totals</strong>: Absolute time spent in a stalled state.</li> <li><strong>Moving Averages</strong>: 10s, 60s, and 300s windows that allow operators to distinguish between transient spikes and sustained resource tension.</li> </ul> <h2 id="proving-stability-performance-testing-at-scale">Proving stability: performance testing at scale<a class="td-heading-self-link" href="#proving-stability-performance-testing-at-scale" aria-label="Heading self-link"></a></h2><p>A common concern when graduating telemetry features is the resource overhead required to collect and serve the metrics. To address this, SIG Node conducted extensive performance validation on high-density workloads (80+ pods) across various machine types.</p> <p>Our testing focused on two primary scenarios to isolate the impact of the Kubelet and kernel-level collection respectively:</p> <ol> <li><strong>Kernel PSI ON / Kubelet Feature OFF</strong> vs <strong>Kernel PSI ON / Kubelet Feature ON</strong> (Kubelet overhead)</li> <li><strong>Kernel PSI OFF / Kubelet Feature ON</strong> vs <strong>Kernel PSI ON / Kubelet Feature ON</strong> (Kernel overhead)</li> </ol> <h4 id="scenario-1-the-kubelet-overhead">Scenario 1: The Kubelet Overhead<a class="td-heading-self-link" href="#scenario-1-the-kubelet-overhead" aria-label="Heading self-link"></a></h4><p>First, we looked at the kubelet usage on 4 core machines (Case 1). For these, the Linux kernel was already tracking pressure on both clusters by default(<code>psi=1</code>), but we toggled the <code>KubeletPSI</code> feature gate to see if the Kubelet actively querying and exposing these metrics impacted the resource usage. The synchronized bursts seen in the graph are practically identical in both magnitude and frequency, confirming that the Kubelet's collection logic is highly lightweight and blends seamlessly into standard housekeeping cycles. There is no issue about the feature affecting the pre-existing resource use, staying within the normal 0.1 cores or <strong>2.5% of the total node capacity</strong>, and is therefore safe for production-scale deployments.</p> <figure> <img src="https://kubernetes.io/images/kubeletPSI_kubelet_cpu_usage_rate_graph.png" alt="A line graph comparing the kubelet CPU usage rate over elapsed time with the Kubelet PSI feature turned off versus on and kernel PSI always on."/> <figcaption> <h4>(Case 1) Kubelet CPU Usage Rate Comparison</h4><p>Figure 2: Kubelet CPU Usage Rate Comparison.</p> </figcaption> </figure> <p>Next, we evaluated the system overhead in the same run. As seen in the following graph, the <strong>System CPU</strong> usage lines for the Kubelet PSI-enabled (red) follows the same pattern as the Kubelet PSI-disabled (blue) clusters, with a slight expected increase from the baseline. This visualizes that once the OS is tracking PSI, at around <strong>2.5 cores</strong>, the act of Kubernetes reading those cgroup metrics is negligible to performance.</p> <figure> <img src="https://kubernetes.io/images/kubeletPSI_sys_cpu_usage_rate_graph.png" alt="A line graph comparing the system CPU usage rate over elapsed time with the PSI feature turned off versus on and kernel PSI default ON."/> <figcaption> <h4>(Case 1) System CPU Usage Rate Comparison</h4><p>Figure 1: Node System CPU Usage Rate Comparison.</p> </figcaption> </figure> <h4 id="scenario-2-the-kernel-overhead">Scenario 2: The Kernel Overhead<a class="td-heading-self-link" href="#scenario-2-the-kernel-overhead" aria-label="Heading self-link"></a></h4><p>Shifting gears, we evaluated the underlying overhead of enabling PSI on the Linux kernel also on a 4 core machine. By comparing a cluster booted with <code>psi=1</code> (COS default) against a cluster with <code>psi=0</code>, we isolated the exact cost of the OS-level bookkeeping. Even under heavy I/O and CPU load at an 80-pod density, the <strong>System CPU</strong> delta between the kernel-enabled and kernel-disabled clusters remained consistently between <strong>0.037 cores</strong> and <strong>0.125 cores</strong> or <strong>0.925% - 3.125%</strong> of the total node capacity. There was a single spike to <strong>0.225 cores</strong>, or <strong>5.6%</strong>, but was controlled back down within a few seconds. This confirms that the internal kernel tracking is highly efficient under load.</p> <figure> <img src="https://kubernetes.io/images/node_sys_cpu_usage_rate_comparison.png" alt="A line graph comparing the Node System (Kernel) CPU usage rate with Kernel PSI ON and OFF over elapsed time."/> <figcaption> <h4>(Case 2) Node System CPU Usage Rate Comparison</h4><p>Figure 3: Node System CPU Usage Rate Comparison.</p> </figcaption> </figure> <p>Figure 4 zooms in on the kubelet process itself, which serves as the primary collector for these metrics. . The results show that even while the kubelet performs periodic <em>sweeps</em> to aggregate data from the cgroup hierarchy, its CPU usage remains remarkably low with interchangeable spikes and nothing exceeding <strong>0.25 cores</strong> or <strong>6.25%</strong> of total capacity for longer than a second.</p> <figure> <img src="https://kubernetes.io/images/kubelet_cpu_usage_rate_comparison.png" alt="A line graph comparing the kubelet CPU usage rate over elapsed time with the Kernel PSI feature turned off versus on."/> <figcaption> <h4>(Case 2) Kubelet CPU Usage Rate Comparison</h4><p>Figure 4: Kubelet CPU Usage Rate Comparison.</p> </figcaption> </figure> <h2 id="improvements-between-beta-1-34-and-stable-1-36">Improvements between beta (1.34) and stable (1.36)<a class="td-heading-self-link" href="#improvements-between-beta-1-34-and-stable-1-36" aria-label="Heading self-link"></a></h2><ul> <li><strong>Smarter Metric Emission for GA:</strong> We improved how the Kubelet handles underlying OS support for PSI. Previously, if the feature was enabled in Kubernetes but the underlying Linux kernel didn't support PSI (<code>psi=0</code>), the Kubelet would emit misleading zero-valued metrics. These could trigger false alarms when read as real metrics instead of missing values. In v1.36, the Kubelet now detects OS-level PSI support via cgroup configurations before reporting. This ensures that pressure metrics are only collected and emitted when they are actually supported by the node, providing cleaner data for monitoring and alerting systems.</li> </ul> <h2 id="getting-started">Getting started<a class="td-heading-self-link" href="#getting-started" aria-label="Heading self-link"></a></h2><p>To use PSI metrics in your Kubernetes cluster, your nodes must meet the following requirements:</p> <ol> <li><strong>Ensure your nodes are running a Linux kernel version 4.20 or later and are using cgroup v2.</strong></li> <li><strong>Ensure PSI is enabled at the OS level</strong> (your kernel must be compiled with <code>CONFIG_PSI=y</code> and must not be booted with the <code>psi=0</code> parameter).</li> </ol> <p>As of v1.36, Kubelet PSI metrics are generally available and you do not need to opt in to any feature gate.</p> <p>Once the OS prerequisites are met, you can start scraping the <code>/metrics/cadvisor</code> endpoint with your Prometheus-compatible monitoring solution or query the Summary API to collect and visualize the new PSI metrics. Note that PSI is a Linux-kernel feature, so these metrics are not available on Windows nodes. Your cluster can contain a mix of Linux and Windows nodes, and on the Windows nodes, the kubelet will simply omit the PSI metrics.</p> <p>If your cluster is running a recent enough version of Kubernetes and you are a privileged node administrator, you can also proxy to the kubelet's HTTP API via the control plane's API server to see real-time pressure data from the Summary API.</p> <blockquote> <p><strong>Caution:</strong> Proxying to the kubelet is a privileged operation. Granting access to it is a security risk, so ensure you have the appropriate administrative permissions before executing these commands.</p></blockquote> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">CONTAINER_NAME</span><span class="o">=</span><span class="s2">"example-container"</span> </span></span><span class="line"><span class="cl">kubectl get --raw <span class="s2">"/api/v1/nodes/</span><span class="k">$(</span>kubectl get nodes -o <span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.items[0].metadata.name}'</span><span class="k">)</span><span class="s2">/proxy/stats/summary"</span> <span class="p">|</span> jq <span class="s1">'.pods[].containers[] | select(.name=="'</span><span class="s2">"</span><span class="nv">$CONTAINER_NAME</span><span class="s2">"</span><span class="s1">'") | {name, cpu: .cpu.psi, memory: .memory.psi, io: .io.psi}'</span> </span></span></code></pre></div><h2 id="further-reading">Further reading<a class="td-heading-self-link" href="#further-reading" aria-label="Heading self-link"></a></h2><p>If you want to dive deeper into how these metrics are calculated and exposed, check out these resources:</p> <ol> <li><a href="https://docs.kernel.org/accounting/psi.html">The official Kernel documentation</a></li> <li><a href="https://kubernetes.io/docs/reference/instrumentation/understand-psi-metrics/">Understanding PSI</a> in the Kubernetes documentation</li> <li><a href="https://github.com/google/cadvisor/blob/master/metrics/prometheus.go">cAdvisor Metrics Implementation</a></li> </ol> <h2 id="acknowledgements">Acknowledgements<a class="td-heading-self-link" href="#acknowledgements" aria-label="Heading self-link"></a></h2><p>Support for PSI metrics was developed through the collaborative efforts of <a href="https://www.kubernetes.dev/community/community-groups/sigs/node/">SIG Node</a>. Special thanks to all contributors who helped design, implement, test, review, and document this feature across its journey from alpha in v1.33, through beta in v1.34, to GA in v1.36.</p> <p>To provide feedback on this feature, join the <a href="https://github.com/kubernetes/community/tree/master/sig-node">Kubernetes Node Special Interest Group</a>, participate in discussions on the <a href="http://slack.k8s.io/">public Slack channel</a> (#sig-node), or file an issue on <a href="https://github.com/kubernetes/kubernetes/issues">GitHub</a>.</p> <h2 id="feedback">Feedback<a class="td-heading-self-link" href="#feedback" aria-label="Heading self-link"></a></h2><p>If you have feedback and want to share your experience using this feature, join the discussion:</p> <ul> <li><a href="https://github.com/kubernetes/community/tree/master/sig-node">SIG Node community page</a></li> <li><a href="http://slack.k8s.io/">Kubernetes Slack</a> in the #sig-node channel</li> <li><a href="https://groups.google.com/forum/#!forum/kubernetes-sig-node">SIG Node mailing list</a></li> </ul> <p>SIG Node would love to hear about your experiences using this feature in production!</p>
🌐 HAR recording on Tracing tracing.startHar() / tracing.stopHar() expose HAR recording as a first-class tracing API, with the same content , mode and urlFilter options as recordHar . The returned Disposable makes it easy to scope a recording with await using : await using har = await context . tracing . startHar ( 'trace.har' ) ; const page = await context . newPage ( ) ; await page . goto ( 'https://playwright.dev' ) ; // HAR is finalized when `har` goes out of scope. 🪝 Drop API New locator.drop() simulates an external drag-and-drop of files or clipboard-like data onto an element. Playwright dispatches dragenter , dragover , and drop with a synthetic [DataTransfer] in the page context — works cross-browser and is great for testing upload zones: await page . locator ( '#dropzone' ) . drop ( { files : { name : 'note.txt' , mimeType : 'text/plain' , buffer : Buffer . from ( 'hello' ) } , } ) ; await page . locator ( '#dropzone' ) . drop ( { data : { 'text/plain' : 'hello world' , 'text/uri-list' : 'https://example.com' , } , } ) ; 🎯 Aria snapshots expect(page).toMatchAriaSnapshot() now works on a Page , in addition to a Locator — equivalent to asserting against page.locator('body') . New boxes option on locator.ariaSnapshot() / page.ariaSnapshot() appends each element's bounding box as [box=x,y,width,height] , useful for AI consumption. 🛑 test.abort() New test.abort() aborts the currently running test from a fixture, hook, or route handler with an optional message. Use it when you have detected an unrecoverable misuse and want to fail the test right away: { await page.route('**/publish', route => { test.abort('Tests must not publish to the shared page. Use the `clone` option.'); return route.abort(); }); // ... });"> test ( 'does not publish to the shared page' , async ( { page } ) => { await page . route ( '**/publish' , route => { test . abort ( 'Tests must not publish to the shared page. Use the `clone` option.' ) ; return route . abort ( ) ; } ) ; // ... } ) ; New APIs Browser, Context and Page Event browser.on('context') — fired when a new context is created on the browser. BrowserContext now mirrors lifecycle events from its pages: browserContext.on('download') , browserContext.on('frameattached') , browserContext.on('framedetached') , browserContext.on('framenavigated') , browserContext.on('pageclose') , browserContext.on('pageload') . Locators and Assertions New option description in page.getByRole() / locator.getByRole() / frame.getByRole() / frameLocator.getByRole() for matching the accessible description . New option pseudo in expect(locator).toHaveCSS() reads computed styles from ::before or ::after . New option style in locator.highlight() applies extra inline CSS to the highlight overlay, plus new page.hideHighlight() to clear all highlights. Network webSocketRoute.protocols() returns the WebSocket subprotocols requested by the page. New option noDefaults in browserType.connectOverCDP() disables Playwright's default overrides on the default context (download behavior, focus emulation, media emulation), so attaching to a user's daily-driver browser doesn't disturb its state. Errors and Reporting New webError.location() mirrors consoleMessage.location() . consoleMessage.location() now exposes line / column properties ( lineNumber / columnNumber are deprecated). New testInfoError.errorContext surfaces additional diagnostic context, such as the aria snapshot of the receiver at the time of an expect(...) matcher failure. reporter.onError() now receives a workerInfo argument with details about the worker for fixture teardown errors. Test runner New {testFileBaseName} token in testProject.snapshotPathTemplate — file name without extension. Test runner now errors when a config tries to override a non-option fixture, and rejects workers: 0 or negative values. 🛠️ Other improvements HTML reporter: npx playwright show-report accepts .zip files directly — no need to unzip first. Steps that contain attachments inside nested children show an indicator on the parent step. The repeatEachIndex is shown in the test header when non-zero. Trace Viewer adds a pretty-print toggle for JSON / form request and response bodies in the network details panel. Breaking Changes ⚠️ Removed long-deprecated APIs: Locator.ariaRef() — use the standard locator.ariaSnapshot() pipeline. handle option on BrowserContext.exposeBinding and Page.exposeBinding . logger option on BrowserType.connect and BrowserType.connectOverCDP — use tracing instead. Context options videosPath / videoSize — use recordVideo instead. Browser Versions Chromium 148.0.7778.96 Mozilla Firefox 150.0.2 WebKit 26.4 This version was also tested against the following stable channels: Google Chrome 147 Microsoft Edge 147
AI Agents & Email deliverability experts on your team Discussion | Link
Turn any text into natural AI voiceovers on iPhone Discussion | Link
The AI agent harness for product design teams Discussion | Link
<p>Volume group snapshots were <a href="https://kubernetes.io/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/">introduced</a> as an Alpha feature with the Kubernetes v1.27 release, moved to <a href="https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/">Beta</a> in v1.32, and to a <a href="https://kubernetes.io/blog/2025/09/16/kubernetes-v1-34-volume-group-snapshot-beta-2/">second Beta</a> in v1.34. We are excited to announce that in the Kubernetes v1.36 release, support for volume group snapshots has reached <strong>General Availability (GA)</strong>.</p> <p>The support for volume group snapshots relies on a set of <a href="https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html#volume-group-snapshot-apis">extension APIs for group snapshots</a>. These APIs allow users to take crash-consistent snapshots for a set of volumes. Behind the scenes, Kubernetes uses a label selector to group multiple <code>PersistentVolumeClaim</code> objects for snapshotting. A key aim is to allow you to restore that set of snapshots to new volumes and recover your workload based on a crash-consistent recovery point.</p> <p>This feature is only supported for <a href="https://kubernetes-csi.github.io/docs/">CSI</a> volume drivers.</p> <h2 id="an-overview-of-volume-group-snapshots">An overview of volume group snapshots<a class="td-heading-self-link" href="#an-overview-of-volume-group-snapshots" aria-label="Heading self-link"></a></h2><p>Some storage systems provide the ability to create a crash-consistent snapshot of multiple volumes. A group snapshot represents <em>copies</em> made from multiple volumes that are taken at the same point-in-time. A group snapshot can be used either to rehydrate new volumes (pre-populated with the snapshot data) or to restore existing volumes to a previous state (represented by the snapshots).</p> <h3 id="why-add-volume-group-snapshots-to-kubernetes">Why add volume group snapshots to Kubernetes?<a class="td-heading-self-link" href="#why-add-volume-group-snapshots-to-kubernetes" aria-label="Heading self-link"></a></h3><p>The Kubernetes volume plugin system already provides a powerful abstraction that automates the provisioning, attaching, mounting, resizing, and snapshotting of block and file storage. Underpinning all these features is the Kubernetes goal of workload portability.</p> <p>There was already a <a href="https://kubernetes.io/docs/concepts/storage/volume-snapshots/">VolumeSnapshot</a> API that provides the ability to take a snapshot of a persistent volume to protect against data loss or data corruption. However, some storage systems support consistent group snapshots that allow a snapshot to be taken from multiple volumes at the same point-in-time to achieve write order consistency. This is extremely useful for applications that contain multiple volumes. For example, an application may have data stored in one volume and logs stored in another. If snapshots for these volumes are taken at different times, the application will not be consistent and will not function properly if restored from those snapshots.</p> <p>While you can quiesce the application first and take individual snapshots sequentially, this process can be time-consuming or sometimes impossible. Consistent group support provides crash consistency across all volumes in the group without the need for application quiescence.</p> <h3 id="kubernetes-apis-for-volume-group-snapshots">Kubernetes APIs for volume group snapshots<a class="td-heading-self-link" href="#kubernetes-apis-for-volume-group-snapshots" aria-label="Heading self-link"></a></h3><p>Kubernetes' support for volume group snapshots relies on three API kinds that are used for managing snapshots:</p> <dl> <dt>VolumeGroupSnapshot</dt> <dd>Created by a Kubernetes user (or automation) to request creation of a volume group snapshot for multiple persistent volume claims.</dd> <dt>VolumeGroupSnapshotContent</dt> <dd>Created by the snapshot controller for a dynamically created VolumeGroupSnapshot. It contains information about the provisioned cluster resource (a group snapshot). The object binds to the VolumeGroupSnapshot for which it was created with a one-to-one mapping.</dd> <dt>VolumeGroupSnapshotClass</dt> <dd>Created by cluster administrators to describe how volume group snapshots should be created, including the driver information, the deletion policy, etc.</dd> </dl> <p>These three API kinds are defined as CustomResourceDefinitions (CRDs). For the GA release, the API version has been promoted to <code>v1</code>.</p> <h2 id="what-s-new-in-ga">What's new in GA?<a class="td-heading-self-link" href="#what-s-new-in-ga" aria-label="Heading self-link"></a></h2><ul> <li>The API version for <code>VolumeGroupSnapshot</code>, <code>VolumeGroupSnapshotContent</code>, and <code>VolumeGroupSnapshotClass</code> is promoted to <code>groupsnapshot.storage.k8s.io/v1</code>.</li> <li>Enhanced stability and bug fixes based on feedback from the beta releases, including the improvements introduced in v1beta2 for accurate <code>restoreSize</code> reporting.</li> </ul> <h2 id="how-do-i-use-kubernetes-volume-group-snapshots">How do I use Kubernetes volume group snapshots<a class="td-heading-self-link" href="#how-do-i-use-kubernetes-volume-group-snapshots" aria-label="Heading self-link"></a></h2><h3 id="creating-a-new-group-snapshot-with-kubernetes">Creating a new group snapshot with Kubernetes<a class="td-heading-self-link" href="#creating-a-new-group-snapshot-with-kubernetes" aria-label="Heading self-link"></a></h3><p>Once a <code>VolumeGroupSnapshotClass</code> object is defined and you have volumes you want to snapshot together, you may request a new group snapshot by creating a <code>VolumeGroupSnapshot</code> object.</p> <p>Label the PVCs you wish to group:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> kubectl label pvc pvc-0 <span class="nv">group</span><span class="o">=</span>myGroup </span></span><span class="line"><span class="cl"><span class="go">persistentvolumeclaim/pvc-0 labeled </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="gp">%</span> kubectl label pvc pvc-1 <span class="nv">group</span><span class="o">=</span>myGroup </span></span><span class="line"><span class="cl"><span class="go">persistentvolumeclaim/pvc-1 labeled </span></span></span></code></pre></div><p>For dynamic provisioning, a selector must be set so that the snapshot controller can find PVCs with the matching labels to be snapshotted together.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">groupsnapshot.storage.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">VolumeGroupSnapshot</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">snapshot-daily-20260422</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">demo-namespace</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumeGroupSnapshotClassName</span><span class="p">:</span><span class="w"> </span><span class="l">csi-groupSnapclass</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">myGroup</span><span class="w"> </span></span></span></code></pre></div><p>The <code>VolumeGroupSnapshotClass</code> is required for dynamic provisioning:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">groupsnapshot.storage.k8s.io/v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">VolumeGroupSnapshotClass</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">csi-groupSnapclass</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">driver</span><span class="p">:</span><span class="w"> </span><span class="l">example.csi.k8s.io</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">deletionPolicy</span><span class="p">:</span><span class="w"> </span><span class="l">Delete</span><span class="w"> </span></span></span></code></pre></div><h3 id="how-to-use-group-snapshot-for-restore">How to use group snapshot for restore<a class="td-heading-self-link" href="#how-to-use-group-snapshot-for-restore" aria-label="Heading self-link"></a></h3><p>At restore time, request a new <code>PersistentVolumeClaim</code> to be created from a <code>VolumeSnapshot</code> object that is part of a <code>VolumeGroupSnapshot</code>. Repeat this for all volumes that are part of the group snapshot.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PersistentVolumeClaim</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">examplepvc-restored-2026-04-22</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">demo-namespace</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">storageClassName</span><span class="p">:</span><span class="w"> </span><span class="l">example-sc</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">dataSource</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">snapshot-0962a745b2bf930bb385b7b50c9b08af471f1a16780726de19429dd9c94eaca0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">VolumeSnapshot</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">apiGroup</span><span class="p">:</span><span class="w"> </span><span class="l">snapshot.storage.k8s.io</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">accessModes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ReadWriteOncePod</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">resources</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">requests</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">storage</span><span class="p">:</span><span class="w"> </span><span class="l">100Mi</span><span class="w"> </span></span></span></code></pre></div><h2 id="as-a-storage-vendor-how-do-i-add-support-for-group-snapshots">As a storage vendor, how do I add support for group snapshots?<a class="td-heading-self-link" href="#as-a-storage-vendor-how-do-i-add-support-for-group-snapshots" aria-label="Heading self-link"></a></h2><p>To implement the volume group snapshot feature, a CSI driver <strong>must</strong>:</p> <ul> <li>Implement a new group controller service.</li> <li>Implement group controller RPCs: <code>CreateVolumeGroupSnapshot</code>, <code>DeleteVolumeGroupSnapshot</code>, and <code>GetVolumeGroupSnapshot</code>.</li> <li>Add group controller capability <code>CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT</code>.</li> </ul> <p>See the <a href="https://github.com/container-storage-interface/spec/blob/master/spec.md">CSI spec</a> and the <a href="https://kubernetes-csi.github.io/docs/">Kubernetes-CSI Driver Developer Guide</a> for more details.</p> <h2 id="how-can-i-learn-more">How can I learn more?<a class="td-heading-self-link" href="#how-can-i-learn-more" aria-label="Heading self-link"></a></h2><ul> <li>The <a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot">design spec</a> for the volume group snapshot feature.</li> <li>The <a href="https://github.com/kubernetes-csi/external-snapshotter">code repository</a> for volume group snapshot APIs and controller.</li> <li>CSI <a href="https://kubernetes-csi.github.io/docs/">documentation</a> on the group snapshot feature.</li> </ul> <h2 id="how-do-i-get-involved">How do I get involved?<a class="td-heading-self-link" href="#how-do-i-get-involved" aria-label="Heading self-link"></a></h2><p>This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. On behalf of SIG Storage, I would like to offer a huge thank you to all the contributors who stepped up over the years to help the project reach GA:</p> <ul> <li>Ben Swartzlander (<a href="https://github.com/bswartz">bswartz</a>)</li> <li>Cici Huang (<a href="https://github.com/cici37">cici37</a>)</li> <li>Darshan Murthy (<a href="https://github.com/darshansreenivas">darshansreenivas</a>)</li> <li>Hemant Kumar (<a href="https://github.com/gnufied">gnufied</a>)</li> <li>James Defelice (<a href="https://github.com/jdef">jdef</a>)</li> <li>Jan Šafránek (<a href="https://github.com/jsafrane">jsafrane</a>)</li> <li>Madhu Rajanna (<a href="https://github.com/Madhu-1">Madhu-1</a>)</li> <li>Manish M Yathnalli (<a href="https://github.com/manishym">manishym</a>)</li> <li>Michelle Au (<a href="https://github.com/msau42">msau42</a>)</li> <li>Niels de Vos (<a href="https://github.com/nixpanic">nixpanic</a>)</li> <li>Leonardo Cecchi (<a href="https://github.com/leonardoce">leonardoce</a>)</li> <li>Rakshith R (<a href="https://github.com/Rakshith-R">Rakshith-R</a>)</li> <li>Raunak Shah (<a href="https://github.com/RaunakShah">RaunakShah</a>)</li> <li>Saad Ali (<a href="https://github.com/saad-ali">saad-ali</a>)</li> <li>Wei Duan (<a href="https://github.com/duanwei33">duanwei33</a>)</li> <li>Xing Yang (<a href="https://github.com/xing-yang">xing-yang</a>)</li> <li>Yati Padia (<a href="https://github.com/yati1998">yati1998</a>)</li> </ul> <p>For those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the <a href="https://github.com/kubernetes/community/tree/master/sig-storage">Kubernetes Storage Special Interest Group</a> (SIG). We always welcome new contributors.</p> <p>We also hold regular <a href="https://github.com/kubernetes/community/tree/master/wg-data-protection">Data Protection Working Group meetings</a>. New attendees are welcome to join our discussions.</p>
Krebs on Security published: Canvas Breach Disrupts Schools & Colleges Nationwide
<p>Dynamic Resource Allocation (DRA) has fundamentally changed how platform administrators handle hardware accelerators and specialized resources in Kubernetes. In the v1.36 release, DRA continues to mature, bringing a wave of feature graduations, critical usability improvements, and new capabilities that extend the flexibility of DRA to native resources like memory and CPU, and support for ResourceClaims in PodGroups.</p> <p>Driver availability continues to expand. Beyond specialized compute accelerators, the ecosystem includes support for networking and other hardware types, reflecting a move toward a more robust, hardware-agnostic infrastructure.</p> <p>Whether you are managing massive fleets of GPUs, need better handling of failures, or simply looking for better ways to define resource fallback options, the upgrades to DRA in 1.36 have something for you. Let's dive into the new features and graduations!</p> <h2 id="feature-graduations">Feature graduations<a class="td-heading-self-link" href="#feature-graduations" aria-label="Heading self-link"></a></h2><p>The community has been hard at work stabilizing core DRA concepts. In Kubernetes 1.36, several highly anticipated features have graduated to Beta and Stable.</p> <h3 id="prioritized-list">Prioritized list (stable)<a class="td-heading-self-link" href="#prioritized-list" aria-label="Heading self-link"></a></h3><p>Hardware heterogeneity is a reality in most clusters. With the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#prioritized-list">Prioritized list</a> feature, you can confidently define fallback preferences when requesting devices. Instead of hardcoding a request for a specific device model, you can specify an ordered list of preferences (e.g., "Give me an H100, but if none are available, fall back to an A100"). The scheduler will evaluate these requests in order, drastically improving scheduling flexibility and cluster utilization.</p> <h3 id="extended-resource">Extended resource support (beta)<a class="td-heading-self-link" href="#extended-resource" aria-label="Heading self-link"></a></h3><p>As DRA becomes the standard for resource allocation, bridging the gap with legacy systems is crucial. The DRA <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#extended-resource">Extended resource</a> feature allows users to request resources via traditional extended resources on a Pod. This allows for a gradual transition to DRA, meaning cluster operators can migrate clusters to DRA but let application developers adopt the ResourceClaim API on their own schedule.</p> <h3 id="partitionable-devices">Partitionable devices (beta)<a class="td-heading-self-link" href="#partitionable-devices" aria-label="Heading self-link"></a></h3><p>Hardware accelerators are powerful, and sometimes a single workload doesn't need an entire device. The <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#partitionable-devices">Partitionable devices</a> feature, provides native DRA support for dynamically carving physical hardware into smaller, logical instances (such as Multi-Instance GPUs) based on workload demands. This allows administrators to safely and efficiently share expensive accelerators across multiple Pods.</p> <h3 id="device-taints">Device taints (beta)<a class="td-heading-self-link" href="#device-taints" aria-label="Heading self-link"></a></h3><p>Just as you can taint a Kubernetes Node, you can apply taints directly to specific DRA devices. <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#device-taints-and-tolerations">Device taints and tolerations</a> empower cluster administrators to manage hardware more effectively. You can taint faulty devices to prevent them from being allocated to standard claims, or reserve specific hardware for dedicated teams, specialized workloads, and experiments. Ultimately, only Pods with matching tolerations are permitted to claim these tainted devices.</p> <h3 id="device-binding-conditions">Device binding conditions (beta)<a class="td-heading-self-link" href="#device-binding-conditions" aria-label="Heading self-link"></a></h3><p>To improve scheduling reliability, the Kubernetes scheduler can use the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#device-binding-conditions">Binding conditions</a> feature to delay committing a Pod to a Node until its required external resources—such as attachable devices or FPGAs—are fully prepared. By explicitly modeling resource readiness, this prevents premature assignments that can lead to Pod failures, ensuring a much more robust and predictable deployment process.</p> <h3 id="device-health-monitoring">Resource health status (beta)<a class="td-heading-self-link" href="#device-health-monitoring" aria-label="Heading self-link"></a></h3><p>Knowing when a device has failed or become unhealthy is critical for workloads running on specialized hardware. With <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#device-health-monitoring">Resource health status</a>, Kubernetes expose device health information directly in the Pod status, giving users and controllers crucial visibility to quickly identify and react to hardware failures. The feature includes support for human-readable health status messages, making it significantly easier to diagnose issues without the need to dive into complex driver logs.</p> <h2 id="new-features">New Features<a class="td-heading-self-link" href="#new-features" aria-label="Heading self-link"></a></h2><p>Beyond stabilizing existing capabilities, v1.36 introduces foundational new features that expand what DRA can do. These are alpha features, so they are behind feature gates that are disabled by default.</p> <h3 id="workload-resourceclaims">ResourceClaim support for workloads<a class="td-heading-self-link" href="#workload-resourceclaims" aria-label="Heading self-link"></a></h3><p>To optimize large-scale AI/ML workloads that rely on strict topological scheduling, the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#workload-resourceclaims">ResourceClaim support for workloads</a> feature enables Kubernetes to seamlessly manage shared resources across massive sets of Pods. By associating ResourceClaims or ResourceClaimTemplates with PodGroups, this feature eliminates previous scaling bottlenecks, such as the limit on the number of pods that can share a claim, and removes the burden of manual claim management from specialized orchestrators.</p> <h3 id="node-allocatable-resources">Node allocatable resources<a class="td-heading-self-link" href="#node-allocatable-resources" aria-label="Heading self-link"></a></h3><p>Why should DRA only be for external accelerators? In v1.36, we are introducing the first iteration of using the DRA APIs to manage <em>node allocatable</em> infrastructure resources (like CPU and memory). By bringing CPU and memory allocation under the DRA umbrella with the DRA <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#node-allocatable-resources">Node allocatable resources</a> feature, users can leverage DRA's advanced placement, NUMA-awareness, and prioritization semantics for standard compute resources, paving the way for incredibly fine-grained performance tuning.</p> <h3 id="resource-pool-status">DRA resource availability visibility<a class="td-heading-self-link" href="#resource-pool-status" aria-label="Heading self-link"></a></h3><p>One of the most requested features from cluster administrators has been better visibility into hardware capacity. The new <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resource-pool-status">Resource pool status</a> feature allows you to query the availability of devices in DRA resource pools. By creating a <code>ResourcePoolStatusRequest</code> object, you get a point-in-time snapshot of device counts — total, allocated, available, and unavailable — for each pool managed by a given driver. This enables better integration with dashboards and capacity planning tools.</p> <h3 id="list-type-attributes">List types for attributes<a class="td-heading-self-link" href="#list-type-attributes" aria-label="Heading self-link"></a></h3><p>ResourceClaim constraint evaluation has changed to work better with scalar and list values: <code>matchAttribute</code> now checks for a non-empty intersection, and <code>distinctAttribute</code> checks for pairwise disjoint values.</p> <p>An <code>includes()</code> function in CEL has also been introduced, that lets device selectors keep working more easily when an attribute changes between scalar and list representations. (The <code>includes()</code> function is only available in DRA contexts for expression evaluation).</p> <h3 id="deterministic-device-selection">Deterministic device selection<a class="td-heading-self-link" href="#deterministic-device-selection" aria-label="Heading self-link"></a></h3><p>The Kubernetes scheduler has been updated to evaluate devices using lexicographical ordering based on resource pool and ResourceSlice names. This change empowers drivers to proactively influence the scheduling process, leading to improved throughput and more optimal scheduling decisions. The ResourceSlice controller toolkit automatically generates names that reflect the exact device ordering specified by the driver author.</p> <h3 id="device-metadata">Discoverable device metadata in containers<a class="td-heading-self-link" href="#device-metadata" aria-label="Heading self-link"></a></h3><p>Workloads running on nodes with DRA devices often need to discover details about their allocated devices, such as PCI bus addresses or network interface configuration, without querying the Kubernetes API. With <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#device-metadata">Device metadata</a>, Kubernetes defines a standard protocol for how DRA drivers expose device attributes to containers as versioned JSON files at well-known paths. Drivers built with the <a href="https://pkg.go.dev/k8s.io/dynamic-resource-allocation/kubeletplugin">DRA kubelet plugin library</a> get this behavior transparently; they just provide the metadata and the library handles file layout, CDI bind-mounts, versioning, and lifecycle. This gives applications a consistent, driver-independent way to discover and consume device metadata, eliminating the need for custom controllers or looking up ResourceSlice objects to get metadata via attributes.</p> <h2 id="what-s-next">What’s next?<a class="td-heading-self-link" href="#what-s-next" aria-label="Heading self-link"></a></h2><p>This release introduced a wealth of new Dynamic Resource Allocation (DRA) features, and the momentum is only building. As we look ahead, our roadmap focuses on maturing existing features toward beta and stable releases while hardening DRA’s performance, scalability, and reliability. A key priority over the coming cycles will be deep integration with <em>workload aware</em> and <em>topology aware scheduling</em>.</p> <p>A big goal for us is to migrate users from Device Plugin to DRA, and we want you involved. Whether you are currently maintaining a driver or are just beginning to explore the possibilities, your input is vital. Partner with us to shape the next generation of resource management. Reach out today to collaborate on development, share feedback, or start building your first DRA driver.</p> <h2 id="getting-involved">Getting involved<a class="td-heading-self-link" href="#getting-involved" aria-label="Heading self-link"></a></h2><p>A good starting point is joining the WG Device Management <a href="https://kubernetes.slack.com/archives/C0409NGC1TK">Slack channel</a> and <a href="https://docs.google.com/document/d/1qxI87VqGtgN7EAJlqVfxx86HGKEAc2A3SKru8nJHNkQ/edit?tab=t.0#heading=h.tgg8gganowxq">meetings</a>, which happen at Americas/EMEA and EMEA/APAC friendly time slots.</p> <p>Not all enhancement ideas are tracked as issues yet, so come talk to us if you want to help or have some ideas yourself! We have work to do at all levels, from difficult core changes to usability enhancements in kubectl, which could be picked up by newcomers.</p>
Two weeks ago we announced that we had identified and fixed an unprecedented number of latent security bugs in Firefox with the help of Claude Mythos Preview and other AI models. In this post, we’ll go into more detail about how we approached this work, what we found, and advice for other projects on making […] The post Behind the Scenes Hardening Firefox with Claude Mythos Preview appeared first on Mozilla Hacks - the Web developer blog .