Good apps aren’t born, they’re guided: Building observable policy as code

As parents in tech, we’ve learned that neither children nor applications thrive without clear boundaries. There are no “good” or “bad” kids, just as there are no inherently “good” or “bad” applications, only behaviors shaped by the guardrails around them. In parenting, we establish rules to encourage safe, responsible decisions while still allowing independence. Modern cloud-native platforms require the same approach. Policy as Code provides those guardrails, defining how applications can operate, what resources they can access, and how they remain compliant at scale without slowing down innovation.

Just as parents establish boundaries to help their children make safe and responsible decisions, platform engineering teams rely on policies to guide application behavior. The challenge is that setting rules alone is not enough; you also need visibility into whether they are being followed, where they are failing, and whether they are creating unintended consequences. Without real-time observability, policy enforcement can become a black box, making it difficult to identify misconfigurations, compliance gaps, or obstacles to developer productivity before they impact production.

This is where Policy as Code and Observability come together. Much like a parent needs feedback to understand how rules are working in practice, platform teams need continuous insight into policy outcomes across their clusters. By combining Kyverno, a Kubernetes-native policy engine, with VictoriaMetrics, an open-source monitoring solution, teams can eliminate policy blind spots and gain a unified view of both compliance and system health, enabling them to scale governance without sacrificing visibility or developer velocity.

Why This Matters: Community Impact and Practitioner Value

Historically, cluster security and infrastructure observability have existed as two distinct silos within platform organizations. Security teams write policy rules, while reliability teams build dashboards.

This framework changes that dynamic by treating policy enforcement as a primary telemetry data source. Instead of relying on passive, text-heavy log files to figure out why a deployment failed, this setup directly transforms admission control webhooks into real-time metric streams. It treats policy as data, allowing organizations to observe compliance trends with the exact same tooling used to monitor CPU usage or network latency.

How It Directly Helps Practitioners

Removes “Invisible Wall” Friction: Developers frequently encounter admission controllers as sudden, unexplained blocks during deployments. By exposing policy metrics on shared Grafana dashboards, developers can instantly see exactly which validation rules are failing across the cluster, transforming security from a roadblock into self-service feedback.

Simplifies Governance via Native Tooling: Practitioners do not need to learn specialized, complex programming languages to audit cluster states. Because Kyverno relies entirely on standard YAML and CEL (Common Expression Language), and VictoriaMetrics natively supports standard PromQL queries, any practitioner can modify policies or adjust monitoring dashboards without deep proprietary expertise.

Prevents Operational Overhead and Budget Bloat: High-volume clusters can trigger thousands of admission requests every minute. For practitioners running traditional monitoring systems, storing these high-cardinality metrics can lead to massive RAM usage and steep infrastructure bills. Using a lightweight time-series backend ensures that teams gain comprehensive policy visibility without incurring high data storage costs.

Operational Challenges Solved

1. Eliminating Specialized Language Overhead

Adopting policy-as-code often introduces a steep learning curve due to specialized, non-native languages. This creates operational bottlenecks where only a small number of engineers can write, maintain, or audit compliance rules. Kyverno addresses this by managing policies as standard Kubernetes manifests using declarative YAML and CEL.

2. Managing High-Cardinality Metrics Scaling

Tracking every admission request, mutation, validation failure, and image verification at scale generates a significant volume of time-series data. VictoriaMetrics is a resource-efficient time-series database that can process millions of data points with low memory overhead. Its native UI Cardinality Explorer helps teams identify which policy labels consume the most storage, ensuring the monitoring infrastructure remains performant.

3. Surface Visibility for Enforced Rules

Policy enforcement can sometimes function as an invisible wall for developers. An admission controller rejects workloads, and there is no central dashboard to track why or how often violations occur. This architecture uses a five-policy implementation that spans the resource lifecycle and forwards raw engine telemetry to real-time dashboards.

Architectural Deployment

Phase 1: Deploying the Core Components

First, install the Kyverno Policy Engine to handle admission control, then install VictoriaMetrics as the time-series database backend.

# 1. Install Kyverno Policy Engine helm repo add kyverno https://kyverno.github.io/kyverno/ helm repo update helm install kyverno kyverno/kyverno -n kyverno --create-namespace # 2. Install VictoriaMetrics Single-Node Server helm repo add vm https://victoriametrics.github.io/helm-charts/ helm repo update helm install vm vm/victoria-metrics-single

Phase 2: Configuring Metric Collection

The vmagent gathers telemetry by scraping the dedicated Kyverno metrics service endpoint (port 8000) and forwarding those data points directly to VictoriaMetrics.

helm upgrade --install vmagent victoriametrics/victoria-metrics-agent \ --set 'remoteWrite[0].url=http://vm-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write' \ --set 'extraArgs.promscrape.config-global=\n scrape_interval: 10s\n\nscrape_configs:\n - job_name: kyverno\n metrics_path: /metrics\n static_configs:\n - targets:\n - kyverno-svc-metrics.kyverno.svc.cluster.local:8000' \ --set 'podLabels.app=vmagent' \ --set 'podLabels.environment=demo' \ --set 'podLabels.team=observability'

Verify that vmagent is utilizing the correct target configuration and successfully scraping endpoints:

curl http://localhost:8429/targets

Expected output response showing state=up, alongside target tracking metrics.

Deploying the good and the bad apps

Just as clear expectations help guide behavior, platform policies help ensure applications follow organizational standards from the moment they are deployed. To demonstrate this, we deploy an application that intentionally violates organizational standards by omitting required Kubernetes labels. This common real-world issue results in missing metadata, inconsistent resource classification, and a lack of ownership information, problems that directly impact observability, cost allocation, governance, and incident response workflows.

apiVersion: apps/v1 kind: Deployment metadata: name: bad-app spec: replicas: 2 selector: matchLabels: app: bad-app template: metadata: labels: app: bad-app random-id: "12345" spec: containers: - name: nginx image: nginx

In contrast, we deploy an application that follows the organization’s labeling and metadata standards. Although both applications run in the same cluster, their behavior from a governance and operational perspective is very different. This is where Policy as Code becomes powerful: by applying policies, we can not only enforce these standards but also observe, measure, and understand their impact across the platform.

apiVersion: apps/v1 kind: Deployment metadata: name: good-app namespace: default spec: replicas: 2 selector: matchLabels: app: good-app template: metadata: labels: app: good-app environment: demo team: platform spec: containers: - name: nginx image: nginx:stable

Before exploring the Kyverno policy lifecycle framework, we access the VictoriaMetrics UI through Kubernetes port forwarding. Kyverno allows us to verify policy status and enforcement behavior, while the VictoriaMetrics vmui provides access to the collected metrics. Within vmui, we can query policy-related metrics and verify that data is being ingested correctly. We can also inspect the Targets page to confirm that Kyverno is being successfully scraped and monitored.

kubectl port-forward svc/vm-victoria-metrics-single-server 8428:8428 Forwarding from [::1]:8428 -> 8428 Handling connection for 8428

kubectl port-forward -n kyverno svc/kyverno-svc-metrics 8000:8000 Forwarding from 127.0.0.1:8000 -> 8000 Forwarding from [::1]:8000 -> 8000 Handling connection for 8000

The Five-Policy Lifecycle Framework

To move beyond simple enforcement and make policy behavior observable, we introduce a structured approach that follows the full lifecycle of a workload in the cluster. Instead of treating policies as isolated rules, we model them as part of a continuous flow of decisions and events that can be inspected, measured, and correlated.

1. Validating Policy

Now, let’s see how policies can help our “bad” application. We create a validating policy that enforces the presence of required labels. Because the policy runs at admission time, every deployment request is evaluated before it is persisted in etcd. This approach prevents configuration drift at the source, eliminates manual review processes, and ensures consistent standards across teams.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-observability-labels spec: validationFailureAction: Enforce background: false rules: - name: check-required-labels match: resources: kinds: - Pod validate: message: "Missing observability labels: app, environment, team" pattern: metadata: labels: app: "?*" environment: "?*" team: "?*"

2. Mutating Policy

To reduce developer friction, we introduce a mutation policy that automatically injects missing compliance labels at resource creation time, guiding workloads toward compliance while preserving the developer experience and avoiding manual intervention.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-safe-observability-labels spec: rules: - name: add-labels-to-pods-only match: any: - resources: kinds: - Pod mutate: patchStrategicMerge: metadata: labels: demo: "true" observability: "enabled" environment: "demo" team: "observability"

The validating policy correctly blocks the mutation attempt. This happens because the mutating policy only adds a non-compliant label (for example, demo) and fails to inject the required metadata, such as app, environment, and team. As a result, the resource still does not meet the policy requirements, and the validation layer enforces the expected constraints.

kubectl run test-pod --image=nginx Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: resource Pod/default/test-pod was blocked due to the following policies require-observability-labels: check-required-labels: 'validation error: Missing observability labels: app, environment, team. rule check-required-labels failed at path /metadata/labels/app/'

However, enforcement alone is not enough. When a non-compliant workload is rejected, platform teams need visibility into when, where, and how often these violations occur. Without an observability layer, policies become a black box, capable of blocking deployments but providing little insight into compliance trends, recurring issues, or their impact on developer workflows.

3. Generating Policy

Generating policies extends governance beyond enforcement by automatically creating required resources at namespace initialization. Instead of requiring teams to remember best practices, the platform provides them by default, establishing standardized “golden paths” for multi-tenant environments, including ConfigMaps, security defaults, and observability configurations.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-configmap spec: rules: - name: generate-config match: any: - resources: kinds: - Namespace generate: apiVersion: v1 kind: ConfigMap name: demo-config namespace: "{{request.object.metadata.name}}" data: data: message: "Hello from Kyverno"

In this example, Kyverno intercepts the namespace creation request, applies the generated policy, and automatically creates a demo-config ConfigMap inside the new namespace. This happens without any manual setup, demonstrating how platform-level automation enforces consistency while reducing operational overhead.

kubectl create namespace demo-ns namespace/demo-ns created kubectl get configmap -n demo-ns NAME DATA AGE demo-config 1 8s kube-root-ca.crt 1 8s

Taken together, this shows how generating policies turns compliance from a manual responsibility into an implicit platform capability, ensuring that every new environment starts from a well-defined and consistent baseline.

4. Validating Image Policy

Securing the software supply chain requires restricting pod creation to trusted container registries. We now enforce this by only allowing trusted container images at deployment time, ensuring that no unverified workloads are executed in the cluster.

This is where image validation policies become critical. They ensure that only approved and trusted images are allowed to run, acting as a first line of defense for supply chain security and compliance requirements.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-image-registries spec: validationFailureAction: Enforce rules: - name: validate-image-registry match: any: - resources: kinds: - Pod validate: message: "Only images from Docker Hub library are allowed" pattern: spec: containers: - image: "nginx*"

Testing the policy is extremely important. In the first instance, we apply an allowed image, nginx, and the pod, named good-pod, is created.

kubectl run good-pod --image=nginx pod/good-pod created

In the second instance, we apply an unapproved image, and the image validation policy blocks the creation of the bad-pod.

kubectl run bad-pod --image=busybox Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: resource Pod/default/bad-pod was blocked due to the following policies restrict-image-registries: validate-image-registry: 'validation error: Only images from Docker Hub library are allowed. rule validate-image-registry failed at path /spec/containers/0/image/'

However, enforcement alone is not enough. Observability completes the picture by showing how these policies behave in practice, how many images are denied, whether teams are improving over time, and what patterns emerge across environments. This turns image security from a static rule into a measurable, continuously improving system.

5. Deleting Policy

Deleting policies are designed to prevent unsafe operations by explicitly restricting high-risk actions, such as deleting critical workloads. For example, we can block DELETE verbs on sensitive resources to protect the cluster from accidental or unauthorized removal.

These policies are particularly important in environments where both human error and automated processes can trigger unintended changes. By mitigating accidental deletions and unsafe automation behavior, they act as a final safety layer in the workload lifecycle.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: protect-deployments spec: validationFailureAction: Enforce background: false rules: - name: block-deletion-of-deployments match: any: - resources: kinds: - Deployment validate: message: "Deletion of Deployments is not allowed in this demo" deny: conditions: any: - key: "{{ request.operation }}" operator: Equals value: DELETE

We can test the policy by creating a deployment, then trying to delete it.

kubectl create deployment test-deploy --image=nginx deployment.apps/test-deploy created

The DELETE policy correctly steps in and stops the deployment from being deleted.

kubectl delete deployment test-deploy Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: resource Deployment/default/test-deploy was blocked due to the following policies protect-deployments: block-deletion-of-deployments: Deletion of Deployments is not allowed in this demo

At the same time, these denial events are not just protective; they are valuable signals. They provide insight into operational risk patterns, highlight recurring mistakes, and help platform teams understand where additional guardrails or developer guidance may be needed.

Dashboard Visualization and Telemetry Analysis

Once metrics are flowing, enforcement events translate directly into real-time Grafana observability dashboards via standardized PromQL queries routed to VictoriaMetrics.

Below is a breakdown of key metrics used in the Grafana dashboard to monitor the behavior of these five Kyverno policies:

Key Monitoring Metrics

Total Webhook Throughput: Tracks the total volume of admission requests. kyverno_admission_requests_total

For example, you can use the kyverno_admission_requests_total{resource_kind="Namespace"} metric to create a panel for the Generate Policy activity, which will show you how often the policy is creating resources. This panel will provide insight into your platform automation.

Another example is to use the kyverno_admission_requests_total{request_webhook="MutatingWebhookConfiguration"} metric to create the Mutating policy activity. This metric is particularly useful for catching friction or adoption improvement, ultimately telling us how often we fix configurations automatically.

Total count of admission requests grouped by whether they were allowed or blocked sum by (request_allowed)(kyverno_admission_requests_total)

This metric will show us two numbers:

Allowed requests (request_allowed="true") – workloads that passed policy checks

Blocked requests (request_allowed="false") – workloads that were rejected

For example, with the above metric

sum by (request_allowed) ( kyverno_admission_requests_total {resource_kind="Pod"} )

We can create the Image Validation Policy to show us the immediate impact, so we can control destructive actions and prevent accidental or unauthorized deletion of critical workloads.

Resource Distribution: Aggregated via sum by (resource_kind), shows which resource kinds are most frequently affected by policy enforcement.

For example, we can create the Delete Policy visualization with the metric.

sum by (resource_kind, request_allowed) ( kyverno_admission_requests_total {resource_request_operation="delete"} )

With this analytical layer active, platform teams can evaluate operational metrics in context to determine whether policies are functioning correctly, whether engineering teams are successfully adapting, and whether enforcement guardrails are overly restrictive.

Managing Metric Cardinality

Tracking exhaustive policy contexts across massive clusters can quickly strain storage infrastructure due to high-cardinality metric explosion.

Using VictoriaMetrics’ integrated Cardinality Explorer inside its UI (vmui), platform teams can instantly isolate which high-volume namespaces or label combinations are consuming excessive resources. For example, the query kyverno_admission_requests_total was requested via the Grafana dashboard; therefore, it appears in vmui. The other queries haven’t been requested yet. If we don’t need specific metrics, then it is best to remove them to avoid latency or high cardinality (this is a recommendation from VictoriaMetrics).

Operational Summary

Combining declarative Kubernetes policies with a high-performance time-series database ensures platform teams scale infrastructure security without creating a visibility blind spot.

Policy Type

Functional Objective

Operational Benefit

Validate

Enforce configuration compliance

Blocks non-compliant resources before cluster entry.

Mutate

Inject missing metadata

Automatically corrects configurations to maintain workflow velocity.

Generate

Bootstrap resources on triggers

Automates repetitive infrastructure provisioning.

Image Validate

Restrict container registries

Enforces supply-chain security baselines.

Delete

Intercept resource removal verbs

Protects vital workloads from accidental termination.

To summarize: policies are not just enforcement mechanisms. They are signals about how your platform behaves in the real world. And when you observe those signals, you transform them into operational intelligence.

Just as effective parenting is not about creating more rules but understanding how those rules shape behavior, effective platform governance is not about writing more policies. It is about knowing whether they are achieving the intended outcome, where they are creating friction, and how they can be improved. Don’t just write policies: observe them, measure them, and continuously refine them. Because the teams that succeed are not the ones with the most policies; they are the ones who truly understand what their policies are doing.

You can further explore the complete configuration manifests and Grafana dashboard files in our open-source repository at https://github.com/didiViking/kyverno-victoriametrics-demo.