Kubeflow SDK evolution- One million downloads and counting

The unified kubeflow-sdk has officially crossed 1 million downloads on PyPI!

This milestone reflects the rapid adoption of this streamlined interface. In this post, we celebrate this community milestone and highlight the core concepts driving the SDK’s design, along with a look at how it simplifies distributed training.

History: From multiple tools to one import kubeflow

Scaling AI workloads shouldn’t require deep expertise in distributed systems and container orchestration. Whether anyone is prototyping on local hardware or deploying to a production Kubernetes cluster, we needed a unified API that abstracts infrastructure complexity while preserving flexibility.

Historically, the typical journey of an ML engineer going to production on Kubeflow was incredibly fragmented: prototype locally on a laptop, rewrite the entire code for distributed training, rebuild container images for every minor change, write complex Kubernetes YAML manifests, wrestle with `kubectl`, and juggle multiple disconnected APIs. At the same time, the community struggled to maintain multiple separate SDKs across individual subprojects like `kubeflow-training`, `kubeflow-katib`, and `model-registry`. Each step demanded different tools and mental models, draining focus and slowing down innovation.

To address these challenges, the Kubeflow community launched the Kubeflow SDK & ML Experience Working Group (WG). KEP-2170: Kubeflow Trainer V2 API rebuilt Trainer’s API with a Python-first client as an explicit design goal rather than an afterthought. That raised an obvious question for the rest of the ecosystem: why should every other component reinvent this? That question led to the Kubeflow SDK & ML Experience Working Group, chartered to build one unified Python interface across Kubeflow instead of another one-off client.

The WG’s work shipped as pip install kubeflow in November 2025, with TrainerClient and OptimizerClient as the first two unified clients. Less than a year later, it’s crossed 1 million downloads.

Under a single import kubeflow, it translates clean Python into the Kubernetes resources your cluster needs (like TrainJob, Experiment, and other CRDs) without you ever touching a manifest.

What Makes It Work: The Design Principles Behind the SDK

The rapid adoption of the SDK is driven by three core design pillars:

1. Pythonic Simplicity — Zero YAML Required

Data scientists configure resources, define hyperparameters, and launch distributed training runs using native Python. No Kubernetes manifest authoring, no YAML indentation debugging.

2. Multi-Backend Portability

The SDK supports three execution backends with the same API, letting you graduate from laptop to cluster without rewriting code:

Backend

When to use

Local Process

Fastest iteration — runs your training function as a Python subprocess, zero infrastructure overhead

Container (Docker/Podman)

Production-like environment on your laptop — same dependencies, same Python version, same system libraries

Kubernetes

Production scale — submits a full distributed TrainJob managed by Kubeflow Trainer, with fault tolerance and resource scheduling

Switching between them is a one-line config change — the training code itself never changes.

3. A Cohesive ML Lifecycle — Built for Two Personas

The SDK draws a clean line between two user types that Kubeflow serves:

AI Practitioners (data scientists, ML engineers) use the SDK to submit jobs and manage workflows entirely in Python, without touching YAML or kubectl

Platform Administrators manage infrastructure — installing components, configuring runtimes, setting resource quotas — nothing on their side changes

For practitioners, the SDK organizes the full AI lifecycle into dedicated client modules. Here’s the current state of ecosystem integration:

Subproject

Integration

Functional Overview

Trainer

✅ Available

Distributed model training and fine-tuning engine

Katib

✅ Available

Automated tuning of model hyperparameters

Model Registry

✅ Available

Centralized tracking of versions and artifacts

Spark

✅ Available

Large-scale data and feature processing

Pipelines

✅ Available

Native orchestration for ML workflow management

Distributed PyTorch training in 15 lines

The following snippet shows how TrainerClient abstracts away the complexity of distributed execution — no manual MASTER_ADDR, WORLD_SIZE, or RANK setup required:

from kubeflow.trainer import TrainerClient, CustomTrainer def train_model(): import torch import torch.nn as nn model = nn.Linear(10, 1) # Your training loop here print("Training job complete!") client = TrainerClient() job_name = client.train( trainer=CustomTrainer( func=train_model, num_nodes=2, resources_per_node={"cpu": "2", "memory": "4Gi"} ) ) print(f"Distributed training job '{job_name}' is running on Kubernetes.")

Behind the scenes, the SDK:

Serializes the training function and its dependencies

Dynamically generates the TrainJob CRD and submits it to the Training Operator

Orchestrates distributed nodes and injects cluster communication environment variables automatically

What used to require dozens of lines of YAML and Kubernetes expertise now fits comfortably in a single Python file.

More Examples

The SDK streamlines the machine learning lifecycle by organizing core functionalities into dedicated, Pythonic client modules:

SparkClient: Handles large-scale data preprocessing. See the Basic Spark Client Example to begin iterating.

TrainerClient: Powering distributed model training across PyTorch, DeepSpeed, and TensorFlow. Explore the PyTorch MNIST Training Notebook or the DeepSpeed Text Summarization Notebook.

OptimizerClient: Automates hyperparameter tuning via Katib experiments with zero YAML overhead.

PipelinesClient: Orchestrates end-to-end automated workflows by chaining preprocessing, training, and tuning into native KFP pipelines. Quick Example

For additional implementation patterns, visit the official Kubeflow SDK Examples Page.

User Survey: What Practitioners Want

To align our roadmap directly with developer needs, we recently ran a community survey (detailed in the Kubeflow SDK User Survey Insights). When asked what they want most, users highlighted three key priorities:

1. Simplified Infrastructure Configuration: Less time setting up compute resources for distributed runs.

2. Better Debugging: Cohesive logs and distributed tracing to troubleshoot remote failures.

3. Faster Iterative Workflows: Speeding up local prototyping before migrating jobs to the cloud.

What we’re building next

Crossing one million downloads motivates us to push further. These insights directly shape the top priorities for our 2026 roadmap:

Kubeflow SDK MCP Server — Exposing the Kubeflow SDK as AI-callable tools via the Model Context Protocol, enabling AI agents and coding assistants to directly orchestrate training jobs, hyperparameter sweeps, and Spark workloads on behalf of developers.

OpenTelemetry Integration — Adding end-to-end observability across the SDK with structured logging and distributed tracing, so platform teams can monitor and debug ML workloads with the same tooling they use for the rest of their infrastructure.

Dynamic LLM Trainer Framework — Purpose-built trainer abstractions for large language model fine-tuning, including GPU support in the container backend and transparent GPU checkpointing via CRIU for resilient long-running training jobs.

For the full list of planned work, see the Kubeflow SDK 2026 Roadmap.

Join the Community

The Kubeflow SDK needs your help and suggestions. We welcome contributions, feedback, and participation from everyone! Join us in any of below ways

Connect with the Community

Join #kubeflow-ml-experience on CNCF Slack

Attend the Kubeflow SDK and ML Experience WG meetings

Check out good first issues to get started

How to get Started

For the installation, follow the installation guide for Kubeflow SDK.

Train your first model with Kubeflow SDK using the quickstart guide.

Resources

Github: https://github.com/kubeflow/sdk

Documentation: https://www.kubeflow.org/docs/components/sdk/

Talks: Streamlining ML Workflows With the Unified Kubeflow SDK – Anna Kramar & Antonin Stefanutti, Red Hat

Demo: Kubeflow SDK Demo – LLM Fine-Tuning with BuiltinTrainer

Thank you to the contributors!