An operator is a controller that encodes operational knowledge. If the runbook says "when a new tenant signs up, create a namespace, provision a database, seed the schema and register a DNS record," an operator is that runbook, written down in a form that also repairs itself.
The mechanics are not difficult. What is difficult — and what separates an operator you trust from one that quietly corrupts state at 3am — is writing a reconcile function that is correct when it is interrupted halfway through and re-run from the top, thirty times in a row.
We will build a TenantDatabase operator: given a declared tenant, it provisions a MongoDB database, generates credentials, and publishes a connection Secret the tenant's application can mount.
The mental model
Kubernetes controllers are not event handlers. They are level-triggered reconcilers, and confusing the two is the root of most operator bugs.
An event handler asks what changed? A reconciler asks what should be true, and what is true? It receives a name, reads the world, and makes the world match the spec. It does not know or care whether it was triggered by a create, an update, a resync timer, or a controller restart.
This has three consequences that shape everything else:
- Reconcile is called many times for one logical change. Every path must be idempotent. Creating something must first check whether it already exists.
- Reconcile can be interrupted anywhere. The process can die between "created the database" and "wrote the Secret." The next invocation must recover from that state without human help.
- You cannot rely on ordering. A delete may be observed before the create you were waiting on. Read current state; never infer it from history.
Scaffolding
mkdir tenant-operator && cd tenant-operator
kubebuilder init \
--domain kinekube.com \
--repo github.com/example/tenant-operator
kubebuilder create api \
--group data \
--version v1alpha1 \
--kind TenantDatabase \
--resource --controllerThat produces the API type in api/v1alpha1/tenantdatabase_types.go and the controller in internal/controller/. Everything else Kubebuilder generates is scaffolding you can read later.
Designing the CRD
The API is the part you cannot easily change. Once someone has resources of this kind in a cluster, every future version has to be convertible from this one. Spend the time here.
type TenantDatabaseSpec struct {
// TenantID is immutable — changing it would orphan the database.
// +kubebuilder:validation:Pattern=`^[a-z0-9][a-z0-9-]{2,38}$`
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="tenantId is immutable"
TenantID string `json:"tenantId"`
// ClusterRef points at the MongoDB deployment that hosts this tenant.
ClusterRef corev1.LocalObjectReference `json:"clusterRef"`
// +kubebuilder:validation:Enum=shared;dedicated
// +kubebuilder:default=shared
Isolation string `json:"isolation,omitempty"`
// +kubebuilder:validation:Minimum=1
// +kubebuilder:default=10
StorageQuotaGB int32 `json:"storageQuotaGB,omitempty"`
// +kubebuilder:validation:Enum=Delete;Retain
// +kubebuilder:default=Retain
ReclaimPolicy string `json:"reclaimPolicy,omitempty"`
}
type TenantDatabaseStatus struct {
// +optional
Conditions []metav1.Condition `json:"conditions,omitempty"`
// +optional
DatabaseName string `json:"databaseName,omitempty"`
// +optional
SecretName string `json:"secretName,omitempty"`
// ObservedGeneration lets callers tell "not started" from "finished".
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
Phase string `json:"phase,omitempty"`
}Four choices worth defending.
Validation lives in the CRD, not the controller. The CEL rule self == oldSelf makes tenantId immutable at admission time. The user gets a clear rejection on kubectl apply rather than a resource that reconciles into a confusing failure.
Defaults live in the CRD too. +kubebuilder:default means the API server fills the value, so every consumer — your controller, kubectl get -o yaml, another operator — sees the same object. Defaulting in the controller means the stored object and the effective object disagree.
ReclaimPolicy defaults to Retain. An operator that deletes customer data because someone ran kubectl delete on the wrong resource is a career event. Make destruction opt-in.
Conditions, not a status string. metav1.Condition is the ecosystem convention, and it gets you kubectl wait --for=condition=Ready for free.
Add printer columns so the resource reads well from the terminal — a small thing that makes an operator feel native:
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Tenant",type=string,JSONPath=`.spec.tenantId`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type TenantDatabase struct { /* ... */ }The +kubebuilder:subresource:status marker matters more than it looks. It splits status into a separate endpoint so writing status never bumps metadata.generation, which is what stops your controller from triggering itself in an infinite loop.
The reconcile loop
Here is the whole shape. Read the ordering carefully — it is the part that is easy to get subtly wrong.
const finalizerName = "data.kinekube.com/tenant-database"
func (r *TenantDatabaseReconciler) Reconcile(
ctx context.Context, req ctrl.Request,
) (ctrl.Result, error) {
log := logf.FromContext(ctx)
var td datav1alpha1.TenantDatabase
if err := r.Get(ctx, req.NamespacedName, &td); err != nil {
// NotFound means it is gone and cleanup already ran. Nothing to do.
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Deletion path: DeletionTimestamp is set, but our finalizer holds
// the object in the API until we say we are done.
if !td.DeletionTimestamp.IsZero() {
return r.reconcileDelete(ctx, &td)
}
// Claim the object so we are guaranteed a chance to clean up.
if !controllerutil.ContainsFinalizer(&td, finalizerName) {
controllerutil.AddFinalizer(&td, finalizerName)
if err := r.Update(ctx, &td); err != nil {
return ctrl.Result{}, err
}
// The Update triggers a fresh reconcile; return rather than
// continuing with a now-stale object.
return ctrl.Result{}, nil
}
if err := r.reconcileDatabase(ctx, &td); err != nil {
meta.SetStatusCondition(&td.Status.Conditions, metav1.Condition{
Type: "Ready",
Status: metav1.ConditionFalse,
Reason: "ProvisioningFailed",
Message: err.Error(),
ObservedGeneration: td.Generation,
})
td.Status.Phase = "Failed"
_ = r.Status().Update(ctx, &td)
// Returning the error requeues with exponential backoff.
return ctrl.Result{}, err
}
if err := r.reconcileCredentialSecret(ctx, &td); err != nil {
return ctrl.Result{}, err
}
meta.SetStatusCondition(&td.Status.Conditions, metav1.Condition{
Type: "Ready",
Status: metav1.ConditionTrue,
Reason: "Provisioned",
Message: "Database and credentials are available",
ObservedGeneration: td.Generation,
})
td.Status.Phase = "Ready"
td.Status.ObservedGeneration = td.Generation
if err := r.Status().Update(ctx, &td); err != nil {
return ctrl.Result{}, err
}
log.Info("reconciled", "tenant", td.Spec.TenantID)
// Drift detection: re-verify the external system periodically, since
// MongoDB will not send us an event when someone drops a database.
return ctrl.Result{RequeueAfter: 10 * time.Minute}, nil
}Why the finalizer is added in its own pass
The Update that adds the finalizer changes the object's resourceVersion. Continuing to use the in-memory copy afterwards risks a conflict on the next write, and conflict errors in a hot loop are how controllers end up hammering the API server. Returning immediately costs one extra reconcile and removes a whole class of race.
Why the periodic requeue
Watches tell you about changes to Kubernetes objects. Nothing tells you about changes to MongoDB. If an administrator drops the tenant database by hand, no event fires and your operator believes everything is fine forever. RequeueAfter closes the loop: the operator re-checks reality on a schedule and repairs drift. This is the difference between a controller and a provisioning script that ran once.
Idempotent external calls
Anything that touches a system outside Kubernetes must be written as check-then-act, and must tolerate the check being wrong.
func (r *TenantDatabaseReconciler) reconcileDatabase(
ctx context.Context, td *datav1alpha1.TenantDatabase,
) error {
dbName := fmt.Sprintf("tenant_%s", strings.ReplaceAll(td.Spec.TenantID, "-", "_"))
client, err := r.Mongo.For(ctx, td.Namespace, td.Spec.ClusterRef.Name)
if err != nil {
return fmt.Errorf("connecting to cluster %s: %w", td.Spec.ClusterRef.Name, err)
}
exists, err := client.DatabaseExists(ctx, dbName)
if err != nil {
return fmt.Errorf("checking database: %w", err)
}
if !exists {
// Still tolerate AlreadyExists: another replica of this operator,
// or a previous interrupted run, may have created it since the check.
if err := client.CreateDatabase(ctx, dbName, td.Spec.StorageQuotaGB); err != nil &&
!errors.Is(err, mongo.ErrDatabaseExists) {
return fmt.Errorf("creating database: %w", err)
}
}
td.Status.DatabaseName = dbName
return nil
}The check-then-act pattern narrows the race window. It does not eliminate it, which is why the AlreadyExists tolerance is not optional. Two reconciles can genuinely run concurrently — during a leader election handover, for instance — and the second one must not fail.
Ownership and garbage collection
Every object your operator creates should be owned by the resource that caused it. Set the owner reference and Kubernetes cleans up for you.
func (r *TenantDatabaseReconciler) reconcileCredentialSecret(
ctx context.Context, td *datav1alpha1.TenantDatabase,
) error {
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-credentials", td.Name),
Namespace: td.Namespace,
},
}
op, err := controllerutil.CreateOrUpdate(ctx, r.Client, secret, func() error {
// Never regenerate an existing password: rotating it on every
// reconcile would break every application already using it.
if secret.Data == nil {
secret.Data = map[string][]byte{}
}
if _, ok := secret.Data["password"]; !ok {
secret.Data["password"] = []byte(generatePassword(32))
}
secret.Data["username"] = []byte(td.Spec.TenantID)
secret.Data["database"] = []byte(td.Status.DatabaseName)
return controllerutil.SetControllerReference(td, secret, r.Scheme)
})
if err != nil {
return err
}
if op != controllerutil.OperationResultNone {
r.Recorder.Eventf(td, corev1.EventTypeNormal, "SecretReconciled",
"credential secret %s %s", secret.Name, op)
}
td.Status.SecretName = secret.Name
return nil
}CreateOrUpdate handles the create-or-patch decision and returns what it did, which is exactly what you want for event emission. The mutate function must be idempotent — note the guard around password generation. Without it, every reconcile rotates the credential and every tenant application starts failing authentication ten minutes later.
SetControllerReference is what makes kubectl delete tenantdatabase acme also delete the Secret. Owner references only work within a namespace and cannot point across them; cluster-scoped children of a namespaced parent need explicit cleanup in your finalizer.
Deletion
func (r *TenantDatabaseReconciler) reconcileDelete(
ctx context.Context, td *datav1alpha1.TenantDatabase,
) (ctrl.Result, error) {
if !controllerutil.ContainsFinalizer(td, finalizerName) {
return ctrl.Result{}, nil
}
if td.Spec.ReclaimPolicy == "Delete" && td.Status.DatabaseName != "" {
client, err := r.Mongo.For(ctx, td.Namespace, td.Spec.ClusterRef.Name)
if err != nil {
// Do not remove the finalizer on a transient failure — that
// would strand the database with no record it ever existed.
return ctrl.Result{}, err
}
if err := client.DropDatabase(ctx, td.Status.DatabaseName); err != nil {
return ctrl.Result{}, err
}
}
controllerutil.RemoveFinalizer(td, finalizerName)
return ctrl.Result{}, r.Update(ctx, td)
}Finalizers are a promise to the API server that you will clean up. They are also the single most common way to wedge a namespace: if your operator is uninstalled while resources still carry its finalizer, those resources cannot be deleted and the namespace hangs in Terminating forever.
Wiring up watches
func (r *TenantDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&datav1alpha1.TenantDatabase{}).
Owns(&corev1.Secret{}).
WithOptions(controller.Options{MaxConcurrentReconciles: 4}).
Complete(r)
}Owns(&corev1.Secret{}) watches Secrets and maps them back to their owner, so if someone deletes a credential Secret by hand, the operator regenerates it within seconds. That self-healing property is most of why you built an operator instead of a Helm chart.
MaxConcurrentReconciles: 4 allows parallel reconciliation across different objects. controller-runtime still guarantees a single object is never reconciled concurrently with itself, so you get throughput without needing your own locking.
Testing what actually breaks
Kubebuilder wires up envtest, which runs a real API server and etcd against your controller. Use it, and test the ugly paths rather than the happy one.
It("is idempotent across repeated reconciles", func() {
for i := 0; i < 5; i++ {
_, err := reconciler.Reconcile(ctx, req)
Expect(err).NotTo(HaveOccurred())
}
var secret corev1.Secret
Expect(k8sClient.Get(ctx, secretKey, &secret)).To(Succeed())
// The password must be identical after five passes.
Expect(secret.Data["password"]).To(Equal(firstPassword))
Expect(fakeMongo.CreateDatabaseCalls()).To(Equal(1))
})Three tests catch the majority of real operator defects:
- Reconcile five times, assert one side effect. Catches non-idempotent creates and credential rotation bugs.
- Kill after the external call, before the status write. Assert the next reconcile recovers. This is the interrupted-halfway case that happens on every deployment.
- Delete with a failing backend. Assert the finalizer stays put and the resource does not vanish, so you never lose track of an orphaned database.
What separates good operators from working ones
The controller-runtime parts are mechanical. The judgement is in four places:
A small, immutable-where-it-matters API. Every field is a support commitment. Fields are easy to add and nearly impossible to remove.
Honest conditions. Ready=True must mean a client can connect right now. If it means "we submitted a request," someone builds automation on it and gets burned.
Errors that name the thing that failed. connecting to cluster orders-prod: dial tcp 10.0.4.11:27017: connection refused ends an investigation. reconcile failed starts one.
Bounded retries. Return the error and let controller-runtime back off. Never time.Sleep inside reconcile — you are holding a worker from a small pool, and sleeping in one blocks every other object waiting behind it.
Build those in and you have something you can hand to a team who will never read this post, which was the point of writing an operator in the first place.