Upgrading AWX on MicroK8s Without Treating the Operator as a Rollback Plan

Historical field note: This article began with an AWX Operator 1.4.0-to-2.0.0 lab upgrade in 2023. The versions are retained as context, not as a current recommendation. Use the official upgrade guidance for the release you intend to run.

Kubernetes operators create an appealing abstraction: upgrade the operator, and let reconciliation move the managed application forward. That is automation, not risk transfer. The operator cannot verify that your backup is restorable, your custom resources remain compatible, or your cluster has enough capacity to complete the transition.

I use the upgrade as a control exercise, not a Helm exercise.

1. Record the starting state

microk8s kubectl get awx --namespace awx
microk8s kubectl get pods --namespace awx
microk8s kubectl get pvc --namespace awx
microk8s helm list --namespace awx

Capture the operator chart version, AWX version, custom-resource specification, image overrides, persistent volumes, and current health. If the baseline is unclear, post-upgrade validation will be guesswork.

2. Back up state and secrets

Use the AWX Operator backup role or another tested method appropriate to the deployment. Protect both the database and the secrets required to decrypt or reconstruct it.

A backup artifact is only evidence that a job ran. A restore test is evidence that the recovery path works.

3. Read every intermediate release note

Review the operator and AWX release notes between the installed and target versions. Pay particular attention to:

• Custom-resource changes
• PostgreSQL and persistent-volume migrations
• Supported Kubernetes versions
• Required manual steps
• Removed or renamed settings
• Backup and restore caveats

The AWX community Helm chart also warns that Helm does not upgrade or delete custom resource definitions automatically. Treat CRD handling as an explicit step in the plan.

4. Verify capacity and access

An upgrade may temporarily run old and new workloads together. Check CPU, memory, and storage headroom before starting. Keep a direct cluster-administration path available in case the AWX interface or its own jobs are unavailable.

5. Upgrade an exact chart version

Refresh the repository and inspect available versions:

microk8s helm repo update
microk8s helm search repo awx-operator/awx-operator --versions

Then use the reviewed version:

microk8s helm upgrade awx-operator awx-operator/awx-operator \
  --namespace awx \
  --version <reviewed-chart-version> \
  --reuse-values

Whether --reuse-values is appropriate depends on how the release is managed. For reproducible environments, a version-controlled values file is clearer:

microk8s helm upgrade awx-operator awx-operator/awx-operator \
  --namespace awx \
  --version <reviewed-chart-version> \
  --values awx-operator-values.yaml

6. Watch reconciliation, not just Helm

Helm reporting deployed means the chart transaction completed. It does not mean the managed AWX application is healthy.

microk8s kubectl get pods --namespace awx --watch
microk8s kubectl logs --follow \
  --namespace awx \
  deployment/awx-operator-controller-manager \
  --container awx-manager

Investigate crash loops, pending pods, migration errors, and storage failures before changing more variables.

7. Validate the service that matters

After reconciliation completes, verify:

• The expected AWX and operator versions
• Administrator and ordinary-user authentication
• Inventories, projects, credentials, schedules, and role assignments
• A representative low-risk job from launch through recorded output
• Webhooks, notifications, and external integrations
• Backup completion under the new version

For a shared platform, keep it in a maintenance state until those checks pass.

Lessons learned

The operator handled reconciliation. It could not tell me whether the starting state was healthy, whether the database and secrets were recoverable, or whether AWX still performed its intended work afterward. That is where automation stops and accountability begins.

I do not count an upgrade as complete until the post-change state has been compared with the baseline: versions, objects, access paths, integrations, a representative job, and a fresh backup. The change also needs a named owner and a tested route back. Until someone has exercised restoration, “rollback” is still a plan on paper.

Those responsibilities start before the maintenance window. A small AWX deployment on MicroK8s still needs control-plane ownership and a recovery path before it becomes responsible for other automation.

Sources and disclosures

• AWX Operator: upgrading
• AWX Operator backup role
• AWX Operator restore role
• AWX Operator Helm chart upgrade guidance

This article preserves a 2023 lab upgrade as historical evidence, not a current version recommendation. It is not affiliated with or endorsed by Ansible, Canonical, or their maintainers. Validate every intermediate release, backup format, custom-resource change, and rollback step for the exact versions in use.