Historical field note: This article began with an AWX Operator 1.4.0-to-2.0.0 upgrade on MicroK8s in 2023. The versions are retained as context, not as a current recommendation. Use the official upgrade guidance for the release you intend to run.
Once AWX orchestrates enterprise patching, vulnerability remediation, or configuration validation, its availability and integrity become part of the security control path. The automation system responsible for patching needs a controlled, recoverable lifecycle itself.
Kubernetes operators create an appealing abstraction: upgrade the operator, and let reconciliation move the managed application forward. That is automation, not risk transfer. The operator cannot verify that your backup is restorable, your custom resources remain compatible, or your cluster has enough capacity to complete the transition.
I use the upgrade as a control exercise, not a Helm exercise.
1. Record the starting state
microk8s kubectl get awx --namespace awxmicrok8s kubectl get pods --namespace awxmicrok8s kubectl get pvc --namespace awxmicrok8s helm list --namespace awxCapture the operator chart version, AWX version, custom-resource specification, image overrides, persistent volumes, and current health. If the baseline is unclear, post-upgrade validation will be guesswork.
2. Back up state and secrets
Use the AWX Operator backup role or another tested method appropriate to the deployment. Protect both the database and the secrets required to decrypt or reconstruct it.
A backup artifact is only evidence that a job ran. A restore test is evidence that the recovery path works.
3. Read every intermediate release note
Review the operator and AWX release notes between the installed and target versions. Pay particular attention to:
- Custom-resource changes
- PostgreSQL and persistent-volume migrations
- Supported Kubernetes versions
- Required manual steps
- Removed or renamed settings
- Backup and restore caveats
The AWX community Helm chart also warns that Helm does not upgrade or delete custom resource definitions automatically. Treat CRD handling as an explicit step in the plan.
4. Verify capacity and access
An upgrade may temporarily run old and new workloads together. Check CPU, memory, and storage headroom before starting. Keep a direct cluster-administration path available in case the AWX interface or its own jobs are unavailable.
5. Upgrade an exact chart version
Refresh the repository and inspect available versions:
microk8s helm repo updatemicrok8s helm search repo awx-operator/awx-operator --versionsThen use the reviewed version:
microk8s helm upgrade awx-operator awx-operator/awx-operator \ --namespace awx \ --version <reviewed-chart-version> \ --reuse-valuesWhether --reuse-values is appropriate depends on how the release is managed. For reproducible environments, a version-controlled values file is clearer:
microk8s helm upgrade awx-operator awx-operator/awx-operator \ --namespace awx \ --version <reviewed-chart-version> \ --values awx-operator-values.yaml6. Watch reconciliation, not just Helm
Helm reporting deployed means the chart transaction completed. It does not mean the managed AWX application is healthy.
microk8s kubectl get pods --namespace awx --watchmicrok8s kubectl logs --follow \ --namespace awx \ deployment/awx-operator-controller-manager \ --container awx-managerInvestigate crash loops, pending pods, migration errors, and storage failures before changing more variables.
7. Validate the service that matters
After reconciliation completes, verify:
- The expected AWX and operator versions
- Administrator and ordinary-user authentication
- Inventories, projects, credentials, schedules, and role assignments
- A representative low-risk job from launch through recorded output
- Webhooks, notifications, and external integrations
- Backup completion under the new version
For a shared platform, keep it in a maintenance state until those checks pass.
Lessons learned
When AWX supports enterprise patching, vulnerability remediation, or other security automation, an unsafe AWX upgrade can weaken the system responsible for carrying out approved work. The operator handled reconciliation. It could not tell me whether the starting state was healthy, whether the database and secrets were recoverable, or whether AWX still performed its intended work afterward. Managing that platform risk is where automation stops and accountability begins.
I do not count an upgrade as complete until the post-change state has been compared with the baseline: versions, objects, access paths, integrations, a representative job, and a fresh backup. The point is not to preserve AWX for its own sake. It is to preserve a trusted path for the automation the organization depends on. The change needs a named owner, a tested route back, and evidence that the patching system can resume its intended role. Until someone has exercised restoration, “rollback” is still a plan on paper.
Those responsibilities start before the maintenance window. An AWX deployment on MicroK8s needs control-plane ownership and a recovery path before it becomes responsible for other automation.
