From AWX to Ansible Automation Platform 2.7: what I learned building a complete PoC

A few years ago I deployed AWX on GKE (Kubernetes on Google Cloud) to automate a large, mixed fleet: Linux servers across several distributions and versions, and Windows servers. It worked, and running it taught me a lot, but it was "my" AWX: virtualenvs built by hand for every combination of versions, credentials that depended on the team's discipline, community upgrades that had to be tested carefully, and a change history rebuilt by hand whenever someone asked what had changed. I wanted to understand, in depth, what Red Hat Ansible Automation Platform (AAP) offers against that today, and the only honest way to find out was to build a complete proof of concept myself, in my own cloud: install it, configure it, break it, and write everything down.

This article covers what I built, what changed since AWX and, above all, what went wrong, because that is where the real learning was.

What I built

• ·AAP 2.7, containerized on RHEL in Google Cloud, in the growth topology (everything on one VM), with three managed servers.
• ·All the infrastructure in Terraform, with security tests that fail if anyone opens a port to the internet.
• ·The whole AAP configuration as code: organization, teams, users, credentials, inventories, templates, workflows, RBAC and Event-Driven Ansible. One command rebuilds the platform.
• ·Seven use cases, each with an automated test that verifies it end to end.

What changed since AWX

1. A single entry point. AAP 2.5 introduced the platform gateway: one UI, one authentication and one RBAC for the controller, the hub and Event-Driven Ansible. My first mistake was assuming the gateway listened on port 8446, because that is how it appears in the installer's documentation. The UI loaded, but everything else returned 404. The real entry point is Envoy on 443; 8446 is an internal backend. I found it with ss -ltnp and a query to the gateway's own API.

2. Execution environments instead of virtualenvs. With AWX on GKE, keeping dependencies for several Linux versions and for Windows (WinRM) was one of the most fragile tasks. Now every job runs in a versioned container image. I built mine with ansible-builder, published it to the private hub, and I can now say exactly which code ran in each job, something I could never state with certainty with virtualenvs.

3. Where it runs. My AWX lived on GKE because the AWX Operator runs on any Kubernetes. Supported AAP only runs on Kubernetes if that Kubernetes is OpenShift; on GKE, EKS or AKS it is not supported. The alternative, and the one I used in the PoC, is the containerized installation on RHEL: the RPM installer is deprecated. Everything now runs on rootless Podman, managed by systemd, and comes back on its own after a reboot.

4. Event-Driven Ansible. This did not exist in AWX. I stopped nginx by hand on a server and, about 90 seconds later, AAP had fixed it on its own, using a service account that only has permission for that remediation.

The seven cases

• ·GCP dynamic inventory without JSON keys: Servers group themselves by label; one that is shut down disappears
• ·CIS hardening with evidence: Measured before and after with OpenSCAP; reports kept in independent storage
• ·Patching with approval: Whoever runs it cannot approve it; if something fails, automatic rollback and production untouched
• ·RBAC by team: Operations runs, Security approves, Audit only reads
• ·Custom execution environment: In the private hub, with no secrets inside the image
• ·Event-driven remediation: A service goes down and is fixed with no human involved
• ·Traceability: The auditor exports who ran and who approved each change

What went wrong (and why I'm glad it did)

These are the failures that taught me the most:

A token exposed in a log. While building the execution environment I passed the Automation Hub token as a command-line argument. At a high verbosity level the tool printed the full command, token included, and the log ended up where it should not have. I contained it (deleted the copies), revoked it and generated a new one. The rule I kept is simple: a secret never goes on the command line. It goes in environment variables or in single-use files, and logs are filtered where they are produced and where they are read.

The auditor who saw nothing. I assigned the "Organization Auditor" role to the audit team. The API answered 200, the test passed... and the auditor saw nothing. AAP 2.7 has two roles with almost the same name: Organization Audit (the one that works) and Organization Auditor. The lesson was not the role but the test: a test that only checks the HTTP status can give a false green. I now check the content.

The rollback that could not go back. I forced a patching failure to test the rollback. The rollback ran... and failed: one of the 18 packages came from a repository that keeps only the latest version. Package rollback depends on the repositories. In production, a disk snapshot before patching is not optional.

The rollback that could do harm. Before anything failed, I designed the rollback to never undo a transaction the patch run did not create. A naive dnf history undo last can revert a legitimate installation from days ago. I tested it explicitly: with no new changes, the rollback touches nothing.

When the documentation does not match the code. One of the official collections ignored the environment variables its own documentation described, and it also read variables without rendering Jinja templates. The answer was in the plugin's code. The temptation was to pass the password with -e on the command line; I did not, because of the token lesson.

My network has four public IPs. I restricted the console to "my IP /32" and from one day to the next it stopped answering. My network leaves through several load-balanced public IPs. I ended up allowing two /28 ranges, validated by Terraform.

Decisions I would take to production as they are

• ·No long-lived keys: cloud identities from the metadata server, and service accounts for the Red Hat registry, the subscription and the integrations.
• ·Idempotent configuration as code: the second run must say changed=0. A false "changed" muddies the audit trail.
• ·Least privilege at object level, and one service account per integration.
• ·Allowlists of services for event-driven remediation, in the rule and again in the playbook.
• ·Evidence outside the platform, in independent storage with retention.

What I would do differently in production

The PoC is one VM. Production is something else: an enterprise topology with two nodes per component, an external database with high availability, a Redis cluster, shared storage for the hub and an internal load balancer. Add to that the corporate directory, the ITSM, the SIEM and a secrets vault. What that topology adds is described at the end of the blueprint's README, and it is my next exercise.

A blueprint to reuse

The whole PoC (Terraform, installation, configuration as code, playbooks, rulebooks, automated tests, the step-by-step guide and what the enterprise topology adds) is published as a reusable, parameterized blueprint, so any team can stand up the same laboratory in its own cloud project: platform-blueprints/automation/aap.

Closing

The most valuable part of this PoC is not that it works, but what broke along the way. The fifteen failures that cost me the most time are in the blueprint's README, each one told as what happens and what to do.

If you are evaluating AAP, my advice is the one I gave myself: build it yourself, break it, and write everything down.