define for which use case to use roles, playbooks, potentially workflows (in Ansible Controller/AWX), and how to split the code you write.

especially when writing automation in a team, it is important to have a certain level of consistency and make sure everybody has the same understanding. By lack of doing so, your automation becomes unreadable and difficult to grasp for new members or even for existing members.

The following is only one example of how to structure your content but has proven robust enough on multiple occasions.

Figure 1. Structure of Automation

Names should clearly communicate purpose without being excessively long. Use the same pattern throughout a project: pick one separator convention (hyphens for repos and files, underscores for AAP resources), one file extension (.yml not mixed .yml and .yaml), and one casing convention per context.

Consistent naming reduces cognitive load, makes resources easier to search and discover, and helps new team members navigate the codebase. Abbreviations should only be used when they are standard and widely understood (e.g. aap, ee, config, dev, prod, qa).

Repository names should follow the pattern {purpose}-{type}, using lowercase with hyphens, and stay under 50 characters.

A predictable repo naming pattern makes it easy to identify the purpose and type of each repository at a glance, especially when managing many repos within an organization.

Playbook files should follow the pattern {verb}-{noun}.yml, clearly describing the action being performed. Some teams use a pb_ prefix (e.g. pb_deploy-webapp.yml) to distinguish playbooks from other YAML files in the repository, which can simplify automated scanning and testing. Group variable files should be named after the category they contain (e.g. credentials.yml, organizations.yml).

Verb-noun naming makes the purpose of each playbook immediately obvious when listing files, and avoids ambiguity about what a playbook does. A consistent prefix like pb_ can be useful when a repository contains many YAML files (inventories, variable files, playbooks) and tooling needs to target only playbooks.

Branch names should follow {type}/{description} using hyphens (e.g. feature/add-webserver-role, fix/inventory-syntax-error, hotfix/security-patch). Commit messages should follow the conventional commits format: {type}: {description} (e.g. feat:, fix:, docs:, chore:, test:, refactor:).

Structured branch and commit naming enables automated tooling (changelog generation, release automation), makes Git history more readable, and helps with code review.

All AAP resource names (organizations, teams, credentials, projects, inventories, job templates, workflow templates, execution environments, schedules) must use lowercase only with underscores as separators. Resource names should include an abbreviated organization prefix (e.g. eng_, plat_, ops_) to ensure uniqueness within the AAP instance.

AAP instances can contain hundreds of resources from multiple teams. Without a consistent naming standard, it becomes impossible to determine ownership, purpose, or scope of resources. Organization prefixes provide a namespace that prevents collisions and makes filtering and searching straightforward.

Collections should be comprised of roles collected either at the type or landscape level. See The Structures Definition

Gathering and publishing collections, rather than individual roles, allows for easier distribution and particularly becomes more important when we discuss Execution Environments.

Often, variables will want to be defined on a collection level, but this can cause issues with roles being able to be reused. By defining collection wide variables and referencing them in roles' defaults variables, this can be made clear and roles can remain reusable. Collection variables are nowhere defined explicitly and are to be documented in the collection’s documentation.

Variables that are shared across collections can cause collisions when roles are reused outside of the original collection. Role variables should continue to be named according to our recommendations for naming variables It still remains possible to overwrite collection variable values for a specific role. Each role has it’s own set of defaults for the variable.

For a collection "mycollection", two roles exist. "alpha" and "beta". For this example, there is no default for the controller_username and would have to be defined in one’s inventory. The no_log variable does have defaults defined, and thus only needs to be defined if the default is being overwritten.

# specific role variables
alpha_job_name: 'some text'
# collection wide variables
alpha_controller_username: "{{ mycollection_controller_username }}"
alpha_no_log: "{{ mycollection_no_log | default('true') }}"

# specific role variables
beta_job_name: 'some other text'
# collection wide variables
beta_controller_username: "{{ mycollection_controller_username }}"
beta_no_log: "{{ mycollection_no_log | default('false') }}"

Include a README file that is in the root of the collection and which contains:

Use https://github.com/ansible-network/collection_prep to generate the documentation for the collection

Include a license file in the root directory Name the license file either LICENSE or COPYING. The contents may be either the text of the applicable license, or a link to the canonical reference for the license on the Internet (such as https://opensource.org/licenses/BSD-2-Clause ) If any file in the collection is licensed differently from the larger collection it is a part of (such as module utilities), note the applicable license in the header of the file.

Don’t put too much logic in your playbook, put it in your roles (or even in custom modules), and try to limit your playbooks to a list of a roles.

Roles are meant to be re-used and the structure helps you to make your code re-usable. The more code you put in roles, the higher the chances you, or others, can reuse it. Also, if you follow the type-function pattern, you can very easily create new (type) playbooks by just re-shuffling the roles. This way you can create a playbook for each purpose without having to duplicate a lot of code. This, in turn, also helps with the maintainability as there is only a single place where necessary changes need to be implemented, and that is in the role

A playbook can contain pre_tasks, roles, tasks and post_tasks sections. Avoid using both roles and tasks sections, the latter possibly containing import_role or include_role tasks.

The order of execution between roles and tasks isn’t obvious, and hence mixing them should be avoided.

Either you need only static importing of roles and you can use the roles section, or you need dynamic inclusion and you should use only the tasks section. Of course, for very simple cases, you can just use tasks without roles.

limit your usage of tags to two aspects:

there is nothing worse than tags which can’t be used alone, they bear the risk to destroy something by being called standalone. An acceptable exception is the pattern to use the role name as tag name, which can be useful while developing the playbook to test, or exclude, individual roles.

Debug messages should have a verbosity defined as appropriate for the message.

Debug messages are useful during testing and development, and can be useful to retain as playbooks go into production for future troubleshooting. However, log messages will clutter your output, which can confuse users with non-relevant information.

A Single Source of Truth (SSOT) is the place where the "ultimate" truth about a certain data is generated, stored and maintained. There can be more than one SSOT, each for a different piece of information, but they shouldn’t overlap and even less conflict. As you create your inventory, you identify these SSOTs and combine them into one inventory using dynamic inventory sources (we’ll see how later on). Only the aspects which are not already provided by other sources are kept statically in your inventory. Doing this, your inventory becomes another source of truth, but only for the data it holds statically, because there is no other place to keep it.

You limit your effort to maintain your inventory to its absolute minimum and you avoid generating potentially conflicting information with the rest of your IT.

You can typically identify three kinds of candidates as SSOTs:

As you combine multiple sources, some will represent:

Mixing up these two kind of information can lead to your automation taking the wrong course of action by thinking that the current situation is aligned with the desired state. That can make your automation go awry and your automation engineers confused. There is a reason why Ansible makes the difference between "facts" (As-Is) and "variables" (To-Be), and so should you. In the end, automation is making sure that the As-Is situation complies to the To-Be description.

The technical tools typically contain a lot of discovered information, like an IP address or the RAM size of a VM. In a typical cloud environment, the IP address isn’t part of the desired state, it is assigned on the fly by the cloud management layer, so you can only get it dynamically from the cloud API and you won’t manage it. In a more traditional environment nevertheless, the IP address will be static, managed more or less manually, so it will become part of your desired state. In this case, you shouldn’t use the discovered information or you might not realize that there is a discrepancy between As-Is and To-Be.

Everybody has started with a single file inventory in ini-format (the courageous ones among us in YAML format), combining list of hosts, groups and variables. An inventory can nevertheless be also a directory containing:

It is the only way to combine simply multiple sources into one inventory, without the trouble to call ansible with multiple -i {inventory_file} parameters, and keep the door open for extending it with dynamic elements.

The following is a complete inventory as described before. You don’t absolutely need to start at this level of complexity, but the experience shows that once you get used to it, it is actually a lot easier to understand and maintain than a single file.

inventory_example/  (1)
├── dynamic_inventory_plugin.yml  (2)
├── dynamic_inventory_script.py  (3)
├── groups_and_hosts  (4)
├── group_vars/  (5)
│   ├── alephs/
│   │   └── capital_letter.yml
│   ├── all/
│   │   └── ansible.yml
│   ├── alphas/
│   │   ├── capital_letter.yml
│   │   └── small_caps_letter.yml
│   ├── betas/
│   │   └── capital_letter.yml
│   ├── greek_letters/
│   │   └── small_caps_letter.yml
│   └── hebrew_letters/
│       └── small_caps_letter.yml
└── host_vars/  (6)
    ├── host1.example.com/
    │   └── ansible.yml
    ├── host2.example.com/
    │   └── ansible.yml
    └── host3.example.com/
        ├── ansible.yml
        └── capital_letter.yml

[all]
host1.example.com
host2.example.com
host3.example.com

[alphas]
host1.example.com

[betas]
host2.example.com

[greek_letters:children]
alphas
betas

[alephs]
host3.example.com

[hebrew_letters:children]
alephs

inventory_satellite/
├── groups_and_hosts
└── host_vars/
    └── sat6.example.com/
        ├── ansible.yml
        └── satellite/
            ├── content_views.yml
            ├── hostgroups.yml
            └── locations.yml

To perform the same task on multiple hosts, don’t create a variable with a list of hosts and loop over it. Instead use as much as possible the capabilities of your inventory, which is already a kind of list of hosts.

There are 4 main reasons for following this advice:

Our first idea could be to define managers and hosts first in an inventory:

[managers]
manager_a
manager_b

[managed_hosts]
host1
host2
host3

---
provision_list_of_hosts:
  - name: host1
    provision_value: uno
  - name: host2
    provision_value: due

- name: Provision hosts in a bad way
  hosts: managers
  gather_facts: false
  become: false
  tasks:
    - name: Create some file to simulate an API call to provision a host
      ansible.builtin.copy:
        content: "{{ item.provision_value }}\n"
        dest: "/tmp/bad_{{ inventory_hostname }}_{{ item.name }}.txt"
        force: true
        owner: root
        group: root
        mode: "0644"
      loop: "{{ provision_list_of_hosts }}"

[managers]
manager_a
manager_b

[managed_hosts_a]
host1
host2

[managed_hosts_b]
host3

[managed_hosts:children]
managed_hosts_a
managed_hosts_b

$ cat inventory_good/host_vars/host1/provision.yml
provision_value: uno
$ cat inventory_good/group_vars/managed_hosts_a/provision.yml
manager_hostname: manager_a

- name: Provision hosts in a good way
  hosts: managed_hosts
  gather_facts: false
  become: false
  tasks:
    - name: Create some file to simulate an API call to provision a host
      ansible.builtin.copy:
        content: "{{ provision_value }}\n"
        dest: "/tmp/good_{{ manager_hostname }}_{{ inventory_hostname }}.txt"
        force: true
        owner: root
        group: root
        mode: "0644"

$ ANSIBLE_STDOUT_CALLBACK=profile_tasks \
	ansible-playbook -i inventory_bad playbook_bad.yml
Saturday 23 October 2021  13:11:45 +0200 (0:00:00.040)       0:00:00.040 ******
Saturday 23 October 2021  13:11:45 +0200 (0:00:00.858)       0:00:00.899 ******
===============================================================================
create some file to simulate an API call to provision a host ------------ 0.86s
$ ANSIBLE_STDOUT_CALLBACK=profile_tasks \
	ansible-playbook -i inventory_good playbook_good.yml
Saturday 23 October 2021  13:11:55 +0200 (0:00:00.040)       0:00:00.040 ******
Saturday 23 October 2021  13:11:56 +0200 (0:00:00.569)       0:00:00.610 ******
===============================================================================
create some file to simulate an API call to provision a host ------------ 0.57s

There are 22 levels of variable precedence. This is almost impossible to keep in mind for a "normal" human and can lead to all kind of weird behaviors if not under control. In addition, the use of play(book) variables is not recommended as it blurs the separation between code and data. The same applies to all constructs including specific variable files as part of the play (i.e. include_vars). By reducing the number of variable types, you end up with a more simple and overseeable list of variables. Together with some explanations why they have their specific precedence, so that they become easier to remember and use wisely:

Don’t use extra vars to define your desired state. Make sure your inventory completely describes how your environment is supposed to look like. Use extra vars only for troubleshooting, debugging or validation purposes.

Inventory variables are typically in some kind of persistent tracked storage (be it a database or Git), and should be your sole source representing your desired state so that you can refer to it non-ambiguously. On the other hand, extra vars are bound to a specific job or ansible-call and disappear together with history.

Don’t use extra vars for the RAM size of VM to create, because this is part of the desired state of your environment, and nobody would know one year down the line if the VM was really created with the proper RAM size according to the state of the inventory. You may use an extra variable to protect a critical part of a destructive playbook, something like are_you_really_really_sure: true/false, which is validated before e.g. a VM is destroyed and recreated to change parameters which can’t be changed on the fly. You can also use extra vars to enforce fact values which can’t be reproduced easily, like overwriting ansible_memtotal_mb to simulate a RAM size fact of terabytes to validate that your code can cope with it.

All plugins, regardless of type, need documentation that describes the input parameters, outputs, and practical examples of how to use it.

See the Ansible Developer Guide sections on Plugin Configuration and Documentation Standards and Module Documenting for more details.

Sphinx (reST) formatted docstring are preferred for Ansible development. This includes all parameters, yields, raises, or returns for all classes, private and public functions written in Python.

PEP-257 states that: "All modules should normally have docstrings, and all functions and classes exported by a module should also have docstrings. Public methods (including the init constructor) should also have docstrings. A package may be documented in the module docstring of the init.py file in the package directory."

Use Python type hints to document variable types. Type hints are supported in Python 3.5 and greater.

Type hints communicate what type a variable can be expected to be in the code. They can be consumed by static analysis tools to ensure that variable usage is consistent within the code base.

Use pytest for writing unit tests for plugins

Pytest is the testing framework used by Ansible Engineering and will provide the best experience for plugin developers. The Ansible Developer Guide section on unit testing has detailed information on when and how to use unit tests.

Ensure a consistent approach to the way complex argument_specs are formatted within a collection.

When hand-writing a complex argspec, the author may choose to build up to data structure from multiple dictionaries or vars. Other authors may choose to implement a complex, nested argspec as a single dictionary. Within a single collection, select one style and use it consistently.

Use of a sngle dictionary

Keep the entry file to a plugin to a minimal and easily maintainable size.

Long and complex code files can be difficult to maintain. Move reusable functions and classes, such as those for data validation or manipulation, to a module_utils/ (for Ansible modules) or plugin_utils/ (for all other plugin types) file and import them into plugins. This keeps the Python code easier to read and maintain.

The ansible.plugin_builder is a tool which helps developers scaffold new plugins.

This will make it easier to troubleshoot failures if they occur

Error messages that communicate specific details of the failure will aid in resolving the problem. Unclear error messages such as "Failed!" are unnecessarily obscure.

Use of whitespace and multi-line indentation makes expressions easier to read.

All AAP objects — organizations, teams, credentials, projects, inventories, job templates, workflow templates, execution environments, schedules, and settings — should be defined as YAML data in a Git repository and applied using the infra.aap_configuration collection. No manual changes should be made through the AAP UI for anything that can be expressed as code.

Configuration as Code provides version control (all changes tracked), reproducibility (rebuild environments from scratch), consistency (same config across Dev/QA/Prod), auditability (clear change history), disaster recovery (restore quickly from Git), and simplified migration between platforms.

The infra.aap_configuration.dispatch role should be used rather than calling individual roles directly. It automatically handles the correct order of operations (organizations before credentials, credentials before projects, projects before job templates, etc.).

Manually ordering role calls is error-prone and creates maintenance burden. The dispatch role ensures dependencies are resolved automatically and configuration is applied idempotently.

AAP CaC repositories should use Ansible inventory with group-based variable files to manage environment-specific configuration. Shared configuration belongs in group_vars/all/ while environment-specific settings go in group_vars/aap_dev/, group_vars/aap_qa/, and group_vars/aap_prod/.

This structure leverages Ansible’s native variable precedence system, where environment-specific group_vars override shared all settings. It keeps the repository organized, makes it clear which configuration is shared versus environment-specific, and makes it easy to use CD pipelines to roll out configuration changes across AAP instances.

AAP resources fall into two categories based on their coupling:

Shared resources would otherwise be duplicated across many files, creating maintenance burden and inconsistency. JT bundles keep tightly coupled resources together so that when one changes (e.g. a new EE version), the related project and job template are updated in the same commit.

CaC credential definitions must never contain plain-text secrets. Use lookup('env', …​) to reference environment variables at runtime, or encrypt files with ansible-vault.

CaC repositories are shared across teams and environments. A secret committed to a CaC repo is visible to everyone with read access to the repository and persists in Git history permanently.

The collections/requirements.yml file must specify an exact version of infra.aap_configuration, never a range or >= constraint.

Collection updates may change behavior, add required parameters, or deprecate features. Pinning to an exact version ensures reproducible builds and prevents unexpected breakage when a new collection version is released. Version upgrades should be deliberate and tested.

After applying CaC configuration, run the playbook again. The second run should report changed=0, confirming that all configuration is already in the desired state. Before applying to production, always run with --check --diff to preview changes without making modifications.

If a CaC playbook is not idempotent, it may create duplicate resources, overwrite unrelated settings, or fail on subsequent runs. Production AAP environments serve live automation workflows, so unexpected changes can break running jobs, remove schedules, or modify credentials.

CaC playbooks should be triggered by CI/CD pipelines (e.g. GitHub Actions, Tekton, Jenkins) on Git events (push, tag creation), not run manually from developer workstations.

Manual execution introduces human error, bypasses quality gates, and creates inconsistent application of configuration. CI/CD pipelines ensure that every configuration change goes through the same validation, testing, and approval process.

All development should target a single main branch. Changes are made through short-lived feature branches that are merged via pull requests after review and CI validation. There should be no long-lived environment branches (e.g. dev, qa, prod).

Long-lived environment branches inevitably diverge, creating merge conflicts and making it unclear what state each environment is in. Trunk-based development with a single main branch provides a single source of truth, reduces merge conflicts through frequent integration, and keeps the codebase in a consistently releasable state.

Feature branches should live for hours to days, not weeks. Break large changes into smaller, incremental pull requests that can be merged frequently. Use feature flags if a feature is incomplete but the code should still be merged.

Long-lived branches accumulate merge conflicts, make code review harder (large diffs), and delay integration. Short-lived branches reduce risk and provide faster feedback on changes.

Two common versioning strategies for automation content are:

SemVer communicates API compatibility (major bump = breaking changes) but requires subjective judgment on each release. CalVer provides instant date visibility, eliminates version bump debates, and pairs naturally with trunk-based development.

Release tags must be immutable: once created, they should never be deleted, moved, or force-pushed. Use annotated tags with descriptive messages that include a summary of changes, testing status, and rollback instructions.

Immutable tags provide a reliable audit trail and enable deterministic promotion across environments. If tags can be moved, it becomes impossible to know what code was actually deployed to a given environment at a given time.

A single release tag (e.g. 26.1.5-0) should promote unchanged through Dev, QA, and Production. The release manifest or deployment tooling tracks which tag is deployed to which environment, rather than creating environment-specific tags.

If different tags are used per environment, there is no guarantee that the code deployed to production is the same code that was tested in QA. Promoting the same immutable tag ensures true atomic promotion — the exact same artifact is tested and deployed everywhere.

Production AAP projects, execution environments, and container images must always reference a specific version tag or image digest, never latest, main, or any mutable reference.

Mutable references make deployments non-reproducible. If production uses latest and someone pushes a new image, production changes without any approval or testing process. This violates every principle of controlled change management.

When releasing, all components (Git tags, collection versions, execution environment image tags, AAP configuration references) should use the same version identifier.

Version mismatches between components are a common source of production incidents. If the playbook expects collection version 26.1.6-0 but the execution environment was built with 26.1.5-0 dependencies, behavior may differ. Keeping all versions synchronized simplifies tracking and debugging.

The main branch should be protected with branch protection rules:

Unprotected main branches allow untested, unreviewed code to enter the codebase. Branch protection ensures every change goes through quality gates before reaching the shared codebase.

When a release contains breaking changes (renamed variables, removed features, changed behavior), document them prominently in the Git tag message and in a CHANGELOG file.

Without CalVer or SemVer major bumps to signal breaking changes, clear documentation becomes the primary mechanism for communicating incompatibilities. Operators need to know about breaking changes before promoting a release to production.

Testing should happen at the earliest feasible point: local development first, then pre-commit hooks, then CI, then integration, and finally validation in production-like environments. Each stage should be faster than the next, providing rapid feedback to developers.

Bugs caught early are cheaper and faster to fix. A syntax error caught in a pre-commit hook takes seconds to fix; the same error caught in a production deployment takes hours to diagnose and remediate.

Run ansible-lint with the production profile to enforce the strictest set of rules. This should be part of both pre-commit hooks and CI pipelines.

The production profile catches issues that less strict profiles allow, such as missing FQCNs, unnamed tasks, deprecated modules, and unsafe practices. Using the strictest profile ensures consistent quality across the codebase.

Every role should have at least one Molecule test scenario that validates the role in an isolated container environment. The full Molecule test sequence includes create, converge, idempotence, verify, and destroy.

Molecule provides a reproducible and isolated testing environment for roles. It tests the complete lifecycle: that the role can be applied, that it is idempotent (running twice produces no changes), and that the desired state is achieved (verification). Without Molecule, role testing requires manual provisioning of test environments.

If a role is designed to support multiple operating system families (e.g. Red Hat and Debian), create Molecule scenarios for each platform and test them.

Package names, service names, file paths, and configuration syntax vary between operating system families. A role that works on RHEL 9 may fail silently on Ubuntu if platform-specific variables or tasks are incorrect. Multi-platform testing catches these issues before they reach production.

Custom Ansible modules, filter plugins, and lookup plugins should have pytest-based unit tests with good coverage. Unit tests should follow the Arrange-Act-Assert pattern and be independent of each other.

Custom Python code in Ansible collections has the same quality requirements as any other software. Unit tests catch regressions, document expected behavior, and enable confident refactoring.

Every role and playbook must be tested for idempotency: running it twice in a row should produce changed=0 on the second run. Molecule’s default test sequence includes an idempotence step. For complex playbooks outside of Molecule, verify idempotency manually.

Non-idempotent automation causes unpredictable behavior, unnecessary restarts, and makes it impossible to use check mode reliably. Testing for idempotency is distinct from writing idempotent code — both are necessary.

Test function and class names should describe what behavior is being tested and what the expected outcome is.

When tests fail, the test name is the first thing developers see. A descriptive name like test_filter_handles_empty_input immediately communicates what broke, while test1 requires reading the test code to understand the failure.

Install and configure pre-commit hooks in every automation repository. Hooks should run syntax validation, linting, secret detection, and formatting checks automatically before each commit.

Pre-commit hooks provide the fastest possible feedback loop — issues are caught before code even reaches the repository, saving CI time and preventing bad code from entering the shared codebase. Without pre-commit hooks, developers rely solely on CI pipelines which are slower and create noisy failure notifications.

CI pipelines should run the same pre-commit hooks as developers run locally, ensuring that even if a developer bypasses local hooks (e.g. git commit --no-verify), the checks still run.

Pre-commit hooks can be bypassed locally, intentionally or accidentally. Additionally, some development environments may not support installing pre-commit at all (e.g. restricted workstations, certain cloud-based editors). CI pipelines serve as the authoritative quality gate that cannot be bypassed, ensuring that no code enters the main branch without passing all checks regardless of the developer’s local setup.

PR validation pipelines should complete in under 5 minutes. Use caching, parallel job execution, conditional execution (only run relevant tests based on changed files), and matrix strategies to keep pipelines fast.

Slow CI pipelines discourage frequent commits and pull requests, lead to larger and riskier changes, and reduce developer productivity. Fast feedback encourages the short-lived branches and frequent integration that trunk-based development requires.

Use CI tools (e.g. GitHub Actions) for testing, linting, and validation — things that run on every push and PR. Use CD tools (e.g. CI/CD pipelines, AAP workflows) for building artifacts, publishing to registries, and deploying to environments.

Testing and building serve different purposes and have different security requirements. Testing needs access to the code but not to production credentials. Building and releasing need access to registries and deployment targets but should only run on approved code (merged to main, tagged).

Each environment promotion (Dev → QA → Prod) should have explicit quality gates that must pass before promotion proceeds:

Quality gates prevent untested or unapproved changes from reaching production. Without explicit gates, the promotion process becomes informal and error-prone, leading to incidents that could have been prevented.

Pull request validation should include automated checks beyond code linting:

Automated PR checks reduce the burden on reviewers by catching mechanical issues before human review begins. They also enforce team conventions consistently without relying on reviewer memory.

Create release manifests that record the exact version of every component (Git commit SHAs, collection versions, execution environment image digests) and which environment each version is deployed to.

Release manifests provide a single source of truth for "what is running in production right now." They enable deterministic rollback (deploy exactly the previous manifest), auditability (who approved what, when), and troubleshooting (quickly identify version mismatches).

Before promoting to production, document the rollback procedure: which previous version to revert to, how to trigger the rollback, and what validation to perform after rollback.

Production incidents happen. Having a pre-documented rollback plan reduces the mean time to recovery (MTTR) from hours to minutes. Without a rollback plan, teams must figure out the recovery process under pressure during an incident.

No passwords, API tokens, private keys, or other sensitive values should ever appear in plain text in any file tracked by Git. This includes playbooks, variable files, templates, scripts, and configuration files.

Secrets committed to Git persist in the repository history permanently, even if subsequently deleted. A single committed secret can compromise entire environments and is nearly impossible to fully remediate (the repository history must be rewritten, all downstream forks and clones are still exposed).

Use automated secret detection tools in both pre-commit hooks and CI pipelines:

Human review alone is insufficient to catch accidentally committed secrets. Automated scanning provides a safety net that catches secrets before they enter the repository, and scanning Git history catches secrets that may have been committed previously.

Most Git hosting platforms offer server-side push protection that blocks pushes containing detected secrets before they ever reach the repository. This is a complementary layer to client-side pre-commit hooks and CI scanning:

Pre-commit hooks run client-side and can be bypassed (git commit --no-verify). CI scanning catches secrets after they have been pushed, meaning the secret is already in the repository history. Push protection is the only mechanism that blocks secrets at the server before they enter the repository at all, closing the gap between client-side hooks and post-push CI scanning.

Every service account, CI pipeline, and AAP credential should have the minimum permissions required to perform its function.

Over-privileged accounts increase the blast radius of security incidents. If a CI pipeline with production credentials is compromised, the attacker gains production access. If that pipeline only has test credentials, the blast radius is limited to the test environment.

AAP provides a credential management system where secrets are stored securely in the platform and injected at runtime. Playbooks should reference credential names, never contain credential values.

AAP credential objects are stored encrypted at rest, access-controlled via RBAC, and audited. Embedding credentials in playbooks or variable files bypasses all of these security controls.

Credentials used by technical and service accounts (passwords, tokens, API keys, SSH keys) should be rotated on a regular schedule. The rotation process should be automated where possible and documented where it cannot be.

Long-lived credentials have a higher probability of being compromised over time. Regular rotation limits the window of exposure if a credential is leaked. Automated rotation ensures the process is consistent and does not depend on human memory.

Every execution environment image should be scanned for known vulnerabilities before it is deployed to any environment. Use tools like Trivy or Grype in the CI/CD pipeline. Additionally, generate a Software Bill of Materials (SBOM) for every release using tools like Syft.

Container images bundle operating system packages, Python libraries, and Ansible collections. Any of these components may have known security vulnerabilities. Scanning catches these issues before they are deployed, reducing the attack surface of automation infrastructure. SBOMs provide a complete inventory of components, enabling vulnerability tracking and compliance reporting.

When all configuration is managed as code (GitOps), Git automatically provides a complete audit trail: who changed what, when, and why (via commit messages and PR descriptions).

Compliance and security frameworks (SOC 2, PCI DSS, HIPAA) require auditable change management. Git provides this automatically when all changes flow through pull requests with reviews and approvals.

Development, QA, and production environments should be network-isolated from each other. Only explicitly permitted traffic (e.g. CI/CD pipelines deploying to a target environment) should cross environment boundaries.

Network segmentation limits the blast radius of security incidents. If the development environment is compromised, network isolation prevents the attacker from pivoting to production. It also prevents accidental cross-environment access (e.g. a dev playbook accidentally targeting production hosts).

Perform periodic reviews of:

Access requirements change over time as team members join, leave, or change roles. Without periodic review, permissions accumulate ("permission creep") and the environment becomes over-privileged.