Ansible in the coriverse

Ansible is an essential tool for managing our data infrastructure. Initially it was only employed to create new linux user accounts during MDA onboarding (see legacy playbooks), but ever since the great RDS Migration of 2023, ansible is the main way that we provision, save and restore data in bulk to the various databases (see PostgreSQL-RDS-Management section in the wiki).

Ansible playbooks are used to manage users on the team’s shared EC2 instance and databases instance within our RDS cluster.

inventory/: stores our ansible hosts (CORI/RISI R server) see 1) for shell env. variable.

playbooks/: contains various playbooks:

playbooks/legacy/: playbooks for managing EC2 instance and adding users. Adding new users worked in January 2023. The old readme was also added here.

playbooks/basic/: provides a playbook to test Ansible and seeing if the connection to some target host works.

playbooks/cori-risi-ad-postgresql/: playbooks to connect to the Active-Directory-secured RDS cluster and manage database instances and users.

playbooks/cori-risi-old-db/: playbooks that were used to connect to the now defunct “old” database (in a terminated RDS cluster).

playbooks/kerberos/: playbook to verify the ability to authenticate with Active Directory

playbooks/postgresql/: before running playbooks against the database instances on the new RDS cluster we tested them on the local PostgreSQL instance running on our shared EC2 instance.

playbooks/queries/: SQL helper/utility scripts.

package.json: npm is also used here as a way to store some procedures and save some time and brain space (see as an example: npm run clean).

Set up Ansible

So far, this works (October 2023):

Assign appropriate values for the ssh user and private key file to the shell environment variables, ANSIBLE_SSH_USER and ANSIBLE_SSH_PRIVATE_KEY_FILE:

# export your name from EC2 instance
export ANSIBLE_SSH_USER=<USER_NAME>
# check ls ~/.ssh to know which key use
export ANSIBLE_SSH_PRIVATE_KEY_FILE=<~/.ssh/SSH_PRIVATE_KEY_FILE>

Run npm install or manually write a simple, local hosts file (hosts):

   [cori_risi_r_server]
   18.235.239.47

Run the basic ansible ping command with command line arguments for the name of the ssh user and the path to the ssh private key file:

ansible cori_risi_r_server -m ping -i hosts --extra-vars "ansible_ssh_user=$ANSIBLE_SSH_USER" --extra-vars "ansible_ssh_private_key_file=$ANSIBLE_SSH_PRIVATE_KEY_FILE"

You should be prompted with:

18.235.239.47 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

Alternatively, this repo also includes a hosts.yml file in the inventory subdirectory, which can be supplied to the ansible commands with -i inventory/hosts.yml:

$ ansible cori_risi_r_server -i inventory/hosts.yml -a "/bin/echo hello"

Notes: To run ansible playbooks on the R server with nohup you can use this sequence of shell commands (from the ansible project directory):

read -s password
******
echo $password > p.temp
nohup bash -c 'ansible-playbook -i inventory/hosts.yml playbooks/cori-risi-ad-postgresql/main.yml --extra-vars "db_database=data" --extra-vars "password_file=$(pwd)/p.temp"' > cori-risi-ad-postgresql.log &

Remember to delete the p.temp file afterwards.