Terraform and NREC: Part IV - Pairing with Ansible¶
Last changed: 2024-05-02
Important
NB! Does not work! Changes to Terraform and Ansible has rendered this documentation obsolete. We will review and update as soon as possible.
This document builds on Terraform Part 1, Part 2 and Part 3. In Part 3 we built an environment in NREC consisting of 4 frontend web servers and 1 backend database server, as well as security groups and an SSH key pair for access. However, we didn’t do anything inside the operating systems of our instances to make them act as web servers or database servers. Terraform only interacts with the Openstack API, and doesn’t do anything inside the instances. Here is where Ansible comes into play. Using Ansible, we’ll configure the web servers and the database server to make a real service.
This is not an introduction to Ansible. It is assumed that the reader has some knowledge and experience in using Ansible.
The files used in this document can be downloaded:
Installing Ansible¶
Ansible is available in most Linux distributions. If possible, use the Ansible version availble from the distribution. For Fedora, and for RHEL or CentOS with EPEL enabled, simply install using yum:
# yum install ansible
For Debian and Ubuntu:
# apt-get install ansible
When using Ansible with NREC and instances created with Terraform,
we’ll often create and destroy instances multiple times. This depends
on your workflow. It may be beneficial to add the following
configuration to your ~/.ansible.cfg
to prevent Ansible from
halting on unknown SSH host keys:
[defaults]
host_key_checking = False
Ansible inventory from Terraform state¶
Terraform maintains a state in the working directory, and is also able
to update its local state against the real resources in NREC. The
local state is stored in terraform.tfstate
, and we’re using a
Python script that reads this file and produces an Ansible inventory
dynamically.
To use the Python script we need to install and set it up:
Download
terraform.py
Put it in
~/.local/bin
and make sure it is executable:$ mv ~/Downloads/terraform.py ~/.local/bin/ $ chmod a+x ~/.local/bin/terraform.py
This installs the Python script terraform.py
into ~/.local/bin
(e.g. your home directory). Usually, this directory should be in your
shell path. If it isn’t, you can add it (for bash):
export PATH=$PATH:~/.local/bin
In your Terraform working directory, create a directory called
“inventory”, and create a symbolic link “hosts” that points to the
terraform.py
script:
$ mkdir inventory
$ ln -s ~/.local/bin/terraform.py inventory/hosts
You can then run ansible from within your Terraform working directory to verify that dynamic inventory works:
$ ansible all -i inventory --list-hosts
hosts (5):
bgo-db-0
bgo-web-0
bgo-web-1
bgo-web-2
bgo-web-3
This only lists the hosts and verifies that the dynamic inventory works. Having Ansible actually connect to the hosts requires additional configuration as described in the next section.
Configuring Ansible connectivity¶
In order for Ansible to function correctly we need to tell Ansible
additional information about the instances. First, we add a map in
variables.tf
to address the SSH user. Ansible needs to know which
SSH user to connect as:
1# Mapping between role and SSH user
2variable "role_ssh_user" {
3 type = map(string)
4 default = {
5 "web" = "centos"
6 "db" = "ubuntu"
7 }
8}
Next, we need to use those variables and add a metadata directive in the compute instance resource. The script terraform.py will use this metadata to correctly create inventory for Ansible. For the web servers (CentOS 8):
1 metadata = {
2 ssh_user = lookup(var.role_ssh_user, "web", "unknown")
3 prefer_ipv6 = 1
4 my_server_role = "web"
5 }
And for the database server (Ubuntu 21.04 LTS):
1 metadata = {
2 ssh_user = lookup(var.role_ssh_user, "db", "unknown")
3 prefer_ipv6 = 1
4 python_bin = "/usr/bin/python3"
5 my_server_role = "database"
6 }
We have added this metadata:
ssh_user
: Using the map variable created in variables.tf (see above).prefer_ipv6
: This tells terraform.py that we want Ansible to use the IPv6 address of the instance. This is needed in our case as we’re using the IPv6 network type in NREC. When using the dualStack network, this is usually not needed.python_bin
: This is only used on the database server (Ubuntu). Ansible needs a working Python binary to function, and in Ubuntu’s case there isn’t a/usr/bin/python
and Ansible needs to be explicitly told which binary to use on the instance.my_server_role
: We use this to control how to identify the web servers and database servers in the Ansible inventory.
With these in place, having applied the configuration with terraform
apply
, we can run terraform.py to view our inventory:
$ terraform.py --root . --hostfile
## begin hosts generated by terraform.py ##
2001:700:2:8301::113b bgo-db-0
2001:700:2:8301::113f bgo-web-0
2001:700:2:8301::100a bgo-web-1
2001:700:2:8301::100b bgo-web-2
2001:700:2:8301::1129 bgo-web-3
## end hosts generated by terraform.py ##
And we run ansible to verify that it is able to reach the instances over the network:
$ ansible all -i inventory -m ping
bgo-web-3 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-web-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-db-0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-web-2 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-web-0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
We can also run a command on the instances to verify that SSH connection is working:
$ ansible all -i inventory -m shell -a 'uname -sr'
bgo-web-3 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-web-1 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-web-2 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-web-0 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-db-0 | CHANGED | rc=0 >>
Linux 4.15.0-70-generic
We have verified that Ansible and dynamic inventory from Terraform state works, and are ready to proceed.
Using Ansible¶
Important
This section includes simple playbooks to show how Ansible can be used for configuring the OS and services. In order to make this into a real service for production use, a lot more work needs to be done.
I order to configure the web and database servers, we have created two
playbooks. They are named web.yaml
and db.yaml
,
respectively. We’ll take a look at web.yaml
first:
1---
2- hosts: os_metadata_my_server_role=web
3 become: true
4
5 tasks:
6 - name: Install Apache
7 yum:
8 name: httpd
9
10 - name: Install php
11 yum:
12 name: php
13
14 - name: Install php-mysql
15 yum:
16 name: php-mysql
17
18 - name: Set httpd_can_network_connect_db SELinux boolean
19 seboolean:
20 name: httpd_can_network_connect_db
21 state: yes
22 persistent: yes
23
24 - name: Make sure Apache is running and enabled
25 systemd:
26 name: httpd
27 state: started
28 enabled: yes
In this playbook, we do the following:
Install the Apache web server, PHP and the PHP MySQL bindings
Make sure that SELinux allows Apache to connect to the database
Make sure that the web service is enabled and running.
Next, lets take a look at db.yaml
which we use to configure the
database server:
1---
2- hosts: os_metadata_my_server_role=database
3 become: true
4
5 tasks:
6 - name: Create XFS filesystem on /dev/sdb
7 filesystem:
8 fstype: xfs
9 dev: /dev/sdb
10
11 - name: Mount volume
12 mount:
13 path: /var/lib/mysql
14 src: /dev/sdb
15 fstype: xfs
16 state: mounted
17
18 - name: Install MariaDB, also starts the service
19 apt:
20 name: mariadb-server
21
22 - name: Set MariaDB bind-address
23 ini_file:
24 path: /etc/mysql/mariadb.conf.d/90-server.cnf
25 section: mysqld
26 option: bind-address
27 value: "{{ ansible_default_ipv4.address }}"
28 backup: yes
29 notify:
30 - restart MariaDB
31
32 - name: Make sure MariaDB is running and enabled
33 systemd:
34 name: mariadb
35 state: started
36 enabled: yes
37
38 - name: Install PyMySQL, needed by Ansible MySQL module
39 apt:
40 name: python3-pymysql
41
42
43 handlers:
44 - name: restart MariaDB
45 systemd:
46 name: "mariadb"
47 state: restarted
In this playbook, we do the following:
Create a filesystem on our volume, available as the
/dev/sdb
device, and mount it as/var/lib/mysql
Install MariaDB (i.e. MySQL)
Set the MariaDB bind address, i.e. the IP address that we want the database server to listen to. We use the internal, private IPv4 address for this. When using the IPv6 network in NREC, instances also get a private IPv4 address. We can use this address for communication between instances, which in our case will be communication between the web servers and the database.
Make sure that the database service is enabled and running.
Install the MySQL bindings for Python. This is needed if we want to use Ansible to communicate with the database server, e.g. for creating databases.
The db.yaml
also includes a handler for restarting MariaDB if we
have done configuration changes which require a restart to take
effect.
The Ansible playbooks above would be run like this, from the Terraform workspace directory:
$ ansible-playbook -i inventory db.yaml
$ ansible-playbook -i inventory web.yaml
Complete example¶
A complete listing of the example files used in this document is provided below.
1# Define required providers
2terraform {
3 required_version = ">= 1.0"
4 required_providers {
5 openstack = {
6 source = "terraform-provider-openstack/openstack"
7 }
8 }
9}
10
11# Configure the OpenStack Provider
12# Empty means using environment variables "OS_*". More info:
13# https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs
14provider "openstack" {}
15
16# SSH key
17resource "openstack_compute_keypair_v2" "keypair" {
18 region = var.region
19 name = "${terraform.workspace}-${var.name}"
20 public_key = file(var.ssh_public_key)
21}
22
23# Web servers
24resource "openstack_compute_instance_v2" "web_instance" {
25 region = var.region
26 count = lookup(var.role_count, "web", 0)
27 name = "${var.region}-web-${count.index}"
28 image_name = lookup(var.role_image, "web", "unknown")
29 flavor_name = lookup(var.role_flavor, "web", "unknown")
30
31 key_pair = "${terraform.workspace}-${var.name}"
32 security_groups = [
33 "default",
34 "${terraform.workspace}-${var.name}-ssh",
35 "${terraform.workspace}-${var.name}-web",
36 ]
37
38 network {
39 name = "IPv6"
40 }
41
42 metadata = {
43 ssh_user = lookup(var.role_ssh_user, "web", "unknown")
44 prefer_ipv6 = 1
45 my_server_role = "web"
46 }
47
48 lifecycle {
49 ignore_changes = [image_name,image_id]
50 }
51
52 depends_on = [
53 openstack_networking_secgroup_v2.instance_ssh_access,
54 openstack_networking_secgroup_v2.instance_web_access,
55 ]
56}
57
58# Database servers
59resource "openstack_compute_instance_v2" "db_instance" {
60 region = var.region
61 count = lookup(var.role_count, "db", 0)
62 name = "${var.region}-db-${count.index}"
63 image_name = lookup(var.role_image, "db", "unknown")
64 flavor_name = lookup(var.role_flavor, "db", "unknown")
65
66 key_pair = "${terraform.workspace}-${var.name}"
67 security_groups = [
68 "default",
69 "${terraform.workspace}-${var.name}-ssh",
70 "${terraform.workspace}-${var.name}-db",
71 ]
72
73 network {
74 name = "IPv6"
75 }
76
77 metadata = {
78 ssh_user = lookup(var.role_ssh_user, "db", "unknown")
79 prefer_ipv6 = 1
80 python_bin = "/usr/bin/python3"
81 my_server_role = "database"
82 }
83
84 lifecycle {
85 ignore_changes = [image_name,image_id]
86 }
87
88 depends_on = [
89 openstack_networking_secgroup_v2.instance_ssh_access,
90 openstack_networking_secgroup_v2.instance_db_access,
91 ]
92}
93
94# Volume
95resource "openstack_blockstorage_volume_v2" "volume" {
96 name = "database"
97 size = var.volume_size
98}
99
100# Attach volume
101resource "openstack_compute_volume_attach_v2" "attach_vol" {
102 instance_id = openstack_compute_instance_v2.db_instance[0].id
103 volume_id = openstack_blockstorage_volume_v2.volume.id
104}
1# Security group SSH + ICMP
2resource "openstack_networking_secgroup_v2" "instance_ssh_access" {
3 region = var.region
4 name = "${terraform.workspace}-${var.name}-ssh"
5 description = "Security group for allowing SSH and ICMP access"
6}
7
8# Security group HTTP
9resource "openstack_networking_secgroup_v2" "instance_web_access" {
10 region = var.region
11 name = "${terraform.workspace}-${var.name}-web"
12 description = "Security group for allowing HTTP access"
13}
14
15# Security group MySQL
16resource "openstack_networking_secgroup_v2" "instance_db_access" {
17 region = var.region
18 name = "${terraform.workspace}-${var.name}-db"
19 description = "Security group for allowing MySQL access"
20}
21
22# Allow ssh from IPv4 net
23resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv4" {
24 region = var.region
25 count = length(var.allow_ssh_from_v4)
26 direction = "ingress"
27 ethertype = "IPv4"
28 protocol = "tcp"
29 port_range_min = 22
30 port_range_max = 22
31 remote_ip_prefix = var.allow_ssh_from_v4[count.index]
32 security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
33}
34
35# Allow ssh from IPv6 net
36resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv6" {
37 region = var.region
38 count = length(var.allow_ssh_from_v6)
39 direction = "ingress"
40 ethertype = "IPv6"
41 protocol = "tcp"
42 port_range_min = 22
43 port_range_max = 22
44 remote_ip_prefix = var.allow_ssh_from_v6[count.index]
45 security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
46}
47
48# Allow icmp from IPv4 net
49resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv4" {
50 region = var.region
51 count = length(var.allow_ssh_from_v4)
52 direction = "ingress"
53 ethertype = "IPv4"
54 protocol = "icmp"
55 remote_ip_prefix = var.allow_ssh_from_v4[count.index]
56 security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
57}
58
59# Allow icmp from IPv6 net
60resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv6" {
61 region = var.region
62 count = length(var.allow_ssh_from_v6)
63 direction = "ingress"
64 ethertype = "IPv6"
65 protocol = "icmp"
66 remote_ip_prefix = var.allow_ssh_from_v6[count.index]
67 security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
68}
69
70# Allow HTTP from IPv4 net
71resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv4" {
72 region = var.region
73 count = length(var.allow_http_from_v4)
74 direction = "ingress"
75 ethertype = "IPv4"
76 protocol = "tcp"
77 port_range_min = 80
78 port_range_max = 80
79 remote_ip_prefix = var.allow_http_from_v4[count.index]
80 security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
81}
82
83# Allow HTTP from IPv6 net
84resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv6" {
85 region = var.region
86 count = length(var.allow_http_from_v6)
87 direction = "ingress"
88 ethertype = "IPv6"
89 protocol = "tcp"
90 port_range_min = 80
91 port_range_max = 80
92 remote_ip_prefix = var.allow_http_from_v6[count.index]
93 security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
94}
95
96# Allow MySQL from IPv4 net
97resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv4" {
98 region = var.region
99 count = length(var.allow_mysql_from_v4)
100 direction = "ingress"
101 ethertype = "IPv4"
102 protocol = "tcp"
103 port_range_min = 3306
104 port_range_max = 3306
105 remote_ip_prefix = var.allow_mysql_from_v4[count.index]
106 security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
107}
108
109# Allow MYSQL from IPv6 net
110resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv6" {
111 region = var.region
112 count = length(var.allow_mysql_from_v6)
113 direction = "ingress"
114 ethertype = "IPv6"
115 protocol = "tcp"
116 port_range_min = 3306
117 port_range_max = 3306
118 remote_ip_prefix = var.allow_mysql_from_v6[count.index]
119 security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
120}
121
122# Allow MYSQL from web servers (IPv4)
123resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv4" {
124 region = var.region
125 count = 1
126 direction = "ingress"
127 ethertype = "IPv4"
128 protocol = "tcp"
129 port_range_min = 3306
130 port_range_max = 3306
131 remote_group_id = openstack_networking_secgroup_v2.instance_web_access.id
132 security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
133}
134
135# Allow MYSQL from web servers (IPv6)
136resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv6" {
137 region = var.region
138 count = 1
139 direction = "ingress"
140 ethertype = "IPv6"
141 protocol = "tcp"
142 port_range_min = 3306
143 port_range_max = 3306
144 remote_group_id = openstack_networking_secgroup_v2.instance_web_access.id
145 security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
146}
1# Variables
2variable "region" {
3}
4
5variable "name" {
6 default = "myproject"
7}
8
9variable "ssh_public_key" {
10 default = "~/.ssh/id_rsa.pub"
11}
12
13variable "network" {
14 default = "IPv6"
15}
16
17variable "volume_size" {
18 default = 20
19}
20
21variable "metadata" {
22 type = list(string)
23 default = []
24}
25
26# Security group defaults
27variable "allow_ssh_from_v6" {
28 type = list(string)
29 default = []
30}
31
32variable "allow_ssh_from_v4" {
33 type = list(string)
34 default = []
35}
36
37variable "allow_http_from_v6" {
38 type = list(string)
39 default = []
40}
41
42variable "allow_http_from_v4" {
43 type = list(string)
44 default = []
45}
46
47variable "allow_mysql_from_v6" {
48 type = list(string)
49 default = []
50}
51
52variable "allow_mysql_from_v4" {
53 type = list(string)
54 default = []
55}
56
57# Mapping between role and image
58variable "role_image" {
59 type = map(string)
60 default = {
61 "web" = "GOLD CentOS 8"
62 "db" = "GOLD Ubuntu 21.04 LTS"
63 }
64}
65
66# Mapping between role and flavor
67variable "role_flavor" {
68 type = map(string)
69 default = {
70 "web" = "m1.small"
71 "db" = "m1.medium"
72 }
73}
74
75# Mapping between role and number of instances (count)
76variable "role_count" {
77 type = map(string)
78 default = {
79 "web" = 4
80 "db" = 1
81 }
82}
83
84# Mapping between role and SSH user
85variable "role_ssh_user" {
86 type = map(string)
87 default = {
88 "web" = "centos"
89 "db" = "ubuntu"
90 }
91}
1# Region
2region = "osl"
3
4# This is needed to access the instance over ssh
5allow_ssh_from_v6 = [
6 "2001:700:100:12::7/128",
7 "2001:700:100:118::67/128"
8]
9allow_ssh_from_v4 = [
10 "129.240.12.7/32",
11 "129.240.118.67/32"
12]
13
14# This is needed to access the instance over http
15allow_http_from_v6 = [
16 "2001:700:200::/48",
17 "2001:700:100::/41"
18]
19allow_http_from_v4 = [
20 "129.177.0.0/16",
21 "129.240.0.0/16"
22]
23
24# This is needed to access the instance over the mysql port
25allow_mysql_from_v6 = [
26 "2001:700:100:118::67/128"
27]
28allow_mysql_from_v4 = [
29 "129.240.118.67/32"
30]
1---
2- hosts: os_metadata_my_server_role=web
3 become: true
4
5 tasks:
6 - name: Install Apache
7 yum:
8 name: httpd
9
10 - name: Install php
11 yum:
12 name: php
13
14 - name: Install php-mysql
15 yum:
16 name: php-mysql
17
18 - name: Set httpd_can_network_connect_db SELinux boolean
19 seboolean:
20 name: httpd_can_network_connect_db
21 state: yes
22 persistent: yes
23
24 - name: Make sure Apache is running and enabled
25 systemd:
26 name: httpd
27 state: started
28 enabled: yes
1---
2- hosts: os_metadata_my_server_role=database
3 become: true
4
5 tasks:
6 - name: Create XFS filesystem on /dev/sdb
7 filesystem:
8 fstype: xfs
9 dev: /dev/sdb
10
11 - name: Mount volume
12 mount:
13 path: /var/lib/mysql
14 src: /dev/sdb
15 fstype: xfs
16 state: mounted
17
18 - name: Install MariaDB, also starts the service
19 apt:
20 name: mariadb-server
21
22 - name: Set MariaDB bind-address
23 ini_file:
24 path: /etc/mysql/mariadb.conf.d/90-server.cnf
25 section: mysqld
26 option: bind-address
27 value: "{{ ansible_default_ipv4.address }}"
28 backup: yes
29 notify:
30 - restart MariaDB
31
32 - name: Make sure MariaDB is running and enabled
33 systemd:
34 name: mariadb
35 state: started
36 enabled: yes
37
38 - name: Install PyMySQL, needed by Ansible MySQL module
39 apt:
40 name: python3-pymysql
41
42
43 handlers:
44 - name: restart MariaDB
45 systemd:
46 name: "mariadb"
47 state: restarted