Deploying OpenStack with Kolla-Ansible

UPDATE

Please see my new article for a much simpler and easier method that explains how to achieve the same using Terraform.

Would you like to set up a complete OpenStack cluster? Then you can easily do this with Kolla-Ansible. Kolla-Ansible is based on Ansible, as the name suggests, and is therefore relatively easy to use especially if you are already familar with Ansible. As usual with Ansible, you create a inventory file in which the individual OpenStack nodes are defined and then simply let Ansible do its magic and within a short time you have a complete OpenStack cluster. It offers many possibilities to extend the cluster with additional components like Cinder (Block Storage), Manila, LBaaS (Octavia), etc. A very good resource is the Kolla-Ansible support matrix. I will go into more detail about the additional modules in further posts and thus extend the OpenStack cluster. Ok that was a lot of Ansible and I can promise you there is more to come. :) Let's get started. My instructions are based on the official Kolla-Ansible Quick Start Guide.

Installation Notes

As usual you can either deploy bare-metal nodes or use my favourite duo of ESXi and Vagrant to deploy the OpenStack Cluster virtually. You should first check out my Blog Post on how to set this up. Of course you can't use the virtualized cluster in production but it is perfect to test new configurations and modules before running them on your production cluster. We will deploy the OpenStack Wallaby release. There is a new release every 6 months and at the time of writing this blog post there is already a newer release called Xena, but I want to test the upgrade procedure in one my future blog posts.

There are basically two ways Kolla-Ansible can set up your cluster. The smallest cluster a.k.a all-in-one will be deployed on a single node and requires at least:

• 2 network interfaces
• 8GB main memory
• 40GB disk space (SSD higly recommended!!!)
• 2 CPU cores

If you have enough resources, you can also deploy a high availability cluster consisting of:

• 1 Seed node
• 3 Control nodes
• 2 Compute nodes

At least my tests showed that for a HA cluster you should have at least the same amount of resouces as for a single node. So simply multiply the resouces above 6 times. Yes, you can deploy just one compute node but this will not allow you to migrate the VMs to another compute node.

Vagrant Deployment

First let's download my Vagrantfile from my Github repo to a folder of your choice. Mine is called kolla-ansible. You have to adjust the IP adresses in the vagrant file. At least for the "public_ip". My local subnet is 192.168.2.0/24. Make sure to adjust this to match yours. Also take a look at the assigned resources RAM and CPU of all the nodes. They might be way over what your ESXi host can provide. Adjust this according to your environment. The resulting VMs will have 4 network cards.

• eth0
• connected to your local subnet
• receive an IP from your local DHCP
• used by vagrant to communicate with the VM

• eth1
• connected to your local subnet
• must not have an IP assigned!!! (my Vagrant file takes care of this)
• used by OpenStack Neutron for provider networks to connect VMs with your local network (neutron_external_interface)

• eth2
• connected to your local subnet
• static IP assigned in Vagrantfile "public_ip"
• used by Kolla-Ansible for communication with the other hosts

• eth3
• should be an isolated network, not accessible from outside!!!
• static IP assigned in Vagrantfile "ip"
• used by the OpenStack services to communicate to each other and the databases

For a production cluster you would define even more networks. You can look up this guide from the official OpenStack Kolla-Ansible Documentation.
By default a HA cluster will be deployed. In case you want the all-in-one set up, simply edit the Vagrantfile and comment out all nodes except the "Seed" node and run "vagrant up".

cd kolla-ansible
vagrant up

If you are deploying the HA cluster and you receive errors during deployment, try to deploy the VMs one after another. It seems that Vagrants ESXi plugin can't handle that many VMs concurrently. I had no problems deploying multiple VMs in Virtualbox.

vagrant destroy -f
vagrant up --no-parallel

Once the VMs are deployed, it is a good idea to take a snapshot.

vagrant snapshot save fresh

Kolla-Ansible Installation

After that we ssh to the seed node and update all of it's packages and reboot it.

vagrant ssh seed
sudo yum -y update && reboot

Log back in on the seed VM and install deps + python3-pip

vagrant ssh seed
sudo dnf install -y python3-devel libffi-devel gcc openssl-devel python3-libselinux python3-pip

Ensure the latest version of pip is installed

sudo pip3 install -U pip

Install EPEL repo

sudo yum -y install epel-release

Install Ansible

sudo pip3 install -U 'ansible<3.0'

Install sshpass

sudo yum -y install sshpass

Install Kolla-ansible

sudo pip3 install --ignore-installed PyYAML kolla-ansible

Kolla-Ansible Configuration

Create the /etc/kolla directory

sudo mkdir -p /etc/kolla
sudo chown $USER:$USER /etc/kolla

Copy globals.yml and passwords.yml to /etc/kolla directory.

cp -r /usr/local/share/kolla-ansible/etc_examples/kolla/* /etc/kolla

Copy all-in-one and multinode inventory files to the current directory.

cp /usr/local/share/kolla-ansible/ansible/inventory/* .

Configure Ansible

For best results, Ansible configuration should be tuned for your environment. For example, add the following options to the Ansible configuration file:

cat <<EOT > ansible.cfg
[defaults]
host_key_checking=False
pipelining=True
forks=100
timeout=40
EOT

Increasing the timeout is necessary to avoid errors like “Timeout (12s) waiting for privilege escalation prompt:”

Configure OpenStack

The main configuration resides in /etc/kolla/globals.yml.

Deploy All-In-One

Use the settings below if you want to deploy a All-In-One cluster on a single node or VM.

sed -i 's/^#kolla_base_distro:.*/kolla_base_distro: "centos"/g' /etc/kolla/globals.yml
sed -i 's/^#kolla_install_type:.*/kolla_install_type: "source"/g' /etc/kolla/globals.yml
sed -i 's/^#network_interface:.*/network_interface: "eth2"/g' /etc/kolla/globals.yml
sed -i 's/^#neutron_external_interface:.*/neutron_external_interface: "eth1"/g' /etc/kolla/globals.yml
sed -i 's/^#openstack_release:.*/openstack_release: "wallaby"/g' /etc/kolla/globals.yml

Deploy HA Cluster

In case you have enough resources, you can simply change three more settings to deploy a HA cluster. You need a free static IP for the external cluster VIP on your local subnet. Make sure to adjust it to match your local subnet.

sed -i 's/^#kolla_internal_vip_address:.*/kolla_internal_vip_address: "192.168.45.222"/g' /etc/kolla/globals.yml
sed -i 's/^#enable_haproxy:.*/enable_haproxy: "yes"/g' /etc/kolla/globals.yml
sed -i 's/^#kolla_external_vip_address:.*/kolla_external_vip_address: "192.168.2.222"/g' /etc/kolla/globals.yml

We can easily lookup all configured settings in globals.yml using:

grep ^[^#] /etc/kolla/globals.yml

Example

[vagrant@seed ~]$ grep ^[^#] /etc/kolla/globals.yml
---
kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "wallaby"
kolla_internal_vip_address: "192.168.45.222"
kolla_external_vip_address: "192.168.2.222"
network_interface: "eth2"
neutron_external_interface: "eth1"
keepalived_virtual_router_id: "222"
enable_haproxy: "yes"

Kolla-Ansible deploys OpenStack in containers and I highly recommend to run a central docker registry for this as otherwise all nodes will download all the container images and this requires a lot of time and disk space. We can run the registry on the seed node in a container. For this we install docker, pull the registry container and start it.

curl -sSL https://get.docker.io | sudo bash
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker
sudo pip3 install docker
sudo mkdir -p /var/lib/registry
sudo docker run -d \
 --name registry \
 --restart=always \
 -p 4000:5000 \
 -v registry:/var/lib/registry \
 registry:2
kolla-ansible -i all-in-one pull

Now we need to push the docker images to our local registry. It seems that Kolla-Ansible is looking for images at URL .../kolla.openstack instead of quay.io/kolla.openstack. That is why I truncated the quay.io from the URL.

sudo -i
docker images | grep kolla | grep -v local | awk '{print $1,$2}' | while read -r image tag; do 
        newimg=`echo ${image} | cut -d / -f2-`
        docker tag ${image}:${tag} localhost:4000/${newimg}:${tag}
        docker push localhost:4000/${newimg}:${tag}
done
exit

Edit globals.yml and add the following, where 192.168.2.210:4000 is the IP address and port on which the registry is listening. Don't forget to  adjust this to the IP of your seed node.

sed -i 's/^#docker_registry:.*/docker_registry: 192.168.2.210:4000/g' /etc/kolla/globals.yml
sed -i 's/^#docker_registry_insecure:.*/docker_registry_insecure: yes/g' /etc/kolla/globals.yml

We can check if the repos have been successfully pushed to our local registry with curl

curl -X GET http://192.168.2.210:4000/v2/_catalog | python3 -m json.tool

We are almost done. If you decided to use the all-in-one deployment, you can simply use the all-in-one ansible hosts file. If you chose to deploy the HA cluster, you have to modify the multinode file.

control[01:03]     ansible_user=vagrant ansible_password=vagrant ansible_become=true api_interface=eth3
compute[01:02]     ansible_user=vagrant ansible_password=vagrant ansible_become=true api_interface=eth3

[control]
# These hostname must be resolvable from your deployment host
control[01:03]

# The above can also be specified as follows:
#control[01:03]     ansible_user=vagrant ansible_password=vagrant ansible_become=true
#compute[01:02]     ansible_user=vagrant ansible_password=vagrant ansible_become=true

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
control[01:03]
#network01
#network02

[compute]
compute[01:02]

[monitoring]
control[01:03]
#monitoring01

# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]
control[01:03]
#storage01

[deployment]
localhost       ansible_connection=local

Try to ansible ping all defined nodes

ansible -i multinode all -m ping

If you fail to ping all the nodes verify the correct IPs in /etc/hosts. Sometimes hostmanager fails to update all the entries.

cat /etc/hosts

Verify that the correct host is reached by checking it's hostname

ansible -i multinode all -m shell -a "uname -n"

Update all control and compute nodes and reboot them

ansible -i multinode compute,control -m shell -a "yum -y update" -b
ansible -i multinode compute,control --forks 1 -m reboot

Check that all nodes have correct date and time

ansible -i multinode all -m shell -a 'date'

If not, set initial time and date

ansible -i multinode all -m shell -a 'date -s "10 OCT 2021 19:21:00"' -b

Enable and start chrony on all nodes

ansible -i multinode all -m shell -a "systemctl enable chronyd; systemctl restart chronyd" -b

Check date and time on all nodes again

ansible -i multinode all -m shell -a "date"

Passwords used in our deployment are stored in the /etc/kolla/passwords.yml file. All passwords are blank in this file and have to be filled either manually or by running random password generator:

kolla-genpwd

If all is fine, you might want to take an additonal snapshot of all nodes before starting the deployment

exit
vagrant snapshot save all_updated
vagrant ssh seed

Actual OpenStack Deployment

The following assumes the use of the multinode inventory. If using a different inventory, such as all-in-one, replace the -i argument accordingly.

Bootstrap servers with kolla deploy dependencies:

This command also has to be re-run if you add a local docker registry!!!

time kolla-ansible -i ./multinode bootstrap-servers

Do pre-deployment checks for hosts:

time kolla-ansible -i ./multinode prechecks

If all checks are fine take another snapshot

exit
vagrant snapshot save all_bootstrpd_checked
vagrant ssh seed            

Finally proceed to actual OpenStack deployment:

time kolla-ansible -i ./multinode deploy

After a while you should be presented with a summary similar to this one, if you deployed a multinode cluster. If you went with the all-in-one your summary will of course only contain one host.

PLAY RECAP *************************************************************************************************************
compute01                  : ok=171  changed=106  unreachable=0    failed=0    skipped=125  rescued=0    ignored=1
compute02                  : ok=165  changed=106  unreachable=0    failed=0    skipped=123  rescued=0    ignored=1
control01                  : ok=290  changed=181  unreachable=0    failed=0    skipped=167  rescued=0    ignored=1
control02                  : ok=198  changed=117  unreachable=0    failed=0    skipped=164  rescued=0    ignored=1
control03                  : ok=198  changed=117  unreachable=0    failed=0    skipped=164  rescued=0    ignored=1
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Congratulations, you just successfully deployed a OpenStack HA Cluster.

Operating OpenStack

Install the OpenStack CLI client:

pip install python-openstackclient

OpenStack requires an openrc file where credentials for admin user are set. To generate this file run:

time kolla-ansible post-deploy
. /etc/kolla/admin-openrc.sh

Depending on how you installed Kolla Ansible, there is a script that will create example networks, images, and so on.Finally take one more snapshot of the fresh deployed OpenStack cluster

exit
vagrant snapshot save deployed
vagrant ssh seed

First Steps

After the deployment has finished it is time to create VMs (instances), networks, security groups etc. To make our Life easier we can simply modify and use the "init-runonce.sh" script. This will do the following:

• Download cirros image (a very small Linux OS image, perfect for connectivity testing)
• Modification of the default security group to allow ICMP traffic and ports 22 (SSH),80 (HTTP) and 8000
• Increase the default quota limit to 40 instances, 40 cores and 96GB RAM
• 2 networks
• a private network demo-net, where the instances will reside
• a provider network public1 that will provide access via floating ips for our instances
• 2 subnets
• demo-subnet (10.0.0.0/24) for the private network
• public1-subnet (192.168.2.0/24) for the provider network
• multiple flavors for the instances (flavors are collections of how many resources, cpu, memory and disk will be availavble in an instance
• a public private key pair for ssh access to the instanceCopy, the init-runonce script:

cp /usr/local/share/kolla-ansible/init-runonce .

Modify it to fit your local flat network

sed -i s'/10.0.2./192.168.2./'g init-runonce

Important!!! By default the script will create a provider network with IPs from 192.168.2.150-199. Make sure to either exclude these from your DHCP server or to adjust the IP range in the script.

Run it

./init-runonce

Once the script is finished, lets see what we got

Example

[vagrant@seed ~]$ openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID                                   | Name     | Subnets                              |
+--------------------------------------+----------+--------------------------------------+
| 28428b87-48f1-4df1-bc74-b450be09d6a4 | demo-net | 0855fa1f-f41d-4dc1-96b3-b13477e99945 |
| 5266fce4-4cd0-4452-8b7e-5e80f8e553b1 | public1  | 2c57a69a-de17-45de-92a8-ddae0499efbc |
+--------------------------------------+----------+--------------------------------------+

[vagrant@seed ~]$ openstack subnet list
+--------------------------------------+----------------+--------------------------------------+----------------+
| ID                                   | Name           | Network                              | Subnet         |
+--------------------------------------+----------------+--------------------------------------+----------------+
| 0855fa1f-f41d-4dc1-96b3-b13477e99945 | demo-subnet    | 28428b87-48f1-4df1-bc74-b450be09d6a4 | 10.0.0.0/24    |
| 2c57a69a-de17-45de-92a8-ddae0499efbc | public1-subnet | 5266fce4-4cd0-4452-8b7e-5e80f8e553b1 | 192.168.2.0/24 |
+--------------------------------------+----------------+--------------------------------------+----------------+

[vagrant@seed ~]$ openstack flavor list
+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1  | m1.tiny   |   512 |    1 |         0 |     1 | True      |
| 2  | m1.small  |  2048 |   20 |         0 |     1 | True      |
| 3  | m1.medium |  4096 |   40 |         0 |     2 | True      |
| 4  | m1.large  |  8192 |   80 |         0 |     4 | True      |
| 5  | m1.xlarge | 16384 |  160 |         0 |     8 | True      |
+----+-----------+-------+------+-----------+-------+-----------+

Once we verified that all the resources have been created, we can go ahead and deploy our first instance

openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1

After a few seconds the instance should be deployed and running

[vagrant@seed ~]$ openstack server list
+--------------------------------------+-------+--------+--------------------+--------+---------+
| ID                                   | Name  | Status | Networks           | Image  | Flavor  |
+--------------------------------------+-------+--------+--------------------+--------+---------+
| 48d63f18-9587-4e93-8a8b-4ea279cbaff2 | demo1 | ACTIVE | demo-net=10.0.0.73 | cirros | m1.tiny |
+--------------------------------------+-------+--------+--------------------+--------+---------+

Access the instance via SSH from our local network

In order to be able to access the new instance from our local network ,we first have to create a floating IP address on the provider virtual network:

openstack floating ip create public1

Example

[vagrant@seed ~]$ openstack floating ip create public1
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2021-09-11T13:08:42Z                 |
| description         |                                      |
| dns_domain          | None                                 |
| dns_name            | None                                 |
| fixed_ip_address    | None                                 |
| floating_ip_address | 192.168.2.166                        |
| floating_network_id | 5266fce4-4cd0-4452-8b7e-5e80f8e553b1 |
| id                  | 148d2ed4-c487-4a49-9f3a-a7136d5e1693 |
| name                | 192.168.2.166                        |
| port_details        | None                                 |
| port_id             | None                                 |
| project_id          | a4478e7100c44825a5b1c41a8bc8f001     |
| qos_policy_id       | None                                 |
| revision_number     | 0                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2021-09-11T13:08:42Z                 |
+---------------------+--------------------------------------+

Now we associate the floating IP address with the instance:

openstack server add floating ip demo1 192.168.2.166

Check the status of your floating IP address:

[vagrant@seed ~]$ openstack server list
+--------------------------------------+-------+--------+-----------------------------------+--------+---------+
| ID                                   | Name  | Status | Networks                          | Image  | Flavor  |
+--------------------------------------+-------+--------+-----------------------------------+--------+---------+
| 48d63f18-9587-4e93-8a8b-4ea279cbaff2 | demo1 | REBOOT | demo-net=10.0.0.73, 192.168.2.166 | cirros | m1.tiny |
+--------------------------------------+-------+--------+-----------------------------------+--------+---------+

Verify connectivity to the instance via floating IP address from the controller node or any host on the provider physical network:

ping 192.168.2.166

Access your instance using SSH from the controller node or any host on the provider physical network:

ssh cirros@192.168.2.166

In case the ping is not working, check that the security rule has been modified to allow icmp traffic:

openstack security group show default

Troubleshooting

If you ever have to completely power off your OpenStack HA Cluster, you might have to run the following command afterwards in order to fix the MariaDB Galera Cluster and bring the OpenStack Cluster back online.

time kolla-ansible -i ./multinode mariadb_recovery