OpenStack Ocata Deployment with OpenStack-Ansible using Ceph

In this tutorial, we will deploy from zero a new OpenStack using OpenStack-Ansible. The purpose is to document all the steps that we did and the issues that we faced. We assume that you already have Ceph installed and configured, for Ceph deployment you can use https://github.com/ceph/ceph-ansible This tutorial is based on https://docs.openstack.org/project-deploy-guide/openstack-ansible/ocata/

Our stack

  • 4 compute/infra nodes
  • 4 ceph nodes
  • OS: Ubuntu 16.04
  • Kernel: 4.4.0-83-generic
  • Deployment Host: compute1

1. Prepare the deployment host, in our case compute1

apt-get update && apt-get dist-upgrade -y
apt-get install -y aptitude build-essential git ntp ntpdate openssh-server python-dev sudo

Configure NTP and reboot the server

2. Configure the network

For more information, you should check https://docs.openstack.org/project-deploy-guide/openstack-ansible/ocata/overview-network-arch.html

Below the file /etc/network/interfaces from compute1 as an example:

auto lo
iface lo inet loopback
    dns-nameservers 172.31.7.243
    dns-search maas

auto eno1
iface eno1 inet manual
    mtu 1500

auto eno2
iface eno2 inet manual
    mtu 1500

auto ens15
iface ens15 inet manual
    bond-master bond0
    mtu 9000

auto ens15d1
iface ens15d1 inet manual
    bond-master bond0
    mtu 9000

auto bond0
iface bond0 inet manual
    bond-lacp-rate 1
    mtu 9000
    bond-mode 802.3ad
    bond-miimon 100
    bond-slaves none
    bond-xmit_hash_policy layer3+4

auto bond0.100
iface bond0.100 inet manual
    vlan-raw-device bond0

auto br-mgmt
iface br-mgmt inet static
    address 10.20.20.10/24
    gateway 10.20.20.2
    bridge_fd 15
    bridge_ports eno2

auto br-storage
iface br-storage inet static
    address 192.168.1.10
    netmask 255.255.255.0
    bridge_fd 15
    mtu 9000
    bridge_ports bond0

auto br-vlan
iface br-vlan inet manual
    bridge_fd 15
    mtu 1500
    bridge_ports eno1

auto br-vxlan
iface br-vxlan inet static
    address 10.30.30.10
    netmask 255.255.255.0
    bridge_fd 15
    mtu 9000
    bridge_ports bond0.100

source /etc/network/interfaces.d/*.cfg

3. Install source and dependencies

git clone -b 15.1.6 https://git.openstack.org/openstack/openstack-ansible /opt/openstack-ansible
/opt/openstack-ansible/scripts/bootstrap-ansible.sh

4. Configure SSH keys

Generate an SSH key on compute1 and copy to all hosts

5. Prepare target hosts

apt-get update
apt-get dist-upgrade -y
apt-get install -y bridge-utils debootstrap ifenslave ifenslave-2.6 lsof lvm2 ntp ntpdate openssh-server sudo tcpdump vlan
echo 'bonding' >> /etc/modules
echo '8021q' >> /etc/modules
service ntp restart

6. Configure Storage

In our case Ceph was already installed with ceph-ansible, below our pools configured:

root@ceph1:~# rados lspools
cinder-volumes
.rgw.root
default.rgw.control
default.rgw.data.root
default.rgw.gc
default.rgw.log
cinder-backup
ephemeral-vms
glance-images
default.rgw.users.uid

7. Playbook time

Copy the contents of the /opt/openstack-ansible/etc/openstack_deploy directory to the /etc/openstack_deploy directory.

cp -rp /opt/openstack-ansible/etc/openstack_deploy /etc/openstack_deploy
cp /etc/openstack_deploy/openstack_user_config.yml.example /etc/openstack_deploy/openstack_user_config.yml

7.1 Service credentials

cd /opt/openstack-ansible/scripts
python pw-token-gen.py --file /etc/openstack_deploy/user_secrets.yml

7.1.1 Configuration files

Below the user_variables.yml and openstack_user_config.yml that we used for this deploy with Ceph.

user_variables.yml
---
# Copyright 2014, Rackspace US, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

###
### This file contains commonly used overrides for convenience. Please inspect
### the defaults for each role to find additional override options.
###

## Debug and Verbose options.
debug: true

## Common Glance Overrides
# Set glance_default_store to "swift" if using Cloud Files backend
# or "rbd" if using ceph backend; the latter will trigger ceph to get
# installed on glance. If using a file store, a shared file store is
# recommended. See the OpenStack-Ansible install guide and the OpenStack
# documentation for more details.
# Note that "swift" is automatically set as the default back-end if there
# are any swift hosts in the environment. Use this setting to override
# this automation if you wish for a different default back-end.
glance_default_store: rbd

## Ceph pool name for Glance to use
glance_rbd_store_pool: glance-images
glance_rbd_store_chunk_size: 8

## Common Nova Overrides
# When nova_libvirt_images_rbd_pool is defined, ceph will be installed on nova
# hosts.
nova_libvirt_images_rbd_pool: ephemeral-vms
nova_force_config_drive: False
nova_nova_conf_overrides:
  libvirt:
    live_migration_uri: qemu+ssh://nova@%s/system?keyfile=/var/lib/nova/.ssh/id_rsa&no_verify=1

# If you wish to change the dhcp_domain configured for both nova and neutron
dhcp_domain: openstack.net

## Common Glance Overrides when using a Swift back-end
# By default when 'glance_default_store' is set to 'swift' the playbooks will
# expect to use the Swift back-end that is configured in the same inventory.
# If the Swift back-end is not in the same inventory (ie it is already setup
# through some other means) then these settings should be used.
#
# NOTE: Ensure that the auth version matches your authentication endpoint.
#
# NOTE: If the password for glance_swift_store_key contains a dollar sign ($),
# it must be escaped with an additional dollar sign ($$), not a backslash. For
# example, a password of "super$ecure" would need to be entered as
# "super$$ecure" below.  See Launchpad Bug #1259729 for more details.
#
# glance_swift_store_auth_version: 3
# glance_swift_store_auth_address: "https://some.auth.url.com"
# glance_swift_store_user: "OPENSTACK_TENANT_ID:OPENSTACK_USER_NAME"
# glance_swift_store_key: "OPENSTACK_USER_PASSWORD"
# glance_swift_store_container: "NAME_OF_SWIFT_CONTAINER"
# glance_swift_store_region: "NAME_OF_REGION"

cinder_ceph_client: cinder
cephx: true

## Common Ceph Overrides
ceph_mons:
  - 192.168.1.100
  - 192.168.1.101
  - 192.168.1.102

## Custom Ceph Configuration File (ceph.conf)
# By default, your deployment host will connect to one of the mons defined above to
# obtain a copy of your cluster's ceph.conf.  If you prefer, uncomment ceph_conf_file
# and customise to avoid ceph.conf being copied from a mon.
#ceph_conf_file: |
#  [global]
#  fsid = 00000000-1111-2222-3333-444444444444
#  mon_initial_members = mon1.example.local,mon2.example.local,mon3.example.local
#  mon_host = 10.16.5.40,10.16.5.41,10.16.5.42
#  # optionally, you can use this construct to avoid defining this list twice:
#  # mon_host = {{ ceph_mons|join(',') }}
#  auth_cluster_required = cephx
#  auth_service_required = cephx


# By default, openstack-ansible configures all OpenStack services to talk to
# RabbitMQ over encrypted connections on port 5671. To opt-out of this default,
# set the rabbitmq_use_ssl variable to 'false'. The default setting of 'true'
# is highly recommended for securing the contents of RabbitMQ messages.
# rabbitmq_use_ssl: false

# RabbitMQ management plugin is enabled by default, the guest user has been
# removed for security reasons and a new userid 'monitoring' has been created
# with the 'monitoring' user tag. In order to modify the userid, uncomment the
# following and change 'monitoring' to your userid of choice.
# rabbitmq_monitoring_userid: monitoring


## Additional pinning generator that will allow for more packages to be pinned as you see fit.
## All pins allow for package and versions to be defined. Be careful using this as versions
## are always subject to change and updates regarding security will become your problem from this
## point on. Pinning can be done based on a package version, release, or origin. Use "*" in the
## package name to indicate that you want to pin all package to a particular constraint.
# apt_pinned_packages:
#   - { package: "lxc", version: "1.0.7-0ubuntu0.1" }
#   - { package: "libvirt-bin", version: "1.2.2-0ubuntu13.1.9" }
#   - { package: "rabbitmq-server", origin: "www.rabbitmq.com" }
#   - { package: "*", release: "MariaDB" }


## Environment variable settings
# This allows users to specify the additional environment variables to be set
# which is useful in setting where you working behind a proxy. If working behind
# a proxy It's important to always specify the scheme as "http://". This is what
# the underlying python libraries will handle best. This proxy information will be
# placed both on the hosts and inside the containers.

## Example environment variable setup:
## (1) This sets up a permanent environment, used during and after deployment:
# proxy_env_url: http://username:pa$$w0rd@10.10.10.9:9000/
# no_proxy_env: "localhost,127.0.0.1,{{ internal_lb_vip_address }},{{ external_lb_vip_address }},{% for host in groups['all_containers'] %}{{ hostvars[host]['container_address'] }}{% if not loop.last %},{% endif %}{% endfor %}"
# global_environment_variables:
#   HTTP_PROXY: "{{ proxy_env_url }}"
#   HTTPS_PROXY: "{{ proxy_env_url }}"
#   NO_PROXY: "{{ no_proxy_env }}"
#   http_proxy: "{{ proxy_env_url }}"
#   https_proxy: "{{ proxy_env_url }}"
#   no_proxy: "{{ no_proxy_env }}"
#
## (2) This is applied only during deployment, nothing is left after deployment is complete:
# deployment_environment_variables:
#   http_proxy: http://username:pa$$w0rd@10.10.10.9:9000/
#   https_proxy: http://username:pa$$w0rd@10.10.10.9:9000/
#   no_proxy: "localhost,127.0.0.1,{{ internal_lb_vip_address }},{{ external_lb_vip_address }}"


## SSH connection wait time
# If an increased delay for the ssh connection check is desired,
# uncomment this variable and set it appropriately.
ssh_delay: 10


## HAProxy
# Uncomment this to disable keepalived installation (cf. documentation)
haproxy_use_keepalived: True
#
# HAProxy Keepalived configuration (cf. documentation)
# Make sure that this is set correctly according to the CIDR used for your
# internal and external addresses.
#haproxy_keepalived_external_vip_cidr: "{{external_lb_vip_address}}/32"
haproxy_keepalived_internal_vip_cidr: "10.20.20.50/24"
#haproxy_keepalived_external_interface:
haproxy_keepalived_internal_interface: br-mgmt

keepalived_use_latest_stable: True

# Defines the default VRRP id used for keepalived with haproxy.
# Overwrite it to your value to make sure you don't overlap
# with existing VRRPs id on your network. Default is 10 for the external and 11 for the
# internal VRRPs
# haproxy_keepalived_external_virtual_router_id:
# haproxy_keepalived_internal_virtual_router_id:

# Defines the VRRP master/backup priority. Defaults respectively to 100 and 20
# haproxy_keepalived_priority_master:
# haproxy_keepalived_priority_backup:

# Keepalived default IP address used to check its alive status (IPv4 only)
# keepalived_ping_address: "193.0.14.129"

# All the previous variables are used in a var file, fed to the keepalived role.
# To use another file to feed the role, override the following var:
# haproxy_keepalived_vars_file: 'vars/configs/keepalived_haproxy.yml'

openstack_service_publicuri_proto: http
openstack_external_ssl: false
haproxy_ssl: false
openstack_user_config.yml
---
cidr_networks:
  container: 10.20.20.0/24
  tunnel: 10.30.30.0/24
  storage: 192.168.1.0/24

used_ips:
  - "10.20.20.1,10.20.20.50"
  - "10.20.20.100,10.20.20.150"
  - "10.30.30.1,10.30.30.50"
  - "10.30.30.100,10.30.30.150"
  - "192.168.1.1,192.168.1.50"
  - "192.168.1.100,192.168.11.50"

global_overrides:
  internal_lb_vip_address: 10.20.20.50
  #
  # The below domain name must resolve to an IP address
  # in the CIDR specified in haproxy_keepalived_external_vip_cidr.
  # If using different protocols (https/http) for the public/internal
  # endpoints the two addresses must be different.
  #
  external_lb_vip_address: vip.openstack.net
  tunnel_bridge: "br-vxlan"
  management_bridge: "br-mgmt"
  provider_networks:
    - network:
        container_bridge: "br-mgmt"
        container_type: "veth"
        container_interface: "eth1"
        ip_from_q: "container"
        type: "raw"
        group_binds:
          - all_containers
          - hosts
        is_container_address: true
        is_ssh_address: true
    - network:
        container_bridge: "br-vxlan"
        container_type: "veth"
        container_interface: "eth10"
        ip_from_q: "tunnel"
        type: "vxlan"
        range: "1:1000"
        net_name: "vxlan"
        group_binds:
          - neutron_linuxbridge_agent
    - network:
        container_bridge: "br-vlan"
        container_type: "veth"
        container_interface: "eth12"
        host_bind_override: "eno1"
        type: "flat"
        net_name: "flat"
        group_binds:
          - neutron_linuxbridge_agent
    - network:
        container_bridge: "br-vlan"
        container_type: "veth"
        container_interface: "eth11"
        type: "vlan"
        range: "1:1"
        net_name: "vlan"
        group_binds:
          - neutron_linuxbridge_agent
    - network:
        container_bridge: "br-storage"
        container_type: "veth"
        container_interface: "eth2"
        ip_from_q: "storage"
        type: "raw"
        group_binds:
          - glance_api
          - cinder_api
          - cinder_volume
          - nova_compute
          - mons

###
### Infrastructure
###

# galera, memcache, rabbitmq, utility
shared-infra_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# repository (apt cache, python packages, etc)
repo-infra_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# load balancer
# Ideally the load balancer should not use the Infrastructure hosts.
# Dedicated hardware is best for improved performance and security.
haproxy_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# rsyslog server
log_hosts:
  infra4:
    ip: 10.20.20.13

###
### OpenStack
###

# keystone
identity_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# cinder api services
storage-infra_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# glance
# The settings here are repeated for each infra host.
# They could instead be applied as global settings in
# user_variables, but are left here to illustrate that
# each container could have different storage targets.
image_hosts:
  infra1:
    ip: 10.20.20.10
#    container_vars:
#      limit_container_types: glance
#      glance_nfs_client:
#        - server: "172.29.244.15"
#          remote_path: "/images"
#          local_path: "/var/lib/glance/images"
#          type: "nfs"
#          options: "_netdev,auto"
  infra2:
    ip: 10.20.20.11
#    container_vars:
#      limit_container_types: glance
#      glance_nfs_client:
#        - server: "172.29.244.15"
#          remote_path: "/images"
#          local_path: "/var/lib/glance/images"
#          type: "nfs"
#          options: "_netdev,auto"
  infra3:
    ip: 10.20.20.12
#    container_vars:
#      limit_container_types: glance
#      glance_nfs_client:
#        - server: "172.29.244.15"
#          remote_path: "/images"
#          local_path: "/var/lib/glance/images"
#          type: "nfs"
#          options: "_netdev,auto"

# nova api, conductor, etc services
compute-infra_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# heat
orchestration_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# horizon
dashboard_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# neutron server, agents (L3, etc)
network_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# ceilometer (telemetry data collection)
metering-infra_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# aodh (telemetry alarm service)
metering-alarm_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# gnocchi (telemetry metrics storage)
metrics_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12

# nova hypervisors
compute_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12
  infra4:
    ip: 10.20.20.13

# ceilometer compute agent (telemetry data collection)
metering-compute_hosts:
  infra1:
    ip: 10.20.20.10
  infra2:
    ip: 10.20.20.11
  infra3:
    ip: 10.20.20.12
  infra4:
    ip: 10.20.20.13

# cinder volume hosts (NFS-backed)
# The settings here are repeated for each infra host.
# They could instead be applied as global settings in
# user_variables, but are left here to illustrate that
# each container could have different storage targets.
storage_hosts:
  stor1:
    ip: 192.168.1.100
    container_vars:
      cinder_backends:
        limit_container_types: cinder_volume
        rbd:
          volume_group: cinder-volumes
          volume_driver: cinder.volume.drivers.rbd.RBDDriver
          volume_backend_name: rbd
          rbd_pool: cinder-volumes
          rbd_ceph_conf: /etc/ceph/ceph.conf
          rbd_user: "{{ cinder_ceph_client }}"
          rbd_secret_uuid: "{{ cinder_ceph_client_uuid }}"

  stor2:
    ip: 192.168.1.101
    container_vars:
      cinder_backends:
        limit_container_types: cinder_volume
        rbd:
          volume_group: cinder-volumes
          volume_driver: cinder.volume.drivers.rbd.RBDDriver
          volume_backend_name: rbd
          rbd_pool: cinder-volumes
          rbd_ceph_conf: /etc/ceph/ceph.conf
          rbd_user: "{{ cinder_ceph_client }}"
          rbd_secret_uuid: "{{ cinder_ceph_client_uuid }}"

  stor3:
    ip: 192.168.1.102
    container_vars:
      cinder_backends:
        limit_container_types: cinder_volume
        rbd:
          volume_group: cinder-volumes
          volume_driver: cinder.volume.drivers.rbd.RBDDriver
          volume_backend_name: rbd
          rbd_pool: cinder-volumes
          rbd_ceph_conf: /etc/ceph/ceph.conf
          rbd_user: "{{ cinder_ceph_client }}"
          rbd_secret_uuid: "{{ cinder_ceph_client_uuid }}"

  stor4:
    ip: 192.168.1.103
    container_vars:
      cinder_backends:
        limit_container_types: cinder_volume
        rbd:
          volume_group: cinder-volumes
          volume_driver: cinder.volume.drivers.rbd.RBDDriver
          volume_backend_name: rbd
          rbd_pool: cinder-volumes
          rbd_ceph_conf: /etc/ceph/ceph.conf
          rbd_user: "{{ cinder_ceph_client }}"
          rbd_secret_uuid: "{{ cinder_ceph_client_uuid }}"

7.2 Check syntax

cd /opt/openstack-ansible/playbooks
openstack-ansible setup-infrastructure.yml --syntax-check

7.3 Setup hosts

openstack-ansible setup-hosts.yml

7.4 Deploy HAProxy

openstack-ansible haproxy-install.yml

7.5 Run setup-infrastructure

openstack-ansible setup-infrastructure.yml

7.6 Check Galera cluster status

ansible galera_container -m shell \
  -a "mysql -h localhost -e 'show status like \"%wsrep_cluster_%\";'"

Good example output:
node3_galera_container-3ea2cbd3 | success | rc=0 >>
Variable_name             Value
wsrep_cluster_conf_id     17
wsrep_cluster_size        3
wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
wsrep_cluster_status      Primary

node2_galera_container-49a47d25 | success | rc=0 >>
Variable_name             Value
wsrep_cluster_conf_id     17
wsrep_cluster_size        3
wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
wsrep_cluster_status      Primary

node4_galera_container-76275635 | success | rc=0 >>
Variable_name             Value
wsrep_cluster_conf_id     17
wsrep_cluster_size        3
wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
wsrep_cluster_status      Primary

7.7 Deploy Openstack

openstack-ansible setup-openstack.yml
***Confirm success with zero items unreachable or failed:***
1 PLAY RECAP *********************************************************
2 deployment_host : ok=XX changed=0 unreachable=0 failed=0

Always check if some step failed.

8.0 Verify everything

8.1 Check the containers

root@openstack1:/opt/openstack-ansible/scripts# ./inventory-manage.py -l
+------------------------------------------------+----------+--------------------------+---------------+----------------+---------------+------------------------+
| container_name                                 | is_metal | component                | physical_host | tunnel_address | ansible_host  | container_types        |
+------------------------------------------------+----------+--------------------------+---------------+----------------+---------------+------------------------+
| infra1_aodh_container-3425c4c7                 | None     | aodh_api                 | infra1        | None           | 10.20.20.51   | None                   |
| infra2_aodh_container-f5657115                 | None     | aodh_api                 | infra2        | None           | 10.20.20.221  | None                   |
| infra3_aodh_container-2b91aaa6                 | None     | aodh_api                 | infra3        | None           | 10.20.20.152  | None                   |
| infra1_ceilometer_api_container-0eac7d0d       | None     | ceilometer_agent_central | infra1        | None           | 10.20.20.92   | None                   |
| infra2_ceilometer_api_container-e17187de       | None     | ceilometer_agent_central | infra2        | None           | 10.20.20.179  | None                   |
| infra3_ceilometer_api_container-586b7e39       | None     | ceilometer_agent_central | infra3        | None           | 10.20.20.244  | None                   |
| infra1                                         | True     | ceilometer_agent_compute | infra1        | None           | 10.20.20.10   | infra1-host_containers |
| infra2                                         | True     | ceilometer_agent_compute | infra2        | None           | 10.20.20.11   | infra2-host_containers |
| infra3                                         | True     | ceilometer_agent_compute | infra3        | None           | 10.20.20.12   | infra3-host_containers |
| infra4                                         | True     | ceilometer_agent_compute | infra4        | None           | 10.20.20.13   | infra4-host_containers |
| infra1_ceilometer_collector_container-8778fcfb | None     | ceilometer_collector     | infra1        | None           | 10.20.20.191  | None                   |
| infra2_ceilometer_collector_container-cbb55c69 | None     | ceilometer_collector     | infra2        | None           | 10.20.20.173  | None                   |
| infra3_ceilometer_collector_container-2835084a | None     | ceilometer_collector     | infra3        | None           | 10.20.20.241  | None                   |
| infra1_cinder_api_container-0936e945           | None     | cinder_api               | infra1        | None           | 10.20.20.229  | None                   |
| infra2_cinder_api_container-b6fe5dba           | None     | cinder_api               | infra2        | None           | 10.20.20.211  | None                   |
| infra3_cinder_api_container-fe0b3819           | None     | cinder_api               | infra3        | None           | 10.20.20.238  | None                   |
| stor1                                          | True     | cinder_backup            | stor1         | None           | 192.168.1.100 | stor1-host_containers  |
| stor2                                          | True     | cinder_backup            | stor2         | None           | 192.168.1.101 | stor2-host_containers  |
| stor3                                          | True     | cinder_backup            | stor3         | None           | 192.168.1.102 | stor3-host_containers  |
| stor4                                          | True     | cinder_backup            | stor4         | None           | 192.168.1.103 | stor4-host_containers  |
| infra1_cinder_scheduler_container-fc83ebc4     | None     | cinder_scheduler         | infra1        | None           | 10.20.20.79   | None                   |
| infra2_cinder_scheduler_container-a757f7fd     | None     | cinder_scheduler         | infra2        | None           | 10.20.20.90   | None                   |
| infra3_cinder_scheduler_container-141627ce     | None     | cinder_scheduler         | infra3        | None           | 10.20.20.178  | None                   |
| infra1_galera_container-3553e5ad               | None     | galera                   | infra1        | None           | 10.20.20.55   | None                   |
| infra2_galera_container-e1d4feff               | None     | galera                   | infra2        | None           | 10.20.20.208  | None                   |
| infra3_galera_container-bc9c86db               | None     | galera                   | infra3        | None           | 10.20.20.240  | None                   |
| infra1_glance_container-0525f6a0               | None     | glance_api               | infra1        | None           | 10.20.20.204  | None                   |
| infra2_glance_container-987db1c7               | None     | glance_api               | infra2        | None           | 10.20.20.248  | None                   |
| infra3_glance_container-ec5c3ea9               | None     | glance_api               | infra3        | None           | 10.20.20.220  | None                   |
| infra1_gnocchi_container-8b274ec4              | None     | gnocchi_api              | infra1        | None           | 10.20.20.71   | None                   |
| infra2_gnocchi_container-91f1915f              | None     | gnocchi_api              | infra2        | None           | 10.20.20.234  | None                   |
| infra3_gnocchi_container-148bed9d              | None     | gnocchi_api              | infra3        | None           | 10.20.20.58   | None                   |
| infra1_heat_apis_container-1f4677cd            | None     | heat_api_cloudwatch      | infra1        | None           | 10.20.20.167  | None                   |
| infra2_heat_apis_container-ace23717            | None     | heat_api_cloudwatch      | infra2        | None           | 10.20.20.214  | None                   |
| infra3_heat_apis_container-65685847            | None     | heat_api_cloudwatch      | infra3        | None           | 10.20.20.60   | None                   |
| infra1_heat_engine_container-490a2be3          | None     | heat_engine              | infra1        | None           | 10.20.20.96   | None                   |
| infra2_heat_engine_container-3b00a93c          | None     | heat_engine              | infra2        | None           | 10.20.20.192  | None                   |
| infra3_heat_engine_container-9890e6ca          | None     | heat_engine              | infra3        | None           | 10.20.20.52   | None                   |
| infra1_horizon_container-3e18da0f              | None     | horizon                  | infra1        | None           | 10.20.20.87   | None                   |
| infra2_horizon_container-c88d4a4f              | None     | horizon                  | infra2        | None           | 10.20.20.160  | None                   |
| infra3_horizon_container-88c85174              | None     | horizon                  | infra3        | None           | 10.20.20.63   | None                   |
| infra1_keystone_container-33cfd6e8             | None     | keystone                 | infra1        | None           | 10.20.20.78   | None                   |
| infra2_keystone_container-b01d0dfa             | None     | keystone                 | infra2        | None           | 10.20.20.217  | None                   |
| infra3_keystone_container-0c3a530e             | None     | keystone                 | infra3        | None           | 10.20.20.153  | None                   |
| infra1_memcached_container-823e0f5c            | None     | memcached                | infra1        | None           | 10.20.20.165  | None                   |
| infra2_memcached_container-b808b561            | None     | memcached                | infra2        | None           | 10.20.20.250  | None                   |
| infra3_memcached_container-6b22d0b1            | None     | memcached                | infra3        | None           | 10.20.20.166  | None                   |
| infra1_neutron_agents_container-2c8cfc26       | None     | neutron_agent            | infra1        | None           | 10.20.20.72   | None                   |
| infra2_neutron_agents_container-da604256       | None     | neutron_agent            | infra2        | None           | 10.20.20.202  | None                   |
| infra3_neutron_agents_container-baf9fc93       | None     | neutron_agent            | infra3        | None           | 10.20.20.209  | None                   |
| infra1_neutron_server_container-3a820c3a       | None     | neutron_server           | infra1        | None           | 10.20.20.68   | None                   |
| infra2_neutron_server_container-1fb85a80       | None     | neutron_server           | infra2        | None           | 10.20.20.235  | None                   |
| infra3_neutron_server_container-357febaa       | None     | neutron_server           | infra3        | None           | 10.20.20.164  | None                   |
| infra1_nova_api_metadata_container-45e5bd4c    | None     | nova_api_metadata        | infra1        | None           | 10.20.20.203  | None                   |
| infra2_nova_api_metadata_container-fbd1f355    | None     | nova_api_metadata        | infra2        | None           | 10.20.20.227  | None                   |
| infra3_nova_api_metadata_container-ca3b0688    | None     | nova_api_metadata        | infra3        | None           | 10.20.20.213  | None                   |
| infra1_nova_api_os_compute_container-bba3453e  | None     | nova_api_os_compute      | infra1        | None           | 10.20.20.54   | None                   |
| infra2_nova_api_os_compute_container-1eea962e  | None     | nova_api_os_compute      | infra2        | None           | 10.20.20.253  | None                   |
| infra3_nova_api_os_compute_container-22b11b8e  | None     | nova_api_os_compute      | infra3        | None           | 10.20.20.237  | None                   |
| infra1_nova_api_placement_container-a6256190   | None     | nova_api_placement       | infra1        | None           | 10.20.20.243  | None                   |
| infra2_nova_api_placement_container-5cc21228   | None     | nova_api_placement       | infra2        | None           | 10.20.20.86   | None                   |
| infra3_nova_api_placement_container-27eda8e8   | None     | nova_api_placement       | infra3        | None           | 10.20.20.74   | None                   |
| infra1_nova_conductor_container-21b30762       | None     | nova_conductor           | infra1        | None           | 10.20.20.177  | None                   |
| infra2_nova_conductor_container-bff97b73       | None     | nova_conductor           | infra2        | None           | 10.20.20.246  | None                   |
| infra3_nova_conductor_container-d384d6eb       | None     | nova_conductor           | infra3        | None           | 10.20.20.65   | None                   |
| infra1_nova_console_container-86f55a54         | None     | nova_console             | infra1        | None           | 10.20.20.206  | None                   |
| infra2_nova_console_container-99dbfa6e         | None     | nova_console             | infra2        | None           | 10.20.20.236  | None                   |
| infra3_nova_console_container-f57a1e27         | None     | nova_console             | infra3        | None           | 10.20.20.89   | None                   |
| infra1_nova_scheduler_container-b84c323a       | None     | nova_scheduler           | infra1        | None           | 10.20.20.212  | None                   |
| infra2_nova_scheduler_container-403753e3       | None     | nova_scheduler           | infra2        | None           | 10.20.20.99   | None                   |
| infra3_nova_scheduler_container-8de84033       | None     | nova_scheduler           | infra3        | None           | 10.20.20.95   | None                   |
| infra1_repo_container-745bad06                 | None     | pkg_repo                 | infra1        | None           | 10.20.20.70   | None                   |
| infra2_repo_container-495c6de1                 | None     | pkg_repo                 | infra2        | None           | 10.20.20.228  | None                   |
| infra3_repo_container-aa03d50a                 | None     | pkg_repo                 | infra3        | None           | 10.20.20.94   | None                   |
| infra1_rabbit_mq_container-7016f414            | None     | rabbitmq                 | infra1        | None           | 10.20.20.83   | None                   |
| infra2_rabbit_mq_container-3a8e985a            | None     | rabbitmq                 | infra2        | None           | 10.20.20.80   | None                   |
| infra3_rabbit_mq_container-385f0d02            | None     | rabbitmq                 | infra3        | None           | 10.20.20.201  | None                   |
| infra4_rsyslog_container-eeff7432              | None     | rsyslog                  | infra4        | None           | 10.20.20.75   | None                   |
| infra1_utility_container-836672cc              | None     | utility                  | infra1        | None           | 10.20.20.170  | None                   |
| infra2_utility_container-31a8813b              | None     | utility                  | infra2        | None           | 10.20.20.197  | None                   |
| infra3_utility_container-dacfde99              | None     | utility                  | infra3        | None           | 10.20.20.188  | None                   |
+------------------------------------------------+----------+--------------------------+---------------+----------------+---------------+------------------------+
root@openstack1:/opt/openstack-ansible/scripts#

8.1 Verify Openstack API

root@openstack1:/opt/openstack-ansible/scripts# ./inventory-manage.py -l|grep util
| infra1_utility_container-836672cc              | None     | utility                  | infra1        | None           | 10.20.20.170  | None                   |
| infra2_utility_container-31a8813b              | None     | utility                  | infra2        | None           | 10.20.20.197  | None                   |
| infra3_utility_container-dacfde99              | None     | utility                  | infra3        | None           | 10.20.20.188  | None                   |
root@openstack1:/opt/openstack-ansible/scripts#

ssh root@infra1_utility_container-836672cc
root@infra1-utility-container-836672cc:~# source
.ansible/      .bash_history  .bashrc        .cache/        .config/       .my.cnf        openrc         .pip/          .profile       .ssh/
root@infra1-utility-container-836672cc:~# source openrc
root@infra1-utility-container-836672cc:~# openstack user list
+----------------------------------+--------------------+
| ID                               | Name               |

+----------------------------------+--------------------+
| 040d6e3b67d7450d880cdb4e77509067 | glance             |
| 13b0477ec4bc4a3d861257bd73f84a05 | ceilometer         |
| 175cd95c55db4e4ea86699f31865ae65 | aodh               |
| 4abb6d992188412399cc8dfe80cb5cb4 | keystone           |
| 4f15a3c4da2440d3a6822c125fff12db | cinder             |
| 6acd0a000f9049feabfd9064c5af4c0c | neutron            |
| 778056b4ff8b40448b777bb423b49716 | stack_domain_admin |
| 89338cee034241f88b1e89171ed0feb4 | heat               |
| 8d89b2e47e5b40efb3bc709f3271f0d1 | admin              |
| aa7634cbf9e541a8b9c7140ccf4c43c8 | gnocchi            |
| b9e06b4525cb4d859305235d0088c1c0 | nova               |
| c32b371de0ed4f438dc28b2fe4df22ac | placement          |
+----------------------------------+--------------------+
root@infra1-utility-container-836672cc:~#

root@infra1-utility-container-836672cc:~# openstack service list
+----------------------------------+------------+----------------+
| ID                               | Name       | Type           |
+----------------------------------+------------+----------------+
| 3219a914279d4ff2b249d0c2f45aad28 | heat-cfn   | cloudformation |
| 3350112667864ba19936dfa2def3e8c1 | cinder     | volume         |
| 4688c1294f504eb8bb0cd092c58b6c19 | nova       | compute        |
| 5ed33166b5c1465f8b2320fd484ff249 | cinderv2   | volumev2       |
| 6b0a5cf4caf94ccf9a84f76efcc96338 | keystone   | identity       |
| 76947efc711a4fd1b1e59255a767b588 | cinderv3   | volumev3       |
| 7a607d25b17a46e1b35b80d6bedbe965 | gnocchi    | metric         |
| 7d5922e395ed4f83a3cf4f17bc0a1cd8 | glance     | image          |
| 812dd3096f7942e6abd66c915f8cfe0d | aodh       | alarming       |
| 8c780a772a434ebab511dbddf402074c | ceilometer | metering       |
| ad41a628c5d245ba93ddc3380f299399 | heat       | orchestration  |
| c2f5b570d25c483d898dc58dea4e9e37 | placement  | placement      |
| f511704a7cf749abac3029c2d439fb9e | neutron    | network        |
+----------------------------------+------------+----------------+
root@infra1-utility-container-836672cc:~#

root@infra1-utility-container-836672cc:~# openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+--------------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                                          |
+----------------------------------+-----------+--------------+----------------+---------+-----------+--------------------------------------------------------------+
| 092102d3ba4a489abfcfb3c9408f6e6e | RegionOne | cinder       | volume         | True    | public    | http://openstack.net:8776/v1/%(tenant_id)s   |
| 0cdfc0ef5e834fac949e9b6130c6728e | RegionOne | nova         | compute        | True    | public    | http://openstack.net:8774/v2.1/%(tenant_id)s |
| 159383c78ce9456d87d38ed2339a0246 | RegionOne | placement    | placement      | True    | admin     | http://10.20.20.50:8780/placement                            |
| 18eb94dbd77c40a480dd7239145bc47c | RegionOne | cinderv2     | volumev2       | True    | internal  | http://10.20.20.50:8776/v2/%(tenant_id)s                     |
| 1b2df9e4704a4882addcb8dba9539ae4 | RegionOne | keystone     | identity       | True    | internal  | http://10.20.20.50:5000/v3                                   |
| 1ddcaeb0663f45ce82edd37c2254980e | RegionOne | cinder       | volume         | True    | internal  | http://10.20.20.50:8776/v1/%(tenant_id)s                     |
| 26715c3ef1054c9584e8c4340f031a3b | RegionOne | nova         | compute        | True    | admin     | http://10.20.20.50:8774/v2.1/%(tenant_id)s                   |
| 2f1f99ccbf9b440ba063c70ee0022e9c | RegionOne | placement    | placement      | True    | public    | http://openstackt:8780/placement          |
| 367a26c8a62c4ded889e2e7083017708 | RegionOne | neutron      | network        | True    | internal  | http://10.20.20.50:9696                                      |
| 3b29dc27da1a469a8f4594d95b16da36 | RegionOne | neutron      | network        | True    | public    | http://openstack.net:9696                    |
| 3ee4b982f303412cbc61d35fe8cc45d2 | RegionOne | gnocchi      | metric         | True    | internal  | http://10.20.20.50:8041                                      |
| 465752ae7af64b1ebb65d0cdbbf26726 | RegionOne | heat         | orchestration  | True    | admin     | http://10.20.20.50:8004/v1/%(tenant_id)s                     |
| 4a48328a3642401d8d64af875c534c01 | RegionOne | cinderv2     | volumev2       | True    | public    | http://openstack.net:8776/v2/%(tenant_id)s   |
| 52d37ab4c49a4c919454be10476ffe8f | RegionOne | heat         | orchestration  | True    | internal  | http://10.20.20.50:8004/v1/%(tenant_id)s                     |
| 57a4e0f36dae4d45ad9321946b47e6a3 | RegionOne | gnocchi      | metric         | True    | admin     | http://10.20.20.50:8041                                      |
| 5a365052eba8484386336b69f48defb3 | RegionOne | aodh         | alarming       | True    | public    | http://openstack.net:8042                    |
| 5adc5052949e409fb3d5b8fd4c768351 | RegionOne | nova         | compute        | True    | internal  | http://10.20.20.50:8774/v2.1/%(tenant_id)s                   |
| 6624f0dd9e884bcb96dcaddb71c0ad9d | RegionOne | keystone     | identity       | True    | public    | http://openstack.net:5000/v3                 |
| 6d30e2a1824c4a98bd163ad1ddc5d3e9 | RegionOne | placement    | placement      | True    | internal  | http://10.20.20.50:8780/placement                            |
| 6fcdf9bf03aa4d4b858b371bb86490e6 | RegionOne | cinderv3     | volumev3       | True    | internal  | http://10.20.20.50:8776/v3/%(tenant_id)s                     |
| 73c141fa9d9a4c8f8b0eba2d48ca4c34 | RegionOne | ceilometer   | metering       | True    | public    | http://openstack.net:8777                    |
| 7bae35ecb0694beaade731b5db1e75d6 | RegionOne | cinderv3     | volumev3       | True    | public    | http://openstack.net:8776/v3/%(tenant_id)s   |
| 824756626b1246ccbd0f9ecd5c086045 | RegionOne | heat         | orchestration  | True    | public    | http://openstack.net:8004/v1/%(tenant_id)s   |
| 85db8bfcf4f14107b6993effe9014021 | RegionOne | heat-cfn     | cloudformation | True    | admin     | http://10.20.20.50:8000/v1                                   |
| 8e0fcc4c280c4dadac8a789bca5e95f7 | RegionOne | aodh         | alarming       | True    | admin     | http://10.20.20.50:8042                                      |
| a9828ebb780e45348e294c43bdee94f2 | RegionOne | gnocchi      | metric         | True    | public    | http://openstack.net:8041                    |
| a9988892a303465d8ddd42874d639849 | RegionOne | glance       | image          | True    | internal  | http://10.20.20.50:9292                                      |
| aa44ab02530c455f8c1a66b4cf6afd84 | RegionOne | heat-cfn     | cloudformation | True    | internal  | http://10.20.20.50:8000/v1                                   |
| af53025f7f894a7bb4317f98dd3cdc58 | RegionOne | keystone     | identity       | True    | admin     | http://10.20.20.50:35357/v3                                  |
| b2b16f433ff842178ffa10ee35fb48f2 | RegionOne | glance       | image          | True    | public    | http://openstack.net:9292                    |
| bafd2cb96b6347ebb51b672f80b180b2 | RegionOne | neutron      | network        | True    | admin     | http://10.20.20.50:9696                                      |
| bbb22e3dc2f74c43a2fda9b835ad52b7 | RegionOne | ceilometer   | metering       | True    | admin     | http://10.20.20.50:8777/                                     |
| d10c4761286541d492e7c156fa3689ae | RegionOne | glance       | image          | True    | admin     | http://10.20.20.50:9292                                      |
| dbd192acd5fe4049b0f87fc346fa5368 | RegionOne | cinderv3     | volumev3       | True    | admin     | http://10.20.20.50:8776/v3/%(tenant_id)s                     |
| e2d7f4369d124fd18e1e555ef0097a29 | RegionOne | cinder       | volume         | True    | admin     | http://10.20.20.50:8776/v1/%(tenant_id)s                     |
| ee613984c72f487fbd928c12b4a8a26a | RegionOne | ceilometer   | metering       | True    | internal  | http://10.20.20.50:8777                                      |
| f4bdf7fcb0884e5f88f0c8957764b538 | RegionOne | cinderv2     | volumev2       | True    | admin     | http://10.20.20.50:8776/v2/%(tenant_id)s                     |
| f52ec71b2c3d43c98679187dcabae0d2 | RegionOne | heat-cfn     | cloudformation | True    | public    | http://openstack.net:8000/v1                 |
| f9b58e818b5b44dc8903ee0a3fe5e5b1 | RegionOne | aodh         | alarming       | True    | internal  | http://10.20.20.50:8042                                      |
+----------------------------------+-----------+--------------+----------------+---------+-----------+--------------------------------------------------------------+
root@infra1-utility-container-836672cc:~#

9. Debugging and Troubleshooting

9.1 Logs

If you need to debug anything, all the logs are inside the Rsyslog container.

root@openstack1:/opt/openstack-ansible/scripts# ./inventory-manage.py -l|grep rsyslog
| infra4_rsyslog_container-eeff7432              | None     | rsyslog                  | infra4        | None           | 10.20.20.75   | None                   |
root@openstack1:/etc/openstack_deploy# ssh infra4_rsyslog_container-eeff7432
root@infra4-rsyslog-container-eeff7432:~# cd /var/log/log-storage/
root@infra4-rsyslog-container-eeff7432:/var/log/log-storage# ls
infra1                                         infra1.openstack.local                         infra2-nova-conductor-container-bff97b73    infra3-nova-api-metadata-container-ca3b0688
infra1-cinder-api-container-0936e945           infra1-rabbit-mq-container-7016f414            infra2-nova-console-container-99dbfa6e      infra3-nova-api-os-compute-container-22b11b8e
infra1-cinder-scheduler-container-fc83ebc4     infra1-repo-container-745bad06                 infra2-nova-scheduler-container-403753e3    infra3-nova-api-placement-container-27eda8e8
infra1-galera-container-3553e5ad               infra2                                         infra2.openstack.local                      infra3-nova-conductor-container-d384d6eb
infra1-glance-container-0525f6a0               infra2-cinder-api-container-b6fe5dba           infra2-rabbit-mq-container-3a8e985a         infra3-nova-console-container-f57a1e27
infra1-heat-apis-container-1f4677cd            infra2-cinder-scheduler-container-a757f7fd     infra2-repo-container-495c6de1              infra3-nova-scheduler-container-8de84033
infra1-heat-engine-container-490a2be3          infra2-galera-container-e1d4feff               infra3                                      infra3.openstack.local
infra1-horizon-container-3e18da0f              infra2-glance-container-987db1c7               infra3-cinder-api-container-fe0b3819        infra3-rabbit-mq-container-385f0d02
infra1-keystone-container-33cfd6e8             infra2-heat-apis-container-ace23717            infra3-cinder-scheduler-container-141627ce  infra3-repo-container-aa03d50a
infra1-neutron-agents-container-2c8cfc26       infra2-heat-engine-container-3b00a93c          infra3-galera-container-bc9c86db            infra4
infra1-neutron-server-container-3a820c3a       infra2-horizon-container-c88d4a4f              infra3-glance-container-ec5c3ea9            infra4.openstack.local
infra1-nova-api-metadata-container-45e5bd4c    infra2-keystone-container-b01d0dfa             infra3-heat-apis-container-65685847         stor1
infra1-nova-api-os-compute-container-bba3453e  infra2-neutron-agents-container-da604256       infra3-heat-engine-container-9890e6ca       stor2
infra1-nova-api-placement-container-a6256190   infra2-neutron-server-container-1fb85a80       infra3-horizon-container-88c85174           stor3
infra1-nova-conductor-container-21b30762       infra2-nova-api-metadata-container-fbd1f355    infra3-keystone-container-0c3a530e          stor4
infra1-nova-console-container-86f55a54         infra2-nova-api-os-compute-container-1eea962e  infra3-neutron-agents-container-baf9fc93
infra1-nova-scheduler-container-b84c323a       infra2-nova-api-placement-container-5cc21228   infra3-neutron-server-container-357febaa
root@infra4-rsyslog-container-eeff7432:/var/log/log-storage#

9.2 Notes

  • Add to /etc/hosts the VIP of openstack on all servers
  • If you encounter the error below:
‘rb’)\nIOError: [Errno 2] No such file or directory: ‘/var/www/repo/pools/ubuntu-16.04-x86_64/ldappool/ldappool-2.1.0-py2.py3-none-any.whl’“, “stdout”: “Collecting aodh from

Enter the repo container that generate the error and run the commands below:

cd /var/www/repo/
mkdir pools/ubuntu-16.04-x86_64
cp -r -p -v ubuntu-16.04-x86_64/* pools/ubuntu-16.04-x86_64/
cd pools/
chown nginx.www-data * -R

9.3 - How to restart all containers

lxc-system-manage system-rebuild

OSA is pretty robust to deploy in production and AiO environments. Considering the how complex an OpenStack deploy can be, they did a good job and it is easy to investigate issues since all the logs are in a single place.