Ever since rolling upgrade has been introduced in the icehouse release of OpenStack. Icehouse enables the Openstack admins to upgrade controller infrastructure first, and subsequently, upgrade individual compute nodes without requiring downtime of the entire cloud to complete the upgrade.
What’s so special about this post, as there are many blogs online to talk about the live upgrades
Yes, of course! But this post mainly focuses on an OpenStack enthusiast, who is so curious about
the rolling upgrade feature and who would like to give a try in the small POC environment
before implementing it in a big scale environment.
Although the procedure is same for upgrading a large scale infrastructure, it requires few
additional steps too. Openstack admins need to handle the high availability / load balancing
environment by removing the nodes from its pool before upgrading each node to avoid the
downtime which is not the scope of this post.
As I mentioned, this post helps the Openstack admin to try the rolling upgrade in a small scale
and even smallest before trying it in the production ready setup. Yes! I mean All-in-one node
You could try the rolling upgrade from any OpenStack release starting from Icehouse. In this
post, I will guide you in upgrading from Openstack Kilo to Openstack Liberty release.
Step 1: BackUp is a Mantra
As you all are aware that the procedure for upgrading the OpenStack is not like “Slide to upgrade
option available for IOS” 🙂 . You need to backup all the possible data from the current setup.
This will help us for rolling back the failed upgrade in worst case scenario.
$ mysqldump -u root -p –all-databases > all-kilo-db_backup.sql
Use the above command to backup the entire database from your MySql DB. Depending on the
size of the DB, this step could take less or more time.
Step 2: Migrate to new package repository
This is where the real upgrading process begins. You must change the package repository from
your current release to the new one.
Follow the below steps to remove the current (Kilo) repo and add the latest liberty repo.
#rm -rf cloudarchive-kilo.list
# apt-get install software-properties-common
# add-apt-repository cloud-archive:liberty
# apt-get update
Repeat the above step2 in each node if you are trying this in multi-node OpenStack setup.
Step 3: Upgrade the packages
Now it’s time to upgrade all the OpenStack packages in one go.
Run the below command to download and install the latest version of package respected to the
newly added cloud-archive.
During this process, the package installation terminal prompt for asking whether to replace the
new configuration file from the package maintainers repo or to keep the existing file as it is.
Select the option to install/replace the package maintainer version of the configuration file.
Selecting this option will replace the existing configuration file by renaming the existing file
with .old suffix as a backup.
If the upgrade process includes a new kernel, reboot your host to activate it.
Step 4: Install the missing Package.
There may be few packages that are newly added in the latest release of the OpenStack.
For example, “python-openstackclient” package is introduced in the liberty release. But the distupgrade
would only process for upgrading the existing packages with its dependencies.
So you need to manually install those missing packages by referring the liberty installation guide
available at http://docs.openstack.org/
Step 5: Update the services configuration files
At this time, you could notice that all the services are in stop status. This is because all the
OpenStack service configuration files are not yet configured.
Again you have to refer the OpenStack installation guide to configure the OpenStack service
starting from keystone, glance, nova, neutron and so on.
Step 6: DB Sync
Before starting the service, you need to run DB Sync command for all the service to populate the
new tables in the database with respect to the upgraded release.
This is because the latest release may contain new features which will get reflected in its DB
schema. So those new tables need to be populated before starting the service of newly upgraded
Here, I have listed the DB-Sync commands for most common services in OpenStack. You could
also refer it from the installation guide.
To populate the Identity service database:
# su -s /bin/sh -c “keystone-manage db_sync” keystone
To Populate the Image service database:
# su -s /bin/sh -c “glance-manage db_sync” glance
To Populate the Compute database:
# su -s /bin/sh -c “nova-manage db sync” nova
To Populate the database:
# su -s /bin/sh -c “neutron-db-manage –config-file /etc/neutron/neutron.conf \
–config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head” neutron
Step 7: Restart the services
After successfully populating the DB, restart all the service. If the services stop working, then
check the /var/log/upstart/ directory for the appropriate service logs. There could be some
dependencies issue you need to fix it.
If there are no errors, it means you have successfully upgraded the Openstack.