Julio's Blog

High Availability and Pacemaker 101!

In this blog post, we will talk a little bit about High Availability and a little bit more about Pacemaker. Here are some of the topics of this post:

Introduction to High Availability (HA) and Clustering.

What is High Availability (HA)?

In IT, High Availability refers to a system or component that is continuously operational for a desirably long length of time.

3 cores principles to HA

How High Availability is implemented on RHEL 7?


What is Clustering?

A cluster is a set of computers working together on a single task. Which task is performed, and how that task is performed, differs from cluster to cluster.

There are four (4) different kinds of clusters:

High-availability clusters: Known as an HA cluster or failover cluster, their function is to keep running services as available as they can be. You could find them in two main configurations:

Load-balancing clusters: All nodes run the same software and perform the same task at the same time and the requests from the clients are distributed between all the nodes.

Compute clusters: Also know as high-performance computing (HPC) cluster. In these clusters, tasks are divided into smaller chunks, which then get computed on different nodes.

Storage clusters: All nodes provide a single cluster file system that will be used by clients to read and write data simultaneously.

Benefits of Highly Available applications

In two words: "Application resiliency".

How HA is implemented on RHEL 7?

Red Hat Enterprise Linux High Availability Add-On.

The High Availability Add-On consists of the following major components:

Cluster infrastructure: Provides fundamental functions for nodes to work together as a cluster: configuration file management, membership management, lock management, and fencing.

High-availability Service Management: Provides failover of services from one cluster node to another.

Cluster administration tools: Configuration and management tools for setting up, configuring, and management.

To provide the above services multiple software components are required on the cluster nodes.


The cluster infrastructure software is provided by Pacemaker and performs the next set of functions:

Cluster software:

pacemaker: It's responsible for all cluster-related activities, such as monitoring cluster membership, managing the services and resources, and fencing cluster members. The RPM contains three (3) important components:

corosync: This is the framework used by Pacemaker for handling communication between the cluster nodes.

pcs: Provides a command-line interface to create, configure, and control every aspect of a Pacemaker/corosync cluster.

Requirements and Support.

Here are some requirements and limits for Pacemaker.

Number of Nodes:

Cluster location:

Single site: A cluster setup where all cluster members are in the same physical location, connected by a local area network. (Supported).

Multisite: Two clusters, one active and one for disaster recovery. Failover for multisite clusters must be managed manually. (Supported).

Strech (or) Geo Clusters: Clusters stretched out over multiple physical locations. (Required architecture review to be supported).

Fencing: Fencing is the process of cutting a node off from shared storage. This can be done by power cycling a node or disabling communication to the storage level.

WARNING: Fencing is required for all nodes in the cluster, either via power fencing, storage fencing, or a combination of both.

NOTE: If the cluster will use integrated fencing devices like ILO or DRAC, the systems acting as cluster nodes must power off immediately when a shutdown signal is received, instead of initiating a clean shutdown.


Virtual Machines supported as nodes and resources.

NOTE: VM as a resource means that virtualization host is participating in a cluster and the virtual machine is a resource that can move between cluster nodes.




RHN Channels Required:

Building a 3 node Apache cluster.

Now, we will build a basic pacemaker cluster serving Apache. To do this we will need 3 systems running RHEL 7.X.

Preparing the systems :

All these actions will happen on the all the cluster nodes.

Configure Firewall

Let's start by configuring FirewallD to allow traffic.

firewall-cmd --permanent --add-service=high-availability  
firewall-cmd --reload  

Install required software

yum install pcs fence-agents-all  

The pcs package requires corosync and pacemaker, so all your software be installed by doing this._

Enable pcsd

pcsd provides cluster configuration sync and the web front end. Needs to be enable in all the servers.

systemctl enable pcsd; systemctl start pcsd  

Set the ha cluster user password

After the software install, a user hacluster will be created. This user will be used for all cluster communication pcsd.

NOTE: You should use the same password across all cluster nodes for this user. If you echo your password like is show below, clear your history afterward :) .

echo password | passwd hacluster --stdin  

Configuring DNS

You should be able to resolve all the nodes in the cluster by name. On this example, we are going to use host files to define our 3 nodes. This is what I added to my hosts files (/etc/hosts).    node1   node1.example.local    node2   node2.example.local    node3   node3.example.local  

Preparing the systems

Authenticate pcsd.
pcsd requires that the cluster nodes authenticate, we are going to use the ha cluster user and password. This actions only needs to happen on one of the nodes.

[[email protected]] pcs cluster auth node1.example.local node2.example.local node3.example.local

Creating the cluster

Let's create the cluster:

pcs cluster setup --name demo-cluster --start node1.example.local node2.example.local node3.example.local  

An important step will be to enable the cluster services on all nodes. By default, if a node is rebooted will not join the cluster until started manually. To avoid this do:

[[email protected]] pcs cluster enable --all

Check the cluster status:

[[email protected]] pcs cluster status

Configuring Fencing

This is a critical step, you must have fencing on the cluster! In this example, I am assuming that we are using KVM for the demo and we will use fence_xvm.

[[email protected]] pcs stonith create fence_node1_vm fence_xvm port="node1" pcmk_host_list="node1.example.local"

[[email protected]] pcs stonith create fence_node2_vm fence_xvm port="node2" pcmk_host_list="node2.example.local"

[[email protected]] pcs stonith create fence_node3_vm fence_xvm port="node3" pcmk_host_list="node3.example.local"

Open the port for the fencing agent fencing:

[[email protected]] for i in `seq 1 3`; do ssh [email protected]$i.example.local firewall-cmd --add-port=1229/tcp --permanent; done

[[email protected]] for i in `seq 1 3`; do ssh [email protected]$i.example.local firewall-cmd reload; done

Check fencing status.

[[email protected]] pcs stonith show


Clustered services consist of one or more resources. A resource can be:

Also usually the resources are a member of resource groups.

Creating the resources for our demo-cluster

First let's create the resource group for our Apache cluster, we are going to name it personal-web and will have a floating IP.

[[email protected]] pcs resource create floatingip IPaddr2 ip= cidr_netmask=24 --group personal-web

Install Apache (httpd)

[[email protected]] yum install httpd -y

Create the web-1 resource using Apache and put it on the personal-web group.

[[email protected]] pcs resource create web-1 apache --group personal-web

Let's check that the created resources are present in the cluster configuration and check the cluster status.

[[email protected]] pcs resource show

[[email protected]] pcs status

Create a file in /var/www/html/ with this content:

echo "Website responding from $HOSTNAME" > /var/www/html/index.html  

Manipulating the cluster

Here are some commands that will help you to manage the cluster.

Start and Stop the cluster

to do it in all nodes use the --all switch.

[[email protected]] pcs cluster start

[[email protected]] pcs cluster stop

Stop the cluster service on a specific remote node

[[email protected]] pcs cluster stop node2.example.com

Disable cluster on reboot on a node

[[email protected]] pcs cluster disable

How to add a node to the cluster

[[email protected]] pcs cluster node add new.example.com

On the new node, you need to Authenticate the rest of the cluster. Also, you will need to add a fence device for it too.

[[email protected]] pcs cluster auth

How to remove a node to the cluster

[[email protected]] pcs cluster node remove new.example.com

[[email protected]] pcs stonith remove fence_newnode new.example.com 

Set a node in standby

(This bans the node from hosting resources).

[[email protected]] pcs cluster node standby new.example.com

Set the cluster in standby

[[email protected]] pcs cluster standby --all

Unset standby

[[email protected]] pcs cluster unstandby --all

Displaying quorum status

[[email protected]] corosync-quorumtool

Slides: https://github.com/juliovp01/txlf-2015-HA-slides

High Availability Add-On overview: https://access.redhat.com/documentation/en-US/RedHatEnterpriseLinux/7/html/HighAvailabilityAdd-OnOverview/

Cluster Labs website: http://clusterlabs.org/

Julio is a Principal Cloud Architect at Red Hat working on Linux, Virtualization, Cloud (OpenStack), and Containers.Julio was born in Cuba but now calls home Austin, TX.