SAPHanaSR-ScaleOut

NAME¶

SAPHanaSR-ScaleOut - Tools for automating SAP HANA system replication in scale-out setups.

DESCRIPTION¶

Overview

The SAPHanaSR-ScaleOut package provides resource agents (RA) and tools for setting up and managing automation of SAP HANA system replication (SR) in scale-out setups. System replication will help to replicate the database data from one site to another site in order to compensate for database failures. With this mode of operation, internal SAP HANA high-availability (HA) mechanisms and the Linux cluster have to work together.

An HANA scale-out setup already is, to some degree, an HA cluster on its own. The HANA is able to replace failing nodes with standby nodes or to restart certain sub-systems on other nodes. As long as the HANA landscape status is not "ERROR" the Linux cluster will not act. The main purpose of the Linux cluster is to handle the take-over to the other site. Only if the HANA landscape status indicates that HANA can not recover from the failure and the replication is in sync, then Linux will act. As an exception, the Linux cluster will react if HANA moves the master nameserver role to another candidate. SAPHanaController is also able to restart former failed worker nodes as standby. In addition to the SAPHanaTopology RA, the SAPHanaSR-ScaleOut solution uses a "HA/DR providers" API provided by HANA to get informed about the current state of the system replication.

Note: To automate SAP HANA SR in scale-up setups, please use the package SAPHanaSR.

Scenarios

In order to illustrate the meaning of the above overview, some important situations are described below. This is not a complete description of all situations.

1. Local start of standby node
The Linux cluster will react if HANA moves any worker node (including the master nameserver role) to another candidate. If the failed node or instance is available in the cluster and switched to HANA standby role, the Linux cluster will restart the SAP HANA local framework so this node could be used for future failovers. This is one exception from the general rule, that the Linux cluster does nothing as long as the HANA landscape status is not "ERROR".

2. Prevention against dual-primary
A primary absolutely must never be started, if the cluster does not know anything about the other site. On initial cluster start, the cluster needs to detect a valid HANA system replication setup, including system replication status (SOK) and last primary timestamp (LPT). This is neccessary to ensure data integrity.

The rational behind this is shown in the following scenario:
1. site_A is primary, site_B is secondary - they are in sync.
2. site_A crashes (remember the HANA ist still marked primary).
3. site_B does the take-over and runs now as new primary.
4. DATA GETS CHANGED ON NODE2 BY PRODUCTION
5. The admin also stops the cluster on site_B (we have two HANAs both
internally marked down and primary now).
6. What, if the admin would now restart the cluster on site_A?
6.1 site_A would take its own CIB after waiting for the initial fencing
time for site_B.
6.2 It would "see" its own (cold) primary and the fact that there was a
secondary.
6.3 It would start the HANA from point of time of step 1.->2.
(the crash), so all data changed inbetween would be lost.
This is why the Linux cluster needs to enforce a restart inhibit.

There are two options to get back both, SAP HANA SR and the Linux cluster, into a fully functional state:
a) The admin starts both nodes again
b) In the situation where the site_B is still down, the admin starts
the primary on site_A manually.
The Linux cluster will follow this administrative decision. In both cases the administrator should register and start a secondary as soon as posible. This avoids a full log partition with consequence of a DATABASE STUCK.

3. Automatic registration as secondary after site failure and takeover
The cluster can be configured to register a former primary database automatically as secondary. If this option is set, the resource agent will register a former primary database as secondary during cluster/resource start.

4. Site take-over not preferred over local re-start
SAPHanaSR-ScaleOut allows to configure, if you prefer to takeover to the secondary after the primary landscape fails. The alternative is to restart the primary landscape, if it fails and only to takeover when no local restart is possible anymore. This can be tuned by SAPHanaController(7) parameters.
The current implementation only allows to takeover in case the landscape status reports 1 (ERROR). The cluster will not takeover, when the SAP HANA still tries to repair a local failure.

5. Recovering from failure of master nameserver
If the master nameserver of an HANA database system fails, the HANA will start the nameserver on another node. Therefore usually up to two nodes are configured as additional nameserver candidates. At least one of them should be a standby node to optimize failover time. The Linux cluster will detect the change and move the IP address to the new active master nameserver.

Implementation

The two HANA database systems (primary and secondary site) are managed by the same single Linux cluster. The maximum number of nodes in that single Linux cluster is given by the Linux cluster limit. An odd number of nodes is needed to handle split-brain situations automatically in stretched clusters.

The HANA consists of two sites with same number of nodes each. There are no HANA nodes outside the Linux cluster. An additional Linux cluster node is used as majority maker for split-brain situations. This dedicated node does not need to have HANA installed and must not run any SAP HANA resources for the same SID.

A common STONITH mechanism is set up for all nodes across all the sites.

Since the IP address of the primary HANA database system is managed by the cluster, only that single IP address is needed to access any nameserver candidate.

Best Practice

* Use two independent corosync rings, at least one of them on bonded network. Resulting in at least three physical links. Unicast is preferred.

* Use Stonith Block Device (SBD), shared LUNs across all nodes on all (three) sites. Of course, together with hardware watchdog.

* Align all timeouts in the Linux cluster with the timeouts of the underlying storage and multipathing.

* Check the installation of OS and Linux cluster on all nodes before doing any functional tests.

* Carefully define, perform, and document tests for all scenarios that should be covered.

* Test HANA HA and SR features without Linux cluster before doing the overall cluster tests.

* Test basic Linux cluster features without HANA before doing the overall cluster tests.

* Be patient. For detecting the overall HANA status, the Linux cluster needs a certain amount of time, depending on the HANA and the configured intervalls and timeouts.

* Before doing anything, always check for the Linux cluster's idle status, left-over migration constraints, and resource failures as well as the HANA landscape status, and the HANA SR status.

REQUIREMENTS¶

For the current version of the package SAPHanaSR-ScaleOut, the support is limited to the following scenarios and parameters:

1. HANA scale-out cluster with system replication. The two HANA database systems (primary and secondary site) are managed by the same single Linux cluster. The maximum number of nodes in that single Linux cluster is given by the Linux cluster limit. An odd number of nodes is needed to handle split-brain situations automatically. A dedicated cluster node might be used as majority maker.

2. Technical users and groups such as sidadm are defined locally in the Linux system.

3. Strict time synchronization between the cluster nodes, f.e. NTP.

4. For scale-out there is no other SAP HANA system (like QA) on the replicating node which needs to be stopped during take-over.

5. Only one system replication for the SAP HANA database.

6. Both SAP HANA database systems have the same SAP Identifier (SID) and Instance Number.

7. Besides SAP HANA you need SAP hostagent to be installed and started on your system.

8. Automated start of SAP HANA database systems during system boot must be switched off.

9. For scale-out, the current resource agent supports SAP HANA in system replication beginning with HANA version 1.0 SPS 11 patch level 112.02. Older versions do not provide the srHook method srConnectionChanged().

10. For scale-out, if the shared storage is implemented with another cluster, that one does not interfere with the Linux cluster. All three clusters (HANA, storage, Linux) have to be aligned.

11. The RAs SAPHanaController and SAPHanaToplogy need to be installed on all cluster nodes, even the majority maker.

12. For scale-out, the current resource agent supports SAP HANA in system replication beginning with HANA version 1.0 SPS 11 patch level 112.

BUGS¶

In case of any problem, please use your favourite SAP support process to open a request for the component BC-OP-LNX-SUSE. Please report any other feedback and suggestions to feedback@suse.com.

AUTHORS¶

F.Herschel, L.Pinne.

COPYRIGHT¶

(c) 2015-2017 SUSE Linux GmbH, Germany.
The package SAPHanaSR-ScaleOut comes with ABSOLUTELY NO WARRANTY.
For details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html

30 Jun 2017

Source file:	SAPHanaSR-ScaleOut.7.en.gz (from resource-agents-sap-hana-scaleout 0.163.2-7.el7_7.1)
Source last updated:	2019-08-17 17:44:26
Converted to HTML:	2023-06-08 08:52:49