11g RAC Transparent Application Failover (TAF)

This example illustrates Transparent Application Failover (TAF) in a two node AIX 11g RAC configuration.

The database name is racdb and the two instances are racdb1 and racdb2.

Node 1 – middba1 which hosts instance racdb1.
Node2 – middba2 which hosts instance racdb2

We have defined two services racdb1 and racdb2. For service racdb1 the preferred instance is racdb1 and
for service racdb2 the preferred instance is racdb2.

The following srvctl commands illustrates the same.

middba1:/u01/oracle/dump> srvctl status service -d racdb -s racdb1
Service racdb1 is running on instance(s) racdb1

middba1:/u01/oracle/dump> srvctl config service -d racdb -s racdb1
racdb1 PREF: racdb1 AVAIL: racdb2

middba1:/u01/oracle/dump> srvctl status service -d racdb -s racdb2
Service racdb2 is running on instance(s) racdb2

middba1:/u01/oracle/dump> srvctl config service -d racdb -s racdb2
racdb2 PREF: racdb2 AVAIL: racdb1

From a SQL*PLUS client, we establish a session as user SYSTEM using the service racdb1. Note the machine name where this service is running from.

testdb:/u01/oracle> sqlplus system/xxx@racdb1

SQL*Plus: Release 11.1.0.6.0 - Production on Fri Aug 14 10:37:03 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters and Real Application Testing options

SQL> select host_name from v$instance;

HOST_NAME
----------------------------------------------------------------
middba1

We then execute a long running SELECT statement

SQL> select * from sh.sales;

While this statement is running we reboot the node middba1 where this service is currently connected to.

We will note that the screen will momentarily freeze while the service gets relocated to the other functional node racdb2 and then the query continues from where it got interrupted. The session will not get disconnected – so for the end user it is transparent from which instance the query is being serviced.

If we check the status of the cluster resources, we will find that the VIP from node middba1 has now relocated to middba2. Other resources that were originally running from node middba1 are now offline.

middba2:/u01/oracle> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    OFFLINE
ora....A1.lsnr application    ONLINE    OFFLINE
ora....ba1.gsd application    ONLINE    OFFLINE
ora....ba1.ons application    ONLINE    OFFLINE
ora....ba1.vip application    ONLINE    ONLINE    middba2
ora....SM2.asm application    ONLINE    ONLINE    middba2
ora....A2.lsnr application    ONLINE    ONLINE    middba2
ora....ba2.gsd application    ONLINE    ONLINE    middba2
ora....ba2.ons application    ONLINE    ONLINE    middba2
ora....ba2.vip application    ONLINE    ONLINE    middba2
ora.racdb.db   application    ONLINE    ONLINE    middba2
ora....cdb1.cs application    ONLINE    ONLINE    middba2
ora....b1.inst application    ONLINE    OFFLINE
ora....db1.srv application    ONLINE    ONLINE    middba2
ora....cdb2.cs application    ONLINE    ONLINE    middba2
ora....b2.inst application    ONLINE    ONLINE    middba2
ora....db2.srv application    ONLINE    ONLINE    middba2

If we check the status of the service, we will find that the service racdb1 is now running on node middba2 connecting to the instance racdb2.

middba2:/u01/oracle> srvctl status service -d racdb -s racdb1
Service racdb1 is running on instance(s) racdb2

Once the node middba1 which was rebooted finally does come online, we will find that the VIP will relocate
to its original node middba1 once the cluster services are brought online after the reboot.

middba2:/u01/oracle> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    middba1
ora....A1.lsnr application    ONLINE    ONLINE    middba1
ora....ba1.gsd application    ONLINE    ONLINE    middba1
ora....ba1.ons application    ONLINE    ONLINE    middba1
ora....ba1.vip application    ONLINE    ONLINE    middba1
ora....SM2.asm application    ONLINE    ONLINE    middba2
ora....A2.lsnr application    ONLINE    ONLINE    middba2
ora....ba2.gsd application    ONLINE    ONLINE    middba2
ora....ba2.ons application    ONLINE    ONLINE    middba2
ora....ba2.vip application    ONLINE    ONLINE    middba2
ora.racdb.db   application    ONLINE    ONLINE    middba2
ora....cdb1.cs application    ONLINE    ONLINE    middba2
ora....b1.inst application    ONLINE    ONLINE    middba1
ora....db1.srv application    ONLINE    ONLINE    middba2
ora....cdb2.cs application    ONLINE    ONLINE    middba2
ora....b2.inst application    ONLINE    ONLINE    middba2
ora....db2.srv application    ONLINE    ONLINE    middba2

Note however, that the service racdb1 will continue running on node middba2 (instance racdb2)even though the original node which it was running on middba1 has been brought online.

We will have to relocate the service back to its original node using the srvctl command

middba1:/u01/oracle> srvctl relocate service -d racdb -s racdb1 -i racdb2 -t racdb1