Steps To Shutdown(stop)/Startup(start) CRS, OHAS, ASM, RDBMS & ACFS Services on a RAC Cluster 11.2 & 12.1 Configuration (Doc ID 1355977.1)

Additional Steps Required When ACFS filesystem are involved: Community Discussions References APPLIES TO: Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.2 [Release 11.2 to 12.1] Information in this document applies to any platform. ***Checked for relevance on 21-Jan-2014*** GOAL The present document provides in detail the steps to shutdown/startup The CRS, OHAS, ASM & RDBMS Services on a RAC Cluster 11.2 & 12.1 configuration: SOLUTION 1) Connect to node #1, then please check if the CRS/OHAS & services are enabled to autostart as follow (repeat this step on each node): # $GRID_ORACLE_HOME/bin/crsctl config crs 2) If not, then you can enable it as follow (repeat this step on each node): # $GRID_ORACLE_HOME/bin/crsctl enable crs 3) Shutdown the services on each node as follow: # $GRID_ORACLE_HOME/bin/crsctl stop crs 4) Verify the services were/are down (repeat this step on each node): # $GRID_ORACLE_HOME/bin/crsctl status resource -t 5) Then start the services on node #1 as follow (only on first node): # $GRID_ORACLE_HOME/bin/crsctl start crs 6) Wait 1 minute, then validate the services started & diskgroups were mounted (only on first node): # $GRID_ORACLE_HOME/bin/crsctl status resource -t 7) Then start the services on node #2 as follow: # $GRID_ORACLE_HOME/bin/crsctl start crs 8) Wait 1 minute, then validate the services started & diskgroups were mounted on node #2: # $GRID_ORACLE_HOME/bin/crsctl status resource -t 9) If there are more nodes in the RAC, then repeat the same steps (7-8). 10) Then check the status of the clusterware globally as follows: # crsctl check cluster -all Sample output: [root@asmgrid1 ~]# crsctl check cluster -all ************************************************************** asmgrid1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** asmgrid2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** Note 1: Alternatively, you can stop and start the cluster globally as follows: Stop: [root@asmgrid1 ~]# crsctl stop cluster -all [-f] Start: [root@asmgrid1 ~]# crsctl start cluster -all Note 2: For RAC ACFS configurations please check the next information: http://docs.oracle.com/cd/E11882_01/readmes.112/e22488/toc.htm#CACCIEID 2.19.3Oracle ACFS and Oracle Clusterware Stack Shut Down When attempting to shut down Oracle Clusterware, the Oracle Clusterware stack may report that it did not successfully stop on selected nodes (reference Bug 8651848). If the database home is on Oracle ACFS, then you may receive the following error: CRS-5014: Agent orarootagent.bin timed out starting process acfsmount for action This error can be ignored. Alternatively, the Oracle Clusterware stack may report that it did not successfully stop on selected nodes due to the inability to shut down the Oracle ACFS resources. If this occurs, take the following steps: Ensure that all file system activity to Oracle ACFS mount points is quiesced by shutting down programs or processes and retry the shutdown. If the ora.registry.acfs resource check function times out, or the resource exhibits a state of UNKNOWN or INTERMEDIATE, then this may indicate an inability to access the Oracle Cluster Registry (OCR). The most common cause of this is a network failure. The commands "acfsutil registry" and "ocrcheck" may give you a better indicator of the specific error. Clear this error and attempt to stop Oracle Clusterware again. Additional Steps Required When ACFS filesystem are involved: Note 3: If ACFS filesystems are associated with this ASM Cluster configuration, then the ACFS filesystems need to be dismounted first, this is because if you try to dismount an ACFS filesystem (or any other regular Unix/Linux filesystem) that is being used/accessed at that time (same as a regular unix/Linux filesystem) then you will get a “Resource Busy” error, therefore “crsctl stop crs” statement will fail as expected. You will need to perform the following additional steps instead: A) Dismount all the ACFS filesystems running in the cluster as follows (as root user): # srvctl stop filesystem -d volume_device_name [-n node_name] [-f] Where: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -d :The Oracle ACFS volume device name ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -n : The name of a node If you do not specify this option, then the utility stops the volume resource on all active nodes in the cluster. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -f : This option stops the file system and also stops any databases or other resources that depend on this file system. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Example: [root@asmgrid1 ~]# df -k | grep asm /dev/asm/volnew1-347 14680064 159700 14520364 2% /u01/app/grid/acfsmounts/acfsdgnew_volnew1 /dev/asm/vol_oh1-115 35651584 4590104 31061480 13% /u04acfs # srvctl stop filesystem -d /dev/asm/volnew1-347 -f # srvctl stop filesystem -d /dev/asm/vol_oh1-115 -f B) Make sure the ACFS filesystem(s) was/were dismounted on all the nodes: # df -k | grep asm C) Then perform the steps described in the following document: =)> Steps To Shutdown/Startup The CRS, OHAS, ASM & RDBMS Services on RAC 11.2 Configuration. (Doc ID 1355977.1) Note: To manually remount the ACFS filesystem back (CRS managed) the next statement needs to be executed as root user: # srvctl start filesystem -d volume_device_name [-n node_name] ============================================================= https://www.youtube.com/watch?v=0iNWl4r8hmo =============== http://www.dbas-oracle.com/2013/07/Shutdown-and-Start-Oracle-Real-Application-Clusters-RAC-Database.html Shutdown and Start Oracle Real Application Clusters Database is any activity, which is performed by DBA for Applying CRS patch, Scheduled maintenance and adding more products in to Real Application Cluster. Since, RAC involves so many components like Database instance, ASM instance, Node application and CRS services, So Start or Shut down Oracle Real Application Clusters Database involved series of steps. An important factor to consider by DBA while start or shutdown Oracle RAC Database is that steps must be preformed in specific suggested order otherwise database can go into inconsistency. In this post, I will explain each step one by one and after performing steps we will also cross check for same operation. Let's start with. How to Shutdown Oracle Real Application Clusters Database ? Here, We will start with stopping resources from Enterprise Manager Grid Control to Oracle Cluster ware or CRS process and will end at stopping CRS services. Here I am taking an example of Two node RAC database to perform all operations. Steps followed are as: 1. Shutdown Oracle Home process accessing database. 2. Shutdown RAC Database Instances on all nodes. 3. Shutdown All ASM instances from all nodes. 4. Shutdown Node applications running on nodes. 5. Shut down the Oracle Cluster ware or CRS. 1. Shutdown Oracle Home process accessing database: There could be oracle and non Oracle application which access database, So first step is to stop all the applications or DBA should inform application owner and he should stop all applications accessing Database. DBA should also stop Oracle application like Enterprise Manager Grid Control and Database Listener which access database for monitoring and database connections. [oracle@database ~]$ emctl stop dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0 Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. https://database.example.com:5500/em/console/aboutApplication Stopping Oracle Enterprise Manager 11g Database Control ... ... Stopped. [grid@node1 bin]$ srvctl stop listener -n node1 [grid@node1 bin]$ srvctl status listener -n node1 Listener LISTENER is enabled on node(s): node1 Listener LISTENER is not running on node(s): node1 2. Shutdown RAC Database Instances on all nodes: Suppose DBA has two node RAC database, So he has to stop all instances from all DB nodes. Here, I am taking an example of two node RAC. First, I am checking on which server database is running and then stopping and verify for the same. Syntax: srvctl stop database -d {databasename} [oracle@node2 ~]$ srvctl status database -d oradb Instance oradb1 is running on node node1 Instance oradb2 is running on node node2 [oracle@node2 ~]$ srvctl stop database -d oradb [oracle@node2 ~]$ srvctl status database -d oradb Instance oradb1 is not running on node node1 Instance oradb2 is not running on node node2 We just need to execute one command from any one of the server having database and it will stop all database instances on all servers. If you have more than one database configured on Nodes, then Database Administrator has to execute this command for each database. 3. Shutdown All ASM instances from all nodes: Next DBA has to shut down an ASM instance which are used to acess database, enter the following command, where node is the name of the node where the ASM instance is running Syntax: srvctl stop asm -n {node} [grid@node2 oracle]# srvctl stop asm -n node1 -f [grid@node2 oracle]# srvctl stop asm -n node2 -f [grid@node2 oracle]# srvctl status asm -n node1 ASM is not running on node1 [grid@node2 oracle]# srvctl status asm -n node2 ASM is not running on node2 Sometimes, Database administrator face some issues in stopping ASM instance, In that case use "-f" option to forcefully shutdown ASM instances. 4. Shutdown Node applications running on nodes: To stop node applications running on a node, enter the following command, where node is the name of the node where the applications are running [grid@node2 oracle]# srvctl stop nodeapps -n node1 -f [grid@node2 oracle]# srvctl status nodeapps -n node1 VIP node1-vip is enabled VIP node1-vip is running on node: node1 Network is enabled Network is running on node: node1 GSD is disabled GSD is not running on node: node1 ONS is enabled ONS daemon is running on node: node1 Repeat same command for all nodes one by one. If you face any issue in stopping node applications use "-f" as force option to stop applications. 5. Shut down the Oracle Clusterware or CRS: In this command all CRS related process will be stopped. This is the only command which needs to be executed by "root" user on all database nodes. [root@node1 bin]# crsctl check cluster -all ************************************************************** node1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************* node2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [root@node1 bin]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1' CRS-2673: Attempting to stop 'ora.crsd' on 'node1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'node1' CRS-2673: Attempting to stop 'ora.node2.vip' on 'node1' ------------------------------------------------- ------------------------------------------------- ------------------------------------------------- CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'node1' CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1' CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@node1 bin]# crsctl check cluster -all CRS-4639: Could not contact Oracle High Availability Services CRS-4000: Command Check failed, or completed with errors. DBA can see Now RAC is completely down. Now you can move ahead with your patching, maintenance activity. Next let's see how to start Oracle RAC cluster database. How to Start Oracle Real Application Clusters Database ? In Starting Real Application Clusters Database, We will also follow a set of sequence. The order will be just opposite to Shutting down the Real Application Clusters Database. 1. Start Oracle Clusterware or CRS. 2. Start Node applications running on nodes. 3. Start All ASM instances from all nodes. 4. Start RAC Database Instances on all nodes. 5. Start Oracle Home process accessing database. 1. Start Oracle Clusterware or CRS: CRS starts automatically when you start or restart Server, but Here DBA has manually shutdown the CRS So, he has to start it manually. This is the only command which needs to be executed by "root" user. Database Administrator should execute this command on all nodes. [root@node1 bin]# crsctl start crs CRS-4123: Oracle High Availability Services has been started [root@node2 bin]# crsctl check cluster -all ************************************************************** node1: CRS-4535: Cannot communicate with Cluster Ready Services CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** node2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** Here, DBA can see "CRS-4639: Could not contact Oracle High Availability Services" or "CRS-4535: Cannot communicate with Cluster Ready Services" messages. Wait 5 minutes and then again check with "crsctl check cluster -all" command. This time Database administrator will get "CRS-4537: Cluster Ready Services is online". If still same issue DBA can start ora.crsd process to resolve this issue. Below is the command [root@node1 bin]# crsctl start res ora.crsd -init [root@node1 bin]# crsctl check cluster -all ************************************************************** node1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** node2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** Now, Cluster is up and running. DBA can also use "ps -ef | grep d.bin" to check cluster status. 2. Start Node applications running on nodes: To start node applications running on a node, enter the following command, where node is the name of the node where the applications are running [grid@node1 bin]$ srvctl start nodeapps -n node1 [grid@node1 bin]$ srvctl status nodeapps -n node1 VIP node1-vip is enabled VIP node1-vip is running on node: node1 Network is enabled Network is running on node: node1 GSD is disabled GSD is not running on node: node1 ONS is enabled ONS daemon is running on node: node1 DBA has to execute this command for each node to start Real Application Clusters Cluster database. 3. Start All ASM instances from all nodes: Next DBA has to start all ASM instances which are used to access database, enter the following command, where node is the name of the node where the ASM instance has to start. [grid@node1 bin]$ srvctl start asm -n node1 [grid@node1 bin]$ srvctl status asm -n node1 ASM is running on node1 DBA has to start ASM instance on all database nodes. 4. Start RAC Database Instances on all nodes: Now, We will start database instances on database nodes to access data. [grid@node1 bin]$ srvctl start database -d oradb [grid@node1 bin]$ srvctl status database -d oradb Instance oradb1 is running on node node1 Instance oradb2 is running on node node2 Now database is up and running on both DB nodes. 5. Start Oracle Home process accessing database: let's start listener and OEM to access database and allow users to connect to the database. [grid@node1 bin]$ srvctl start listener -n node1 [grid@node1 bin]$ srvctl status listener -n node1 Listener LISTENER is enabled on node(s): node1 Listener LISTENER is running on node(s): node1 [oracle@database ~]$ emctl start dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0 Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. https://database.example.com:5500/em/console/aboutApplication Starting Oracle Enterprise Manager 11g Database Control .......................... . started. ------------------------------------------------------------------ Logs are generated in directory /etc/oracle/oracle/database.example.com_orcl/sysman/log Now Database is ready to user. Since, shutting down and starting up RAC database is a series of steps. SO, I tried to make a one place repository for Shutdown and Start Oracle Real Application Clusters Database. If you wants to Shutdown or Restart Oracle RAC database Server. See How to Shutdown or Restart Oracle RAC Database Server in 6 Steps ?