runcluvfy.sh Prerequisite check for Oracle12c RAC Cluster Installation

/TapeBackup/GRID_SOFTWARE/GRID_12.1.0.2/grid/runcluvfy.sh stage -pre crsinst -n mof-pidmdb01,mof-pidmdb02 -verbose /TapeBackup/GRID_SOFTWARE/GRID_12.1.0.2/grid/runcluvfy.sh stage -post hwos -n mof-pidmdb01,mof-pidmdb02 -verbose http://myorastuff.blogspot.qa/2016/03/prerequisite-check-for-oracle12c-rac.html Prerequisite check for Oracle12c RAC Cluster Installation I am about to install Oracle12c Cluster on two node RAC. I am checking all the prerequisite before i start the cluster installation. I thought, i might be helpful for others if i post the things to be checked before we start installing the cluster software. The RAC nodes are already built and ready to install the cluster. My hostnames are ractest4 & racttest5. Here is my ip info. Step 1 : Make sure you have either root password or Oracle account with sudo to root privileges to install cluster software. It ask to enter the password in the middle of the installation. I carry root password for my installation. Step 2 : Make sure you are able to ping public ip and private ip between two nodes. Public IP [oracle@RACTEST4 grid]$ ping 192.168.56.212 PING 192.168.56.212 (192.168.56.212) 56(84) bytes of data. 64 bytes from 192.168.56.212: icmp_seq=1 ttl=64 time=1.49 ms 64 bytes from 192.168.56.212: icmp_seq=2 ttl=64 time=0.416 ms 64 bytes from 192.168.56.212: icmp_seq=3 ttl=64 time=0.411 ms 64 bytes from 192.168.56.212: icmp_seq=4 ttl=64 time=0.479 ms ^C --- 192.168.56.212 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3362ms rtt min/avg/max/mdev = 0.411/0.699/1.493/0.459 ms [oracle@RACTEST4 grid]$ [root@RACTEST5 ~]# ping 192.168.56.211 PING 192.168.56.211 (192.168.56.211) 56(84) bytes of data. 64 bytes from 192.168.56.211: icmp_seq=1 ttl=64 time=0.324 ms 64 bytes from 192.168.56.211: icmp_seq=2 ttl=64 time=0.425 ms 64 bytes from 192.168.56.211: icmp_seq=3 ttl=64 time=0.468 ms 64 bytes from 192.168.56.211: icmp_seq=4 ttl=64 time=0.436 ms ^C --- 192.168.56.211 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3469ms rtt min/avg/max/mdev = 0.324/0.413/0.468/0.055 ms [root@RACTEST5 ~]# Private IP [oracle@RACTEST4 grid]$ ping 192.168.1.212 PING 192.168.1.212 (192.168.1.212) 56(84) bytes of data. 64 bytes from 192.168.1.212: icmp_seq=1 ttl=64 time=0.387 ms 64 bytes from 192.168.1.212: icmp_seq=2 ttl=64 time=0.403 ms 64 bytes from 192.168.1.212: icmp_seq=3 ttl=64 time=0.384 ms 64 bytes from 192.168.1.212: icmp_seq=4 ttl=64 time=0.410 ms ^C --- 192.168.1.212 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3245ms rtt min/avg/max/mdev = 0.384/0.396/0.410/0.010 ms [oracle@RACTEST4 grid]$ [root@RACTEST5 ~]# ping 192.168.1.211 PING 192.168.1.211 (192.168.1.211) 56(84) bytes of data. 64 bytes from 192.168.1.211: icmp_seq=1 ttl=64 time=1.76 ms 64 bytes from 192.168.1.211: icmp_seq=2 ttl=64 time=0.443 ms 64 bytes from 192.168.1.211: icmp_seq=3 ttl=64 time=0.348 ms ^C --- 192.168.1.211 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2393ms rtt min/avg/max/mdev = 0.348/0.853/1.768/0.648 ms [root@RACTEST5 ~]# Step 3 : Make sure you see the ASM disk on both the nodes. [root@RACTEST4 ~]# oracleasm listdisks DISK1 [root@RACTEST4 ~]# [root@RACTEST5 ~]# oracleasm listdisks DISK1 [root@RACTEST5 ~]# Step 4 : Run the cluster verification utility. The runcluvfy.sh script found in the grid directory, the same directory where you find runinstaller. This utility helps resolve problems before running runInstaller script. runcluvfy.sh stage -pre crsinst -n ractest4,ractest5 By default the CVU reports in non verbose mode by only reporting the summary of a test, we can obtain detailed output by using the -verbose argument. The -verbose argument produces detailed output of individual checks and where applicable shows results for each node in a tabular layout. runcluvfy.sh stage -pre crsinst -n ractest4,ractest5 -verbose [oracle@RACTEST4 grid]$ ls -l runcluvfy.sh -rwxr-xr-x. 1 root root 5085 Dec 20 2013 runcluvfy.sh [oracle@RACTEST4 grid]$ ./runcluvfy.sh stage -pre crsinst -n ractest4,ractest5 Performing pre-checks for cluster services setup Checking node reachability... Node reachability check passed from node "ractest4" Checking user equivalence... PRVG-2019 : Check for equivalence of user "oracle" from node "ractest4" to node "ractest5" failed PRKC-1044 : Failed to check remote command execution setup for node ractest5 using shells /usr/bin/ssh and /usr/bin/rsh File "/usr/bin/rsh" does not exist on node "ractest5" No RSA host key is known for ractest5 and you have requested strict checking.Host key verification failed. PRVG-2019 : Check for equivalence of user "oracle" from node "ractest4" to node "ractest4" failed PRKC-1044 : Failed to check remote command execution setup for node ractest4 using shells /usr/bin/ssh and /usr/bin/rsh File "/usr/bin/rsh" does not exist on node "ractest4" No RSA host key is known for ractest4 and you have requested strict checking.Host key verification failed. ERROR: User equivalence unavailable on all the specified nodes Verification cannot proceed Pre-check for cluster services setup was unsuccessful on all the nodes. [oracle@RACTEST4 grid]$ It seems ssh is not setup between two nodes. Let me setup the ssh. Step 5 : Setup the ssh between two nodes. Login to sshsetup directory and run the below commands. My sshsetup location is /backup/software/cluster/grid/sshsetup ./sshUserSetup.sh -user oracle -hosts "ractest4 ractest5" -noPromptPassphrase [oracle@RACTEST4 sshsetup]$ ./sshUserSetup.sh -user oracle -hosts "ractest4 ractest5" -noPromptPassphrase The output of this script is also logged into /tmp/sshUserSetup_2016-03-17-09-19-03.log Hosts are ractest4 ractest5 user is oracle Platform:- Linux Checking if the remote hosts are reachable PING RACTEST4.localdomain (192.168.56.211) 56(84) bytes of data. 64 bytes from RACTEST4.localdomain (192.168.56.211): icmp_seq=1 ttl=64 time=0.012 ms 64 bytes from RACTEST4.localdomain (192.168.56.211): icmp_seq=2 ttl=64 time=0.032 ms 64 bytes from RACTEST4.localdomain (192.168.56.211): icmp_seq=3 ttl=64 time=0.029 ms 64 bytes from RACTEST4.localdomain (192.168.56.211): icmp_seq=4 ttl=64 time=0.030 ms 64 bytes from RACTEST4.localdomain (192.168.56.211): icmp_seq=5 ttl=64 time=0.029 ms --- RACTEST4.localdomain ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.012/0.026/0.032/0.008 ms PING RACTEST5.localdomain (192.168.56.212) 56(84) bytes of data. 64 bytes from RACTEST5.localdomain (192.168.56.212): icmp_seq=1 ttl=64 time=0.382 ms 64 bytes from RACTEST5.localdomain (192.168.56.212): icmp_seq=2 ttl=64 time=0.407 ms 64 bytes from RACTEST5.localdomain (192.168.56.212): icmp_seq=3 ttl=64 time=0.371 ms 64 bytes from RACTEST5.localdomain (192.168.56.212): icmp_seq=4 ttl=64 time=0.418 ms 64 bytes from RACTEST5.localdomain (192.168.56.212): icmp_seq=5 ttl=64 time=0.416 ms --- RACTEST5.localdomain ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.371/0.398/0.418/0.031 ms Remote host reachability check succeeded. The following hosts are reachable: ractest4 ractest5. The following hosts are not reachable: . All hosts are reachable. Proceeding further... firsthost ractest4 numhosts 2 The script will setup SSH connectivity from the host RACTEST4.localdomain to all the remote hosts. After the script is executed, the user can use SSH to run commands on the remote hosts or copy files between this host RACTEST4.localdomain and the remote hosts without being prompted for passwords or confirmations. NOTE 1: As part of the setup procedure, this script will use ssh and scp to copy files between the local host and the remote hosts. Since the script does not store passwords, you may be prompted for the passwords during the execution of the script whenever ssh or scp is invoked. NOTE 2: AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE directories. Do you want to continue and let the script make the above mentioned changes (yes/no)? yes The user chose yes User chose to skip passphrase related questions. Creating .ssh directory on local host, if not present already Creating authorized_keys file on local host Changing permissions on authorized_keys to 644 on local host Creating known_hosts file on local host Changing permissions on known_hosts to 644 on local host Creating config file on local host If a config file exists already at /home/oracle/.ssh/config, it would be backed up to /home/oracle/.ssh/config.backup. Removing old private/public keys on local host Running SSH keygen on local host with empty passphrase Generating public/private rsa key pair. Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: 94:de:17:5e:0e:08:fb:3f:eb:6f:f8:0f:3f:3f:6e:f2 oracle@RACTEST4.localdomain The key's randomart image is: +--[ RSA 1024]----+ | . | | + . | | + . o . | | o o . = | | S o o . | | o | | o.. | | .+.* | | .o+BEB| +-----------------+ Creating .ssh directory and setting permissions on remote host ractest4 THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT. The script would create ~oracle/.ssh/config file on remote host ractest4. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup. The user may be prompted for a password here since the script would be running SSH on host ractest4. Warning: Permanently added 'ractest4,192.168.56.211' (RSA) to the list of known hosts. oracle@ractest4's password: Done with creating .ssh directory and setting permissions on remote host ractest4. Creating .ssh directory and setting permissions on remote host ractest5 THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT. The script would create ~oracle/.ssh/config file on remote host ractest5. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup. The user may be prompted for a password here since the script would be running SSH on host ractest5. Warning: Permanently added 'ractest5,192.168.56.212' (RSA) to the list of known hosts. oracle@ractest5's password: Done with creating .ssh directory and setting permissions on remote host ractest5. Copying local host public key to the remote host ractest4 The user may be prompted for a password or passphrase here since the script would be using SCP for host ractest4. oracle@ractest4's password: Done copying local host public key to the remote host ractest4 Copying local host public key to the remote host ractest5 The user may be prompted for a password or passphrase here since the script would be using SCP for host ractest5. oracle@ractest5's password: Done copying local host public key to the remote host ractest5 cat: /home/oracle/.ssh/known_hosts.tmp: No such file or directory cat: /home/oracle/.ssh/authorized_keys.tmp: No such file or directory SSH setup is complete. ------------------------------------------------------------------------ Verifying SSH setup =================== The script will now run the date command on the remote nodes using ssh to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP, THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR PASSWORDS. If you see any output other than date or are prompted for the password, ssh is not setup correctly and you will need to resolve the issue and set up ssh again. The possible causes for failure could be: 1. The server settings in /etc/ssh/sshd_config file do not allow ssh for user oracle. 2. The server may have disabled public key based authentication. 3. The client public key on the server may be outdated. 4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle. 5. User may not have passed -shared option for shared remote users or may be passing the -shared option for non-shared remote users. 6. If there is output in addition to the date, but no password is asked, it may be a security alert shown as part of company policy. Append the additional text to the /sysman/prov/resources/ignoreMessages.txt file. ------------------------------------------------------------------------ --ractest4:-- Running /usr/bin/ssh -x -l oracle ractest4 date to verify SSH connectivity has been setup from local host to ractest4. IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR. Thu Mar 17 09:21:15 EDT 2016 ------------------------------------------------------------------------ --ractest5:-- Running /usr/bin/ssh -x -l oracle ractest5 date to verify SSH connectivity has been setup from local host to ractest5. IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR. Thu Mar 17 09:21:15 EDT 2016 ------------------------------------------------------------------------ SSH verification complete. [oracle@RACTEST4 sshsetup]$ Step 6 : Re run the runcluvfy.sh again and see any issues. [oracle@RACTEST4 grid]$ ./runcluvfy.sh stage -pre crsinst -n ractest4,ractest5 Performing pre-checks for cluster services setup Checking node reachability... Node reachability check passed from node "ractest4" Checking user equivalence... User equivalence check passed for user "oracle" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "192.168.56.0" with node(s) ractest5,ractest4 TCP connectivity check passed for subnet "192.168.56.0" Node connectivity passed for subnet "192.168.1.0" with node(s) ractest5,ractest4 TCP connectivity check passed for subnet "192.168.1.0" Interfaces found on subnet "192.168.56.0" that are likely candidates for VIP are: ractest5 eth0:192.168.56.212 ractest4 eth0:192.168.56.211 Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are: ractest5 eth3:192.168.1.212 ractest4 eth1:192.168.1.211 WARNING: Could not find a suitable set of interfaces with the same name for the private interconnect Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.56.0". Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Checking ASMLib configuration. Check for ASMLib configuration passed. Total memory check failed Check failed on nodes: ractest5,ractest4 Available memory check passed Swap space check passed Free disk space check passed for "ractest5:/usr,ractest5:/var,ractest5:/etc,ractest5:/sbin,ractest5:/tmp" Free disk space check passed for "ractest4:/usr,ractest4:/var,ractest4:/etc,ractest4:/sbin,ractest4:/tmp" Check for multiple users with UID value 60321 passed User existence check passed for "oracle" Group existence check passed for "oinstall" Group existence check passed for "dba" Membership check for user "oracle" in group "oinstall" [as Primary] passed Membership check for user "oracle" in group "dba" passed Run level check passed Hard limits check passed for "maximum open file descriptors" Soft limits check passed for "maximum open file descriptors" Hard limits check passed for "maximum user processes" Soft limits check passed for "maximum user processes" System architecture check passed Kernel version check passed Kernel parameter check passed for "semmsl" Kernel parameter check passed for "semmns" Kernel parameter check passed for "semopm" Kernel parameter check passed for "semmni" Kernel parameter check passed for "shmmax" Kernel parameter check passed for "shmmni" Kernel parameter check passed for "shmall" Kernel parameter check passed for "file-max" Kernel parameter check passed for "ip_local_port_range" Kernel parameter check passed for "rmem_default" Kernel parameter check passed for "rmem_max" Kernel parameter check passed for "wmem_default" Kernel parameter check passed for "wmem_max" Kernel parameter check passed for "aio-max-nr" PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ractest5" PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ractest4" Kernel parameter check passed for "panic_on_oops" Package existence check passed for "binutils" Package existence check passed for "compat-libcap1" Package existence check passed for "compat-libstdc++-33(x86_64)" Package existence check passed for "libgcc(x86_64)" Package existence check passed for "libstdc++(x86_64)" Package existence check passed for "libstdc++-devel(x86_64)" Package existence check passed for "sysstat" Package existence check passed for "gcc" Package existence check passed for "gcc-c++" Package existence check passed for "ksh" Package existence check passed for "make" Package existence check passed for "glibc(x86_64)" Package existence check passed for "glibc-devel(x86_64)" Package existence check passed for "libaio(x86_64)" Package existence check passed for "libaio-devel(x86_64)" Package existence check passed for "nfs-utils" Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)" Port availability check passed for ports "6200,6100" Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)" Port availability check passed for ports "42424" Check for multiple users with UID value 0 passed Current group ID check passed Starting check for consistency of primary group of root user Check for consistency of root user's primary group passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP configuration file "/etc/ntp.conf" existence check passed No NTP Daemons or Services were found to be running PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s): ractest5,ractest4 Clock synchronization check using Network Time Protocol(NTP) failed Core file name pattern consistency check passed. User "oracle" is not part of "root" group. Check passed Default user file creation mask check passed Checking integrity of file "/etc/resolv.conf" across nodes "domain" and "search" entries do not coexist in any "/etc/resolv.conf" file All nodes have same "search" order defined in file "/etc/resolv.conf" The DNS response time for an unreachable node is within acceptable limit on all nodes Check for integrity of file "/etc/resolv.conf" passed Time zone consistency check passed Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ... All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf" Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed Checking daemon "avahi-daemon" is not configured and running Daemon not configured check passed for process "avahi-daemon" Daemon not running check passed for process "avahi-daemon" Starting check for /dev/shm mounted as temporary file system ... Check for /dev/shm mounted as temporary file system passed Starting check for /boot mount ... Check for /boot mount passed Starting check for zeroconf check ... ERROR: PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "RACTEST5.localdomain" PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "RACTEST4.localdomain" Check for zeroconf check failed Pre-check for cluster services setup was unsuccessful on all the nodes. [oracle@RACTEST4 grid]$ Now ssh issue resolved! but still there are four more issues appeared. Issue 1 The /etc/udev/rules.d/70-persistent-net.rules file has some incorrect naming convention. I fixed this file. WARNING: Could not find a suitable set of interfaces with the same name for the private interconnect Issue 2 Added below entry in /etc/sysctl.conf file on both node kernel.panic_on_oops = 1 Run the sysctl -p on both nodes. PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ractest5" PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ractest4" Issue 3The below issue can be ignored. Since we are configuring NTP protocol. CSSD will take care of time synchronization. Clock synchronization check using Network Time Protocol(NTP) failed Issue 4 Added below entry in /etc/sysconfig/network file on both nodes and restart the network service. NOZEROCONF=yes ERROR: PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "RACTEST5.localdomain" PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "RACTEST4.localdomain" Check for zeroconf check failed Step 7 : Re run the runcluvfy.sh again and see any issues. [oracle@RACTEST4 grid]$ ./runcluvfy.sh stage -pre crsinst -n ractest4,ractest5 Performing pre-checks for cluster services setup Checking node reachability... Node reachability check passed from node "ractest4" Checking user equivalence... User equivalence check passed for user "oracle" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "192.168.56.0" with node(s) ractest5,ractest4 TCP connectivity check passed for subnet "192.168.56.0" Node connectivity passed for subnet "192.168.1.0" with node(s) ractest5,ractest4 TCP connectivity check passed for subnet "192.168.1.0" Interfaces found on subnet "192.168.56.0" that are likely candidates for VIP are: ractest5 eth0:192.168.56.212 ractest4 eth0:192.168.56.211 Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are: ractest5 eth1:192.168.1.212 ractest4 eth1:192.168.1.211 Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.56.0". Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Checking ASMLib configuration. Check for ASMLib configuration passed. Total memory check failed Check failed on nodes: ractest5,ractest4 Available memory check passed Swap space check passed Free disk space check passed for "ractest5:/usr,ractest5:/var,ractest5:/etc,ractest5:/sbin,ractest5:/tmp" Free disk space check passed for "ractest4:/usr,ractest4:/var,ractest4:/etc,ractest4:/sbin,ractest4:/tmp" Check for multiple users with UID value 60321 passed User existence check passed for "oracle" Group existence check passed for "oinstall" Group existence check passed for "dba" Membership check for user "oracle" in group "oinstall" [as Primary] passed Membership check for user "oracle" in group "dba" passed Run level check passed Hard limits check passed for "maximum open file descriptors" Soft limits check passed for "maximum open file descriptors" Hard limits check passed for "maximum user processes" Soft limits check passed for "maximum user processes" System architecture check passed Kernel version check passed Kernel parameter check passed for "semmsl" Kernel parameter check passed for "semmns" Kernel parameter check passed for "semopm" Kernel parameter check passed for "semmni" Kernel parameter check passed for "shmmax" Kernel parameter check passed for "shmmni" Kernel parameter check passed for "shmall" Kernel parameter check passed for "file-max" Kernel parameter check passed for "ip_local_port_range" Kernel parameter check passed for "rmem_default" Kernel parameter check passed for "rmem_max" Kernel parameter check passed for "wmem_default" Kernel parameter check passed for "wmem_max" Kernel parameter check passed for "aio-max-nr" Kernel parameter check passed for "panic_on_oops" Package existence check passed for "binutils" Package existence check passed for "compat-libcap1" Package existence check passed for "compat-libstdc++-33(x86_64)" Package existence check passed for "libgcc(x86_64)" Package existence check passed for "libstdc++(x86_64)" Package existence check passed for "libstdc++-devel(x86_64)" Package existence check passed for "sysstat" Package existence check passed for "gcc" Package existence check passed for "gcc-c++" Package existence check passed for "ksh" Package existence check passed for "make" Package existence check passed for "glibc(x86_64)" Package existence check passed for "glibc-devel(x86_64)" Package existence check passed for "libaio(x86_64)" Package existence check passed for "libaio-devel(x86_64)" Package existence check passed for "nfs-utils" Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)" Port availability check passed for ports "6200,6100" Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)" Port availability check passed for ports "42424" Check for multiple users with UID value 0 passed Current group ID check passed Starting check for consistency of primary group of root user Check for consistency of root user's primary group passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP configuration file "/etc/ntp.conf" existence check passed No NTP Daemons or Services were found to be running PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s): ractest5,ractest4 Clock synchronization check using Network Time Protocol(NTP) failed Core file name pattern consistency check passed. User "oracle" is not part of "root" group. Check passed Default user file creation mask check passed Checking integrity of file "/etc/resolv.conf" across nodes "domain" and "search" entries do not coexist in any "/etc/resolv.conf" file All nodes have same "search" order defined in file "/etc/resolv.conf" The DNS response time for an unreachable node is within acceptable limit on all nodes Check for integrity of file "/etc/resolv.conf" passed Time zone consistency check passed Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ... All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf" Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed Checking daemon "avahi-daemon" is not configured and running Daemon not configured check passed for process "avahi-daemon" Daemon not running check passed for process "avahi-daemon" Starting check for /dev/shm mounted as temporary file system ... Check for /dev/shm mounted as temporary file system passed Starting check for /boot mount ... Check for /boot mount passed Starting check for zeroconf check ... Check for zeroconf check passed Pre-check for cluster services setup was unsuccessful on all the nodes. [oracle@RACTEST4 grid]$ Now all set! It is ready to start installing the cluster software! During the cluster software installation, i found cvuqdisk package missing. The package is existing in software location under grid/rpm directory. I installed this package on both nodes. [root@RACTEST4 rpm]# pwd /backup/software/cluster/grid/rpm [root@RACTEST4 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%] [root@RACTEST4 rpm]# [root@RACTEST5 cluster]# rpm -ivh cvuqdisk-1.0.9-1.rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%] [root@RACTEST5 cluster]# Again i clicked Check Again button and now the issue is disappeared. I am not worried about physical memory now.. NTP protocol can be ignored. since CSSD will take care of time synchronization. We are good now!!!