Installing OEM 13c and deployment of targets on Oracle Enterprise Linux

This is a step by step document of installing OEM 13c Release 2 and how to deploy targets in Linux environment.

Note:Oracle Database 12.1.0.2 is already installed, the steps for which are not covered here.

Download OEM 13c from edelivery website https://edelivery.oracle.com

Unzip the files.

[oracle@oem13c oem_binaries]$ chmod u+x em13200_linux64.bin

[oracle@oem13c oem_binaries]$ ./em13200_linux64.bin

** The directory "/tmp" does not have enough space. At least 10241 MB of free space are required.

Please input another directory or [Exit]: /u01/oem/oracle/tmp

0%................................................................./u01/oem/oracle/tmp/sfx_jhKt1s/Disk5/stage/Components/

....100%

Launcher log file is /u01/oem/oracle/tmp/OraInstall2018-09-20_09-30-47PM/launcher2018-09-20_09-30-47PM.log.

Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 2266.441 MHz    Passed

Checking monitor: must be configured to display at least 256 colors.   Actual 16777216    Passed

Checking swap space: must be greater than 512 MB.   Actual 8799 MB    Passed

Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)

Preparing to launch the Oracle Universal Installer from /u01/oem/oracle/tmp/OraInstall2018-09-20_09-30-47PM

====Prereq Config Location main===

/u01/oem/oracle/tmp/OraInstall2018-09-20_09-30-47PM/stage/prereq

EMGCInstaller args -scratchPath

EMGCInstaller args /u01/oem/oracle/tmp/OraInstall2018-09-20_09-30-47PM

EMGCInstaller args -sourceType

EMGCInstaller args network

EMGCInstaller args -timestamp

EMGCInstaller args 2018-09-20_09-30-47PM

EMGCInstaller args -paramFile

EMGCInstaller args /u01/oem/oracle/tmp/sfx_jhKt1s/Disk1/install/linux64/oraparam.ini

EMGCInstaller args -nocleanUpOnExit

DiskLoc inside SourceLoc/u01/oem/oem_binaries

EMFileLoc:/u01/oem/oracle/tmp/OraInstall2018-09-20_09-30-47PM/oui/em/

ScratchPathValue :/u01/oem/oracle/tmp/OraInstall2018-09-20_09-30-47PM

[root@oem13c middleware]# pwd

/u01/oem/oracle/middleware

[root@oem13c middleware]# sh allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/oem/oracle/middleware/root.sh ......

/etc exist

Creating /etc/oragchomelist file...

/u01/oem/oracle/middleware

Finished product-specific root actions.

/etc exist

Finished execution of  /u01/oem/oracle/middleware/root.sh ......

Starting to execute /u01/oem/oracle/agent/agent_13.2.0.0.0/root.sh ......

Finished product-specific root actions.

/etc exist

Finished execution of  /u01/oem/oracle/agent/agent_13.2.0.0.0/root.sh ......

[root@oem13c middleware]#

Login to the OEM url using sysman user

Create the env variable in the OMS host.

[oracle@oem13c oem]$ cat oem.env

export OMS_HOME=/u01/oem/oracle/middleware

export AGENT_HOME=/u01/oem/oracle/agent/agent_inst

[oracle@oem13c oem]$

 Next, we will push the agents to the respective hosts.

Click on "add target manually" as show above in the screenshot.

Next,

Click on install agent on host

Click on "+ Add" button and provide the required session name of your choice

Choose the host which should have the fully qualified name and the specific platform

Note: The host to be added should be reachable from OMS host. Thus before proceeding, check if  you are able to do a telnet

telnet oem13c.localdomain 3872--check if the client server is able to telnet to the oms server

Trying 192.168.56.6...

Connected to oem13c.localdomain (192.168.56.6).

Escape character is '^]'.

^CConnection closed by foreign host.

Once the deployment completes, we will be able to see the added host.

Till now, only the host monitoring has been enabled. OEM gives the advantage of adding the different types of targets for monitoring. Here, we have shown only the database monitoring part. 

Click on Add using guided process

Choose the respective host

provide the dbsnmp credentials

As we can see now that database is now available

For adding Oracle E-Business Suite as the target, we need to install the AMP plugins first, for which we need to follow the note:

• Getting Started with Oracle Application Management Suite for Oracle E-Business Suite Release 13.1.1.1.0 (Note 2045552.1)

This concludes installing OEM 13c and deploying targets in OEM. In the further posts, i will cover how to create metrics extensions and how to create incident rules in OEM 13c.

Hope, this post will help someone.

OEM 13c secure agent is failing with javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

Today, i was doing the installation of OEM 13c Release 2 and as part of that installing the agent in the new host was error'ed óut with the below

Manually running the secure agent was also failing.

[oracle@rac2 agent]$ cd /u01/oem/oracle/agent/agent_inst/bin

[oracle@rac2 bin]$ ./emctl secure agent

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.

Agent is already stopped...   Done.

Securing agent...   Started.

Enter Agent Registration Password :

Securing agent...   Failed.

Thus i checked the secure.log and found the message as "

2018-09-05 08:45:19,938 [main] ERROR agent.SecureAgentCmd main.353 - Failed to secure the Agent:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

        at sun.security.ssl.SSLSessionImpl.getPeerCertificateChain(SSLSessionImpl.java:470)

        at oracle.sysman.emctl.secure.agent.SecureAgentCmd.checkUpload(SecureAgentCmd.java:665)

        at oracle.sysman.emctl.secure.agent.SecureAgentCmd.secureAgent(SecureAgentCmd.java:450)

        at oracle.sysman.emctl.secure.agent.SecureAgentCmd.main(SecureAgentCmd.java:346)

2018-09-05 08:45:19,941 [main] INFO  agent.SecureAgentCmd main.355 - Re-trying. Trials left:0

[05-09-2018 08:45:19] USERINFO ::Securing agent...   Failed.""

Solution:-The time in the client machine as not correct. Changed it using the root user

[root@rac2 agent]# date -s "22:04:00 OCT 27, 2018"

Sat Oct 27 22:04:00 EDT 2018

[root@rac2 agent]# date

Sat Oct 27 22:04:06 EDT 2018

Thereafter Secure agent command got a successful run.

[oracle@rac2 bin]$ ./emctl secure agent

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.

Agent is already stopped...   Done.

Securing agent...   Started.

Enter Agent Registration Password :

Securing agent...   Successful.

Hope it helps someone.

Creating Physical Standby Database for Oracle E-Business Suite Release 12.2.7 Using Oracle 12c (12.1.0.2)

This post describes how to configure an Oracle E-Business Suite Release 12.2.x environment to use Oracle Database 12c Release 1 (12.1.0.2) as a physical standby. Also the role transition switchover/Switchback tests have been explained in details. 

Overview:

Standby Database

A standby database is a transactionally consistent copy of the primary database. Using a backup copy of the primary database, you can create up to 30 standby databases and incorporate them in Oracle Data Guard configuration.

There are three types of standby databases:

• Physical standby Provides a physically identical copy of the primary database, with on-disk database structures that are identical to the primary database on a block-for-block basis. The database schema, including indexes, is the same. A physical standby database is kept synchronized with the primary database by recovering the redo data received from the primary database.
• Logical standby 
  Contains the same logical information as the primary database, although the physical organization and structure of the data can be different. It is kept synchronized with the primary database by transforming the data in the redo logs received from the primary database into SQL statements and then executing the SQL statements on the standby database.
• Snapshot standby
  A fully updatable standby database. Like a physical or logical standby database, a snapshot standby database receives and archives redo data from a primary database. Unlike a physical or logical standby database, a snapshot standby database does not apply the redo data that it receives.

Oracle Data Guard

Oracle Data Guard is a set of services that create, manage, and monitor one or more standby databases to enable a primary database to survive disasters and data corruption. If the primary database becomes unavailable due to a planned or an unplanned outage, Oracle Data Guard can switch a standby database to the primary role, minimizing the downtime.

Data Guard offers three modes of data protection:

• Maximum Protection
  This mode offers the highest level of data protection. Data is synchronously transmitted to the standby database from the primary database, and transactions are not committed on the primary database unless the redo data is available on at least one standby database configured in this mode. If the last standby database configured in this mode becomes unavailable, processing stops on the primary database. This mode guarantees no data loss.
• Maximum Availability
  This mode is similar to the maximum protection mode, including no data loss. However, if a standby database becomes unavailable (for example, due to network connectivity problems), processing continues on the primary database. When the fault is corrected, the standby database is resynchronized with the primary database. If there is a need to failover before the standby database is resynchronized, some data may be lost.
• Maximum Performance
  This mode offers slightly less data protection on the primary database, but higher performance than maximum availability mode. In this mode, as the primary database processes transactions, redo data is asynchronously shipped to the standby database. The commit operation on the primary database does not wait for the standby database to acknowledge receipt of redo data before completing write operations on the primary database. If any standby destination becomes unavailable, processing continues on the primary database, and there is little effect on primary database performance.

This is based on the classic/traditional architecture of an Oracle E-Business Suite 12.2 Data Guard configuration, where the systems are replicated across the primary and standby sites and primary application tier is connected with primary database and standby application tier is connected with standby database as seen in the diagram below:

Now lets jump to the steps.

Enable Forced Logging

To protect against unlogged direct writes in the primary database that cannot be propagated to the standby database, turn on FORCE LOGGING at the primary database before performing data file backups that will be used to create the standby database.

Use the following SQL command to set FORCE LOGGING on the primary database:

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> ALTER DATABASE FORCE LOGGING;

Database altered.

Configure Oracle Net Communication To/From Standby System

PROD-Primary Server              Host-rac2.localdomain

PRODDR-Standby server         Host-dr.localdomain

In the standby server

Create a static listener which will be used in active duplication.

listener.ora

SID_LIST_LISTENER =

  (SID_LIST =

    (SID_DESC =

      (SID_NAME = PROD)

      (ORACLE_HOME = /u01/database/oracledb/product/12c/db_1)

    )

  )

LISTENER =

  (DESCRIPTION_LIST =

    (DESCRIPTION =

      (ADDRESS = (PROTOCOL = TCP)(HOST =dr.localdomain )(PORT = 1521))

    )

  )

ADR_BASE_LISTENER= /u01/database/oracledb/diag

cat tnsnames.ora

IFILE=/u01/database/oracledb/product/12c/db_1/network/admin/PRODDR_rac2_ifile.ora

[oracle@dr admin]$

[oracle@dr admin]$ cat PRODDR_rac2_ifile.ora

PRODDR=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (SERVER=DEDICATED)

            )

        )

PROD=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (INSTANCE_NAME=PROD)

            )

        )

[oracle@dr admin]

Now set the environment file.

[oracle@dr database]$ cat database.env

export ORACLE_SID=PROD

export ORACLE_HOME=/u01/database/oracledb/product/12c/db_1

export PATH=$PATH:/u01/database/oracledb/product/12c/db_1/bin

start the listener now

lsnrctl start

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 02-SEP-2018 11:47:38

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/oracledb/product/12c/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production

System parameter file is /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora

Log messages written to /u01/database/oracledb/diag/diag/tnslsnr/dr/listener/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias                     LISTENER

Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production

Start Date                02-SEP-2018 11:47:40

Uptime                    0 days 0 hr. 0 min. 1 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora

Listener Log File         /u01/database/oracledb/diag/diag/tnslsnr/dr/listener/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Services Summary...

Service "PROD" has 1 instance(s).

  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...

The command completed successfully

tnsping PRODDR

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 02-SEP-2018 12:07:14

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=PROD) (SERVER=DEDICATED)))

OK (0 msec)

In the primary server

go to $TNS_ADMIN

under the sqlnet_ifile.ora

add the below

NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME)

SQLNET.EXPIRE_TIME= 10

tcp.validnode_checking =no

tcp.invited_nodes=(dr.localdomain)

SQLNET.INBOUND_CONNECT_TIMEOUT =60

SQLNET.ALLOWED_LOGON_VERSION_SERVER=8

Create the context ifile and add the below

PRODDR=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (SERVER=DEDICATED)

            )

        )

PROD=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (INSTANCE_NAME=PROD)

            )

        )

Test the tnsping from both the nodes.

Set Primary Database Initialization Parameters

In the standby 

Created a temporary init file and started the db in no mount.

[oracle@dr dbs]$ cat initPRODDR.ora

DB_NAME=PROD

DB_UNIQUE_NAME=PRODDR

DB_BLOCK_SIZE=8192

and start the dr server in nomount

[oracle@dr dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 2 14:47:12 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/u01/database/oracledb/product/12c/db_1/dbs/initPRODDR.ora';

ORACLE instance started.

Total System Global Area  218103808 bytes

Fixed Size                  2922712 bytes

Variable Size             159385384 bytes

Database Buffers           50331648 bytes

Redo Buffers                5464064 bytes

SQL> exit;

Set Up Secure Connections

orapwd file=orapwPROD password=pwd entries=10 ignorecase=y

Copy the same to the standby server.

Enable Archive Logging on Primary System

In my case my primary was already archive enabled

Add Standby Redo Logs

SQL> select bytes/1024/1024 as MB from v$log;

        MB

----------

      1000

      1000

Now add the standby redo logs

estimate the number of redo logs

SQL> select GROUP#,MEMBER from v$logfile;

    GROUP#

----------

MEMBER

--------------------------------------------------------------------------------

         2

/u01/database/PROD/data/log02a.dbf

         2

/u01/database/PROD/data/log02b.dbf

         1

/u01/database/PROD/data/log01a.dbf

    GROUP#

----------

MEMBER

--------------------------------------------------------------------------------

         1

/u01/database/PROD/data/log01b.dbf

now add the standby logs

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 3 '/u01/database/PROD/data/log03a.dbf' size 1000M;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 '/u01/database/PROD/data/log04.dbf' size 1000M;

Database altered.

Create a physical standby using active duplication method

Add the below parameters to the existing init file in the primary server

*.fal_server='PROD'

*.log_archive_config='dg_config=(PROD,PRODDR)'

*.log_archive_dest_2='service=PRODDR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=PRODDR'

Ensure the primary server is started with a spfile.

In  primary server:

Create rman duplicate script

run {

 allocate channel pri1 type disk;

 allocate auxiliary channel stby type disk;

 duplicate target database

  for standby

  from active database

  dorecover

  spfile

    set db_unique_name='PRODDR'

    set control_files= ' /u01/database/oracledb/data/cntrl01.dbf',' /u01/database/oracledb/data /cntrl02.dbf',' /u01/database/oracledb/data /cntrl03.dbf'

    set fal_server='PROD'

    set standby_file_management='AUTO'

    set log_archive_config='dg_config=(PROD,PRODDR)'

    set log_archive_dest_1='LOCATION= /u01/database/oracledb/data/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=PRODDR'

    set log_archive_dest_2='service=PRODDR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=PRODDR'

set log_file_name_convert=' /u01/database/PROD/data/',' /u01/database/oracledb /data/'

set  db_file_name_convert=' /u01/database/PROD/data/',' /u01/database/oracledb/data/'

nofilenamecheck;

  }

Now issue the rman duplicate session

rman target sys/pwd@PROD auxiliary sys/pwd@PRODDR

Recovery Manager: Release 12.1.0.2.0 - Production on Sun Sep 2 14:52:02 2018

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PROD (DBID=391996181)

connected to auxiliary database: PROD (not mounted)

RMAN>@duplicate.rmn

This completes the creation of physical standby database PRODDR and it will be in mount state.

Now in the standby server, start the mrp process

SQL> alter database recover managed standby database disconnect from session;

Database altered.

In Primary server

verify the parameter

SQL> show parameter log_archive_dest_state_2;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_state_2             string      enable

In the standby server

SQL> SELECT 'Last Applied : ' Logs,

TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#

FROM v$archived_log

WHERE sequence# =

(SELECT MAX(sequence#) FROM v$archived_log WHERE applied='YES'

)

UNION

SELECT 'Last Received : ' Logs,

TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#

FROM v$archived_log

WHERE sequence# =(SELECT MAX(sequence#) FROM v$archived_log );

  2    3    4    5    6    7    8    9   10   11

LOGS             TIME                  THREAD#  SEQUENCE#

---------------- ------------------ ---------- ----------

Last Applied :   03-SEP-18:09:38:45          1         67

Last Received :  03-SEP-18:09:38:45          1         67

In the primary server

> archive log list;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /u01/database/PROD/data

Oldest online log sequence     67

Next log sequence to archive   68

Current log sequence           68

Thus we can see from above that the archive logs are being shipped from primary server to standby server.

*************************************Configuration : Standby Application Tier(s) Connected to the Standby Database******************

run adpreclone in the run file system of the application tier.

[applmgr@rac2 scripts]$ perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.22

Enter the APPS User Password:

Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /u01/application/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:

perl /u01/application/PROD/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/u01/application/PROD/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/u01/application/PROD/fs1/EBSapps/comn/clone component=appsTier method= appctx=/u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml showProgress

 Setting the wls environment

Beginning application tier Stage - Mon Sep  3 10:25:56 2018

/u01/application/PROD/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u01/application/PROD/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/u01/application/PROD/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/u01/application/PROD/fs1/EBSapps/comn/java/classes:/u01/application/PROD/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/u01/application/PROD/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/u01/application/PROD/fs1/FMW_Home/webtier/oui/jlib/share.jar:/u01/application/PROD/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u01/application/PROD/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/u01/application/PROD/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/application/PROD/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml -stage /u01/application/PROD/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/log/clone/StageAppsTier_09031025.log

  \     20% completed

Completed Stage...

Mon Sep  3 10:52:41 2018

As, run file system points to fs1, so we have created the directories fs1 and fs2 in dr and will copy the EBSapps directory from run file system of the primary to dr fs1.

Now create the appsutil directory in the dr server.

In the primary application server

[applmgr@rac2 PROD]$ perl $AD_TOP/bin/admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/log/Make                                                                                        AppsUtil_09032132.log

output located at /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/out/appsut                                                                                        il.zip

MakeAppsUtil completed successfully.

[applmgr@rac2 PROD]$ scp /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/out/appsutil.zip oracle@192.168.X.X:/u01/database/oracledb/product/12c/db_1

oracle@192.168.X.X's password:

appsutil.zip                                                                                                                          100% 3685KB   3.6MB/s   00:00

[applmgr@rac2 PROD]$

in the dr server

[oracle@dr db_1]$ unzip -o appsutil.zip

Archive:  appsutil.zip

   creating: appsutil/

thus appsutil directory is now created.

Now before creating the context file, we should convert the database to snapshot standby.

SQL> alter system set db_recovery_file_dest_size=32g scope=both;

System altered.

SQL> alter system set db_recovery_file_dest='/u01/database/flas_recovery_area' scope=both;

System altered.

SQL> alter system set db_flashback_retention_target=1440 scope=both;

System altered.

Stop redo apply on the physical standby database:

SQL> alter database recover managed standby database cancel;

Database altered.

Turn on flashback logging:

SQL> alter database flashback on;

Database altered.

Convert the standby database into a snapshot standby database:

SQL> alter database convert to snapshot standby;

Database altered.

SQL>

SQL> shut immediate;

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL> startup;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             452987000 bytes

Database Buffers         1677721600 bytes

Redo Buffers               13848576 bytes

Database mounted.

Database opened.

remove the application config info

SQL*Plus: Release 12.1.0.2.0 Production on Mon Sep 3 21:55:00 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Mon Sep 03 2018 10:25:27 -04:00

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> exec ad_zd_fixer.clear_valid_nodes_info;

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL>

now create the context file. before that Copy the jre from primary database server $ORACLE_HOME/appsutil/jre

[oracle@dr bin]$ perl adbldxml.pl appsuser=apps appspass=welcome123 template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp out=$ORACLE_HOME/appsutil/PROD_dr.xml

Starting context file generation for db tier..

Using JVM from /u01/database/oracledb/product/12c/db_1/appsutil/jre/bin/java to execute java programs..

The log file for this adbldxml session is located at:

/u01/database/oracledb/product/12c/db_1/appsutil/log/adbldxml_09032214.log

AC-20010: Error: File - listener.ora could not be found at the location:

        /u01/app/oracledb/product/12c/db_1/network/admin/listener.ora

indicated by TNS_ADMIN. Context file can not be generated.

Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: dr

Enter Port of Database server: 1521

Enter SID of Database server: PROD

Enter Database Service Name: PROD

Enter the value for Display Variable: dr:1.0

The context file has been created at:

/u01/database/oracledb/product/12c/db_1/appsutil/PROD_dr.xml

run autoconfig now:

[oracle@dr PROD_dr]$ sh adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: /u01/database/oracledb/product/12c/db_1/appsutil/log/PROD_dr/09032312/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.

        Using ORACLE_HOME location : /u01/database/oracledb/product/12c/db_1

        Classpath                   : :/u01/database/oracledb/product/12c/db_1/jdbc/lib/ojdbc6.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java/xmlparserv2.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java:/u01/database/oracledb/product/12c/db_1/jlib/netcfg.jar:/u01/database/oracledb/product/12c/db_1/jlib/ldapjclnt12.jar

        Using Context file          : /u01/database/oracledb/product/12c/db_1/appsutil/PROD_dr.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121

Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.

Now configure the application tier

[applmgr@dr bin]$ perl adcfgclone.pl appsTier dualfs

                     Copyright (c) 2002, 2015 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.60

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] : no

Running: Context clone...

Log file located at /u01/application/fs1/EBSapps/comn/clone/bin/CloneContext_0904023938.log

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [dr] :

Target System Database SID : PROD

Target System Database Server Node [dr] :

Target System Database Domain Name [localdomain] :

Target System Base Directory : /u01/application

Target System Base Directory set to /u01/application

Target System Current File System Base set to /u01/application/fs1

Target System Other File System Base set to /u01/application/fs2

Target System Fusion Middleware Home set to /u01/application/fs1/FMW_Home

Target System Other File System Fusion Middleware Home set to /u01/application/fs2/FMW_Home

Target System Web Oracle Home set to /u01/application/fs1/FMW_Home/webtier

Target System Other File System Web Oracle Home set to /u01/application/fs2/FMW_Home/webtier

Target System Appl TOP set to /u01/application/fs1/EBSapps/appl

Target System Other File System Appl TOP set to /u01/application/fs2/EBSapps/appl

Target System COMMON TOP set to /u01/application/fs1/EBSapps/comn

Target System Other File System COMMON TOP set to /u01/application/fs2/EBSapps/comn

Target System Instance Home Directory [/u01/application] :

Target System Current File System Instance Top set to /u01/application/fs1/inst/apps/PROD_dr

Do you want to preserve the Display [rac2:0.0] (y/n)  : n

Target System Display [dr:0.0] :

Target System Root Service [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 1

Checking the port pool 1

done: Port Pool 1 is free

Report file located at /u01/application/fs1/inst/apps/PROD_dr/admin/out/portpool.lst

RC-40201: Unable to connect to Database PROD.

Enter the Database listener port [1522] : 1521

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/tmp

2. /usr/tmp

3. /u01/database/PROD/12.1.0/appsutil/outbound/PROD_rac2

4. /usr/tmp

Choose a value which will be set as APPLPTMP value on the target node [1] : 1

The new APPL_TOP context file has been created :

  /u01/application/fs1/inst/apps/PROD_dr/appl/admin/PROD_dr.xml

Check Clone Context logfile /u01/application/fs1/EBSapps/comn/clone/bin/CloneContext_0904023938.log for details.

Creating Patch file system context file.....

Log file located at /u01/application/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0904024546.log

Target System Other File System Instance Top set to /u01/application/fs2/inst/apps/PROD_dr

Target System Port Pool [0-99] : 2

Checking the port pool 2

done: Port Pool 2 is free

Report file located at /u01/application/fs2/inst/apps/PROD_dr/admin/out/portpool.lst

The new APPL_TOP context file has been created :

  /u01/application/fs2/inst/apps/PROD_dr/appl/admin/PROD_dr.xml

Check Clone Context logfile /u01/application/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0904024546.log for details.

FMW Pre-requisite check log file location : /u01/application/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Running: FMW pre-req check...

Configuring: Run file system....

LogFile located at /u01/application/fs1/inst/apps/PROD_dr/admin/log/clone/run/RCloneApplyAppstier_09040246.log

Configuring: Patch file system....

LogFile located at /u01/application/fs1/inst/apps/PROD_dr/admin/log/clone/patch/RCloneApplyAppstier_09040359.log

Do you want to startup the Application Services for PROD? (y/n) [n] : n

Services not started

this completes configuring apps tier configuration in dr server.

Revert the Physical Standby Database to its Original State

Change the standby database from snapshot mode back into standby mode using the following commands:

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing opt                                                                                        ions

SQL> startup mount force;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             486541432 bytes

Database Buffers         1644167168 bytes

Redo Buffers               13848576 bytes

Database mounted.

SQL> alter database convert to physical standby;

Database altered.

SQL> shutdown immediate;

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL> startup nomount;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             486541432 bytes

Database Buffers         1644167168 bytes

Redo Buffers               13848576 bytes

SQL> alter database mount standby database;

Database altered.

SQL>

enable redo log apply

SQL> alter database recover managed standby database using current logfile disconnect;

Database altered.

Performing role transitions

I have illustrated the switchover and switchback steps. If you are familiar with these steps, then failover can also be done easily which is not in the scope of this post.

Prepare for a switchover to the standby server

1. Verify the primary database instance is open and the standby database instance is mounted.
    
2. Shut down all the application tier services on all nodes on the primary site.
   
   $ adstpall.sh <appsuser>/<appspass>
3. Verify there are no active users connected to the database. Shut down all the sessions on the primary database.
4. Use the following command on both the primary and standby databases to ensure that the last redo data transmitted from the primary database has been applied on the standby database. If necessary, perform a log file switch before proceeding further:.
   SQL>select sequence#,applied from v$archived_log;

Check whether the primary is ready for a switch. Query the switchover_status column of the v$database fixed view to determine whether the database is ready to switch modes

SQL> select switchover_status from v$database;

SWITCHOVER_STATUS

--------------------

TO STANDBY

 Initiate a switchover on the primary database

Connect as sysdba and issue the following SQL command:

SQL> alter database commit to switchover to physical standby with session shutdown;

Database altered.

When this statement has completed, the primary database will have been converted to a standby database. During the statement's execution, the current control file is backed up to the current SQL session's trace file - this makes it possible to reconstruct the control file should the need arise.

Shut down and mount the primary database

To complete the transition, the database must be shut down and restarted in a mounted state. Change the value of LOG_ARCHIVE_DEST_STATE_2 to defer, in addition, recovery can be started in preparation for transition.

SQL> shutdown immediate;

ORA-01012: not logged on

SQL> exit;

Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

[oracle@rac2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 4 04:08:40 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>

SQL> startup nomount;

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             452987000 bytes

Database Buffers         1677721600 bytes

Redo Buffers               13848576 bytes

SQL> alter database mount standby database;

Database altered.

SQL> alter system set LOG_ARCHIVE_DEST_STATE_2=defer scope=both sid='*';

System altered.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> exit;

Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

[oracle@rac2 dbs]$ ps -ef|grep mrp

oracle   11858     1  1 04:10 ?        00:00:00 ora_mrp0_PROD

oracle   11865  8370  0 04:10 pts/2    00:00:00 grep mrp

[oracle@rac2 dbs]$

At this point in the switchover, both databases are standby databases.

Verify the switchover status on the standby server

As the ORACLE user on the standby-to-become-the-primary database server, verify that it is ready to switch to the primary:

SQL> select switchover_status from v$database;

SWITCHOVER_STATUS

--------------------

TO PRIMARY

SQL> alter database commit to switchover to primary with session shutdown;

Database altered.

Adjust the network parameters:

In the context ifile.ora change the host name in the primary as well as in the standby database.

cat PROD_rac2_ifile.ora

PRODDR=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (SERVER=DEDICATED)

            )

        )

PROD=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (INSTANCE_NAME=PROD)

            )

        )

in the new primary server

PRODDR=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (SERVER=DEDICATED)

            )

        )

PROD=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))

            (CONNECT_DATA=

                (SERVICE_NAME=PROD)

                (INSTANCE_NAME=PROD)

            )

On the current primary, change the LOG_ARCHIVE_DEST_STATE_2 to enable using the following command:

SQL> alter system set LOG_ARCHIVE_DEST_STATE_2=enable scope=both sid='*';

System altered.

To complete the transition, the database must be shut down and re-started:

SQL> shut immediate;

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL> startup;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             486541432 bytes

Database Buffers         1644167168 bytes

Redo Buffers               13848576 bytes

Database mounted.

Database opened.

Complete the database configuration

1. Connect to the new primary database using SQL*Plus as the APPS user, and execute the following commands:

SQL> exec fnd_net_services.remove_system ('PROD');

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> exec ad_zd_fixer.clear_valid_nodes_info;

PL/SQL procedure successfully completed.

SQL>

As the ORACLE user on the new primary database server, use AutoConfig to complete the configuration for the primary operations, providing the APPS password when prompted

[oracle@dr PROD_dr]$ sh adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: /u01/database/oracledb/product/12c/db_1/appsutil/log/PROD_dr/09041329/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.

        Using ORACLE_HOME location : /u01/database/oracledb/product/12c/db_1

        Classpath                   : :/u01/database/oracledb/product/12c/db_1/jdbc/lib/ojdbc6.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java/xmlparserv2.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java:/u01/database/oracledb/product/12c/db_1/jlib/netcfg.jar:/u01/database/oracledb/product/12c/db_1/jlib/ldapjclnt12.jar

        Using Context file          : /u01/database/oracledb/product/12c/db_1/appsutil/PROD_dr.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121

Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.

stop and start the listener on the new primary database server:

[oracle@dr database]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 14:11:21

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))

The command completed successfully

[oracle@dr database]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 14:11:27

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/oracledb/product/12c/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production

System parameter file is /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora

Log messages written to /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias                     PROD

Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production

Start Date                04-SEP-2018 14:11:27

Uptime                    0 days 0 hr. 0 min. 0 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora

Listener Log File         /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Services Summary...

Service "PROD" has 1 instance(s).

  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...

The command completed successfully

[oracle@dr database]$

On the new standby server, stop and start the listener for the standby services

[oracle@rac2 database]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 06:36:55

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))

The command completed successfully

[oracle@rac2 database]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 06:37:01

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/PROD/12.1.0/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production

System parameter file is /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora

Log messages written to /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias                     PROD

Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production

Start Date                04-SEP-2018 06:37:01

Uptime                    0 days 0 hr. 0 min. 0 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora

Listener Log File         /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))

Services Summary...

Service "PROD" has 1 instance(s).

  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...

The command completed successfully

Configure the Application Tier After Role Transition (Switchover)

Once the primary database configuration has completed and its listeners have started, log in to each new-primary application tier server as the APPLMGR user, and run the final configuration steps on the run file systems providing the APPS password when prompted::

[applmgr@dr scripts]$ sh adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: /u01/application/fs1/inst/apps/PROD_dr/admin/log/09041433/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.

        Using CONFIG_HOME location     : /u01/application/fs1/inst/apps/PROD_dr

        Classpath                   : /u01/application/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/application/fs1/EBSapps/comn/java/classes

        Using Context file          : /u01/application/fs1/inst/apps/PROD_dr/appl/admin/PROD_dr.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...

        Configuring AD_TOP........COMPLETED

        Configuring FND_TOP.......COMPLETED

        Configuring ICX_TOP.......COMPLETED

        Configuring MSC_TOP.......COMPLETED

        Configuring IEO_TOP.......COMPLETED

        Configuring BIS_TOP.......COMPLETED

        Configuring CZ_TOP........COMPLETED

        Configuring SHT_TOP.......COMPLETED

        Configuring AMS_TOP.......COMPLETED

        Configuring CCT_TOP.......COMPLETED

        Configuring WSH_TOP.......COMPLETED

        Configuring CLN_TOP.......COMPLETED

        Configuring OKE_TOP.......COMPLETED

        Configuring OKL_TOP.......COMPLETED

        Configuring OKS_TOP.......COMPLETED

        Configuring CSF_TOP.......COMPLETED

        Configuring IBY_TOP.......COMPLETED

        Configuring JTF_TOP.......COMPLETED

        Configuring MWA_TOP.......COMPLETED

        Configuring CN_TOP........COMPLETED

        Configuring CSI_TOP.......COMPLETED

        Configuring WIP_TOP.......COMPLETED

        Configuring CSE_TOP.......COMPLETED

        Configuring EAM_TOP.......COMPLETED

        Configuring GMF_TOP.......COMPLETED

        Configuring PON_TOP.......COMPLETED

        Configuring FTE_TOP.......COMPLETED

        Configuring ONT_TOP.......COMPLETED

        Configuring AR_TOP........COMPLETED

        Configuring AHL_TOP.......COMPLETED

        Configuring IES_TOP.......COMPLETED

        Configuring OZF_TOP.......COMPLETED

        Configuring CSD_TOP.......COMPLETED

        Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.

[applmgr@dr scripts]$

On the patch file system change to $AD_TOP/bin/ and run the following command:

[applmgr@dr bin]$ perl adconfig.pl contextfile=/u01/application/fs2/inst/apps/PROD_dr/appl/admin/PROD_dr.xml -syncctx

Enter the APPS user password:

The log file for this session is located at: /u01/application/fs2/inst/apps/PROD_dr/admin/log/09041447/adconfig.log

Option specified      : Synchronize context file

Only context file synchronization will be performed

        Classpath                   : /u01/application/fs2/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/application/fs2/EBSapps/comn/java/classes

        Using ContextFile     : /u01/application/fs2/inst/apps/PROD_dr/appl/admin/PROD_dr.xml

        Synchronizing the context file ......COMPLETED

AutoConfig completed successfully.

[applmgr@dr bin]$

Update the Host Name in the FND_CONCURRENT_REQUESTS and FND_CONC_REQ_OUTPUTS tables

In order to synchronize your concurrent manager log and out directories, you must change the host name in the fnd_concurrent_requests table to the standby server name:

SQL> update apps.fnd_concurrent_requests

set logfile_node_name ='DR',

outfile_node_name = 'DR'

where logfile_node_name ='RAC2'

and outfile_node_name ='RAC2';  2    3    4    5

638 rows updated.

SQL> commit;

Commit complete

:rsync the concurrent log and out files from old primary application server to new ones:

rsync -avz log applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc
rsync -avz out applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc

Now start the application services in the new primary servers run file system and redirect the user to the new url

http://dr.localdomain:8001/OA_HTML/AppsLocalLogin.jsp

Now we will do the switchback which will make primary to standby and standby to primary

a.Verify that the primary database at the standby site is open, and the standby database at the primary site is mounted.

b. Verify all the redo logs are transferred to standby and are applied.

c. On the primary site, check whether the switchover_status from v$database is showing: TO STANDBY

in primary

SQL> select switchover_status from v$database;

SWITCHOVER_STATUS

--------------------

TO STANDBY

On the primary database, issue the command:

SQL> alter database commit to switchover to physical standby;

Database altered.

So this has shut down my database.

[oracle@dr dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 5 04:38:42 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             503318648 bytes

Database Buffers         1627389952 bytes

Redo Buffers               13848576 bytes

SQL> alter database mount standby database;

Database altered.

SQL> alter system set LOG_ARCHIVE_DEST_STATE_2=defer scope=both sid='*';

System altered.

SQL>

in the standby

SQL> select switchover_status from v$database;

SWITCHOVER_STATUS

--------------------

TO PRIMARY

SQL> select database_role from v$database;

DATABASE_ROLE

----------------

PHYSICAL STANDBY

in the standby init file

changed the below parameters

*.log_archive_config='dg_config=(PROD,PRODDR)'

*.log_archive_dest_1='LOCATION=/u01/database/PROD/data'

*.log_archive_dest_2='service=PRODDR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=PRODDR'

*.log_archive_dest_state_2='enable

now in the standby

SQL> alter database commit to switchover to primary with session shutdown;

Database altered.

SQL> exit;

Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

[oracle@rac2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 4 21:17:08 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from v$database;

OPEN_MODE            DATABASE_ROLE

-------------------- ----------------

MOUNTED              PRIMARY

As the ORACLE user on both the primary and standby database servers, change the host name in the standby service definitions to point to the new standby server in both the <CONTEXT_NAME>_ifile.ora and <STNDBY_CONTEXT>_ifile.ora (which are located in the <TNS_ADMIN> directory(already done in the above sections, you can refer it to interchange the host names again)

now in the primary

[oracle@rac2 dbs]$ pwd

/u01/database/PROD/12.1.0/dbs

[oracle@rac2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 4 21:21:55 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/u01/database/PROD/12.1.0/dbs/initPROD.ora';

ORA-32006: SEC_CASE_SENSITIVE_LOGON initialization parameter has been deprecated

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             452987000 bytes

Database Buffers         1677721600 bytes

Redo Buffers               13848576 bytes

Database mounted.

SQL> create spfile from pfile;

File created.

SQL> shut immediate;

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL> startup;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             452987000 bytes

Database Buffers         1677721600 bytes

Redo Buffers               13848576 bytes

Database mounted.

Database opened.

SQL> select database_role,open_mode from v$database;

DATABASE_ROLE    OPEN_MODE

---------------- --------------------

PRIMARY          READ WRITE

in the stanbdy started the mrp process

[oracle@dr admin]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 5 04:57:37 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> alter database recover managed standby database disconnect from session;

Database altered.

Now verify if the archives are getting shipped or not

in the primary

SQL> alter system switch logfile;

System altered.

SQL> archive log list;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /u01/database/PROD/data

Oldest online log sequence     128

Next log sequence to archive   129

Current log sequence           129

SQL>

in the stanbdy

SQL> SELECT 'Last Applied : ' Logs,

TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#

FROM v$archived_log

WHERE sequence# =

(SELECT MAX(sequence#) FROM v$archived_log WHERE applied='YES'

)

UNION

SELECT 'Last Received : ' Logs,

TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#

FROM v$archived_log

WHERE sequence# =(SELECT MAX(sequence#) FROM v$archived_log );    2    3    4    5    6    7    8    9   10   11

LOGS             TIME                  THREAD#  SEQUENCE#

---------------- ------------------ ---------- ----------

Last Applied :   04-SEP-18:21:29:46          1        128

Last Received :  04-SEP-18:21:29:46          1        128

Connect to the new primary database using SQL*Plus as the APPS user, and execute the following commands:

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Tue Sep 04 2018 16:03:42 -04:00

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> exec fnd_net_services.remove_system ('PROD');

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> exec ad_zd_fixer.clear_valid_nodes_info;

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL>

run autoconfig

[oracle@rac2 PROD_rac2]$ sh adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: /u01/database/PROD/12.1.0/appsutil/log/PROD_rac2/09042141/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.

        Using ORACLE_HOME location : /u01/database/PROD/12.1.0

        Classpath                   : :/u01/database/PROD/12.1.0/jdbc/lib/ojdbc6.jar:/u01/database/PROD/12.1.0/appsutil/java/xmlparserv2.jar:/u01/database/PROD/12.1.0/appsutil/java:/u01/database/PROD/12.1.0/jlib/netcfg.jar:/u01/database/PROD/12.1.0/jlib/ldapjclnt12.jar

        Using Context file          : /u01/database/PROD/12.1.0/appsutil/PROD_rac2.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121

Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.

[oracle@rac2 PROD_rac2]$

in the primary server and standby server

start stop listener

[oracle@rac2 PROD_rac2]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 21:42:42

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))

The command completed successfully

[oracle@rac2 PROD_rac2]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 21:42:46

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/PROD/12.1.0/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production

System parameter file is /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora

Log messages written to /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias                     PROD

Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production

Start Date                04-SEP-2018 21:42:46

Uptime                    0 days 0 hr. 0 min. 0 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora

Listener Log File         /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))

Services Summary...

Service "PROD" has 1 instance(s).

  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...

The command completed successfully

[oracle@rac2 PROD_rac2]$

in standby

[oracle@dr admin]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 05-SEP-2018 05:15:33

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))

The command completed successfully

[oracle@dr admin]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 05-SEP-2018 05:15:38

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/oracledb/product/12c/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production

System parameter file is /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora

Log messages written to /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias                     PROD

Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production

Start Date                05-SEP-2018 05:15:38

Uptime                    0 days 0 hr. 0 min. 0 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora

Listener Log File         /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Services Summary...

Service "PROD" has 1 instance(s).

  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...

The command completed successfully

[oracle@dr admin]$

Configure the Application Tier After Role Transition switchback

in the run file system, run autoconfig

[applmgr@rac2 scripts]$ sh adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/log/09042147/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.

        Using CONFIG_HOME location     : /u01/application/PROD/fs1/inst/apps/PROD_rac2

        Classpath                   : /u01/application/PROD/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/application/PROD/fs1/EBSapps/comn/java/classes

        Using Context file          : /u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...

        Configuring AD_TOP........COMPLETED

        Configuring FND_TOP.......COMPLETED

        Configuring ICX_TOP.......COMPLETED

        Configuring MSC_TOP.......COMPLETED

        Configuring IEO_TOP.......COMPLETED

        Configuring BIS_TOP.......COMPLETED

        Configuring CZ_TOP........COMPLETED

        Configuring SHT_TOP.......COMPLETED

        Configuring AMS_TOP.......COMPLETED

        Configuring CCT_TOP.......COMPLETED

        Configuring WSH_TOP.......COMPLETED

        Configuring CLN_TOP.......COMPLETED

        Configuring OKE_TOP.......COMPLETED

        Configuring OKL_TOP.......COMPLETED

        Configuring OKS_TOP.......COMPLETED

        Configuring CSF_TOP.......COMPLETED

        Configuring IBY_TOP.......COMPLETED

        Configuring JTF_TOP.......COMPLETED

        Configuring MWA_TOP.......COMPLETED

        Configuring CN_TOP........COMPLETED

        Configuring CSI_TOP.......COMPLETED

        Configuring WIP_TOP.......COMPLETED

        Configuring CSE_TOP.......COMPLETED

        Configuring EAM_TOP.......COMPLETED

        Configuring GMF_TOP.......COMPLETED

        Configuring PON_TOP.......COMPLETED

        Configuring FTE_TOP.......COMPLETED

        Configuring ONT_TOP.......COMPLETED

        Configuring AR_TOP........COMPLETED

        Configuring AHL_TOP.......COMPLETED

        Configuring IES_TOP.......COMPLETED

        Configuring OZF_TOP.......COMPLETED

        Configuring CSD_TOP.......COMPLETED

        Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.

in the patch file system

[applmgr@rac2 PROD]$ cd $AD_TOP/bin

[applmgr@rac2 bin]$ echo $CONTEXT_FILE

/u01/application/PROD/fs2/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml

[applmgr@rac2 bin]$ perl adconfig.pl contextfile=/u01/application/PROD/fs2/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml -syncctx

Update the host names for concurrent requests log and out files to point to new primary after switchback

update apps.fnd_concurrent_requests set logfile_node_name ='RAC2',outfile_node_name ='RAC2' where logfile_node_name ='DR' and outfile_node_name ='DR';

645 rows updated.

SQL> commit;

Commit complete.

rsync the concurrent log and out files between primary and secondary

rsync -avz log applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc

rsync -avz out applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc

start the application services and redirect the user to new url

http://rac2.localdomain:8000/OA_HTML/AppsLocalLogin.jsp

Once you get comfortable with switchover/Switchback process, then failover steps can also be easily done. At this moment, failover is out of scope for this post.

Hope this post will help someone. Happy learning..:)