Monday, September 17, 2018

Creating Physical Standby Database for Oracle E-Business Suite Release 12.2.7 Using Oracle 12c (12.1.0.2)

This post describes how to configure an Oracle E-Business Suite Release 12.2.x environment to use Oracle Database 12c Release 1 (12.1.0.2) as a physical standby. Also the role transition switchover/Switchback tests have been explained in details. 


Overview:

Standby Database
A standby database is a transactionally consistent copy of the primary database. Using a backup copy of the primary database, you can create up to 30 standby databases and incorporate them in Oracle Data Guard configuration.
There are three types of standby databases:
  • Physical standby Provides a physically identical copy of the primary database, with on-disk database structures that are identical to the primary database on a block-for-block basis. The database schema, including indexes, is the same. A physical standby database is kept synchronized with the primary database by recovering the redo data received from the primary database.
  • Logical standby 
    Contains the same logical information as the primary database, although the physical organization and structure of the data can be different. It is kept synchronized with the primary database by transforming the data in the redo logs received from the primary database into SQL statements and then executing the SQL statements on the standby database.
  • Snapshot standby
    A fully updatable standby database. Like a physical or logical standby database, a snapshot standby database receives and archives redo data from a primary database. Unlike a physical or logical standby database, a snapshot standby database does not apply the redo data that it receives.

Oracle Data Guard
Oracle Data Guard is a set of services that create, manage, and monitor one or more standby databases to enable a primary database to survive disasters and data corruption. If the primary database becomes unavailable due to a planned or an unplanned outage, Oracle Data Guard can switch a standby database to the primary role, minimizing the downtime.
Data Guard offers three modes of data protection:
  • Maximum Protection
    This mode offers the highest level of data protection. Data is synchronously transmitted to the standby database from the primary database, and transactions are not committed on the primary database unless the redo data is available on at least one standby database configured in this mode. If the last standby database configured in this mode becomes unavailable, processing stops on the primary database. This mode guarantees no data loss.
  • Maximum Availability
    This mode is similar to the maximum protection mode, including no data loss. However, if a standby database becomes unavailable (for example, due to network connectivity problems), processing continues on the primary database. When the fault is corrected, the standby database is resynchronized with the primary database. If there is a need to failover before the standby database is resynchronized, some data may be lost.
  • Maximum Performance
    This mode offers slightly less data protection on the primary database, but higher performance than maximum availability mode. In this mode, as the primary database processes transactions, redo data is asynchronously shipped to the standby database. The commit operation on the primary database does not wait for the standby database to acknowledge receipt of redo data before completing write operations on the primary database. If any standby destination becomes unavailable, processing continues on the primary database, and there is little effect on primary database performance.


This is based on the classic/traditional architecture of an Oracle E-Business Suite 12.2 Data Guard configuration, where the systems are replicated across the primary and standby sites and primary application tier is connected with primary database and standby application tier is connected with standby database as seen in the diagram below:








Now lets jump to the steps.


Enable Forced Logging
To protect against unlogged direct writes in the primary database that cannot be propagated to the standby database, turn on FORCE LOGGING at the primary database before performing data file backups that will be used to create the standby database.
Use the following SQL command to set FORCE LOGGING on the primary database:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> ALTER DATABASE FORCE LOGGING;

Database altered.



Configure Oracle Net Communication To/From Standby System
PROD-Primary Server              Host-rac2.localdomain
PRODDR-Standby server         Host-dr.localdomain

In the standby server

Create a static listener which will be used in active duplication.

listener.ora

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PROD)
      (ORACLE_HOME = /u01/database/oracledb/product/12c/db_1)
    )
  )


LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST =dr.localdomain )(PORT = 1521))
    )
  )

ADR_BASE_LISTENER= /u01/database/oracledb/diag

cat tnsnames.ora

IFILE=/u01/database/oracledb/product/12c/db_1/network/admin/PRODDR_rac2_ifile.ora

[oracle@dr admin]$

[oracle@dr admin]$ cat PRODDR_rac2_ifile.ora
PRODDR=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (SERVER=DEDICATED)
            )
        )

PROD=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (INSTANCE_NAME=PROD)
            )
        )
[oracle@dr admin]


Now set the environment file.

[oracle@dr database]$ cat database.env
export ORACLE_SID=PROD
export ORACLE_HOME=/u01/database/oracledb/product/12c/db_1
export PATH=$PATH:/u01/database/oracledb/product/12c/db_1/bin


start the listener now

lsnrctl start

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 02-SEP-2018 11:47:38

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/oracledb/product/12c/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora
Log messages written to /u01/database/oracledb/diag/diag/tnslsnr/dr/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                02-SEP-2018 11:47:40
Uptime                    0 days 0 hr. 0 min. 1 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora
Listener Log File         /u01/database/oracledb/diag/diag/tnslsnr/dr/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))
Services Summary...
Service "PROD" has 1 instance(s).
  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully



tnsping PRODDR

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 02-SEP-2018 12:07:14

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=PROD) (SERVER=DEDICATED)))
OK (0 msec)


In the primary server

go to $TNS_ADMIN

under the sqlnet_ifile.ora

add the below

NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME)
SQLNET.EXPIRE_TIME= 10
tcp.validnode_checking =no
tcp.invited_nodes=(dr.localdomain)
SQLNET.INBOUND_CONNECT_TIMEOUT =60
SQLNET.ALLOWED_LOGON_VERSION_SERVER=8



Create the context ifile and add the below

PRODDR=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (SERVER=DEDICATED)
            )
        )

PROD=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (INSTANCE_NAME=PROD)
            )
        )



Test the tnsping from both the nodes.


Set Primary Database Initialization Parameters

In the standby 

Created a temporary init file and started the db in no mount.
[oracle@dr dbs]$ cat initPRODDR.ora
DB_NAME=PROD
DB_UNIQUE_NAME=PRODDR
DB_BLOCK_SIZE=8192

and start the dr server in nomount
[oracle@dr dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 2 14:47:12 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/u01/database/oracledb/product/12c/db_1/dbs/initPRODDR.ora';
ORACLE instance started.

Total System Global Area  218103808 bytes
Fixed Size                  2922712 bytes
Variable Size             159385384 bytes
Database Buffers           50331648 bytes
Redo Buffers                5464064 bytes
SQL> exit;


Set Up Secure Connections

orapwd file=orapwPROD password=pwd entries=10 ignorecase=y

Copy the same to the standby server.

Enable Archive Logging on Primary System
In my case my primary was already archive enabled

Add Standby Redo Logs



SQL> select bytes/1024/1024 as MB from v$log;

        MB
----------
      1000
      1000
Now add the standby redo logs
estimate the number of redo logs
SQL> select GROUP#,MEMBER from v$logfile;

    GROUP#
----------
MEMBER
--------------------------------------------------------------------------------
         2
/u01/database/PROD/data/log02a.dbf

         2
/u01/database/PROD/data/log02b.dbf

         1
/u01/database/PROD/data/log01a.dbf


    GROUP#
----------
MEMBER
--------------------------------------------------------------------------------
         1
/u01/database/PROD/data/log01b.dbf

now add the standby logs

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 3 '/u01/database/PROD/data/log03a.dbf' size 1000M;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 '/u01/database/PROD/data/log04.dbf' size 1000M;

Database altered.


Create a physical standby using active duplication method

Add the below parameters to the existing init file in the primary server
*.fal_server='PROD'
*.log_archive_config='dg_config=(PROD,PRODDR)'
*.log_archive_dest_2='service=PRODDR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=PRODDR'



Ensure the primary server is started with a spfile.

In  primary server:

Create rman duplicate script

run {
 allocate channel pri1 type disk;
 allocate auxiliary channel stby type disk;
 duplicate target database
  for standby
  from active database
  dorecover
  spfile
    set db_unique_name='PRODDR'
    set control_files= ' /u01/database/oracledb/data/cntrl01.dbf',' /u01/database/oracledb/data /cntrl02.dbf',' /u01/database/oracledb/data /cntrl03.dbf'
    set fal_server='PROD'
    set standby_file_management='AUTO'
    set log_archive_config='dg_config=(PROD,PRODDR)'
    set log_archive_dest_1='LOCATION= /u01/database/oracledb/data/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=PRODDR'
    set log_archive_dest_2='service=PRODDR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=PRODDR'
set log_file_name_convert=' /u01/database/PROD/data/',' /u01/database/oracledb /data/'
set  db_file_name_convert=' /u01/database/PROD/data/',' /u01/database/oracledb/data/'
nofilenamecheck;
  }


Now issue the rman duplicate session

rman target sys/pwd@PROD auxiliary sys/pwd@PRODDR

Recovery Manager: Release 12.1.0.2.0 - Production on Sun Sep 2 14:52:02 2018

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PROD (DBID=391996181)
connected to auxiliary database: PROD (not mounted)

RMAN>@duplicate.rmn


This completes the creation of physical standby database PRODDR and it will be in mount state.


Now in the standby server, start the mrp process
SQL> alter database recover managed standby database disconnect from session;

Database altered.

In Primary server
verify the parameter
SQL> show parameter log_archive_dest_state_2;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_state_2             string      enable





In the standby server

SQL> SELECT 'Last Applied : ' Logs,
TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#
FROM v$archived_log
WHERE sequence# =
(SELECT MAX(sequence#) FROM v$archived_log WHERE applied='YES'
)
UNION
SELECT 'Last Received : ' Logs,
TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#
FROM v$archived_log
WHERE sequence# =(SELECT MAX(sequence#) FROM v$archived_log );
  2    3    4    5    6    7    8    9   10   11
LOGS             TIME                  THREAD#  SEQUENCE#
---------------- ------------------ ---------- ----------
Last Applied :   03-SEP-18:09:38:45          1         67
Last Received :  03-SEP-18:09:38:45          1         67


In the primary server

> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u01/database/PROD/data
Oldest online log sequence     67
Next log sequence to archive   68
Current log sequence           68


Thus we can see from above that the archive logs are being shipped from primary server to standby server.

*************************************Configuration : Standby Application Tier(s) Connected to the Standby Database******************

run adpreclone in the run file system of the application tier.
[applmgr@rac2 scripts]$ perl adpreclone.pl appsTier

                     Copyright (c) 2011, 2014 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adpreclone Version 120.31.12020000.22

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

 Checking the status of the Oracle WebLogic Administration Server....

 Running perl /u01/application/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /u01/application/PROD/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/u01/application/PROD/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/u01/application/PROD/fs1/EBSapps/comn/clone component=appsTier method= appctx=/u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml showProgress



 Setting the wls environment

Beginning application tier Stage - Mon Sep  3 10:25:56 2018

/u01/application/PROD/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u01/application/PROD/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/u01/application/PROD/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/u01/application/PROD/fs1/EBSapps/comn/java/classes:/u01/application/PROD/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/u01/application/PROD/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/u01/application/PROD/fs1/FMW_Home/webtier/oui/jlib/share.jar:/u01/application/PROD/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u01/application/PROD/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/u01/application/PROD/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/application/PROD/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.StageAppsTier -e /u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml -stage /u01/application/PROD/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM   -showProgress -nopromptmsg

Log file located at /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/log/clone/StageAppsTier_09031025.log

  \     20% completed

Completed Stage...
Mon Sep  3 10:52:41 2018


As, run file system points to fs1, so we have created the directories fs1 and fs2 in dr and will copy the EBSapps directory from run file system of the primary to dr fs1.






Now create the appsutil directory in the dr server.

In the primary application server

[applmgr@rac2 PROD]$ perl $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/log/Make                                                                                        AppsUtil_09032132.log
output located at /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/out/appsut                                                                                        il.zip
MakeAppsUtil completed successfully.



[applmgr@rac2 PROD]$ scp /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/out/appsutil.zip oracle@192.168.X.X:/u01/database/oracledb/product/12c/db_1
oracle@192.168.X.X's password:
appsutil.zip                                                                                                                          100% 3685KB   3.6MB/s   00:00
[applmgr@rac2 PROD]$

in the dr server
[oracle@dr db_1]$ unzip -o appsutil.zip
Archive:  appsutil.zip
   creating: appsutil/
thus appsutil directory is now created.




Now before creating the context file, we should convert the database to snapshot standby.
SQL> alter system set db_recovery_file_dest_size=32g scope=both;

System altered.

SQL> alter system set db_recovery_file_dest='/u01/database/flas_recovery_area' scope=both;

System altered.

SQL> alter system set db_flashback_retention_target=1440 scope=both;

System altered.
Stop redo apply on the physical standby database:
SQL> alter database recover managed standby database cancel;

Database altered.

Turn on flashback logging:
SQL> alter database flashback on;


Database altered.

Convert the standby database into a snapshot standby database:

SQL> alter database convert to snapshot standby;

Database altered.

SQL>
SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             452987000 bytes
Database Buffers         1677721600 bytes
Redo Buffers               13848576 bytes
Database mounted.
Database opened.
remove the application config info

SQL*Plus: Release 12.1.0.2.0 Production on Mon Sep 3 21:55:00 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Mon Sep 03 2018 10:25:27 -04:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> exec ad_zd_fixer.clear_valid_nodes_info;

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL>



now create the context file. before that Copy the jre from primary database server $ORACLE_HOME/appsutil/jre

[oracle@dr bin]$ perl adbldxml.pl appsuser=apps appspass=welcome123 template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp out=$ORACLE_HOME/appsutil/PROD_dr.xml

Starting context file generation for db tier..
Using JVM from /u01/database/oracledb/product/12c/db_1/appsutil/jre/bin/java to execute java programs..

The log file for this adbldxml session is located at:
/u01/database/oracledb/product/12c/db_1/appsutil/log/adbldxml_09032214.log
AC-20010: Error: File - listener.ora could not be found at the location:
        /u01/app/oracledb/product/12c/db_1/network/admin/listener.ora
indicated by TNS_ADMIN. Context file can not be generated.

Could not Connect to the Database with the above parameters, Please answer the Questions below


Enter Hostname of Database server: dr

Enter Port of Database server: 1521

Enter SID of Database server: PROD

Enter Database Service Name: PROD

Enter the value for Display Variable: dr:1.0

The context file has been created at:
/u01/database/oracledb/product/12c/db_1/appsutil/PROD_dr.xml


run autoconfig now:

[oracle@dr PROD_dr]$ sh adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /u01/database/oracledb/product/12c/db_1/appsutil/log/PROD_dr/09032312/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u01/database/oracledb/product/12c/db_1
        Classpath                   : :/u01/database/oracledb/product/12c/db_1/jdbc/lib/ojdbc6.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java/xmlparserv2.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java:/u01/database/oracledb/product/12c/db_1/jlib/netcfg.jar:/u01/database/oracledb/product/12c/db_1/jlib/ldapjclnt12.jar

        Using Context file          : /u01/database/oracledb/product/12c/db_1/appsutil/PROD_dr.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.


Now configure the application tier
[applmgr@dr bin]$ perl adcfgclone.pl appsTier dualfs

                     Copyright (c) 2002, 2015 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.60

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] : no


Running: Context clone...

Log file located at /u01/application/fs1/EBSapps/comn/clone/bin/CloneContext_0904023938.log

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [dr] :

Target System Database SID : PROD

Target System Database Server Node [dr] :

Target System Database Domain Name [localdomain] :

Target System Base Directory : /u01/application

Target System Base Directory set to /u01/application

Target System Current File System Base set to /u01/application/fs1

Target System Other File System Base set to /u01/application/fs2

Target System Fusion Middleware Home set to /u01/application/fs1/FMW_Home
Target System Other File System Fusion Middleware Home set to /u01/application/fs2/FMW_Home

Target System Web Oracle Home set to /u01/application/fs1/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u01/application/fs2/FMW_Home/webtier

Target System Appl TOP set to /u01/application/fs1/EBSapps/appl
Target System Other File System Appl TOP set to /u01/application/fs2/EBSapps/appl

Target System COMMON TOP set to /u01/application/fs1/EBSapps/comn
Target System Other File System COMMON TOP set to /u01/application/fs2/EBSapps/comn

Target System Instance Home Directory [/u01/application] :

Target System Current File System Instance Top set to /u01/application/fs1/inst/apps/PROD_dr

Do you want to preserve the Display [rac2:0.0] (y/n)  : n

Target System Display [dr:0.0] :

Target System Root Service [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 1

Checking the port pool 1
done: Port Pool 1 is free
Report file located at /u01/application/fs1/inst/apps/PROD_dr/admin/out/portpool.lst
RC-40201: Unable to connect to Database PROD.

Enter the Database listener port [1522] : 1521

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/tmp
2. /usr/tmp
3. /u01/database/PROD/12.1.0/appsutil/outbound/PROD_rac2
4. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 1
The new APPL_TOP context file has been created :
  /u01/application/fs1/inst/apps/PROD_dr/appl/admin/PROD_dr.xml
Check Clone Context logfile /u01/application/fs1/EBSapps/comn/clone/bin/CloneContext_0904023938.log for details.

Creating Patch file system context file.....

Log file located at /u01/application/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0904024546.log

Target System Other File System Instance Top set to /u01/application/fs2/inst/apps/PROD_dr

Target System Port Pool [0-99] : 2

Checking the port pool 2
done: Port Pool 2 is free
Report file located at /u01/application/fs2/inst/apps/PROD_dr/admin/out/portpool.lst
The new APPL_TOP context file has been created :
  /u01/application/fs2/inst/apps/PROD_dr/appl/admin/PROD_dr.xml
Check Clone Context logfile /u01/application/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0904024546.log for details.

FMW Pre-requisite check log file location : /u01/application/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...

Configuring: Run file system....
LogFile located at /u01/application/fs1/inst/apps/PROD_dr/admin/log/clone/run/RCloneApplyAppstier_09040246.log
Configuring: Patch file system....
LogFile located at /u01/application/fs1/inst/apps/PROD_dr/admin/log/clone/patch/RCloneApplyAppstier_09040359.log


Do you want to startup the Application Services for PROD? (y/n) [n] : n

Services not started

this completes configuring apps tier configuration in dr server.


Revert the Physical Standby Database to its Original State

Change the standby database from snapshot mode back into standby mode using the following commands:
Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing opt                                                                                        ions

SQL> startup mount force;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             486541432 bytes
Database Buffers         1644167168 bytes
Redo Buffers               13848576 bytes
Database mounted.
SQL> alter database convert to physical standby;

Database altered.

SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup nomount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             486541432 bytes
Database Buffers         1644167168 bytes
Redo Buffers               13848576 bytes
SQL> alter database mount standby database;

Database altered.

SQL>

enable redo log apply
SQL> alter database recover managed standby database using current logfile disconnect;

Database altered.




Performing role transitions

I have illustrated the switchover and switchback steps. If you are familiar with these steps, then failover can also be done easily which is not in the scope of this post.

Prepare for a switchover to the standby server

  1. Verify the primary database instance is open and the standby database instance is mounted.
     
  2. Shut down all the application tier services on all nodes on the primary site.
    $ adstpall.sh <appsuser>/<appspass>
  3. Verify there are no active users connected to the database. Shut down all the sessions on the primary database.
  4. Use the following command on both the primary and standby databases to ensure that the last redo data transmitted from the primary database has been applied on the standby database. If necessary, perform a log file switch before proceeding further:.
    SQL>select sequence#,applied from v$archived_log;

Check whether the primary is ready for a switch. Query the switchover_status column of the v$database fixed view to determine whether the database is ready to switch modes

SQL> select switchover_status from v$database;

SWITCHOVER_STATUS
--------------------

TO STANDBY


 Initiate a switchover on the primary database

Connect as sysdba and issue the following SQL command:

SQL> alter database commit to switchover to physical standby with session shutdown;

Database altered.
When this statement has completed, the primary database will have been converted to a standby database. During the statement's execution, the current control file is backed up to the current SQL session's trace file - this makes it possible to reconstruct the control file should the need arise.

Shut down and mount the primary database

To complete the transition, the database must be shut down and restarted in a mounted state. Change the value of LOG_ARCHIVE_DEST_STATE_2 to defer, in addition, recovery can be started in preparation for transition.
SQL> shutdown immediate;
ORA-01012: not logged on
SQL> exit;
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@rac2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 4 04:08:40 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>
SQL> startup nomount;
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             452987000 bytes
Database Buffers         1677721600 bytes
Redo Buffers               13848576 bytes
SQL> alter database mount standby database;

Database altered.

SQL> alter system set LOG_ARCHIVE_DEST_STATE_2=defer scope=both sid='*';

System altered.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> exit;
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@rac2 dbs]$ ps -ef|grep mrp
oracle   11858     1  1 04:10 ?        00:00:00 ora_mrp0_PROD
oracle   11865  8370  0 04:10 pts/2    00:00:00 grep mrp
[oracle@rac2 dbs]$


At this point in the switchover, both databases are standby databases.

Verify the switchover status on the standby server

As the ORACLE user on the standby-to-become-the-primary database server, verify that it is ready to switch to the primary:
SQL> select switchover_status from v$database;

SWITCHOVER_STATUS
--------------------
TO PRIMARY


SQL> alter database commit to switchover to primary with session shutdown;

Database altered.


Adjust the network parameters:

In the context ifile.ora change the host name in the primary as well as in the standby database.

cat PROD_rac2_ifile.ora
PRODDR=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (SERVER=DEDICATED)
            )
        )

PROD=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (INSTANCE_NAME=PROD)
            )
        )



in the new primary server
PRODDR=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (SERVER=DEDICATED)
            )
        )

PROD=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (INSTANCE_NAME=PROD)
            )


On the current primary, change the LOG_ARCHIVE_DEST_STATE_2 to enable using the following command:

SQL> alter system set LOG_ARCHIVE_DEST_STATE_2=enable scope=both sid='*';

System altered.
To complete the transition, the database must be shut down and re-started:

SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             486541432 bytes
Database Buffers         1644167168 bytes
Redo Buffers               13848576 bytes
Database mounted.
Database opened.


Complete the database configuration

  1. Connect to the new primary database using SQL*Plus as the APPS user, and execute the following commands:
SQL> exec fnd_net_services.remove_system ('PROD');

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> exec ad_zd_fixer.clear_valid_nodes_info;

PL/SQL procedure successfully completed.

SQL>

As the ORACLE user on the new primary database server, use AutoConfig to complete the configuration for the primary operations, providing the APPS password when prompted


[oracle@dr PROD_dr]$ sh adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /u01/database/oracledb/product/12c/db_1/appsutil/log/PROD_dr/09041329/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u01/database/oracledb/product/12c/db_1
        Classpath                   : :/u01/database/oracledb/product/12c/db_1/jdbc/lib/ojdbc6.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java/xmlparserv2.jar:/u01/database/oracledb/product/12c/db_1/appsutil/java:/u01/database/oracledb/product/12c/db_1/jlib/netcfg.jar:/u01/database/oracledb/product/12c/db_1/jlib/ldapjclnt12.jar

        Using Context file          : /u01/database/oracledb/product/12c/db_1/appsutil/PROD_dr.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.

stop and start the listener on the new primary database server:
[oracle@dr database]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 14:11:21

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))
The command completed successfully
[oracle@dr database]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 14:11:27

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/oracledb/product/12c/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora
Log messages written to /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     PROD
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                04-SEP-2018 14:11:27
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora
Listener Log File         /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))
Services Summary...
Service "PROD" has 1 instance(s).
  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@dr database]$

On the new standby server, stop and start the listener for the standby services
[oracle@rac2 database]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 06:36:55

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))
The command completed successfully
[oracle@rac2 database]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 06:37:01

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/PROD/12.1.0/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora
Log messages written to /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     PROD
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                04-SEP-2018 06:37:01
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora
Listener Log File         /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))
Services Summary...
Service "PROD" has 1 instance(s).
  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Configure the Application Tier After Role Transition (Switchover)

Once the primary database configuration has completed and its listeners have started, log in to each new-primary application tier server as the APPLMGR user, and run the final configuration steps on the run file systems providing the APPS password when prompted::
[applmgr@dr scripts]$ sh adautocfg.sh
Enter the APPS user password:

The log file for this session is located at: /u01/application/fs1/inst/apps/PROD_dr/admin/log/09041433/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.
        Using CONFIG_HOME location     : /u01/application/fs1/inst/apps/PROD_dr
        Classpath                   : /u01/application/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/application/fs1/EBSapps/comn/java/classes

        Using Context file          : /u01/application/fs1/inst/apps/PROD_dr/appl/admin/PROD_dr.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...
        Configuring AD_TOP........COMPLETED
        Configuring FND_TOP.......COMPLETED
        Configuring ICX_TOP.......COMPLETED
        Configuring MSC_TOP.......COMPLETED
        Configuring IEO_TOP.......COMPLETED
        Configuring BIS_TOP.......COMPLETED
        Configuring CZ_TOP........COMPLETED
        Configuring SHT_TOP.......COMPLETED
        Configuring AMS_TOP.......COMPLETED
        Configuring CCT_TOP.......COMPLETED
        Configuring WSH_TOP.......COMPLETED
        Configuring CLN_TOP.......COMPLETED
        Configuring OKE_TOP.......COMPLETED
        Configuring OKL_TOP.......COMPLETED
        Configuring OKS_TOP.......COMPLETED
        Configuring CSF_TOP.......COMPLETED
        Configuring IBY_TOP.......COMPLETED
        Configuring JTF_TOP.......COMPLETED
        Configuring MWA_TOP.......COMPLETED
        Configuring CN_TOP........COMPLETED
        Configuring CSI_TOP.......COMPLETED
        Configuring WIP_TOP.......COMPLETED
        Configuring CSE_TOP.......COMPLETED
        Configuring EAM_TOP.......COMPLETED
        Configuring GMF_TOP.......COMPLETED
        Configuring PON_TOP.......COMPLETED
        Configuring FTE_TOP.......COMPLETED
        Configuring ONT_TOP.......COMPLETED
        Configuring AR_TOP........COMPLETED
        Configuring AHL_TOP.......COMPLETED
        Configuring IES_TOP.......COMPLETED
        Configuring OZF_TOP.......COMPLETED
        Configuring CSD_TOP.......COMPLETED
        Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.
[applmgr@dr scripts]$
On the patch file system change to $AD_TOP/bin/ and run the following command:
[applmgr@dr bin]$ perl adconfig.pl contextfile=/u01/application/fs2/inst/apps/PROD_dr/appl/admin/PROD_dr.xml -syncctx
Enter the APPS user password:

The log file for this session is located at: /u01/application/fs2/inst/apps/PROD_dr/admin/log/09041447/adconfig.log

Option specified      : Synchronize context file
Only context file synchronization will be performed

        Classpath                   : /u01/application/fs2/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/application/fs2/EBSapps/comn/java/classes

        Using ContextFile     : /u01/application/fs2/inst/apps/PROD_dr/appl/admin/PROD_dr.xml

        Synchronizing the context file ......COMPLETED

AutoConfig completed successfully.
[applmgr@dr bin]$

Update the Host Name in the FND_CONCURRENT_REQUESTS and FND_CONC_REQ_OUTPUTS tables

In order to synchronize your concurrent manager log and out directories, you must change the host name in the fnd_concurrent_requests table to the standby server name:
SQL> update apps.fnd_concurrent_requests
set logfile_node_name ='DR',
outfile_node_name = 'DR'
where logfile_node_name ='RAC2'
and outfile_node_name ='RAC2';  2    3    4    5

638 rows updated.

SQL> commit;

Commit complete


:rsync the concurrent log and out files from old primary application server to new ones:

rsync -avz log applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc
rsync -avz out applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc

Now start the application services in the new primary servers run file system and redirect the user to the new url


http://dr.localdomain:8001/OA_HTML/AppsLocalLogin.jsp


Now we will do the switchback which will make primary to standby and standby to primary

a.Verify that the primary database at the standby site is open, and the standby database at the primary site is mounted.
b. Verify all the redo logs are transferred to standby and are applied.
c. On the primary site, check whether the switchover_status from v$database is showing: TO STANDBY


in primary
SQL> select switchover_status from v$database;

SWITCHOVER_STATUS
--------------------
TO STANDBY
On the primary database, issue the command:

SQL> alter database commit to switchover to physical standby;

Database altered.

So this has shut down my database.
[oracle@dr dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 5 04:38:42 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             503318648 bytes
Database Buffers         1627389952 bytes
Redo Buffers               13848576 bytes
SQL> alter database mount standby database;

Database altered.

SQL> alter system set LOG_ARCHIVE_DEST_STATE_2=defer scope=both sid='*';

System altered.

SQL>


in the standby
SQL> select switchover_status from v$database;

SWITCHOVER_STATUS
--------------------
TO PRIMARY

SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PHYSICAL STANDBY


in the standby init file
changed the below parameters
*.log_archive_config='dg_config=(PROD,PRODDR)'
*.log_archive_dest_1='LOCATION=/u01/database/PROD/data'
*.log_archive_dest_2='service=PRODDR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=PRODDR'
*.log_archive_dest_state_2='enable

now in the standby


SQL> alter database commit to switchover to primary with session shutdown;

Database altered.

SQL> exit;
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@rac2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 4 21:17:08 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from v$database;

OPEN_MODE            DATABASE_ROLE
-------------------- ----------------
MOUNTED              PRIMARY


As the ORACLE user on both the primary and standby database servers, change the host name in the standby service definitions to point to the new standby server in both the <CONTEXT_NAME>_ifile.ora and <STNDBY_CONTEXT>_ifile.ora (which are located in the <TNS_ADMIN> directory(already done in the above sections, you can refer it to interchange the host names again)


now in the primary
[oracle@rac2 dbs]$ pwd
/u01/database/PROD/12.1.0/dbs
[oracle@rac2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 4 21:21:55 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/u01/database/PROD/12.1.0/dbs/initPROD.ora';
ORA-32006: SEC_CASE_SENSITIVE_LOGON initialization parameter has been deprecated
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             452987000 bytes
Database Buffers         1677721600 bytes
Redo Buffers               13848576 bytes
Database mounted.
SQL> create spfile from pfile;

File created.

SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size             452987000 bytes
Database Buffers         1677721600 bytes
Redo Buffers               13848576 bytes
Database mounted.
Database opened.
SQL> select database_role,open_mode from v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PRIMARY          READ WRITE



in the stanbdy started the mrp process
[oracle@dr admin]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Sep 5 04:57:37 2018

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> alter database recover managed standby database disconnect from session;

Database altered.

Now verify if the archives are getting shipped or not
in the primary
SQL> alter system switch logfile;

System altered.

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u01/database/PROD/data
Oldest online log sequence     128
Next log sequence to archive   129
Current log sequence           129
SQL>

in the stanbdy
SQL> SELECT 'Last Applied : ' Logs,
TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#
FROM v$archived_log
WHERE sequence# =
(SELECT MAX(sequence#) FROM v$archived_log WHERE applied='YES'
)
UNION
SELECT 'Last Received : ' Logs,
TO_CHAR(next_time,'DD-MON-YY:HH24:MI:SS') TIME,thread#,sequence#
FROM v$archived_log
WHERE sequence# =(SELECT MAX(sequence#) FROM v$archived_log );    2    3    4    5    6    7    8    9   10   11

LOGS             TIME                  THREAD#  SEQUENCE#
---------------- ------------------ ---------- ----------
Last Applied :   04-SEP-18:21:29:46          1        128
Last Received :  04-SEP-18:21:29:46          1        128
Connect to the new primary database using SQL*Plus as the APPS user, and execute the following commands:

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Tue Sep 04 2018 16:03:42 -04:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> exec fnd_net_services.remove_system ('PROD');

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> exec ad_zd_fixer.clear_valid_nodes_info;

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL>
run autoconfig
[oracle@rac2 PROD_rac2]$ sh adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /u01/database/PROD/12.1.0/appsutil/log/PROD_rac2/09042141/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u01/database/PROD/12.1.0
        Classpath                   : :/u01/database/PROD/12.1.0/jdbc/lib/ojdbc6.jar:/u01/database/PROD/12.1.0/appsutil/java/xmlparserv2.jar:/u01/database/PROD/12.1.0/appsutil/java:/u01/database/PROD/12.1.0/jlib/netcfg.jar:/u01/database/PROD/12.1.0/jlib/ldapjclnt12.jar

        Using Context file          : /u01/database/PROD/12.1.0/appsutil/PROD_rac2.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.
[oracle@rac2 PROD_rac2]$

in the primary server and standby server
start stop listener
[oracle@rac2 PROD_rac2]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 21:42:42

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))
The command completed successfully
[oracle@rac2 PROD_rac2]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 04-SEP-2018 21:42:46

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/PROD/12.1.0/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora
Log messages written to /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     PROD
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                04-SEP-2018 21:42:46
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/database/PROD/12.1.0/network/admin/PROD_rac2/listener.ora
Listener Log File         /u01/database/PROD/12.1.0/admin/PROD_rac2/diag/tnslsnr/rac2/prod/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2.localdomain)(PORT=1521)))
Services Summary...
Service "PROD" has 1 instance(s).
  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@rac2 PROD_rac2]$
in standby
[oracle@dr admin]$ lsnrctl stop PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 05-SEP-2018 05:15:33

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))
The command completed successfully
[oracle@dr admin]$ lsnrctl start PROD

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 05-SEP-2018 05:15:38

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/database/oracledb/product/12c/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora
Log messages written to /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     PROD
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                05-SEP-2018 05:15:38
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/database/oracledb/product/12c/db_1/network/admin/listener.ora
Listener Log File         /u01/database/oracledb/product/12c/db_1/admin/PROD_dr/diag/tnslsnr/dr/prod/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dr.localdomain)(PORT=1521)))
Services Summary...
Service "PROD" has 1 instance(s).
  Instance "PROD", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@dr admin]$


Configure the Application Tier After Role Transition switchback

in the run file system, run autoconfig
[applmgr@rac2 scripts]$ sh adautocfg.sh
Enter the APPS user password:

The log file for this session is located at: /u01/application/PROD/fs1/inst/apps/PROD_rac2/admin/log/09042147/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.
        Using CONFIG_HOME location     : /u01/application/PROD/fs1/inst/apps/PROD_rac2
        Classpath                   : /u01/application/PROD/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/application/PROD/fs1/EBSapps/comn/java/classes

        Using Context file          : /u01/application/PROD/fs1/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...
        Configuring AD_TOP........COMPLETED
        Configuring FND_TOP.......COMPLETED
        Configuring ICX_TOP.......COMPLETED
        Configuring MSC_TOP.......COMPLETED
        Configuring IEO_TOP.......COMPLETED
        Configuring BIS_TOP.......COMPLETED
        Configuring CZ_TOP........COMPLETED
        Configuring SHT_TOP.......COMPLETED
        Configuring AMS_TOP.......COMPLETED
        Configuring CCT_TOP.......COMPLETED
        Configuring WSH_TOP.......COMPLETED
        Configuring CLN_TOP.......COMPLETED
        Configuring OKE_TOP.......COMPLETED
        Configuring OKL_TOP.......COMPLETED
        Configuring OKS_TOP.......COMPLETED
        Configuring CSF_TOP.......COMPLETED
        Configuring IBY_TOP.......COMPLETED
        Configuring JTF_TOP.......COMPLETED
        Configuring MWA_TOP.......COMPLETED
        Configuring CN_TOP........COMPLETED
        Configuring CSI_TOP.......COMPLETED
        Configuring WIP_TOP.......COMPLETED
        Configuring CSE_TOP.......COMPLETED
        Configuring EAM_TOP.......COMPLETED
        Configuring GMF_TOP.......COMPLETED
        Configuring PON_TOP.......COMPLETED
        Configuring FTE_TOP.......COMPLETED
        Configuring ONT_TOP.......COMPLETED
        Configuring AR_TOP........COMPLETED
        Configuring AHL_TOP.......COMPLETED
        Configuring IES_TOP.......COMPLETED
        Configuring OZF_TOP.......COMPLETED
        Configuring CSD_TOP.......COMPLETED
        Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.
in the patch file system
[applmgr@rac2 PROD]$ cd $AD_TOP/bin
[applmgr@rac2 bin]$ echo $CONTEXT_FILE
/u01/application/PROD/fs2/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml
[applmgr@rac2 bin]$ perl adconfig.pl contextfile=/u01/application/PROD/fs2/inst/apps/PROD_rac2/appl/admin/PROD_rac2.xml -syncctx



Update the host names for concurrent requests log and out files to point to new primary after switchback

update apps.fnd_concurrent_requests set logfile_node_name ='RAC2',outfile_node_name ='RAC2' where logfile_node_name ='DR' and outfile_node_name ='DR';

645 rows updated.

SQL> commit;

Commit complete.


rsync the concurrent log and out files between primary and secondary

rsync -avz log applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc

rsync -avz out applmgr@192.168.x.x:/u01/application/fs_ne/inst/PROD_dr/logs/appl/conc

start the application services and redirect the user to new url

http://rac2.localdomain:8000/OA_HTML/AppsLocalLogin.jsp



Once you get comfortable with switchover/Switchback process, then failover steps can also be easily done. At this moment, failover is out of scope for this post.


Hope this post will help someone. Happy learning..:)






 










4 comments:

  1. Hi samrat,
    Thanks for sharing your knowledge,i'am going to clone my EBS 12.2.5 from single instance to exadata RAC environment, hope you can help in the steps.

    Thanks
    Abdulqader

    ReplyDelete
  2. Hi Sam , How does Application on DR get sync up with primary application ?
    Any context file changes?

    ReplyDelete
  3. Oracle Apps Dba Pieces: Creating Physical Standby Database For Oracle E-Business Suite Release 12.2.7 Using Oracle 12C (12.1.0.2) >>>>> Download Now

    >>>>> Download Full

    Oracle Apps Dba Pieces: Creating Physical Standby Database For Oracle E-Business Suite Release 12.2.7 Using Oracle 12C (12.1.0.2) >>>>> Download LINK

    >>>>> Download Now

    Oracle Apps Dba Pieces: Creating Physical Standby Database For Oracle E-Business Suite Release 12.2.7 Using Oracle 12C (12.1.0.2) >>>>> Download Full

    >>>>> Download LINK 3m

    ReplyDelete