Abort a patch applied to run file system via Hotpatch in EBS 12.2.x

In EBS 12.2.x, we may get this error sometimes while applying the patch using adop.

APPL_TOP is set to /ERPAPP/TEST/fs1/EBSapps/appl
[STATEMENT] [START 2016/07/04 12:36:16] Determining admin node
[STATEMENT] [END 2016/07/04 12:36:18] Determining admin node
[STATEMENT] Recommended Worker Count: 2 Max Allowed Workers: 9
[STATEMENT] [START 2016/07/04 12:36:22] Acquiring lock on sessions table
[STATEMENT] [END 2016/07/04 12:36:23] Acquiring lock on sessions table
[STATEMENT] [START 2016/07/04 12:36:23] Checking for any pending sessions
[STATEMENT] There is already a session which is incomplete. Details are:
[STATEMENT] Session Id: 2
[STATEMENT] Prepare phase status: Y
[STATEMENT] Apply phase status: P
[STATEMENT] Cutover phase status: N
[STATEMENT] Abort phase status: N
[STATEMENT] Session status: F
[STATEMENT] [Note: Y denotes that the phase is done
[STATEMENT] N denotes that the phase has not been completed
[STATEMENT] X denotes that the phase is not applicable
[STATEMENT] R denotes that the phase is running (in progress) or ran
[STATEMENT] F denotes that the phase has failed
[STATEMENT] P (is applicable only to APPLY phase) denotes atleast
[STATEMENT] one patch is already applied for the session id
[ERROR] Cannot apply hotpatch as another online patching cycle is going on
[ERROR] Unrecoverable error occured. Exiting the current session.


The reason for this error is because another patching session is active.
[applmgr@cvm-928 ~]$ adop -status
Enter the APPS username: apps
Enter the APPS password:

Current Patching Session ID: 2


Node Name       Node Type       Phase       Status          Started                        Finished                       Elapsed
--------------- --------------- ----------- --------------- ------------------------------ ------------------------------ ------------
cvm-928         master          APPLY       FAILED          30-JUN-16 01:44:53 +05:30      30-JUN-16 01:48:01 +05:30      0:03:08
                                PREPARE     COMPLETED       30-JUN-16 12:07:26 +05:30      30-JUN-16 12:37:40 +05:30      0:30:14
                                CUTOVER     NOT STARTED
                                CLEANUP     NOT STARTED

File System Synchronization Used in this Patching Cycle: None

Solution: Abort the patching cycle and run full cleanup.

adop phase=abort

adop phase=cleanup cleanup_mode=full

Patch can be rolled back till we reached the cutover phase. Once we reach the cutover phase, we cannot abort a phase. Thus the question comes, what will we do, if some problem occur which eventually lead to failure of cutover phase.Taking a backup is always recommended before applying a patch but it is too costly. We cannot take a cold back up of EBS application filesystema and Database everytime just to apply a single patch. There is a time constraint, downtime and cost involved to keep the backup.

Thus we have to go for flashback option for database where we can roll back the database just before the cutover stage. To enable the flashback option, execute the below statements


Connect to the oracle database

1. sqlplus / as sysdba

alter system set db_recovery_file_dest='/oracle/stage/backups' scope=BOTH SID='*';

alter system set db_recovery_file_dest_size=200GB scope=BOTH SID='*';

alter system set db_flashback_retention_target=300;(this is in minutes, set it according to the environment)

alter database flashback on;


2. Stop the ebs services, conc processing and other outgoing interface just to ensure that there won't be any loss of data.

3.start the patching cycle

adop phase=prepare,apply,finalize

4. Create a restore point

alter system switch logfile;
create restore point ABc guarantee flashback database;
alter system switch logfile;

5. execute adop phase=cutover.

if we face any issue, we can flashback the database to the restore point.

shutdown immediate;
startup mount;
flashback database to restore point ABC;
alter database open read only;

shutdown immediate;
startup mount;
alter database open resetlogs;

6.
Disable flashback
alter database flashback off;
drop restore point ABC;
Restore point dropped.
alter system set db_recovery_file_dest='';
System altered.
select FLASHBACK_ON from v$database;


now we also have to revert back the appliction file system. By checking the cutover logs, we can check that if the cutover has failed before the cutover happened or after the cutover. If the cutover has not happened, then just clean shutdown and startup the ebs services.

if the cutover has happened and file system has switched, then we have to switch the filesystem again.

shutdown the ebs services
perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -action=ctxupdate -contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>

and start the services from the old file system

In a multi-node environment, repeat the preceding two steps on all nodes, starting with the admin node and then proceeding to the slave nodes.

Options and Next Steps
After the restore is complete, you have two basic options for proceeding:
Abort the current patching cycle, if the issue that required you to restore was caused by the patches you were attempting to apply.

Identify and fix any other issues in the current patching cycle, and proceed with patching.


Reference-Document 1584097.1, Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover.

https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531065.htm