Sunday, January 22, 2017

Auditing in EBS(Though Old but still Gold)

The online monitoring of user activity within Oracle Applications is achieved via the Monitor Users form (Form Name: FNDSCMON.fmx). The navigation path within the Sysadmin responsibility is Security > User > Monitor. In order to activate the capability of this form one has to select an appropriate option for the profile option Sign-On: Audit Level Profile Option. The available options are: -
NONE - No monitoring.
USER - Will only show you a list of logged in users.
RESPONSIBILITY - Will show you the user logged in and the responsibility they are using.
FORM - Will go down to the lowest level of detail and show you the User, Responsibility and Form being accessed.
The screen shot below indicates what you would see if you had chosen the FORM option for the profile option in question.
Log in with System Administrator Responsibility
Navigate to Security: Users > Monitor

Click (CTL + F11) keys to display the result.






This is a very useful screen since it tells you exactly which users are logged in and what are they doing in the system at any point in time. One may check this screen before bouncing or restarting the system to make sure all users are logged out.


It is a good practice to set the Sign-On: Audit Level profile option to “Form” since it gives the most detailed information above other choices but it will impact the system performance since it collects a lot of information, so you have keep that in mind.


Viewing Monitoring Reports about Users and their activity:
Depending on what audit level you have selected for the profile option under discussion you may also generate various reports as indicated below: –
Sign-On Audit Concurrent Requests: View information about who is requesting what concurrent requests and from which responsibilities and forms.
Sign-On Audit Forms:View who is navigating to what form and when they do it.
Sign-On Audit Responsibilities: Used to view who is selecting what responsibility and when they are doing it.
Sign-On Audit Users:Used to view who signs on and for how long.
Sign-On Audit Unsuccessful: Show audit information about unsuccessful logins to Oracle Applications.

To view any of the given reports monitoring user activity use following steps:
Navigate to System Administrator Responsibility > Concurrent > Requests Or from the Menu Bar go to View > Requests
Choose Submit New Request > Single Request
Select report you want from the 4 requests given above.
Click Submit > Find
Select report you choose and click View output button to view the report.
Notifying Users of Unsuccessful Logins to their accounts:

Sign-On Audit can track user logins and provide users with a warning message if anyone has made an unsuccessful attempt to sign on with their application username since their last sign-on. This warning message appears after a user signs on. You do not have to audit the user with Sign-On Audit to use this notification feature.

To inform users about unsuccessful logins to their account, you can set the “Sign-On: Notification” profile option to Yes. To do that from System Administrator Responsibility > Profile > System > Find the profile option “Sign-On: Notification” and change its value to Yes.

Page access tracking:
Page Access Tracking in Oracle Applications Manager reports aggregate data for OAF- and JTF-based applications, with Sign-on Audit data gathered for Forms-based applications. Consequently, you can view usage flow paths that span all three technology stacks.
Page Access Tracking allows administrators to track application usage statistics and perform Web site traffic analysis, aggregate data about user flows as well as drill down to a particular user session.
Page Access Tracking transparently captures application-rich context information for every user click. In terms of performance, the data capture has negligible overhead.


Enable page tracking.
Go to OAM>Site Map > Monitoring > Applications Usage Reports > Page Access Tracking and Sign-on Audit Configuration>>Click on Configurations




And then at the bottom, click on apply. Next Migrating Page Access Tracking Data
Page Access Tracking data is logged in the database in a staging area. It needs to be migrated and mined before the UI reports are refreshed. How frequently Page AccessTracking Data needs to be migrated and mined depends on how up-to-date the UI reports need to be. Generally speaking, the recommendation is to run the concurrent program at least once a day (for example at midnight) for the reports to be relatively up to date.
The "Page Access Tracking Data Migration" can be scheduled either in Oracle Forms, or within the OAM responsibility.
Schedule the concurrent program "Page Access Tracking Data Migration" using Concurrent Manager:

After the Concurrent Program completes, you should be able to view the latest data reports and statistic


To view the output:


Click on the respective pointers to know more about in details. For example, I clicked on one of the pointers under Responsibility for 16th Jan 2017




We can also click on the respective pointers to drill down further.


Debug Logging (Unexpected Logging)
The Oracle E-Business Suite Debug Log is used for a variety of purposes. As the name suggests, it is often used to diagnosing and debugging problems encountered in Oracle E-Business Suite code. This log can also assist with diagnosing security problems and detecting security attacks. It can also be leveraged for post-attack / forensic analysis.
The debug log is documented in the Oracle E-Business Suite Maintenance Guide - Logging. The current recommendation in this chapter is to set this to “UNEXPECTED”. This is important from a security auditing perspective, since this is the level at which many of the security errors are written out to the log.
The default configuration (and the current recommendation) for Debug Logging is set to log information to the database. This makes it easier to correlate and maintain logs in a multi-tier application environment.
The Debug Logging mechanism also supports logging to the file system using the following profile:
AFLOG_FILENAME
FND: Debug Log Filename
"/path/to/apps.log"
Customers may want to consider enabling logging on the file system, as there are some security benefits to maintaining the log on the file system. File system logging will generally provide better protection against an attacker being able to modify logging records via a SQL injection attack.

Example for collecting OAF Log files:



Enable Diagnostics -- Set User (should use same user to reproduce the issue) level system profile values as follows:
FND: Debug Log Level = Statement
FND: Debug Log Enabled = Yes
FND: Debug Log Module = %
FND: Debug Filename = /tmp/trace.log - Should have proper write permissions.
FND: Diagnostics = Yes
Step 2 
The Diagnostics button should now be visible on the page - next to Close window/Preferences/Help/Personalise Page
1. Obtain the Trace File
a: Click on the Diagnostics button and choose "Show Log".
b: Under Categories, verify all checkboxes are checked; also, enter "%" for Module.
c: Under Sequence, verify "Other" is checked, and "%" is entered in the field next to it.
d: The Search should return all logged statements in the database.
e: Click on the Export button to export the log.

Please upload the trace file obtained to Oracle support

2. From Filename specified in profile FND: Debug Filename.

3. From fnd_log_messages
After reproducing the issue we can also get the logs from database table
fnd_log_messages this way.
Run this query with login id in place of <USER_NAME> in query to know the USER_ID of the person logging in. 
    select USER_ID from FND_USER where USER_NAME like '<USER_NAME>'
Before logging in (after setting the profiles) run the query below with user_id in place of <USER_ID> and check the count. 
   select session_id, count(*)
   from fnd_log_messages
   where user_id = <USER_ID>
   group by session_id;

After logging in run the query again and check that the count has increased.
Also get the SESSION_ID of the present login session from the new row.

Use the USER_ID and SESSION_ID to query from fnd_log_messages or run the spool script to get the log file log.lst 
set termout off
set pages 30000
set lines 4000

col MESSAGE_TEXT format a3950
col TIMESTAMP format a22
col LOG_LEVEL format a2

spool log.lst

select
T1.LOG_LEVEL
||'^'|| to_char(T1.TIMESTAMP, 'dd-mon-yyyy hh24:mi:ss')
||'^'|| T1.MESSAGE_TEXT
from
( select
rownumrownumber, substr(message_text,1,3950) message_text
,module,log_level, timestamp
from fnd_log_messages
where user_id = <USER_ID>
and session_id = <SESSION_ID>
order by timestamp asc
) T1;

spool off
 

If there are no Self service log messages returned, please run the following
select
rownumrownumber, log_level, substr(message_text,1,3950) message_text, timestamp,module
from fnd_log_messages
where user_id = <USER_ID>
and session_id = <SESSION_ID>
and module like '%selfservice%'
order by timestamp asc
  


Enabling Audit in Database.

Setting up or enabling the standard audit trail is quite easy and is outlined in the 2-Day+ Security Guide as follows:
1.  Start Database Control
2.  Log in as SYS with SYSDBA privileges
3.  Click Server to display the Server subpage
4.  Click Initialization Parameters in the Database Configuration section
5.  Click SPFile to display the SPFile subpage. If you don’t use anSPFile just continue to the next step
6.  Enter audit_trail in the Name field to find the AUDIT_TRAIL parameter and then click Go
7.  Enter the type of auditing you want in the Value field where value can be:
DB – Enables Database Auditing where records will primarily be written to the sys.aud$
OS – Enables auditing records to be written to the operating system where you must also specify AUDIT_FILE_DEST which tells Oracle where to write the audit record.
NONE – Will disable standard auditing
DB, EXTENDED – does all that the type of DB does, plus populating the SQL bind and SQL text CLOB-type columns of the sys.aud$ table when available
XML – will write to the operating system audit records in XML format
EXTENDED – will write to the operating system audit records in XML format but also populates the SQL bind and SQL text CLOB-types columns of the sys.aud$ table when available
8.  Click Apply
9.  Restart the Oracle instance
While setting up standard auditing from the control panel is quite acceptable, this is one of the tasks that can also easily be accomplished through the standard PL*SQL interface.
1.  Log in as SYS with SYSDBA privileges
Sqlplus / as sysdba
2.  Check the current settings
SQL> show parameter audit
NAME                                 TYPE        VALUE
---------------------- ----------- ------------------------------
audit_file_dest        string      /opt/app/oracle/admin/db11/adump
audit_sys_operationsboolean     FALSE
audit_syslog_level     string
audit_trail            string      DB
3.  Set the type of auditing you want by setting the audit_trail parameter
SQL> alter system set audit_trail=OS scope=spfile;
System altered.
4.  Set the auditing destination
SQL> alter system set audit_file_dest='/opt/app/oracle/admin/db11/adump' scope=spfile;
System altered.
5.  Restart the Oracle instance
SQL> SHUTDOWN
SQL> STARTUP
Also note that there are two additional parameters audit_sys_operations and audit_syslog_level that you should consider setting if you are concerned about the SYS account activity.
1.  audit_sys_operations - this initialization parameter tells Oracle to turn on auditing of the SYS connections, and users connecting with the SYSDBA or SYSOPER privilege. It has either a TRUE or FALSE value
2.  audit_syslog_level – this initialization parameter enables SYS and standard OS auditing records to be written to the system using the SYSLOG utility
Now that we’ve enables operating system auditing, it is always nice to see exactly what we’ve accomplished. While this is not an exhaustive example, it does touch the surface of what we are trying to accomplish here. Suppose now that we have an un-authorized access attempt (failed login) to our database through the use of the SCOTT/TIGER account.
- Advertisement -
1. we’ve locked this account and any attempts will be now be met with ‘the account is locked’ message:
[oracle@ludwigadump]$ sqlplusscott/tiger
SQL*Plus: Release 11.1.0.6.0 - Production on Thu Dec 10 06:51:43 2009
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
ERROR:
ORA-28000: the account is locked
2.  And now, because of auditing, this attempt will not go unnoticed and will be reported in the audit_file_dest location:
[oracle@ludwig ~]$ cd $ORACLE_HOME/adump
[oracle@ludwigadump]$ more ora_3817.aud
Audit file /opt/app/oracle/admin/db11/adump/ora_3817.aud
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /opt/app/oracle/product/11.1.0/db_1
System name:    Linux
Node name:      ludwig
Release:        2.6.18-8.el5
Version:        #1 SMP Thu Mar 15 19:57:35 EDT 2007
Machine:        i686
Instance name: db11
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 3817, image: oracle@ludwig (TNS V1-V3)

Thu Dec 10 06:48:46 2009
SESSIONID: "280057" ENTRYID: "1" STATEMENT: "1" USERID: "SCOTT" USERHOST: "ludwig" TERMINAL: "pts/1" ACTION: "100"
RETURNCODE: "28000" COMMENT$TEXT: "Authenticated by: DATABASE" OS$USERID: "oracle"
Now that you have the settings changed, you can start monitoring the activity on the SYS account. Remember standard auditing can monitor and record various user database actions such as SQL statements, privileges, schemas, objects, and network and multitier activity. Now is the time to dig in, see how far standard auditing can take you to gain compliancy and then fill in the gaps. The issue becomes whether you can provide auditors everything they need to pass your audit. If you can monitor all database traffic and take appropriate action then you will pass, otherwise you are in for a lot of work. But more on that latter.


Database Audit and Audit trail purging
DBMS_AUDIT_MGMT.
There are several reason to enable database audit and at least as much reason to not enable database audit.

Enable Audit

Ok, first of all we have to set the corresponding init.ora parameter and enable the database audit. In my case I’ll also enable audit of all sys operation. This is not necessary when you plan to use standard audit. I just did it because I restarted the database anyway. ok lame excuse, I will come back again later on.
Set the init.ora parameter:
ALTER system SET audit_sys_operations=TRUE scope=spfile;

ALTER system SET audit_trail=db_extended scope=spfile;

startup force;

SHOW parameter audit
Initialize the audit infrastructure. In this example I’ve initialized the standard audit and set a default interval of 10 days.
BEGIN
  DBMS_AUDIT_MGMT.INIT_CLEANUP(
    AUDIT_TRAIL_TYPE => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
    DEFAULT_CLEANUP_INTERVAL => 240 /*hours*/
  );
END;
/

commit;
Create a dedicated tablespace for the audit data and change the location of the audit trail. If this is not done AUD$ will remain in the SYSAUX tablespace.
CREATE tablespaceaudit_datadatafile '/u01/oradata/TDB03/audit01TDB03.dbf'
SIZE 100M autoextend ON NEXT 50M maxsize 5G;

BEGIN
  DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_LOCATION(
    AUDIT_TRAIL_TYPE => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
    AUDIT_TRAIL_LOCATION_VALUE => 'AUDIT_DATA'
  );
END;
/

commit;
Review the audit settings
col PARAMETER_NAME FOR a30
col PARAMETER_VALUE FOR a15
col AUDIT_TRAIL FOR a20
SELECT PARAMETER_NAME, PARAMETER_VALUE, AUDIT_TRAIL
FROM DBA_AUDIT_MGMT_CONFIG_PARAMS
WHERE audit_trail = 'STANDARD AUDIT TRAIL';

PARAMETER_NAME                 PARAMETER_VALUE AUDIT_TRAIL
------------------------------ --------------- --------------------
DEFAULT CLEAN UP INTERVAL      12              STANDARD AUDIT TRAIL
DB AUDIT TABLESPACE            AUDIT_DATA      STANDARD AUDIT TRAIL
DB AUDIT CLEAN BATCH SIZE      10000           STANDARD AUDIT TRAIL

col segment_name FOR a10
SELECT owner,segment_name,tablespace_name FROM dba_segments WHERE segment_name ='AUD$';

OWNER                          SEGMENT_NA TABLESPACE_NAME
------------------------------ ---------- ------------------------------
SYS                            AUD$       AUDIT_DATA
Ok, the audit infrastructure has been setup and configured. Now lets enable audit for certain system privilege and objects. The first part of the following audit statements have been taken from $ORACLE_HOME/rdbms/admin/secconf.sql. This script is used in 11g to enable audit during database creation.
audit ALTER any TABLE BY access;
audit CREATE any TABLE BY access;
audit DROP any TABLE BY access;
audit CREATE any PROCEDURE BY access;
audit DROP any PROCEDURE BY access;
audit ALTER any PROCEDURE BY access;
audit GRANT any privilege BY access;
audit GRANT any object privilege BY access;
audit GRANT any ROLE BY access;
audit audit system BY access;
audit CREATE external job BY access;
audit CREATE any job BY access;
audit CREATE any library BY access;
audit CREATE public DATABASE link BY access;
audit exempt access policy BY access;
audit ALTER USER BY access;
audit CREATE USER BY access;
audit ROLE BY access;
audit CREATE SESSION BY access;
audit DROP USER BY access;
audit ALTER DATABASE BY access;
audit ALTER system BY access;
audit ALTER profile BY access;
audit DROP profile BY access;
audit DATABASE link BY access;
audit system audit BY access;
audit profile BY access;
audit public synonym BY access;
audit system GRANT BY access;

audit DELETE ON sys.aud$;
audit ALTER ON DEFAULT;
audit GRANT ON DEFAULT;
There are a few data dictionary views to review which object and system privilege has audit enabled.
SELECT  * FROM DBA_STMT_AUDIT_OPTS;

USER_NAME  PROXY_NAME AUDIT_OPTION                 SUCCESS    FAILURE
---------- ---------- ---------------------------- ---------- ----------
                      ALTER SYSTEM                 BY ACCESS  BY ACCESS
                      SYSTEM AUDIT                 BY ACCESS  BY ACCESS
                      CREATE SESSION               BY ACCESS  BY ACCESS
                      CREATE USER                  BY ACCESS  BY ACCESS
                      ALTER USER                   BY ACCESS  BY ACCESS
                      DROP USER                    BY ACCESS  BY ACCESS
                      PUBLIC SYNONYM               BY ACCESS  BY ACCESS
                      DATABASE LINK                BY ACCESS  BY ACCESS
                      ROLE                         BY ACCESS  BY ACCESS
                      PROFILE                      BY ACCESS  BY ACCESS
                      CREATE ANY TABLE             BY ACCESS  BY ACCESS
                      ALTER ANY TABLE              BY ACCESS  BY ACCESS
                      DROP ANY TABLE               BY ACCESS  BY ACCESS
                      GRANT TABLE                  BY ACCESS  BY ACCESS
                      CREATE PUBLIC DATABASE LINK  BY ACCESS  BY ACCESS
                      GRANT ANY ROLE               BY ACCESS  BY ACCESS
                      SYSTEM GRANT                 BY ACCESS  BY ACCESS
                      ALTER DATABASE               BY ACCESS  BY ACCESS
                      CREATE ANY PROCEDURE         BY ACCESS  BY ACCESS
                      ALTER ANY PROCEDURE          BY ACCESS  BY ACCESS
                      DROP ANY PROCEDURE           BY ACCESS  BY ACCESS
                      ALTER PROFILE                BY ACCESS  BY ACCESS
                      DROP PROFILE                 BY ACCESS  BY ACCESS
                      GRANT ANY PRIVILEGE          BY ACCESS  BY ACCESS
                      CREATE ANY LIBRARY           BY ACCESS  BY ACCESS
                      EXEMPT ACCESS POLICY         BY ACCESS  BY ACCESS
                      GRANT ANY OBJECT PRIVILEGE   BY ACCESS  BY ACCESS
                      CREATE ANY JOB               BY ACCESS  BY ACCESS
                      CREATE EXTERNAL JOB          BY ACCESS  BY ACCESS

SELECT * FROM DBA_PRIV_AUDIT_OPTS;

USER_NAME  PROXY_NAME PRIVILEGE                    SUCCESS    FAILURE
---------- ---------- ---------------------------- ---------- ----------
                      CREATE EXTERNAL JOB          BY ACCESS  BY ACCESS
                      CREATE ANY JOB               BY ACCESS  BY ACCESS
                      GRANT ANY OBJECT PRIVILEGE   BY ACCESS  BY ACCESS
                      EXEMPT ACCESS POLICY         BY ACCESS  BY ACCESS
                      CREATE ANY LIBRARY           BY ACCESS  BY ACCESS
                      GRANT ANY PRIVILEGE          BY ACCESS  BY ACCESS
                      DROP PROFILE                 BY ACCESS  BY ACCESS
                      ALTER PROFILE                BY ACCESS  BY ACCESS
                      DROP ANY PROCEDURE           BY ACCESS  BY ACCESS
                      ALTER ANY PROCEDURE          BY ACCESS  BY ACCESS
                      CREATE ANY PROCEDURE         BY ACCESS  BY ACCESS
                      ALTER DATABASE               BY ACCESS  BY ACCESS
                      GRANT ANY ROLE               BY ACCESS  BY ACCESS
                      CREATE PUBLIC DATABASE LINK  BY ACCESS  BY ACCESS
                      DROP ANY TABLE               BY ACCESS  BY ACCESS
                      ALTER ANY TABLE              BY ACCESS  BY ACCESS
                      CREATE ANY TABLE             BY ACCESS  BY ACCESS
                      DROP USER                    BY ACCESS  BY ACCESS
                      ALTER USER                   BY ACCESS  BY ACCESS
                      CREATE USER                  BY ACCESS  BY ACCESS
                      CREATE SESSION               BY ACCESS  BY ACCESS
                      AUDIT SYSTEM                 BY ACCESS  BY ACCESS
                      ALTER SYSTEM                 BY ACCESS  BY ACCESS

SELECT * FROM DBA_OBJ_AUDIT_OPTS;

OWNER OBJECT_NAME          OBJECT_TYPE ALT AUD COM DEL GRA IND INS LOC REN SEL UPD REF EXE CRE REA WRI FBK
----- -------------------- ----------- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
SYS   AUD$                 TABLE       -/- -/- -/- S/S -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/-
SYS   VERIFY_FUNCTION      PROCEDURE   -/- -/- -/- -/- S/S -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/-
SYS   VERIFY_FUNCTION_11G  PROCEDURE   -/- -/- -/- -/- S/S -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/- -/-

SELECT * FROM ALL_DEF_AUDIT_OPTS;

ALT AUD COM DEL GRA IND INS LOC REN SEL UPD REF EXE FBK REA
--- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
S/S -/- -/- -/- S/S -/- -/- -/- -/- -/- -/- -/- -/- -/- -/-
Audit trail housekeeping
So far we’ve spend quite some time on setting up the audit. Let’s come to the actual part of the post, the housekeeping. The plan is to have regular database job, which purge the audit data as required. For this I defined the following steps:
Create a report of your audit trail (optional)
Export your audit trail information for longterm archiving (optional)
Set the archive timestamp as soon as audit data has been archived
Purge archived audit data
I the current example I’ll skip the first to steps.
Create daily archive timestamp job. This job will be a regular dbms_scheduler job.
BEGIN
  DBMS_SCHEDULER.CREATE_JOB (
    job_name   => 'DAILY_AUDIT_ARCHIVE_TIMESTAMP',
    job_type   => 'PLSQL_BLOCK',
    job_action => 'BEGIN DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP(AUDIT_TRAIL_TYPE =>
                   DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,LAST_ARCHIVE_TIME => sysdate-10); END;',
    start_date => sysdate,
    repeat_interval => 'FREQ=HOURLY;INTERVAL=24',
    enabled    =>  TRUE,
    comments   => 'Create an archive timestamp'
  );
END;
/
Create daily purge job
BEGIN
  DBMS_AUDIT_MGMT.CREATE_PURGE_JOB(
    AUDIT_TRAIL_TYPE           => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
    AUDIT_TRAIL_PURGE_INTERVAL => 24 /* hours */,
    AUDIT_TRAIL_PURGE_NAME     => 'Daily_Audit_Purge_Job',
    USE_LAST_ARCH_TIMESTAMP    => TRUE
  );
END;
/

commit;
The defined AUDIT JOB’s are then visible in DBA_AUDIT_MGMT_CLEANUP_JOBS.
col JOB_NAME FOR a30
col JOB_FREQUENCY FOR a40
SELECT JOB_NAME,JOB_STATUS,AUDIT_TRAIL,JOB_FREQUENCY FROM DBA_AUDIT_MGMT_CLEANUP_JOBS;

JOB_NAME               JOB_STAT AUDIT_TRAIL          JOB_FREQUENCY
---------------------- -------- -------------------- ------------------------
DAILY_AUDIT_PURGE_JOB  ENABLED  STANDARD AUDIT TRAIL FREQ=HOURLY;INTERVAL=24
As we can see in dba_scheduler_jobs, DBMS_SCHEDULER.CREATE_JOB is just creating a other scheduler job.
col next_run_date FOR a50
SELECT job_name, next_run_date, state, enabled FROM dba_scheduler_jobs WHERE job_name LIKE '%AUDIT%';

JOB_NAME                       NEXT_RUN_DATE                                      STATE           ENABL
------------------------------ -------------------------------------------------- --------------- -----
DAILY_AUDIT_ARCHIVE_TIMESTAMP  31-MAY-11 01.46.58.000000 PM +02:00                SCHEDULED       TRUE
DAILY_AUDIT_PURGE_JOB          31-MAY-11 01.49.34.900000 PM EUROPE/VIENNA         SCHEDULED       TRUE


















Monday, January 9, 2017

EBS DB 11.1.0.7 upgrade to 12c 12.1.0.2

Due to some restricitons in uploading the Word document here, i have uploaded the document in Google Drive.

Please refer to my link.

EBS_DB_Upgrade

Friday, January 6, 2017

How To Create, export, import and Evolve a SQL Plan Baseline

Introduction: The query optimizer normally uses information like object and system statistics, bind values and so on to determine the best plan for a SQL statement. In some cases, defects in either these inputs or the optimizer can lead to a sub-optimal plan. A SQL plan baseline for a SQL statement consists of a set of accepted plans. When the statement is parsed, the optimizer will only select the best plan from among this set. If a different plan is found using the normal cost-based selection process, the optimizer will add it to the plan history but this plan will not be used until it is verified to perform better than the existing accepted plan and is evolved


Parameters

Two parameters allow you to control SPM.The first, optimizer_capture_sql_plan_baselines, which is FALSE by default, allows you to automatically capture plans. SPM will start managing every repeatable SQL statement that is executed and will create a plan history for it. The first plan that is captured will be automatically accepted. Subsequent plans for these statements will not be accepted until they are evolved.

The second parameter, optimizer_use_sql_plan_baselines, is TRUE by default. It allows the SPM aware optimizer to use the SQL plan baseline if available when compiling a SQL statement. If you set this parameter to FALSE, the SPM aware optimizer will be disabled and you will get the regular cost-based optimizer which will select the best plan based on estimated cost.

Note:Setting optimizer_capture_sql_plan_baselines = true permanently will result in a SQL plan baseline being created for every repeatable SQL statement on the system. That includes all of the recursive SQL Oracle executes on your behalf as well as every 'select sysdate from dual;' or 'select * from v$sql;' the DBA may do on the system. This could lead to a very large number of SQL plan baselines being captured and an increased foot print in the SYSAUX tablespace. There are some customers who do have optimizer_capture_sql_plan_baselines = true and they have not encountered any problem other than a large SYSAUX tablespace!



Scenario where a query is working fine in one instance and working bad in another instance. Thus i will exporting/importing the baseline from one instance to another.

SYS@SAM AS SYSDBA> show parameter baseline;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_capture_sql_plan_baselines boolean     FALSE
optimizer_use_sql_plan_baselines     boolean     TRUE


Identify the problematic sql text using awr, statspack etc.


Get the sql_id for the sql_text-  "Select R.Rowid From Fnd_Concur%"

SELECT sql_id, hash_value, SUBSTR(sql_text,1,40) Text FROM v$sql WHERE sql_text LIKE '%Select R.Rowid From Fnd_Concur%';

5ugwm6cpfpr7b


create a tuning set

SYS@SAM AS SYSDBA> begin
 DBMS_SQLTUNE.CREATE_SQLSET(SQLSET_NAME => '5ugwm6cpfpr7b_STS01', DESCRIPTION => 'TEST SQL TUNE SET');
 END;
 /  2    3    4

PL/SQL procedure successfully completed.

SYS@SAM AS SYSDBA>


Populate sql tuning set

SYS@SAM AS SYSDBA> DECLARE
 CUR SYS_REFCURSOR;
 BEGIN
 OPEN CUR FOR
 SELECT VALUE(P) FROM TABLE (DBMS_SQLTUNE.SELECT_CURSOR_CACHE('sql_id = ''5ugwm6cpfpr7b''')) p;
 DBMS_SQLTUNE.LOAD_SQLSET(SQLSET_NAME=> '5ugwm6cpfpr7b_STS01', POPULATE_CURSOR=>CUR);
 CLOSE CUR;
 END;
 /
  2    3    4    5    6    7    8    9

PL/SQL procedure successfully completed.

SYS@SAM AS SYSDBA> SYS@SAM AS SYSDBA>


List out sql tuning set


SYS@SAM AS SYSDBA>
SELECT plan_hash_value FROM TABLE(DBMS_SQLTUNE.SELECT_SQLSET(SQLSET_NAME => '5ugwm6cpfpr7b_STS01'));

PLAN_HASH_VALUE
---------------
     2836784050



LOAD DESIRED PLAN FROM ‘SQL TUNING SET’ AS SQL PLAN BASELINE

SYS@SAM AS SYSDBA> DECLARE
 MY_PLANS PLS_INTEGER;
 BEGIN
 MY_PLANS := DBMS_SPM.LOAD_PLANS_FROM_SQLSET(SQLSET_NAME=>'5ugwm6cpfpr7b_STS01', BASIC_FILTER=> 'PLAN_HASH_VALUE=

''2836784050''' );
 END;
 /  2    3    4    5    6

PL/SQL procedure successfully completed.

SYS@SAM AS SYSDBA>


 VERIFY IF SQL PLAN BASELINE GOT CREATED SUCCESSFULLY

select * from dba_sql_plan_baselines;

select sql_handle from dba_sql_plan_baselines;
SQL_HANDLE
------------------------------
SQL_a00583ac0188a3d6




========================================================================


Now we have to pack this sql baseline.

Create a staging table in system schema. It cannot be under sys schema

BEGIN
  DBMS_SPM.CREATE_STGTAB_BASELINE(table_name => 'stage1');
END;
/


pack the sql baselines

DECLARE
  my_plans number;
BEGIN
  my_plans := DBMS_SPM.PACK_STGTAB_BASELINE(
    table_name => 'stage1',
    enabled => 'yes',
    SQL_HANDLE => 'SQL_a00583ac0188a3d6');
END;
/



Now export the sql baselines

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SYS@SAM AS SYSDBA> CREATE OR REPLACE DIRECTORY test_dir AS '/xxx_out/dump';

Directory created.

SYS@SAM AS SYSDBA> GRANT READ, WRITE ON DIRECTORY test_dir TO system;

Grant succeeded.

SYS@SAM AS SYSDBA> exit;
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@SAM2 /xxx_out> expdp system/*** tables=stage1 directory=TEST_DIR dumpfile=stage.dmp logfile=stage.log

compression=all

Export: Release 11.2.0.3.0 - Production on Thu Jan 5 06:00:15 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Starting "SYSTEM"."SYS_EXPORT_TABLE_01":  system/******** tables=stage1 directory=TEST_DIR dumpfile=stage.dmp

logfile=stage.log compression=all
Estimate in progress using BLOCKS method...
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 192 KB
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/PRE_TABLE_ACTION
Processing object type TABLE_EXPORT/TABLE/POST_INSTANCE/PROCACT_INSTANCE
Processing object type TABLE_EXPORT/TABLE/POST_INSTANCE/PROCDEPOBJ
. . exported "SYSTEM"."STAGE1"                           8.164 KB       1 rows
Master table "SYSTEM"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYSTEM.SYS_EXPORT_TABLE_01 is:
  /xxx_out/dump/stage.dmp
Job "SYSTEM"."SYS_EXPORT_TABLE_01" successfully completed at 06:09:


--------------------------------------------------------------------------------------------
Import the dump to the target database.


check first if there any baselines present or not.


select * from dba_sql_plan_baselines;


Import the sql baseline

oracle@SAM1 /oracle/stage> impdp system/***** tables=stage1  dumpfile=stage.dmp

Import: Release 11.2.0.3.0 - Production on Thu Jan 5 06:17:23 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Master table "SYSTEM"."SYS_IMPORT_TABLE_01" successfully loaded/unloaded
Starting "SYSTEM"."SYS_IMPORT_TABLE_01":  system/******** tables=stage1 dumpfile=stage.dmp
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
. . imported "SYSTEM"."STAGE1"                           8.164 KB       1 rows
Job "SYSTEM"."SYS_IMPORT_TABLE_01" successfully completed at 06:17:33

oracle@ /oracle/stage>

unpack the sql baseline


sqlplus system/<password>

SET SERVEROUTPUT ON
SYSTEM> DECLARE
  l_plans_unpacked  PLS_INTEGER;
BEGIN
  l_plans_unpacked := DBMS_SPM.unpack_stgtab_baseline(
    table_name      => 'STAGE1',
    table_owner     => 'SYSTEM');
DBMS_OUTPUT.put_line('Plans Unpacked: ' || l_plans_unpacked);
END;  2    3    4    5    6    7    8
  9  /
Plans Unpacked: 1

PL/SQL procedure successfully completed.

SYSTEM@SAM1>

check if the baselines are there or not

SYSTEM@SAM1 > select sql_handle from dba_sql_plan_baselines;

SQL_HANDLE
------------------------------
SQL_a00583ac0188a3d6


check if the plan is accepted or not.

SYS@SAM1 AS SYSDBA> select sql_handle, plan_name, sql_text, enabled, accepted, fixed from dba_sql_plan_baselines;



If the plan is not accepted, then we have to eveolve it, which is discussed later in the post.

========================================================================

Scenario-2 -Force the optimizer to choose a plan. The first query runs against an unindexed column, does a full scan. And then an index have been added but the optimizer still chooses to do a full table scan. By Evolving the baseline, we forced the optimizer to choose the index and at last we removed the original plan which was doing a full table scan.

oracle@SAM2 ~> sqlplus test/welcome123

SQL*Plus: Release 11.2.0.3.0 Production on Fri Jan 6 02:57:07 2017

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

TEST@SAM >

TEST@SAM > create table t as select * from applsys.FND_OBJECTS;

Table created.

TEST@SAM >


TEST@SAM > exec DBMS_STATS.GATHER_SCHEMA_STATS ('test');

PL/SQL procedure successfully completed.



TEST@SAM > show parameter baselines

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_capture_sql_plan_baselines boolean     FALSE
optimizer_use_sql_plan_baselines     boolean     TRUE


TEST@SAM > ALTER SESSION SET OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES = TRUE;

Session altered.



TEST@SAM > select count(*) from t;

  COUNT(*)
----------
      1307


TEST@SAM > /

  COUNT(*)
----------
      1307

TEST@SAM >


ALTER SESSION SET OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES = FALSE;



SYS@SAM AS SYSDBA> select sql_handle, plan_name, sql_text, enabled, accepted, fixed from dba_sql_plan_baselines;

SQL_HANDLE           PLAN_NAME                                  SQL_TEXT                                   ENA ACC FIX
-------------------- ------------------------------------------ ------------------------------------------ --- --- ---
SQL_793213869456f9be SQL_PLAN_7kchmhua5dydy3fdbb376             select count(*) from t                     YES YES NO



Lets create an index and create a new baseline

TEST@SAM > create index t_idx on t (OBJECT_ID);

Index created.

TEST@SAM >TEST@SAM > exec dbms_stats.gather_schema_stats ('test');

PL/SQL procedure successfully completed.


SYS@SAM AS SYSDBA> alter system flush shared_pool;

System altered.

SYS@SAM AS SYSDBA>


TEST@SAM > ALTER SESSION SET OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES = TRUE;

Session altered.

TEST@SAM >


TEST@SAM > select count(*) from t;

  COUNT(*)
----------
      1307

TEST@SAM > /

  COUNT(*)
----------
      1307

TEST@SAM >


TEST@SAM > ALTER SESSION SET OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES = FALSE;

Session altered.

TEST@SAM >


TEST@SAM > set autotrace on;
TEST@SAM > select count(*) from t;

  COUNT(*)
----------
      1307


Execution Plan
----------------------------------------------------------
Plan hash value: 2966233522

-------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Cost (%CPU)| Time     |
-------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |    30   (4)| 00:00:01 |
|   1 |  SORT AGGREGATE    |      |     1 |            |          |
|   2 |   TABLE ACCESS FULL| T    |  1307 |    30   (4)| 00:00:01 |
-------------------------------------------------------------------

Note
-----
   - SQL plan baseline "SQL_PLAN_7kchmhua5dydy3fdbb376" used for this statement



SYS@SAM AS SYSDBA> select sql_handle, plan_name, sql_text, enabled, accepted, fixed from dba_sql_plan_baselines;

SQL_HANDLE                     PLAN_NAME
------------------------------ ------------------------------
SQL_TEXT                                                                         ENA ACC FIX
-------------------------------------------------------------------------------- --- --- ---
SQL_793213869456f9be           SQL_PLAN_7kchmhua5dydy3fdbb376
select count(*) from t                                                           YES YES NO

SQL_793213869456f9be           SQL_PLAN_7kchmhua5dydy89596ecb
select count(*) from t                                                           YES NO  NO



check the execution plan for this baseline.


SYS@SAM AS SYSDBA> select * from table(dbms_xplan.display_sql_plan_baseline('SQL_793213869456f9be'));

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------
SQL handle: SQL_793213869456f9be
SQL text: select count(*) from t
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SQL_PLAN_7kchmhua5dydy3fdbb376         Plan id: 1071362934
Enabled: YES     Fixed: NO      Accepted: YES     Origin: AUTO-CAPTURE
--------------------------------------------------------------------------------

Plan hash value: 2966233522

-------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Cost (%CPU)| Time     |
-------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |    30   (4)| 00:00:01 |
|   1 |  SORT AGGREGATE    |      |     1 |            |          |
|   2 |   TABLE ACCESS FULL| T    |  1307 |    30   (4)| 00:00:01 |
-------------------------------------------------------------------

--------------------------------------------------------------------------------
Plan name: SQL_PLAN_7kchmhua5dydy89596ecb         Plan id: 2304339659
Enabled: YES     Fixed: NO      Accepted: NO      Origin: AUTO-CAPTURE
--------------------------------------------------------------------------------

Plan hash value: 995313729

------------------------------------------------------------------
| Id  | Operation        | Name  | Rows  | Cost (%CPU)| Time     |
------------------------------------------------------------------
|   0 | SELECT STATEMENT |       |     1 |     4   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE  |       |     1 |            |          |
|   2 |   INDEX FULL SCAN| T_IDX |  1307 |     4   (0)| 00:00:01 |
------------------------------------------------------------------

35 rows selected.

SYS@SAM AS SYSDBA>


Create the evolve report but do not actually change the ACCEPTED flag yet by setting commit=>no


SYS@SAM AS SYSDBA> set serveroutput on
  declare evolve_out CLOB;
  begin
  evolve_out := DBMS_SPM.EVOLVE_SQL_PLAN_BASELINE ( SQL_HANDLE => 'SQL_793213869456f9be', COMMIT => 'NO' );
  dbms_output.put_line(evolve_out);
  end;
/SYS@SAM AS SYSDBA>   2    3    4    5    6

-------------------------------------------------------------------------------


Evolve SQL Plan Baseline
Report
-------------------------------------------------------------------------------

Inputs:
----
---
  SQL_HANDLE = SQL_793213869456f9be
  PLAN_NAME  =
  TIME_LIMIT = DBMS_SPM.AUTO_LIMIT
  VERIFY
= YES
  COMMIT     = NO

Plan: SQL_PLAN_7kchmhua5dydy89596ecb
------------------------------------


Plan was verified: Time used .12 seconds.
  Plan passed performance criterion: 60.7 times better
than baseline plan.

                            Baseline Plan      Test Plan       Stats Ratio


-------------      ---------       -----------
  Execution Status:              COMPLETE
COMPLETE
  Rows Processed:                       1              1
  Elapsed Time(ms):
2.817          2.538              1.11
  CPU Time(ms):                     2.888          2.555
1.13
  Buffer Gets:                        258              4              64.5
  Physical Read
Requests:               0              0
  Physical Write Requests:              0              0


Physical Read Bytes:                  0              0
  Physical Write Bytes:                 0
0
  Executions:                           1
1

-------------------------------------------------------------------------------


Report
Summary
-------------------------------------------------------------------------------
Number of
plans verified: 1
Number of plans accepted: 0



PL/SQL procedure successfully completed.

SYS@SAM AS SYSDBA>


Now evolve the plan with commit option as YES.

SYS@SAM AS SYSDBA> set serveroutput on
  declare evolve_out CLOB;
  begin
  evolve_out := DBMS_SPM.EVOLVE_SQL_PLAN_BASELINE ( SQL_HANDLE => 'SQL_793213869456f9be', COMMIT => 'YES' );
  dbms_output.put_line(evolve_out);
  end;
/SYS@SAM AS SYSDBA>   2    3    4    5    6

-------------------------------------------------------------------------------


Evolve SQL Plan Baseline
Report
-------------------------------------------------------------------------------

Inputs:
----
---
  SQL_HANDLE = SQL_793213869456f9be
  PLAN_NAME  =
  TIME_LIMIT = DBMS_SPM.AUTO_LIMIT
  VERIFY
= YES
  COMMIT     = YES

Plan: SQL_PLAN_7kchmhua5dydy89596ecb
------------------------------------

Plan was verified: Time used .07 seconds.
  Plan passed performance criterion: 60.7 times better
than baseline plan.
  Plan was changed to an accepted plan.

                            Baseline
Plan      Test Plan       Stats Ratio
                            -------------      ---------
-----------
  Execution Status:              COMPLETE       COMPLETE
  Rows Processed:
1              1
  Elapsed Time(ms):                 2.804          2.519              1.11
  CPU
Time(ms):                     2.888          2.555              1.13
  Buffer Gets:
258              4              64.5
  Physical Read Requests:               0              0


Physical Write Requests:              0              0
  Physical Read Bytes:                  0
0
  Physical Write Bytes:                 0              0
  Executions:                           1
1

-------------------------------------------------------------------------------


Report
Summary
-------------------------------------------------------------------------------
Number of
plans verified: 1
Number of plans accepted: 1



PL/SQL procedure successfully completed.

SYS@SAM AS SYSDBA>

SYS@SAM AS SYSDBA> select sql_handle, plan_name, sql_text, enabled, accepted, fixed from dba_sql_plan_baselines;

SQL_HANDLE                     PLAN_NAME
------------------------------ ------------------------------
SQL_TEXT                                                                         ENA ACC FIX
-------------------------------------------------------------------------------- --- --- ---
SQL_793213869456f9be           SQL_PLAN_7kchmhua5dydy3fdbb376
select count(*) from t                                                           YES YES NO

SQL_793213869456f9be           SQL_PLAN_7kchmhua5dydy89596ecb
select count(*) from t                                                           YES YES NO

SQL_a00583ac0188a3d6           SQL_PLAN_a01c3ph0sj8yq562c7608
SELECT sql_id, hash_value, SUBSTR(sql_text,1,40) Text FROM v$sql WHERE sql_tex   YES YES NO
t LIKE '%Select R.Rowid From Fnd_Concur%' order by L
AST_LOAD_TIME asc




TEST@SAM > set autotrace on
TEST@SAM > select count(*) from t;

  COUNT(*)
----------
      1307


Execution Plan
----------------------------------------------------------
Plan hash value: 995313729

------------------------------------------------------------------
| Id  | Operation        | Name  | Rows  | Cost (%CPU)| Time     |
------------------------------------------------------------------
|   0 | SELECT STATEMENT |       |     1 |     4   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE  |       |     1 |            |          |
|   2 |   INDEX FULL SCAN| T_IDX |  1307 |     4   (0)| 00:00:01 |
------------------------------------------------------------------

Note
-----
   - SQL plan baseline "SQL_PLAN_7kchmhua5dydy89596ecb" used for this statement

TEST@SAM >

If we need to drop the original plan, we can do by

declare
 drop_result pls_integer;
 begin
 drop_result := DBMS_SPM.DROP_SQL_PLAN_BASELINE(
 sql_handle => 'SQL_a00583ac0188a3d6',
 plan_name => 'SQL_PLAN_a01c3ph0sj8yq562c7608');
 dbms_output.put_line(drop_result);  
 end;
/

=========================*************************============
Another method especially in EBS

- 1. Create a Baseline from cursor cache (based on SQL_ID and PHV)
DECLARE
  l_plans_loaded  PLS_INTEGER;
BEGIN
  l_plans_loaded := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(
                      sql_id => 'a29c10n7x9gvt',
                      plan_hash_value => 749062863);
END;

-- 2. Create TMP table for baseline export
BEGIN
  DBMS_SPM.CREATE_STGTAB_BASELINE(table_name => 'STG_SPM_TRANSFER', table_owner => 'APPS');
END;

-- 3. Pack the Baseline into the TMP table

DECLARE
    v_sql_handle VARCHAR2(30);
    v_packed     PLS_INTEGER;
BEGIN
    SELECT DISTINCT sql_handle 
    INTO v_sql_handle
    FROM dba_sql_plan_baselines 
    WHERE signature IN (SELECT exact_matching_signature 
                        FROM v$sql 
                        WHERE sql_id = 'a29c10n7x9gvt');

    v_packed := DBMS_SPM.PACK_STGTAB_BASELINE(
        table_name  => 'STG_SPM_TRANSFER',
        table_owner => 'APPS',
        sql_handle  => v_sql_handle
    );
    
    DBMS_OUTPUT.PUT_LINE('SQL_HANDLE found: ' || v_sql_handle);
    DBMS_OUTPUT.PUT_LINE('Plans packed: ' || v_packed);
EXCEPTION
    WHEN NO_DATA_FOUND THEN
        DBMS_OUTPUT.PUT_LINE('Error: Baseline not found. Make sure Step 1 (LOAD) was successful.');
    WHEN OTHERS THEN
        RAISE;
END;
/

-- On  DB server
expdp apps/password tables=APPS.STG_SPM_TRANSFER directory=DATA_PUMP_DIR dumpfile=spm_fix.dmp




References:http://www.oracle.com/technetwork/articles/database/create-sql-plan-baseline-2237506.html
           https://blogs.oracle.com/optimizer/entry/sql_plan_management_part_1_of_4_creating_sql_plan_baselines