AutoPatch error: The worker should not have status 'Running' or 'Restarted' at this point.

Description of Issue

R12.2 ADOP: AutoPatch error: The worker should not have status 'Running' or 'Restarted' at this point. AutoPatch error: Error running SQL and EXEC commands in parallel

Cause

ADOP Patch Session got terminated abnormally due to network issue or patch hung or Lost connectivity. Error encountered when tried to re-start adop patch.

Solution

1. Start Adctrl and look at the worker status.  Are workers running or started?   Select option: 1. Show Worker Status then 4. Tell manager that a worker failed its job   2. Restart the adop patch from the begining. Use below options: - If you want to restart a failed patch from the very beginning, you need to specify below options with adop restart=no abandon=yes - If you want to restart a failed patch from where it left off, you only need to specify below options with adop restart=yes abandon=no

ORA-00600: internal error code, arguments: [kcbgcur_6] 4096 4194 4193 4197 on database crash

Description of Issue

ORCL Instance Crashed due to ORA-00600: internal error code, arguments: [600], [ORA-00600: internal error code, arguments: [kcbgcur_6], [583], [4294967295], [4096], [0], [], [], [] , [], [], [], [], [], []

Tried to restart the ORCL database, we were able to mount the ORCL DB however unable to open
the database.

Below errors observed in alert log:

Errors in Alert Log

SMON: enabling cache recovery Errors in file /u01/ORCL/app/oraORCL/diag/rdbms/ORCL/ORCL/trace/ORCL_ora_32464.trc  (incident=322482): ORA-00600: internal error code, arguments: [kcbgcur_6], [583], [4294967295], [4096], [0], [], [], [] Incident details in: /u01/ORCL/app/oraORCL/diag/rdbms/ORCL/ORCL/incident/incdir_322482/ ORCL_ora_32464_i322482.trc Errors in file /u01/ORCL/app/oraORCL/diag/rdbms/ORCL/ORCL/trace/ORCL_ora_32464.trc: ORA-00600: internal error code, arguments: [kcbgcur_6], [583], [4294967295], [4096], [0], [], [], [] Error 600 happened during db open, shutting down database USER (ospid: 32464): terminating the instance due to error 600 Instance terminated by USER, pid = 32464 ORA-1092 signalled during: ALTER DATABASE OPEN... ORA-1092 : opiodr aborting process unknown ospid (32464_1) Mon Aug 29 02:26:31 2016 ORA-1092 : opitsk aborting process

Cause

This issue generally occurs when there is a power outage or hardware failure that initially crashes the database. On startup, the database does the normal roll forward (redo) and then rollback (undo), this is where the error is generated on the rollback.

Solution

1. Shutdown the instance 2. Startup in mount mode and set the following parameters SQL> startup mount; SQL> alter system set undo_management=MANUAL scope=SPFILE; 3. After setting the above parameter we are able to open the database SQL> shutdown immediate; SQL> startup mount; SQL> alter database open; 4. Check the status of all undo segments SQL> select tablespace_name, status, segment_name from dba_rollback_segs where status != 'OFFLINE'; This is critical - we are looking for all undo segments to be offline - System will always be online. If any are 'PARTLY AVAILABLE' or 'NEEDS RECOVERY' - Please open an issue with Oracle Support or update the current SR.  There are many options from this moment and Oracle Support Analyst can offer different solutions for the bad undo segments. If all offline then continue to the next step 5. Create new undo tablespace - example SQL>create undo tablespace <new undo tablespace> datafile <datafile> size 2000M; 6. Drop old undo tablespace (You can drop this later on as well) SQL> drop tablespace <old undo tablespace> including contents and datafiles; 7. Modify the spfile with the new undo tablespace name and change the undo management to AUTO SQL> alter system set undo_tablespace = '<new tablespace created in step 5>' scope=spfile; SQL> alter system set undo_management=AUTO scope=SPFILE; 8. Restart Database SQL> shutdown immediate; SQL> startup;

Concurrent Requests stay in Pending/Standby and Inactive/No Manager status forever

In Oracle E Business Suite, It has been observed that when attempting to submit any request, request fails with Inactive No Manager Phase/Status Combination and sometimes concurrent manager does not pick up any request it stays in Pending/Standby status

Review the following points when the concurrent request is in Inactive phase with No Manager status and pending standby status


Verify that Internal Concurrent Manager(ICM) is up and running.

Use any one navigation mentioned below to check the status details of Internal Manager.

i. Oracle Applications Manager(OAM) > Site Map > Monitoring > Availability 

   > Internal Concurrent Manager > View Status.
                          OR
ii. System Administrator Responsibility > Concurrent > Manager > Administer

If Concurrent Managers are up and running fine however not processing any requests then there can be few reasons:

1. Specialization rules prohibits processing any requests

 i.Verify if Include Specialization Rule Exists for the Standard Manager.

if exists then remove the same as below:

1. Log into Applications.
2. Navigate: System Administrator / Concurrent / Manager / Define.
3. Query for 'Standard Manager'.
4. Click Specialization Rules.
5. Remove any Include Rules from the Specialization Rules for Standard Manager.
6. Restart the Managers to test.

***Any "Include" entry is highly unadvisable for Standard Manager***

ii.Verify that there is at least one active concurrent manager with/without specialization rules that allow the concurrent program to run.

Run the following query to check whether any specialization rule defined for any concurrent manager that includes/excludes the concurrent program

SELECT    'Concurrent program '
       || fcp.concurrent_program_name
       || ' is '
       || DECODE (fcqc.include_flag, 'I', 'included in ', 'E', 'excluded from ')
       || fcqv.user_concurrent_queue_name specialization_rule_details
  FROM fnd_concurrent_queues_vl fcqv, fnd_concurrent_queue_content fcqc, fnd_concurrent_programs fcp
 WHERE fcqv.concurrent_queue_id = fcqc.concurrent_queue_id 
  AND fcqc.type_id = fcp.concurrent_program_id 
  AND fcp.concurrent_program_name = '<PROGRAM_SHORT_NAME>';

Make sure that Concurrent Manager whose specialization rule includes the concurrent program is up and running.


2. Program with Run alone flag is submitted with pending or running status

Run alone means exactly what it says. It means that it runs alone, no other requests can be running. It also means that it cannot start until no other requests are running.

Run the Below query to identify the programs that are Run alone flag enabled.

select request_id from fnd_concurrent_requests
where CONCURRENT_PROGRAM_ID in (
SELECT CONCURRENT_PROGRAM_ID
FROM FND_CONCURRENT_PROGRAMS_VL
WHERE RUN_ALONE_FLAG='Y');  

Remove Run alone: Y for the Concurrent Program returned in above query


3. All the Managers are all busy

Verify if manager is busy, increase the processes for CM if needed

4. If nothing worked above, then perform below action plan if requests are in inactive no manager status

1. Deactivate manager
2. Remove all specialization rules
 a. Login to Oracle Applications as System Administrator responsibility
 b. Navigate to: Concurrent > Manager > Define > Query for desired 
manager
 c. Click Specialization Rules tab 
 d. Delete entries
3. Activate manager 
4. Add specialization rules back
 a. Login to Oracle Applications as System Administrator responsibility
 b. Navigate to: Concurrent > Manager > Define > Query for desired 
manager
 c. Click Specialization Rules tab 
 d. Add entries

FYI:

The concurrent request queue view is used internally to map requests to managers. This view would be regenerated when concurrent managers are created, or specialization rules are altered.

"Build Concurrent Request Queue View" concurrent request internally gets submitted when concurrent managers are created, or specialization rules are altered.

However, If required we can manually regenerate the concurrent request queue view for concurrent managers by entering the following command as an applmgr user at operating system prompt.

FNDLIBR FND FNDCPBWV apps/<pwd> SYSADMIN 'System Administrator' SYSADMIN

Troubleshooting Oracle E Business Suite Workflow Mailer and Notification issues

1. Check status of workflow mailer

SELECT component_name, component_status
FROM fnd_svc_components
WHERE component_type = 'WF_MAILER';

2. Check the log file of workflow mailer

Please run the following query to locate all current Workflow Mailer Service log:

set linesize 155; 
set pagesize 200; 
set verify off; 
column MANAGER format a15; 
column MEANING format a15; 
SELECT concurrent_queue_name manager, fcp.last_update_date, fcp.concurrent_process_id pid, meaning, fcp.logfile_name 
FROM fnd_concurrent_queues fcq, fnd_concurrent_processes fcp, fnd_lookups flkup 
WHERE concurrent_queue_name in ('WFMLRSVC') 
AND fcq.concurrent_queue_id = fcp.concurrent_queue_id 
AND fcq.application_id = fcp.queue_application_id 
AND flkup.lookup_code=fcp.process_status_code 
AND lookup_type ='CP_PROCESS_STATUS_CODE' 
AND meaning='Active';

After setting app env file run the following command to collect the logs

grep ":ERROR:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrerr.log
grep "Exception:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrexc.log
grep ":UNEXPECTED:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrunexp.log

3. Check If the notification for the user is going to discard folder

- First check, If the notifications approved through email are going to Discard folder for that user.
- Try to approve the notification from workflow worklist from Oracle EBS ERP and see if its going to Discard folder or processing successfully.

Please run below command to see notifications moved to Discard folder

grep ":DISCARD:"  $APPLCSF/$APPLLOG/FNDCPGSC*.txt  > mlrdiscard.log 
grep "Approver:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt  > mlrapprover.log 

4. Check below queries if the messages are in ready state

select NVL(substr(wfe.corrid,1,50),'NULL - No Value') corrid,
decode(wfe.state,
 0,' 0 = Ready',
 1,'1 = Delayed',
 2,'2 = Retained',
 3,'3 = Exception',
 to_char(substr(wfe.state,1,12))) State,
 count(*) COUNT
from applsys.wf_notification_out wfe
group by wfe.corrid, wfe.state;

select tab.msg_state, count(*) from applsys.aq$wf_notification_out tab group by tab.msg_state ;

select count(*) from APPLSYS.AQ$WF_NOTIFICATION_OUT
where msg_state in ('READY','WAIT');

Oracle Database Startup Hangs or taking long time On Solaris SPARC Operating System

Issue:

Seeing slow SGA allocation on database startup (even in startup nomount taking long time appears hung/hang)

Alert Log stuck at "DISM Started" Line as shown below:

TRUSS output shows startup nomount is sleeping... looping and Err#11 EAGAIN at semtimedop  and sleeping... at pollsys

When truss ( truss -o connect.out -f sqlplus / as sysdba ) the startup or shutdown, or truss the SQL*Plus connection, they all will show spinning on the same semtimedop command.

This is OS level issues.

Cause:

The underlying problem in this specific case was an OS Bug.

If DISM is enabled in a Local Domain / Local Zone configuration the SGA is put into a memory segment which is larger than the actual size of the SGA.

Solution #1:

Disable the use of DISM by the database on Oracle Solaris in one of two ways:

1.Unset the SGA_MAX_SIZE / MEMORY_MAX_TARGET / MEMORY_TARGET parameters.
2.Ensure SGA_MAX_SIZE is set to the same value as SGA_TARGET parameter or equal to the sum of all SGA components in the instance.

Solution #2:

Need to check with OS admin to determine the underlying cause of the semtimedop call failure.

Please apply official patch 150400-02 or later at OS Level.

Workaround is available by setting the following in /etc/system:

set disable_ism_large_pages=0x74

A reboot is required to make the change effective.

Unable to find an Output Post Processor service to post-process request in PCP environment

Problem Description:

Concurrent requests getting completed in warning status which are running on CM Admin node 2 in PCP environment.

Error in Log File:

POST-PROCESSING ---------+ Unable to find an Output Post Processor service to post-process request 13343094. Check that the Output Post Processor service is running.

Cause:

OPP manager does not exist on secondary admin node.

Solution:

In a Parallel Concurrent Processing (PCP) environment an need to create additional Output Post Processing (OPP) concurrent managers in nodes other than the primary node.

Reference:

How to Create a Second OPP Concurrent Manager in a Node Different Than The Primary Node (Doc ID 729883.1)

Intermittent Login issue R12.1 - 500 Internal Server Error

Problem Description:

Users are unable to get application login page.

Users are frequently getting below error while accessing the EBS application URL.

Cause:

This issue occurred due to low oacore heap size.

Solution:

Increase the oacore heap size and restart application services.

Change the parameters in $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

of both form nodes as shown below:

From:

-Xmx512M -Xms128M -XX:MaxPermSize=160M

To:

-Xmx1536M -Xms128M -XX:MaxPermSize=160M

Reference:

JVM: Guidelines to setup the Java Virtual Machine in Apps E-Business Suite 11i and R12 (Doc ID 362851.1)

Intermittent Login issue 11i - 500 Internal Server Error

Check Logs for Error Details:

[appltest@appnode jvm]$ pwd
/oraclesw/testora/iAS/Apache/Jserv/logs/jvm
[appltest@appnode jvm]$

[appltest@appnode jvm]$ tail -15f OACoreGroup.0.stderr
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
ApacheJServ/1.1.2: An error occurred listening to the port: java.lang.OutOfMemoryError
-------------------------------
Thu Nov 12 13:50:09 PST 2015

Increase the JVM Memory:

To increase the memory for oacore JVM,
edit the file   $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties

From:

wrapper.bin.parameters=-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB

To:
wrapper.bin.parameters=-verbose:gc -Xmx1024M -Xms512M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB

Bounce Apache Services:

./adapcctl.sh stop
./adapcctl.sh start

Verify in below log file to confirm JVM started with updated memory parameter:

[appltest@appnode jvm]$ pwd
/oraclesw/testora/iAS/Apache/Jserv/logs/jvm

[appltest@appnode jvm]$ tail -f OACoreGroup.0.stdout

-------------------------------
Mon Nov 23 00:44:25 PST 2015
-------------------------------
/usr/java/j2sdk1.4.2_15/jre/bin/java -DCLIENT_PROCESSID=11905 -Xmx1024M -Xms512M -XX:NewSize=60M -XX:MaxPermSize=128M -XX:MaxNewSize=120M -Xrs

Also Modify the JVM parameter in CONTEXT_FILE, so next time when we run the autoconfig this will make changes permanently:

Parameter : 
<jvm_options oa_var="s_jvm_options" osd="Linux">-Xmx1024M -Xms512M -XX:NewSize=60M -XX:MaxPermSize=128M -XX:MaxNewSize=120M -Xrs</jvm_options>

How to Open Oracle EBS 11.5.10.2 Forms on Windows 10(64-bit) with IE 11

Accessing Oracle EBS 11i Forms on Windows 10(64-bit) with IE 11

This Post covers the steps, required browser and JRE settings to access Oracle EBS 11.5.10.2 Forms (having JRE Sun Plugin  1.6 update 17 on EBS application server) on Windows 10 (64-bit) with Internet Explorer 11.

       1.    Desktop Windows 10 / IE 11 Versions:

         
           2.    Download Java SE Runtime Environment 8u73 (jre-8u73-windows-x64.exe)

                Link to Download->

                http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html

          3.      Start the installation of  jre-8u73-windows-x64.exe

            Click -> Install

            4.      Make sure Java 8 Update 73 (64-bit) installed : Go to -> Control Panel -> Programs and                               Features

               5.      Now add Application URL  http://hostname.domain.com  to :

                     Internet Options -> Security -> Trusted Sites

                      (In screenshot I have striked out the hostname and domain name due to security                                             reasons.)

         6.      Open EBS application -> http://hostname.domain.com:8020/

                      This will prompt below screen -> Click on click here to download J2SE Plugin                                                 1.6.0_17 when prompted.

                Click on -> Save

           Once the Download is completed - > Click on Run

           Complete the installation as shown below

            7.      Make sure Java 6 Update 17 (32-bit) installed : Go to -> Control Panel -> Programs                                        and Features

               8.      Now Go to -> Compatibility View settings

           Add -> domain.com

            9.  Now you will see the applet is loading which takes sometime around 1-2 mins

             10.      Forms opened successfully!!

Users unable to see the assigned responsibilities On E-Business Suite Login Navigator

Issue:

Sometimes we face the issue that responsibilities does not show up on E-Business Suite Login Navigator after assigning to the user

Cause:

User data in the FND_USER, FND_RESPONSIBILITY, WF_LOCAL_USER_ROLES, and WF_USER_ROLE_ASSIGNMENTS tables is not valid.

Solution:

To synchronize data follow the steps below:

1.  Navigate to the define user form (FNDSCAUS).

2.  Query up the user (that is having the issue) and insure the responsibility is NOT end dated.

3.  END DATE the USER (not the responsibility) and save the record.

4.  UNEND DATE the USER and save the record.

5.  Run the request 'Sync responsibility role data into the WF table'.

6.  Run the request "Workflow Directory Services User/Role Validation" (with parameters 10000, yes, yes, yes).

7.  Have the user log off and back on and verify that the responsibility appears.

Ref. Note ID:433466.1

For "Workflow Directory Services User/Role Validation" Request "Workflow Agent Listener Service" must be running

Ref. Note ID:733335.1

To Start the "Workflow Agent Listener Service"

Login as SYSADMIN in OAM then navigate to:

1. Sitemap > Generic Services > Status Overview > View All > Generic Service Component Container > select Workflow Agent Listener Service > Start 

Unable to query some FND users in EBS 11i from front end "Users" Form

Issue:

Recently I came across into the strange issue where some FND users are unable to query from "Users" form.

Using the EBS 11i instance, and querying a user (System Administrator > Security > User > Define), the application is spinning.

Cause:

The instance was earlier configured with SSO however EBS SSO Users not changed to EBS Local User.

DBMS_LDAP: LDAP operation - waiting event due to OID was configured earlier and trying to use DBMS_LDAP while querying the user.

select USER_GUID from FND_USER where USER_NAME='KBROCK';

USER_GUID
--------------------------------
AC76C1E60F002181E040A8C00C0A063C


--Query to check session wait:

set lines 120
set pages 1000
col event for a30
select sid, event, p1, p2, p3, p1raw from v$session_wait
where event not like '%messag%' and event not in ('pipe get','PL/SQL lock timer','Streams AQ: qmn slave idle wait','Streams AQ: waiting for time management or cleanup tasks','Streams AQ: qmn coordinator idle wait')
and (wait_time=0 or state='WAITING')
order by 2
/


--Ouptut :

       SID EVENT                                  P1         P2         P3 P1RAW
---------- ------------------------------ ---------- ---------- ---------- --------
       424 DBMS_LDAP: LDAP operation               0          0          0 00
       430 DBMS_LDAP: LDAP operation               0          0          0 00
       600 pmon timer                            300          0          0 0000012C
       594 smon timer                            300          0          0 0000012C

Solution:

1. Unlink a FND_USER user account in Oracle EBusiness Suite that is linked to an SSO Account in Oracle Internet Directory

[approd@oradb ~]$ cd $FND_TOP/patch/115/sql/
[approd@oradb sql]$ ls -ltr fndssouu.sql

-rwxr-xr-x 1 approd dba 2102 Dec  1  2010 fndssouu.sql
[approd@oradb sql]$
[approd@oradb sql]$ sqlplus apps/apps

SQL*Plus: Release 8.0.6.0.0 - Production on Wed Oct 7 22:17:53 2015

(c) Copyright 1999 Oracle Corporation.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> @fndssouu.sql KBROCK

PL/SQL procedure successfully completed.


Commit complete.

    2.  Kill the earlier DBMS_LDAP: LDAP operation sessions

    3.  Reset user_guid in bulk

--Script for all users having user_guid:

spool userguid.sh
SELECT 'sqlplus apps/apps  @$FND_TOP/patch/115/sql/fndssouu.sql  '||user_name ||' ;'
FROM fnd_user
WHERE user_guid IS NOT NULL
spool off;
/

$ sh userguid.sh > userguid.log

    4. Retest the issue.

Reference Metalink Notes:

• How to unlink a FND_USER user account in Oracle EBusiness Suite that is linked to an SSO   Account in Oracle Internet Directory (Doc ID 429375.1)
• Customer RecommendedHow To Change An EBS SSO User Change To EBS LOCAL User (Doc ID 1931776.1)