kkpo_rcinfo_defstg

UPDATE: bug 10373381 is fixed in PSU2 ( 11.2.0.2.2 ) - see below

ORA-00600[KKPO_RCINFO_DEFSTG:OBJNOTFOUND] ON 11.2 DATABASES AFTER UPGRADE [ID 1308400.1]

Applies to:

Oracle Database Products
Information in this document applies to any platform.

Symptoms

This article is specifically written for V11202 databases that are encountering
on basic operations :-

ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:objnotfound],[xxxxx], []

The error will be reported at the user session level and within the alert log.

The argument [xxxxx] will represent an object_id that can be queried from dba_objects.

The article however might still be applicable for 11201 databases and we will report the
known circumstances.

The following symptoms can be attributed :-

a) ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:objnotfound],[xxxxx], []
will be seen at session level and within alert log

b) The ORA-00600 will be seen on various basic DML/DDL operations e.g.

select /*+ all_rows */ count(1) from "A"."B" where sourceid=49;

select * from DBSNMP.AQ_MNTR_MSGS_QUEUE where rownum <=1;

c) The sessions seeing the error can vary e.g. SQLPLUS, Grid Control

d) Running hcheck.sql on the database it will show messages like :-

Problem: Orphaned IND$ (no SEG$) - See unpublished Note.65987.1 (Bug:624613/3655873)
ORPHAN IND$: OBJ=701427 DOBJ=701427 TS=13 RFILE/BLOCK=0 0 BO#=701321 SegType=

100's of objects are likely to be reported, the OBJ/DOBJ values will correlate to
the arguments reported in the ORA-00600 error

e) select count(*) from deferred_stg$;

This SQL will likely return zero rows

f) select to_char(action_time,'DD-MON-YYYY HH24:MI'),
action, version, id, comments
from registry$history;

This will report an entry similar to :-

29-MAR-2011 09:16 UPGRADE 11.2.0.2.0
Upgraded from 11.2.0.2.0

Informing that CATUPGRD.SQL has been run

All of these symptoms would be expected to be present for the article to be
applicable. If the error is seen without all symptoms being observed this article
can still be considered and a SR raised to Support if assistance is needed.

Changes

The error is primarily associated with various SQL operations failing at session
level after the incorrect running of CATUPGRD.SQL
and is related to missing deferred segment references in the data dictionary.

The error can also be associated with upgrading an earlier <11.2 database to 11.2.0.1
and then subsequently to 11.2.0.2 but again the likely root cause is still linked to
how CATUPGRD.SQL has been run. 

Cause

The cause of the error varies but two primary causes are identified :-

11202 Databases : If database is installed as V11201 and an 'Out-of-place' upgrade
is performed such that 11202 is installed into a new ORACLE_HOME and then
the V11201 CATUPGRD.SQL is inadvertently run it is highly likely this article is applicable

As mentioned this article is specifically written for the above scenario, however :-

11201 Databases : If database is installed as V11201 and the V11201 CATUPGRD.SQL
is inadvertently run it is also highly likely this article is applicable

These are the 'confirmed' scenarios, others might exist and might be assessable
by reviewing the 'Symptoms'

For the meaning of 'Out-of-place' upgrade please refer to the V11202PSR README Document
that will be available when the PSR is downloaded.

The ORA-600 itself is related to deferred segments introduced into 11.2

Solution

If the ORA-600 has been hit and it is shown that CATUPGRD.SQL was run
incorrectly :-

11202 Databases : Restore from a valid backup and perform upgrade/patching again after
ensuring Patch:10373381 is applied to the 11202 ORACLE_HOME

11201 Databases : Restore from a valid backup. No fix is available for this
situation

This article has been created to inform customers of known problems in running
CATUPGRD.SQL incorrectly.

Patch:10373381 will not fix any 'current' problem, it will only prevent the accidental
running of the wrong CATUPGRD.SQL script on an 11202 database.

Patch:10373381 can be applied immediately after installing the software into its
own ORACLE_HOME, this may also act as a preventative measure before any
post installation upgrade steps are followed.

For 11202 Patch:10373381 exists

This is a generic fix that should be applied to the 11202 ORACLE_HOME

If the error is not accounted for by this article and the error is seen only
on a few objects it might be possible to recreate the deferred object/s
associated with the ORA-00600, particularly if  the table is empty and is a
non-ORACLE table.

To determine what object the ORA-600 relates to :-

select * from dba_objects where object_id='XXXXX'

Where 'XXXXX' is the first argument reported from the ORA-600

If the error relates to an ORACLE object and only a few
objects are affected a SR should be raised to Support.  Such objects
should never be dropped from the database unless the dependencies and
DDL recreation is known.   Expectations need to be set in advance in that the
ORA-600 trace does not tell us what caused the error,  it only shows the consequence
and we cannot guarantee cleanup steps for all situations.

If upgrading from a <11.2 RDBMS to 11.2 release it might be useful to plan for upgrade
to 11202 as opposed to 11201

References

BUG:10373381 - ORA-600 [KKPO_RCINFO_DEFSTG:OBJNOTFOUND] AFTER RERUNNING CATUPGRD.SQL


PSU 11.2.0.2.2 contains all fixes previously released in PSU 11.2.0.2.1 and the following new fixes:

Automatic Storage Management

10084145 - dbmv2: ora-600 [1427] mounting diskgroup after all cells restarted
10094201 - disk offline is slow
10102506 - disk resync takes a long time even with no stale extents
10222719 - dbmv2: asm instance hangs with rbal process waits on "no free buffer"
10227288 - dg forcibly dismounted after one fg lost due to "could not read pst for grp 4"
10228151 - asm diskgroups not getting mounted
10230571 - tb:x:reboot one cell node, rbal hit ora-600[17183]
10245086 - ora-01210 during create tablespace
10299224 - tb:x:pivoting an extent on an offline disk can cause stale xmaps in rdbms
10329146 - marking different sr bits from multiple dbws can cause a lost write
10332589 - tb:x:mount normal redundancy dg, failed with ora-00600:[kfcinitrq20]
10356513 - disk offlined with non zero timeout expelled immediately
11067567 - 11202_gibtwo: kept generating "elapsed time did not advance " in asm alert log

Buffer Cache Management

9651350 - ora-00308 and ora-27037 when ora-8103 without event 10736 been set
10110863 - trace files is still generated after applying patch:9651350
10205230 - tb_x64: hit ora-00600: [kclwcrs_6]
10332111 - sql running long in active standby

Generic

9591812 - incorrect wait events in 11.2 ("cursor: mutex s" instead of "cursor: mutex x")
9905049 - ebr: ora-00600: internal error code, arguments: [kqlhdlod-bad-base-objn]
10052141 - exadata database crash with ora-7445 [_wordcopy_bwd_dest_aligned] and ora-600 [2
10052956 - ora-7445 [kjtdq()+176]
10157402 - lob segment has null data after long to lob conversion in parallel mode
10187168 - obsolete parent cursors if version count exceeds a threshold
10217802 - alter user rename raises ora-4030
10229719 - qrmp:12.2:ora-07445 while performing complete database import on solaris sparc
10264680 - incorrect version_number reported after patch for 10187168 applied
10411618 - add different wait schemes for mutex waits
11069199 - ora-600 [kksobsoletecursor:invalid stub] quering pq when pq is disabled
11818335 - additional changes when wait schemes for mutex waits is disabled

High Availability

10018789 - dbmv2-bigbh:spin in kgllock caused db hung and high library cache lock
10129643 - appsst gsi11g m9000: ksim generic wait event
10170431 - ctwr consuming lots of cpu cycles
10190642 - sageetl_dbmv: ora-00600: [1433] followed by instance crash with asm on exadata

Oracle Space Management

6523037 - et11.1dl: ora-600 [kddummy_blkchk] [6110] on update
9724970 - pdml fails with ora-600 [4511]. ora-600 [kdblkcheckerror] by block check
10218814 - dbmv2: ora-00600:[3020] data block corruption on standby
10219576 - ora-600 [ktsl_allocate_disp-fragment]

Oracle Virtual Operating System Services

10127360 - dg4msql size increasing to 1.5gb after procedure executed 250 times

Oracle Transaction Management

10358019 - invalid results from flashback_transaction_query after applying patch:10322043

Oracle Utilities

10373381 - ora-600 [kkpo_rcinfo_defstg:objnotfound] after rerunning catupgrd.sql

Server Manageability

11699057 - ora-00001: unique constraint (sys.wri$_sqlset_plans_tocap_pk) violated