SBL-SVR-00052: Internal: Invalid proc handle

Applies to:

Siebel CRM Service, SPE - Version 8.0 [20405] to 8.0 [20405] [Release V8]
Information in this document applies to any platform.

Symptoms

Statement of what the issue is

After a fresh installation, the Server Request Broker component fails to initialize at startup

ERROR generated in Siebel Enterprise log
-------------------------------------------------
ServerLog ComponentUpdate 2 0000290248210580:0 2008-05-07 14:18:31 SCBroker INITIALIZED Component has initialized.

GenericLog GenericError 1 00007988482104d8:0 2008-05-07 14:18:34 (scirkey.cpp (1142) err=1310772 sys=2) SBL-SVR-00052: Internal: Invalid proc handle

ServerLog ProcessExit 1 00007988482104d8:0 2008-05-07 14:18:34 SRBroker 3584 TERMINATED Process 3584 was terminated

GenericLog GenericError 1 0000290248210580:0 2008-05-07 14:23:26


ERROR generated in Server Request Broker log
---------------------------------------------------------
ServerLog LstnObjInherit 3 0000000248210e00:0 2008-05-07 14:18:26 Inherited listening object for port 49177

ServerLog LstnObjPrivCreate 3 0000797f482104d8:0 2008-05-07 14:18:26 Created port 49180 for Server Request Broker

DBCLog DBCLogError 1 0000797f482104d8:0 2008-05-07 14:18:29 [DataDirect][ODBC Oracle driver][Oracle]ORA-12541: TNS:no listener


GenericLog GenericError 1 0000797f482104d8:0 2008-05-07 14:18:29 (srbthrd.cpp (3997) err=2097168 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to connect to DB (data ops)

GenericLog GenericError 1 0000797f482104d8:0 2008-05-07 14:18:29 (srbmtsrv.cpp (86) err=2097168 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to connect to DB (data ops)

SrbLayerLog Error 1 0000797f482104d8:0 2008-05-07 14:18:29 Main Init fails

GenericLog GenericError 1 0000797f482104d8:0 2008-05-07 14:18:29 (smimtsrv.cpp (1187) err=2097168 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to connect to DB (data ops)

SmiLayerLog Error 1 0000797f482104d8:0 2008-05-07 14:18:29 Terminate process due to unrecoverable error: 2097168. (Main Thread)

Cause

After reviewing the logs provided, it was determined that possible problem areas include (a) an incorrectly defined ODBC data source and/or (b) problems with the the Listener configuration; although wider Database connectivity problems cannot be discounted.

DBCLog DBCLogError 1 0000797f482104d8:0 2008-05-07 14:18:29 [DataDirect][ODBC Oracle driver][Oracle]ORA-12541: TNS:no listener
"ORA-12541: TNS:no listener" is usually logged when a connection request can not be completed because the listener is not running (or is simply unavailable) at the specified location - see Note:21475.1 on Metalink

Corrective Action is described as follows:
Ensure that the supplied destination address matches one of the addresses used by the listener - compare the TNSNAMES.ORA entry with the appropriate LISTENER.ORA file (or TNSNAV.ORA if the connection is to go by way of an Interchange). Start the listener on the remote machine.

Solution

Based on a review of the logs recently uploaded, the most probable cause of the initialization problem is an incorrectly defined ODBC Data Source or a misconfigured listerner

It would therefore be of some help it you could carry out the following measures and provide proof on the results obtained at each stage (screen shots, *.cfg file (3)), if further assistance is required.

(1) Verify the ODBC Data Source is correctly defined as shown in this bookshelf reference: 'Siebel Installation Guide for Microsoft Windows > Configuring Siebel Enterprise Server and Related Components > Verifying the ODBC Data Source > Verifying the ODBC Data Source for Oracle'

and/or

(2) Test connectivity to the database, using the ODBC DataSource tool and also verify the hostname of the Server hosting the Database Listener by comparing TNSNAMES.ORA entry with appropriate LISTENER.ORA file.

and/or

(3) Ensure that the ODBC Data Source name also matches the Server connect string specified in siebel.cfg on the Siebel Server.

Applies to:

Siebel System Software - Version 8.0.0.8[20430] to 8.1.1.7 [21238] [Release V8]
Information in this document applies to any platform.

Symptoms

Customer had installed 8.0.0.9 patchset to production enviroment. Previously customer had Siebel 8.0.0.8 version and was installed on the IBM AIX 5.3 (64-bit). After installation of 8.0.0.9 patchset siebel server was not coming up.

Error:
-------
Siebel server Logs:
--------------------
GenericLog GenericError 1 000069d14e0b30c0:0 2011-06-29 18:15:09 (sccnmsys.cpp (1305) err=2883609 sys=0) SBL-SCC-00025: No value found in the Gateway. Default value in the repository is used.
GenericLog GenericError 1 000069e04e0b30c0:0 2011-06-29 18:15:09 (sccnmsys.cpp (1305) err=2883609 sys=0) SBL-SCC-00025: No value found in the Gateway. Default value in the repository is used.

Enterprise Logs:
------------------
GenericLog GenericError 1 000076ee4deb1046:0 2011-06-05 13:31:39 (scirkey.cpp (1142) err=1310772 sys=0) SBL-SVR-00052: Internal: Invalid proc handle
ServerLog ProcessExit 1 000076ee4deb1046:0 2011-06-05 13:31:39 SRBroker 594106 SBL-GEN-00255 Process 594106 exited with error - Internal: Error during exec()

Changes

Applies patchset 8.0.0.9 on top of 8.0.0.8.

Cause

After Verification it was found installable library files from previous installation 8.0.0.8 got corrupted and hence installation of 8.0.0.9 is not getting through.

Further verification showed, release timestamp for some library files from 'siebsrvr/lib' were not set to patch release date but patch install date.
In Ideal case the timestamp should correspond to patch release date, thing is that the release time stamp is only set at the very end. If something breaks before then files remain on disk with install date.

In case of customers environment, some library files has timestamp of April2011 which is patch installation date, in correct scenario it should have been patch 8.0.0.8 release date (Oct2009).
Which shows incorrect deployment of 8.0.0.8 patchset.

Solution

Below are some prominent steps for re-installation of gateway and Siebel Servers:

Reinstalling the Gateway Server:
-----------------------------------------------------------
1) Stop the Gateway service
2) Uninstall the Gateway server.
3) Delete the Gateway server directory structure.
4) Reboot the machine.
5) Install the Gateway server.

Reinstalling the Siebel Server:
-----------------------------------------------------------
1) Make backups of all the .cfg files present in \siebsrvr\bin and the siebns.dat file present in \gtwysrvr\ADMIN.
2) Make backup of the .srf file present in \siebsrvr\OBJECTS.
3) Make backups of the following folders present in the \siebsrvr directory to take care of any customizations:
IDCENTRIC, MSGTEMPL, REPORTS, WEBTEMPL.
4) I am not sure if you have remote users in your environment, If there are remote users working off this application server, backup the following folders to avoid reextracting them:
DOCKING, DBTEMPL
5) Stop the Siebel server service
6) Uninstall the Siebel server
7) Delete the Siebel server directory structure.
8) Remove Siebel Server from the Gateway by removing configuration from wizard.
9) Reboot the machine.
10) Start the Gateway server service if not running already.
11) Install the Siebel server using same installation parameters as before.
12) Copy the backed up folders, .cfg files and the .srf file to the corresponding directory where it belongs.
13) Stop and restart the Gateway and Siebel server services.
14) From the Server Administration screen enable the needed component groups.

There is a note out there on MOS with the correct steps. It refers to Windows but same steps apply to AIX as well:
Reinstalling the Siebel Servers (Doc ID 476317.1)

Note: This solution was specific to one environment/scenario, there is possibility that same solution may not work for other environments.

Applies to:

Siebel System Software - Version: 8.0 [20405] and later   [Release: V8 and later ]
z*OBSOLETE: Microsoft Windows 2000
Product Release: V7 (Enterprise)
Version: 7.5.3.15 [16279]
Database: Oracle 9.2.0.8
Application Server OS: Microsoft Windows 2000 Server SP 4
Database Server OS: Red Hat Linux 4.0

This document was previously published as Siebel SR 38-3355796733.

Symptoms

Customer reported the following:
We just completed the Siebel upgrade from version 7.5.3.15 to version 8.0. Presently we are testing the application and when we go to Admin-> Server Management screen, we are sometimes getting the following errors:
>>>
We dedected an Error which may have occurred for one or more of the following reasons:
28*SBL-SCM-00028: Key not found1*012*sccconfg.cpp4*26880*0*0*0*
<<<

and
>>>
We detected an Error which may have occured for one or more of the following reasons:
SBL-SCM-00028: Key not found
<<<

>>
>>

Cause

The Problem was caused by a corrupted enterprise entry in the enterprise configuration file siebns.dat.
As a result of this case, document bug 10524274 was logged to address the fact that there is no explicit documentation in the Siebel CRM bookshelf how to remove an enterprise from the configuration.

SBL-SVR-00052

Solution

For the benefit of other readers:

After reviewing the siebns.dat file it could be noticed that there are two enterprise entries. One enterprise called 'PRODUCTION' and the other one 'test'. The 'test' enterprise had no sub key entries which is an indication that this enterprise entry is corrupt.

To remove the corrupted enterprise entry from the gateway name server configuration and thus from the siebns.dat file the customer used the following steps:
1> stopped all SWSE web server services
2> stopped all Siebel server services
3> used the enterprise configuration wizard to remove the enterprise 'test'.
4> restarted the gateway name server service
5> started the Siebel server services
6> started the SWSE web server service
After that the customer logged in to the Siebel Callcenter application using a Siebel web client (thin client) and was able to navigate to the Siebel system administration screens.

Applies to:

Siebel CRM - Version: 8.1.1 SIA [21111] and later   [Release: V8 and later ]
Information in this document applies to any platform.
***Checked for relevance on 17-Feb-2012***

Symptoms

The Gateway daemon for Siebel Industry Applications version 8.1.1 running on IBM AIX 5.3 against Oracle 11g shows high CPU. This can be observed after starting a Siebel server, which has a duration of 2-3 minutes. After the Siebel server has started the CPU of the Gateway daemon reduces and stabilizes, however the Siebel Workflow components fail after startup. If these failed components are restarted manually via Siebel Server Manager then the CPU increases on the Gateway as before.

ERROR MESSAGES

1. SBL-DAT-00173: An invalid key was entered.
2. SBL-NET-01218: The connection was refused by server %1. No component is listening on port %2.
3. SBL-SRB-00040: Cannot open asynchronous SISNAPI connection.
4. SBL-GEN-05009: Unable to connect to the gateway server.
5. SBL-SVR-00052: Internal: Invalid proc handle

Cause

Issue caused by extremely large SIEBNS.DAT which can be seen by the high CPU in the gateway process originating from the reads performed by the NSClient.

Bug 10643323 has been logged to address a Product Enhancement Request so that the size of the SIEBNS.DAT does not grow excessively and if it does then to provide means to reduce the size - in particular removing event log level entries.

The large amounts of reads being performed by the NSClient are directly related to the high CPU observed on the gateway

Solution

To reduce the size of the SIEBNS.DAT, the following can be employed to resolve the issue:
1. Restore backup of SIEBNS.DAT after changing event log levels for large amount of server components
2. Remove unnessary component definitions

Applies to:

Siebel Assignment Manager - Version 8.0.0.5 SIA [20420] and later
Information in this document applies to any platform.

Symptoms

Customer was calling Assignment Mgr component from Siebel workflow using synchronous 'Server Request'. They were seeing Assignment Mgr failures and errors in Server Request Broker logs.

The issue / error message still happened even after the enterprise restart/reboot this morning. This issue had caused Leads exceptions when ever they ran the Assignment Manager job with a couple of hundred records to over a thousand. The issue had been intermittent and not reproducible in any other environments.

1) The following error was found in the Enterprise log error:

“SBL-SVR-00052: Internal: Invalid proc handle”. Then AsgnSrvr is terminated. So it never stays online. Thus SR Broker can never see it, and Assignment Manager does not work.

2) The following errors were prevalent in the Workflow, SRBroker and Assignment Manager logs:

"LM Automation Process Failed: Error invoking service 'Server Requests', method 'SubmitRequest' at step 'Invoke Assignment Rules.'.(SBL-BPR-00162)
--
"GenericError 1 0000123d4b9425bc:0 2010-03-12 00:10:30 (asgnsrvr.cpp (914) err=851969 sys=0) SBL-OSD-00001: Internal: Function call timed out (0)"

SBL-SRB-00061: Process AsgnSrvr on Siebel Server AMSPROINT1 terminated
SBL-BPR-00162: Error invoking service 'Server Requests', method 'SubmitRequest' at step 'Invoke Assignment Rules.'.

“13400: [TCPIP-server] bind() to INADDR_ANY:49152 failed for sd=5464 (err=10048 | Address is already in use (TCP/IP address/port).)”

“SBL-SVR-09016: Failed to get task instance: task number 1516240902. The task may have exited or does not exist.”

Cause

There is one AsgnSrvr parameter caught Oracle’s attention: "Refresh people skills interval" = 60 (seconds). The setting is too low for typical usage. If the refresh process takes too much time to complete, it may prevent AsgnSrvr to proceed. it may worth for further investigation.

This is parameter is also called "MaxSkillsAge". The customer had the AsgnSrvr component parameter 'MaxSkillsAge=60': 60 seconds is too short. While reloading Skills, AsgnSrvr cannot process any request. If the skill data doesn't change frequently, 1 day may be more appropriate (which would be 86,400 seconds), depending on how often employee skills are updated.

Solution

After changing MaxSkillsAge parameter to 0, customer did not see Assignment Manager error in any of the servers. They set it to zero instead of 1 day because the customer uses the Release button in the application that clears the rules from the system cache, re-evaluates the skills and regenerate the rules.

Questions to PM:

Question 1: Could you please double confirm (if required) from Engineering that Clear Rules will re-evaluate the skills too? This will confirm that the MaxSkillsAge parameter can be set to 0 (default value - disabled).
Answer: Yes. By clicking the Clear Rules (OOB it is called “Release” button), the rulecache will be regenerated and the skills and skill items will be reloaded into memory from the database.

Question 2: If the MaxSkillsAge is disabled (value set to 0), how does it impact the Dynamic Assignment functionality where the user does not get to control "Clear Rules" button in the application? If the Skills are modified on daily basis or hourly basis and MaxSkillsAge is disabled, how are the skills evaluated in Dynamic Assignment jobs?
Answer: When AsgnSrvr component is started the skills will be loaded into memory, and this will be used for Dynamic assignment until there is a reload either due to “Clear Rules” button or because MaxSkillsAge is set to a non zero value. So if you set MaxSkillsAge to zero and the Clear Rules button is not clicked then the values stored in memory during startup will be used.

Applies to:

Siebel Finance - Version: 8.1.1.3[21219] and later   [Release: V8 and later ]
Information in this document applies to any platform.

Symptoms

When "start_server all" command is excuted on already runnign server, Its stopping the already running server on AIX.

Siebel Server "edcasbld309" (Enterprise "ccp8s1") is already running.
started at Thu Oct 20 15:02:34 2011, pid: 1155072, autostart: no

Cause

Tested it internally by running start_server all command on already running server on AIX,
And when i checked the enterprise log, components are terminating with below error in enterprise log.

GenericLog GenericError 1 00000f6f4ec64046:0 2011-11-21 21:49:38 (scirkey.cpp (1142) err=1310772 sys=0) SBL-SVR-00052: Internal: Invalid proc handle
ServerLog ProcessExit 1 00000f6f4ec64046:0 2011-11-21 21:49:38 SRBroker 835728 TERMINATED Process 835728 was terminated

Applies to:

Siebel CRM - Version 8.1.1.4 SIA [21225] and later
Information in this document applies to any platform.

Symptoms

The SRBroker component is crashing at Siebel Server startup
The Siebel Enterprise log shows the crash when the Siebel server is starting :
ServerLog ComponentUpdate 2 00002e305004003e:0 2012-07-17 00:02:40 SrvrSched INITIALIZED Component has initialized (no spawned procs).
ServerLog ProcessCreate 1 00002e305004003e:0 2012-07-17 00:02:40 Multithreaded-Server-Prozess (OS-PID= 13959346 ) für SRBroker erstellt
ServerLog ProcessCreate 1 00002e305004003e:0 2012-07-17 00:02:40 Server-Prozess (OS-PID= 24051950 ) für SCBroker erstellt
ServerLog ProcessCreate 1 00002e305004003e:0 2012-07-17 00:02:40 Server-Prozess (OS-PID= 28770536 ) für SCBroker erstellt
ServerLog ComponentUpdate 2 00002e305004003e:0 2012-07-17 00:02:45 SCBroker INITIALIZED Component has initialized.
GenericLog GenericError 1 00002e315004003e:0 2012-07-17 00:02:49 (scirkey.cpp (1142) err=1310772 sys=0) SBL-SVR-00052: Intern: Ungültiger Prozess-Handle
ServerLog ProcessExit 1 00002e315004003e:0 2012-07-17 00:02:49 SRBroker 13959346 SBL-OSD-02011   Process 13959346 exited with error - Prozess wegen einer Segment-Verletzung beendet (SIGSEGV)
ServerLog ProcessCreate 1 00002e305004003e:0 2012-07-17 00:02:50 Multithreaded-Server-Prozess (OS-PID= 39518450 ) für SRBroker erstellt
ServerLog ComponentUpdate 2 00002e305004003e:0 2012-07-17 00:03:00 SRBroker INITIALIZED Component has initialized.

Looking at the core file for this crash in dbx debugger ... crashing thread:
(dbx) where
pth_signal.pthread_kill(??, ??) at 0xd05098c0
pth_signal._p_raise(??) at 0xd0508d28
raise.raise(??) at 0xd01373e0
_SigBusSegvIotHandler(int,int,__sigcontext*)(??, ??, ??) at 0xd0f18020
CSSNSClient::SubTreeRead(const wchar_t*,NSScope,NSReqObj*&,unsigned int&)(??, ??, ??, ??, ??) at 0xd2492588
scmNSClient::ExecRequest(scmRequestObj*,scmExecReqFlags)(0xd05d9bbc, 0x3002cd50, 0x3002ce78) at 0xd1da51b8
sccQueryObj::Execute()(??) at 0xd282905c
sccQueryObj::FindObj(const wchar_t*,sccObject*&)(??, ??, ??) at 0xd2828be8
sccComponent::FindConnStr(const wchar_t*,sccConnectStr*&)(??, ??, ??) at 0xd27f9c04
SRBRouter::_GetOneServerInfo(sccServer*,NSSrvrInfo*)(??, ??, ??) at 0xd3e1e358
SRBRouter::_GetAllServerInfo(sccEnterprise*,SRBQueue*&)(??, ??, ??) at 0xd3e1c3dc
SRBRouter::CacheNS(int)(??, ??) at 0xd3e07c20
SRBRouter::Init()(??) at 0xd3e071f4
SRBMTServer::MainInit()(??) at 0xd388d390
smimtsrv.smiMainThread::CompMainInit()(??) at 0x100135f0
smimtsrv.smiMainThread::CompMainInit()(??) at 0x1000a028
smiMainThread::Startup()(??) at 0x10006cb8
wmain(??, ??) at 0x100035f4
main(??, ??) at 0x100032c4

Threads in the debugger ...
(dbx) th
thread  state-k     wchan    state-u    k-tid   mode held scope function
>$t1     run                  running  93717039     k   no   sys  pthread_kill
$t2     run                  running  79298627     u   no   sys  select
$t3     run                  blocked  41353839     u   no   sys  _event_sleep
$t4     run                  blocked  78315649     u   no   sys  _event_sleep
$t5     run                  blocked  87818473     u   no   sys  _event_sleep
$t6     run                  blocked  12583097     u   no   sys  _event_sleep

Cause

This crash occurs when multiple Siebel Servers are started in short order. A race condition is leading to a crash of the SRBroker on some of the Siebel Servers.

Solution

A bug has been raised for the forthcoming 8.1.1.9 Fix Pack:
BUG 13976875: SRBROKER CRASHES AT STARTUP

A bug has been raised for a Quick Fix for the 8.1.1.8 Fix Pack :
BUG 14380887: SRBROKER CRASHES AT STARTUP

The current workaround is to stop and restart the Siebel Server if this SRBroker crash is seen when the server starts.  The restart of the server at this point means that the server is not in the short order sequence, hence the SRBroker should not encounter this defect.