SBL-SMI-00062: Internal: No more process (multithreaded server) slots available

Applies to:

Siebel System Software - Version: 7.8.2.3 SIA [19221] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Enterprise)
Version: 7.8.2.3 [19221] Com/Med
Database: Oracle 9.2.0.6
Application Server OS: Microsoft Windows 2003 Server
Database Server OS: IBM AIX 5L 5.2

This document was previously published as Siebel SR 38-3474473723.

Symptoms

SBL-CSR-00102, SBL-SMI-00062, SBL-GEN-28758, SBL-SCB-00005- CALL STACK -
siebmtshmw +0xffc00000
sssasos +0x489e = CompCleanup() +0x1f1e
siebmtshmw +0x527e
siebmtshmw +0x20347 = SmiInProcMsgHandler() +0xad7
siebmtshmw +0x22d16 = SmiInProcMsgHandler() +0x34a6
siebmtshmw +0x15a13 = SmiCleanupDetTask() +0xddd3
siebmtshmw +0x16fd7 = GetSmiWorkQObj() +0x1497
siebmtshmw +0xca33 = SmiCleanupDetTask() +0x4df3
sslcosd +0x20f2 = OSDThreadPrivIsInit() +0x1d0
sslcosd +0x217e = OSDThreadPrivIsInit() +0x25c
MSVCR70 +0x412f = beginthreadex() +0xba
kernel32 +0x2608b = GetModuleFileNameA() +0xeb

Cause

Configuration/ Setup

Solution

Message 1

For the benefit of other readers.

The crash with call stack above happened very frequently in the customer.

In the OM logs right before the crash there was this error that suggested the communication between Siebel Web Server and Siebel App Server was terminated:

SisnTcpIp    SisnSockError    1    0    2007-10-22 12:50:25     11712: [TCPIP-server] send() failed for sd=3768 (err=10054 | An existing connection was forcibly closed by the remote host (peer).)

Because of the crash other subsequent messages were seen in SCBroker log. But they are just a consequence of the crash:

SBL-SCB-00005: Connection timeout waiting for message.
SBL-SCB-00011: Failed to connect to pipe (SEBL_3_5608) on process 5608.

Further, this was last actions captured by *.fdr file. But again this was just a consequence and it was not related to the root cause of the crash:

Communications Client Business Service    PushToolbarMessage   
Communications Client Business Service    PushToolbarMessage   
Fire Service Event    Communications Client    WaitForCmd

Solution:

After disabled SNMP protocol to diminish the retransmission of the TCP packages, this crash was no longer seen.

Applies to:

Siebel Workflow - Version: 7.7.2.10 [18385] to 8.0 [20405] - Release: V7 to V8
Information in this document applies to any platform.
** Checked for relevance on 27 February 2012**

Symptoms

Customer is facing recent and intermittent unavailability of WfProcMgr component installed in one server. Customer has noticed one occurrence this Sunday 02/15 at 10:30 AM and  02/16 3:20 AM. When it occurred they were able to restart the server. Customer is not able to always restart the siebel server because the application does not respond in a timely manner.

Since WfProcMgr seems unavailable for some reason SISNAPI errors in SRBroker logs were found also as a consequence of this component being unavailable it seems that records in S_ESCL_REC are increasing.

The records in S_ESCL_REC seems to be associated to triggers created for assignment of campaigns. No recent changes were made on wf policies or configuration.



In the enterprise log and confirmed that WfProcMgr became unavailable.

From the Enterprise logs the related WfProcMgr logs and this analysis resulted in 3 groups of enterprise +wfprocmgr logs:

In WfProcMgr_58975.3.log  can also find the error SBL-SMI-00140: Internal: The MT Server has been disabled.
Customer can see from WfProcMgr_80330.log that the same instance'1-6E1YM0' was still not able to be resumed even after several hours > SBL-BPR-00124: Cannot resume process instance '1-6E1YM0'. Verify that it does exist and has a 'Waiting', 'Suspended' or 'In Error' status.

A3, B3: For the Workflow Process Manager created by the Siebel Server Scheduler at 2009-02-16 03:19:36 with task id 80973 and exited at 2009-02-16 03:20:37.

Finally here Customer were able to find the error SBL-SMI-00062: Internal: No more process (multithreaded server) slots available in WfProcMgr_80973.log.


Customer checked the memory limits and found that memory limits are not being reached (or reached only 20% below the limit) which indicate that the cause does not seem memory related.

Cause

There were 2 problems associated to this workflow component crash;hang:

1. Associated to the jobs executed by workflow process SMCC - List Import is consuming too many threads. At some point max tasks limit is reached and as a consequence no more threads can be created to continue trying to process these records. This would explain why we saw the error SBL-BPR-00124: Cannot resume process instance '1-6E1YM0'.

2. After this problem was solved it seems that a crash was found. This crash was associated to the workflow process "SMCC Sync PEM Status - Contact Phone Status" fails.
The error seems to happen when the method "'SyncPEMStatus" updates a business component field with a picklist with a value that is not available in this picklist. The pick lists name is "Response Outcome PickList".

Solution

1. By stopping the instances in loop from the workflow instance view have helped to free up some threads and avoid the Max Tasks limit to be reached. This means that this job that is processing List Import records must be reviewed in order to ensure that it will not keep in loop and allocating a high number of threads until the limit is reached.

2. The suggestion here is to ensure that the value being used (by workflow process "SMCC Sync PEM Status - Contact Phone Status") to update the field (by the method "'SyncPEMStatus") exists in the picklist associated to this field ("Response Outcome PickList").

Applies to:

Siebel CRM - Version: 8.0.0.5 [20420] - Release: V8
Information in this document applies to any platform.

Goal

If eChannelObjMgr_enu PID crashes the server creates one process for eChannelObjMgr_enu + 1 additional process for SrvrSched.
that's when we hit this error "No more process (multithreaded server) slots available".

Can you please explain the cause of this error?


Below is the log snippet:

ServerLog ProcessExit 1 00001d5f4b2617ee:0 2009-12-16 11:19:18 eChannelObjMgr_enu 17140 SBL-OSD-02006 Process 17140 exited with error - Process exited because it received signal SIGABRT.

ServerLog ProcessCreate 1 00001d5d4b2617ee:0 2009-12-16 11:19:18 Created server process (OS pid = 5745 ) for eChannelObjMgr_enu
ServerLog ProcessCreate 1 00001d5d4b2617ee:0 2009-12-16 11:19:18 Created server process (OS pid = 5759 ) for SrvrSched

ServerLog ProcessExit 1 00001d5f4b2617ee:0 2009-12-16 11:29:21 eChannelObjMgr_enu 5745 SBL-SMI-00062 Process 5745 exited with error - Internal: No more process (multithreaded server) slots available
ServerLog ProcessExit 1 00001d5f4b2617ee:0 2009-12-16 11:29:21 eChannelObjMgr_enu 5759 SBL-SMI-00062 Process 5759 exited with error - Internal: No more process (multithreaded server) slots available

ServerLog ProcessCreate 1 00001d5d4b2617ee:0 2009-12-16 11:29:21 Created server process (OS pid = 7246 ) for eChannelObjMgr_enu

Solution

After an OM failure new MT Server PID is started and SrvrSched also starts a PID (as a result some more processes are spawned).

The AOM tuning does recommend you to set MaxMTServer=MinMTServers to avoid overhead time for starting the new process.
However the defect CR# 10642132 causes these error messages appear in the log under following scenario:

When you have MaxMTServer=MinMTServers and OM crash occurs then component starts a process as it does not have the requisite MinMTServers running.
The siebsrvr (Scheduler) also starts a process.

If the new process that started sees that it has more than MinMtServers, it will exit.
These message do not imply problem in the above case.

Hence it would help to set MinMTServers to a lower value than MaxMTServers so that at any given time you have higher than MinMTServers running but lower than MaxMTServers.

If the issue persists after thatinvistigate the resource availability aspect of the server and the concurrent OM loads.









 

Applies to:

Siebel System Software - Version: 7.5.3 [16157] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Professional)
Version: 7.7.2.1 [18353]
Database: Oracle 9.2.0.5
Application Server OS: Microsoft Windows 2003 Server
Database Server OS: Sun Solaris 8

This document was previously published as Siebel SR 38-2943624441.

Symptoms

SBL-SVR-00028, SBL-SMI-00062 We did a reconfigure on our Call Center Object Manager and see that afterwards we get errors in the log

SCCObjMgr_enu   8213      SBL-SMI-00062   Process exited with error - Internal: No more process (multithreaded server) slots available

I suspect that we may not have something set correctly to effectively use the reconfigure option.

Solution

In reviewing the log filles the error "SBL-SVR-00028: No more tasks available for this component" was also found. Both errors are usually related to the MaxTasks and MaxMTServers values on the server. The customer was advised to view the document "Tuning AOM Components for CPU and Memory Utilization" and Technical Note 388 on SupportWeb, these give details on how to best calculate the values.

The customer updated the values of MaxTasks and MaxMTServers, as well as MemoryBasedRecycle and MemoryLimit, and this resolved the issue.