SBL-ADM-02047: Could not send SISNAPI message

Applies to:

Product Release: V7 (Enterprise)
Version: 7.5.3.9 [16194]
Database: Oracle 9.2.0.2
Application Server OS: Sun Solaris 8
Database Server OS: Sun Solaris 8

This document was previously published as Siebel SR 38-2884485569.

Symptoms

SBL-SVR-00027

we lost accesses to the Server Admin screen and srvrmgr on multiple environment (dev, test, staging, etc).

when accessing the server admin screen, we got error msg:
SBL-NET-01034: The SISNAPI connection was closed by the peer

when accessing srvrmgr, we got msg:
Connected to 0 server(s) out of a total of 1 server(s) in the enterprise

Failed to connect server s3_intf_s1: Handshake failed

The srvrmgr.llog file shows:
2021 2006-02-07 10:11:35 2006-02-07 10:12:58 -0600 00000004 001 001f 0001 09
srvrmgr 29510 1 /u00/siebel/siebsrvr/log/srvrmgr.log 7.5.3.9 [16194] ENU
GenericLog      GenericError    1       2006-02-07 10:11:35     (sasess.cpp 9(66
4) err=1801034 sys=0) SBL-NET-01034: The SISNAPI connection was closed by the pe
er
GenericLog      GenericError    1       2006-02-07 10:11:35     (sasess.cpp 9(66
4) err=902047 sys=0) SBL-ADM-02047: Could not send SISNAPI message
GenericLog      GenericError    1       2006-02-07 10:12:58     (sacmd.cpp 10(38
8) err=902049 sys=0) SBL-ADM-02049: There is no connected server targeted for th
at command
GenericLog      GenericError    1       2006-02-07 10:12:58     (sacmdl.cpp 29(6
71) err=902049 sys=0) SBL-ADM-02049: There is no connected server targeted for t
hat command
$


(to be continued)

Solution

Message 1

(continuing...)

The enterprise log file shows:
...
ServerLog       ProcessExit     1       2005-12-07 14:31:27     S3SuspectAssigne
d46085     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
ServerLog       ProcessExit     1       2005-12-07 14:31:27     S3RouteNational
46090     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
ServerLog       ProcessExit     1       2005-12-07 14:31:27     IBSNCSTM
46088     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
ServerLog       ProcessExit     1       2005-12-07 14:31:28     IBSNEscalate
46087     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
...


(to be continued)

Message 2

(continuing ...)

The PSTACK of the coredump shows:
eagnmnsu257(ROOT)# pstack -F core.global.sun4u.eagnmnsu257.1138992838.5252.900.siebsess.7025
core 'core.global.sun4u.eagnmnsu257.1138992838.5252.900.siebsess.7025' of 7025: siebsess 44 1 /u00/siebel/siebsrvr/admin/s3_dev_es.s3_dev_s1.shm 263 e
----------------- lwp# 1 / thread# 1 --------------------
b98c538c __1cJqeLicPath6FpkCpC_v_ (62656c2f, 62656c2f, 73696562, 73727672, 2f6d772f, 6c69623a) + 2c
----------------- lwp# 2 / thread# 2 --------------------
bfb1f090 _signotifywait (bf02c000, bfbe0b80, 0, bf6c0ef8, 2f, 7efefeff) + 8
bfb1a534 thr_errnop (0, 0, 0, 0, 0, 0) + 20
----------------- lwp# 3 --------------------------------
bf019300 private___lwp_cond_wait (bed65d98, bf02cd74, bf02c000, 0, 0, 4) + 8
bfb1cc8c _door_return (bed65cd8, bf00a380, 0, 0, 0, 0) + 68
----------------- lwp# 4 --------------------------------
bfb1cc34 _door_return (25, bf02c000, bf02d678, 3, bf02c000, 1) + 10
bf00a380 _lwp_start (bea0bd98, 0, 0, 0, 0, 0) + 18
bfb1a534 thr_errnop (0, 0, 0, 0, 0, 0) + 20
-------------------------- thread# 3 --------------------
bf00d9e0 _reap_wait (bf030988, 1e8fc, 0, bf02c000, 0, 0) + 38
bf00d738 _reaper (bf02ce08, bf032710, bf030988, bf02cde0, 1, fe400000) + 38
bf01b11c _thread_start (0, 0, 0, 0, 0, 0) + 40

(to be continued)

Message 3

Output of a truss on the srvrmgr is attached to this SR for your review.

Also, when we ran the odbcsql, we also got a core dump, the pstack is the same as the pstack for srvrmgr.

we are able to start the siebel servers and the callcenter OM, and access screens other than the server admin screen. But the OM's with SIGBUS do not work any more.

Even more confusing is that we we cold start (reboot the machines), the error does not appear. However, this is not a acceptable solution. Because we are running on Solaris machines, reboot the production won't be thinkable.

If you can assist us in find and resolve the problems before we move to production, it's great.

Thanks,

Message 4

(1/2)

For the benefit of other readers:

It was found in the truss output and in the siebsrvr_xx.log logs located in siebsrvr/log directory the error SBL-SVR-00027 error message found. The error may be due to lack of file descriptors.

Based on above, the customer was recommended to follow the instructions on the bookshelf below and adjust the kernel settings appropriately.

Siebel Server Installation Guide for UNIX > Tuning UNIX Operating Systems for Siebel Installation > Tuning Siebel eBusiness Applications for Solaris > Tuning Kernel Settings for Solaris

The kernel adjustments alleviated the problem for a while, but the SIGBUS error and core dumps start happening again.

Please, see below the findings that I have found about in the truss.out file provided, within this file there is the _LD_LIBRARY_PATH value used, as you can see below there is no reference for the /u00/oracle/product/920/v64/lib32 directory, also it is repeating some path directories some times. This is exactly what I would like to avoid in the tests that I have asked you.

The LD_LIBRARY_PATH was reviewed and it was found a reference to the Oracle 64 bit library directory. The customer was asked to remove the reference to the Oracle 64 bit library keeping only the Oracle 32 bit library as it is the only supported version of Oracle client libraries.

After removing the Oracle 64 bit library path from the LD_LIBRARY_PATH the SIGBUS error and core dumps did not appear anymore.

Message 5

(2/2)

The following is found in the Bookshelf > SIEBEL SERVER INSTALLATION GUIDE FOR UNIX VERSION 7.5.3 JULY 2003 12-FN585F > Installing the Siebel Server.

NOTE: Siebel applications support the Oracle 32-bit client, but if you have installed the Oracle 64-bit client on your Siebel Server, you need to add $ORACLE_HOME/lib32 instead of $ORACLE_HOME/lib to your LD_LIBRARY_PATH (Solaris),SHLIB_PATH (HP-UX), or LIBPATH (AIX).

Kind Regards,
Siebel Technical Support