Applies to:
Siebel System Software - Version: 7.5.2.100 [15252] to 8.1.1.1 SIA [21211] - Release: V7 to V8Generic UNIX
Generic Linux
Product Release: V7 (Enterprise)
Version: 7.5.2.100 [15252]
Database: IBM DB2 8.1 FixPack 6b
Application Server OS: IBM AIX 5L 5.1
Database Server OS: IBM AIX 5L 5.1
This document was previously published as Siebel SR 38-3078571521.
Symptoms
SBL-GEN-00255, SBL-OSD-00220, SBL-OSD-02006, SBL-OSD-02011, SBL-OSD-00204, SBL-ADM-02049, SBL-GEN-00001, OSDmprotectThe Siebel Server was running fine until a few weeks ago. Now we are unable to start the Siebel Server. The Siebel Gateway Name Server is running fine.
When we start the Siebel Server by using start_server all command, the main thread starts (siebsrvr) but when it spawns the child threads (siebmtsh), they are not staying alive.
No Siebel multithreaded server processes are started after running start_server all and no Application Object Manager log files are being generated.
The SiebSrvr.log under siebsrvr/log directory may contain the following error:
SBL-OSD-00220: Internal: pthread_mutex_trylock () failed with error. (22)
In some situations the following error may also be seen:
OSDmprotect: mprotect(0xf7400000, 0, 0x00000001) failed with error 22
In addition to SBL-OSD-00220 error messages in SiebSrvr.log log file, error messages like the following are being logged in the Enterprise Server log file (<enterprise_name>.<server_name>.log) under $SIEBEL_ROOT/enterprises/<enterprise_name>/<server_name>/log directory:
<NoCompName> 64527 SBL-GEN-00002 Process exited with error - Error code SBL-GEN-00002
Changes
Cause
A) This situation has been reported when the Unix machine has been shutdown without having run the stop_server script. The stop_server script should remove the osdf.<enterprise_name>.<server_name> file that resides in the $SIEBEL_ROOT/siebsrvr/sys directory.B) This situation has also been reported when the svc.siebsrvr.<enterprise_name>:<server_name> was duplicated under $SIEBEL_ROOT/siebsrvr/sys directory.
When you specify the ALL argument, the start_server script searches for all files under $SIEBEL_ROOT/siebsrvr/sys in the format svc.siebsrvr.* and parses it to identify the Enterprise name and the Server name.
The first string after svc.siebsrvr. is the Enterprise name and the second one is the Server name.
The start_server script skips all filenames ended with the .bak extension, because they are considered backup files.
Then it starts the service and creates the OSDF file based on these parsed names.
Customer has three svc* files on his system:
- $SIEBEL_ROOT/siebsrvr/sys/svc.siebsrvr.<enterprise_name>:<server_name>
- $SIEBEL_ROOT/siebsrvr/sys/svc.siebsrvr.<enterprise_name>:<server_name>.bak
- $SIEBEL_ROOT/siebsrvr/sys/svc.siebsrvr.<enterprise_name>:<server_name>.OLD1
The second file is considered as a backup file by start_server script.
However, the other two files are used by start_server to start up the Siebel Server, and both point to the same Enterprise and Server names.
Therefore, the start_server script is starting the same Siebel Server twice.
The problem is that, when the second instance is started, the OSDF file is overwritten, and the information used by Siebel about the memory structures used by the first instance is lost. The second instance will conflict with the first one, and the processes end up failing.
Solution
In order to fix this behavior and guarantee you have a clean environment, please do the following:1. Shutdown the Siebel Server and run the following commands to clean all structures that may remain:
% stop_server ALL
% reset_server -e <enterprise_name> <server_name>
% siebclean -f $SIEBEL_ROOT/siebsrvr/admin/<enterprise_name>.<server_name>.shm -q
% cleansync -f $SIEBEL_ROOT/siebsrvr/sys/osdf.<enterprise_name>.<server_name> -d
% siebctl -r $SIEBEL_ROOT -S siebsrvr -i <enterprise_name>:<server_name> -k -q
% mwcleanup -silent 2>/dev/null
2. If using AIX, log into UNIX as root and run the following command:
% /usr/sbin/slibclean
3. Ensure no Siebel processes are running and check that there are no semaphores or shared memory segments allocated for the UNIX user who owns the Siebel Server installation:
% ps -ef | grep <username>
% ipcs -b
4. If there are any processes, shared memory segments or semaphores stuck in memory, they can be killed by using the following commands, respectively:
% kill -9 <process_ID>
% ipcrm -m <shared_memory_segment_ID>
% ipcrm -s <semaphore_ID>
5. Remove the following files if they still exist:
$SIEBEL_ROOT/siebsrvr/sys/osdf.<enterprise_name>.<server_name>
$SIEBEL_ROOT/siebsrvr/admin/<enterprise_name>.<server_name>.shm
6. Remove all svc* files or move all of them out from $SIEBEL_ROOT/siebsrvr/sys directory.
7. Recreate the correct svc* file:
For Siebel 8.0.x and earlier please use the comand:
% siebctl -S siebsrvr -i "<enterprise_name>:<server_name>" -a -g "-g <gateway_name> -e <enterprise_name> -s <server_name>"
For Siebel 8.1.x and later please use the comand:
% siebctl -S siebsrvr -i "<enterprise_name>:<server_name>" -a -g "-g <gateway_name> -e <enterprise_name> -s <server_name> -u <sadmin_user>" -e <sadmin_password>please note that the siebel server service needs user and password for gateway authentication starting with Siebel version 8.1
8. Restart the Siebel Server:
% start_server all
Customers should note that if there are still any processes, semaphores or shared memory segments stuck in memory, the machine will need to be rebooted before recreating the svc* file (between steps 6 and 7).
Customers should also note that recreating the svc* file (steps 6 and 7) is not necessary in all situations, and in most cases step 1 only is sufficient to guarantee a clean startup.
KEYWORDS: start_server all, failure to start Siebel on Unix server, OSDmprotect, err=1300220 sys=22 sys=0 SBL-GEN-00001 SBL-GEN-00002 SBL-GEN-00003 SBL-GEN-00004 SBL-GEN-00005 SBL-GEN-00006 SBL-GEN-00007 pthread_mutex_trylock AIX slibclean ipcrm
Applies to:
Siebel System Software - Version: 7.5.3 SIA [16157] and later [Release: V7 and later ]IBM AIX on POWER Systems (64-bit)
Product Release: V7 (Enterprise)
Version: 7.5.3 [16157] Cons Sec
Database: IBM DB2/UDB 7.2
Application Server OS: IBM AIX 5L 5.1
Database Server OS: IBM AIX 5L 5.1
This document was previously published as Siebel SR 38-1095528661.
Symptoms
SBL-GEN-00255, SBL-OSD-02006The object manager and several important server components are unavailable or partially online - this is holding up Production load!!!The following is appearing in the Enterprise server log:
ServerLog ProcessExit 1 2003-11-03 22:41:58 <NoCompName> 17427 SBL-GEN-00255 Process exited with error - Internal: Error during exec()
Cause
Enhancement request 12-IYEB5WSolution
Message 1
For the benefit of others:The following components are showing as UNAVAILABLE via srvrmgr and SIGBART in the enterprise log.
Business Integration Batch Manager
Business Integration Manager
Workflow Process Batch Manager
Workflow Process Manager
eConsumerSectorObjMgr_enu
Multiple occurrences of the following error was seen in the enterprise server log:
ServerLog ProcessExit 1 2003-11-03 22:41:58 <NoCompName> 17427 SBL-GEN-00255 Process exited with error - Internal: Error during exec()
As part of the troubleshooting:
Reset some parameter values for one of the components which is unavailable, stop and restart the Siebel server via the UNIX command line and see what error messages may be displayed after the start up.
Tried the following for eConsumerObjMgr_enu on server, siebdvsrvr
1. Use srvrmgr to set the following:
change param foreground=true for comp eConsumerObjMgr_enu server siebdvsrvr
change evtloglvl %=5 for comp eConsumerObjMgr_enu server siebdvsrvr
change param errorflags=-1 for server siebdvsrvr
change param traceflags=-1 for server siebdvsrvr
2. Stop the Siebel server.
3. As root login, run the AIX command slibclean
4. Start the Siebel server using the following command (do not use the start_server script and ensure siebenv.sh has already been invoked):
siebsrvr /g <gateway_host> /e <enterprise> /s <server>
eg. siebsrvr /g gateway_host_name /e siebdvent /s siebdvsrvr
(cont)
Message 2
(cont)5. Do you see any error messages on the screen following the siebsrvr startup?
You can control C out of that screen and then revert back to restarting the Siebel server via start_server script.
To revert back with the parameters you can reset:
change param foreground=FALSE for comp eConsumerObjMgr_enu server siebdvsrvr
change evtloglvl %=1 for comp eConsumerObjMgr_enu server siebdvsrvr
From Step 5 when running the siebsrvr command, the following message displayed:
Fatal Error: The file /siebapp/sieb75/siebsrvr/mw/system/registry.bin is not a valid registry file
Fatal Error: Could not load the HKLM hive (/siebapp/sieb75/siebsrvr/mw/system/registry.bin).
Resolution:
Checked the date and file size of $SIEBEL_ROOT/mw/system/registry.bin.*
registry.bin is 245760 dated 9/29
registry.bin.old is 819200 dated 9/25
The backup copy of registry.bin.old is the correct size. Shutdown the Gateway and Siebel Servers and copied in this backup copy to be the current registry.bin via:
mv $SIEBEL_ROOT/mw/system/registry.bin $SIEBEL_ROOT/mw/system/registry.bin.current
mv $SIEBEL_ROOT/mw/system/registry.bin.old $SIEBEL_ROOT/mw/system/registry.bin
Restarted the servers and all components are now coming online available.
Enhancement request 12-IYEB5W was logged to provide a more descriptive error message to identify a corrupt registry.bin
Applies to:
Siebel eCommunications Call Center - Version: 8.0 [20405] and later [Release: V8 and later ]Information in this document applies to any platform.
Symptoms
Customer tried to install and configure the swse with Oracle Application Server but always failed with below error:GenericLog GenericError 1 0000000249a74757:0 2009-02-27 16:13:33 Executing step: ApplyValuescfg
GenericLog GenericError 1 0000000249a74757:0 2009-02-27 16:13:33 Executing step: RestartWebServer
GenericLog GenericError 1 0000000249a74757:0 2009-02-27 16:13:33 Executing step: ShutdownApacheServer
GenericLog GenericError 1 0000000249a74757:0 2009-02-27 16:13:35 (ossystem.cpp (96) err=255 sys=0) SBL-GEN-00255: Internal: Error during exec()
GenericLog GenericError 1 0000000249a74757:0 2009-02-27 16:13:35 Step ShutdownApacheServer: failed to run program %%WebServerInstance%%%%OSDirSeparator%%bin%%OSDirSeparator%%apachectl with cmdline stop
GenericLog GenericError 1 0000000249a74757:0 2009-02-27 16:13:35 Failed during Execution, err: 255
Cause
There are 2 possible causes:
- Initially they installed the complete Oracle Application Server, not the one from companion CD.
- They then reinstalled using Oracle HTTP Server 10.1.3/Apache v2.x, from the Oracle Application Server 10.1.3 companion CD, but the "Web Server Instance Location" was not entered correctly.
The "Web Server Instance Location" should be the directory where the Oracle Application Server is installed.
The customer supplied it with:
Web Server Instance Location : /data1/siebel_sweapp/https-one5all
while the Oracle Application Server was installed in /data1/oracle/app10gR3/ohs (httpd.conf exists in conf directory of the Web Server, /data1/oracle/app10gR3/ohs/conf/httpd.conf)
After correcting the above parameter, customer was able to proceed and complete the swse installation
Oracle Application Server 10.1.3 companion CD can be downloaded from:
http://www.oracle.com/technology/software/products/ias/htdocs/101310.html
Solution
It was suggested to customer to rerun the configuration with the correct location.
The ”Web Server Instance Location” should be: /data1/oracle/app10gR3/ohs.
The installation/configuration of the swse was then completed successfully.
Applies to:
Siebel System Software - Version: 8.0.0.8[20430] and later [Release: V8 and later ]Information in this document applies to any platform.
Symptoms
Customer had installed 8.0.0.9 patchset to production enviroment.
Previously customer had Siebel 8.0.0.8 version and was installed on the
IBM AIX 5.3 (64-bit). After installation of 8.0.0.9 patchset siebel
server was not coming up.
Error:
-------
Siebel server Logs:
--------------------
GenericLog GenericError 1 000069d14e0b30c0:0 2011-06-29 18:15:09 (sccnmsys.cpp (1305) err=2883609 sys=0) SBL-SCC-00025: No value found in the Gateway. Default value in the repository is used.
GenericLog GenericError 1 000069e04e0b30c0:0 2011-06-29 18:15:09 (sccnmsys.cpp (1305) err=2883609 sys=0) SBL-SCC-00025: No value found in the Gateway. Default value in the repository is used.
Enterprise Logs:
------------------
GenericLog GenericError 1 000076ee4deb1046:0 2011-06-05 13:31:39 (scirkey.cpp (1142) err=1310772 sys=0) SBL-SVR-00052: Internal: Invalid proc handle
ServerLog ProcessExit 1 000076ee4deb1046:0 2011-06-05 13:31:39 SRBroker 594106 SBL-GEN-00255 Process 594106 exited with error - Internal: Error during exec()
Changes
Applies patchset 8.0.0.9 on top of 8.0.0.8.Cause
After Verification it was found installable library files from previous installation 8.0.0.8 got corrupted and hence installation of 8.0.0.9 is not getting through.Further verification showed, release timestamp for some library files from 'siebsrvr/lib' were not set to patch release date but patch install date.
In Ideal case the timestamp should correspond to patch release date, thing is that the release time stamp is only set at the very end. If something breaks before then files remain on disk with install date.
In case of customers environment, some library files has timestamp of April2011 which is patch installation date, in correct scenario it should have been patch 8.0.0.8 release date (Oct2009).
Which shows incorrect deployment of 8.0.0.8 patchset.
Solution
Below are some prominent steps for re-installation of gateway and Siebel Servers:Reinstalling the Gateway Server:
-----------------------------------------------------------
1) Stop the Gateway service
2) Uninstall the Gateway server.
3) Delete the Gateway server directory structure.
4) Reboot the machine.
5) Install the Gateway server.
Reinstalling the Siebel Server:
-----------------------------------------------------------
1) Make backups of all the .cfg files present in \siebsrvr\bin and the siebns.dat file present in \gtwysrvr\ADMIN.
2) Make backup of the .srf file present in \siebsrvr\OBJECTS.
3) Make backups of the following folders present in the \siebsrvr directory to take care of any customizations:
IDCENTRIC, MSGTEMPL, REPORTS, WEBTEMPL.
4) I am not sure if you have remote users in your environment, If there are remote users working off this application server, backup the following folders to avoid reextracting them:
DOCKING, DBTEMPL
5) Stop the Siebel server service
6) Uninstall the Siebel server
7) Delete the Siebel server directory structure.
8) Remove Siebel Server from the Gateway by removing configuration from wizard.
9) Reboot the machine.
10) Start the Gateway server service if not running already.
11) Install the Siebel server using same installation parameters as before.
12) Copy the backed up folders, .cfg files and the .srf file to the corresponding directory where it belongs.
13) Stop and restart the Gateway and Siebel server services.
14) From the Server Administration screen enable the needed component groups.
There is a note out there on MOS with the correct steps. It refers to Windows but same steps apply to AIX as well:
Reinstalling the Siebel Servers (Doc ID 476317.1)
Note: This solution was specific to one environment/scenario, there is possibility that same solution may not work for other environments.
References
NOTE:476145.1 - Failure When Installing Siebel Patches on the AIX Operating SystemNOTE:971849.1 - Error "SBL-SVR-01031: Invalid Parameter" is generated in the SRProc log files
Applies to:
Siebel Server Extns for Solaris Operating Envrmnt - Version: 8.1.1.5 [21229] and later [Release: V8 and later ]Information in this document applies to any platform.
Symptoms
SWSE configuration fails with error on step SetMimeTypes
2011-08-19 14:22:34 Looking for executable
/usr3/siebiut/siebel/siebsrvr/bin//usr3/siebiut/siebel/siebsrvr/install_script/install/UpdateMimeTypes
2011-08-19
14:22:34
/usr3/siebiut/siebel/siebsrvr/bin//usr3/siebiut/siebel/siebsrvr/install_script/install/UpdateMimeTypes
not found
2011-08-19 14:22:36 (ossystem.cpp (96) err=255 sys=0) SBL-GEN-00255: Internal: Error during exec()
2011-08-19 14:22:36 OSSystem returned 255
2011-08-19
14:22:36 Step SetMimeTypes: failed to run program
%%SiebelRoot%%%%OSDirSeparator%%install_script%%OSDirSeparator%%install%%OSDirSeparator%%UpdateMimeTypes
with cmdline %%SiebelRoot%% %%WebServerInstance%%
2011-08-19 14:22:36 Failed during Execution, err: 255
2011-08-19 14:39:31 Exiting configuration. Cause
SWSERoot was incorrectly defined prior to launching the configuration wizard
Verbose log file contains the following entries:
Value key SiebelRoot resolves to /usr3/siebiut/siebel/siebsrvr
Value key SWSERoot resolves to /usr3/siebiut/siebel/siebsrvr
Solution
Verify the directory containing your SWEApp installation and then source the correct environment variables using cfgenv.sh from it, before running the configuration wizard once more.
No comments:
Post a Comment