Performing a Basic Cobalt Server Health Check

Overview

At times, you might need to verify the system's health (usually) after a hardware replacement. 

This article provides the steps to perform a basic Cobalt system health check and verify if the Cobalt server is operating correctly.

Process

Perform the following steps as a smsadm user:

1. Verify that the SMSC status is Online:
   
   

   sms_status

2. Check the start time for all the processes and compare them with the SMSC start time:
   
   

   ps -ef | grep smsadm

3. Verify that all major SMPP clients are in the Connected state (SMPP Client status):
   
   

   display-scthruput:;

4. Verify the number of messages, status, and violations in the current hour (SMPP Server status):
   
   

   display-ssthruput:;

5. Ensure that there are no violations:
   
   

    display-sviolation:;

6. Verify the license utilization in SMServer:
   
   

   sm_lmd -l

7. Check the status for all processes:
   
   

   ebs_ps

8. Verify all the error codes:
   
   You may need to execute this command several times.
   
   

    sms_operator->query_db

9. Check for the important traffic counters from the recent traffic files:
   
   For example, counters like Submit SM-Rcvd, Delivered SM, and SS7-MO-Rcvd indicate that the messages are coming to the SMSC, and the SMSC is delivering the messages.
   
   

   egrep "Time_Stamp|Submit SM-Rcvd|Delivered SM|SS7-MO-Rcvd" TRF.<timestamp>

10. Observe and confirm that there is no core file:
    
    

    find $EBSHOME/* -name "*core.*"

    • If you observe a core file, follow the next two steps as a root user.
    1. Note down the timestamp for the core file by running the following command:
       
       

       ls -l core.*

    2. Note down the process that generated the core file by running the following command:
       
       

       file core

11. Ensure that there is no error or related log:
    
    

    • grep -i "exited (FAIL)" $EBSHOME/SMS/$VERSION/RUN/mlog/MLog*

    • grep -i "exited (KILL)" $EBSHOME/SMS/$VERSION/RUN/mlog/MLog*

12. Ensure that there is no error or related logs.
    (The log in the below command indicates that some process went to the blocked state.)
    
    

    grep -i "IPC message failed" $EBSHOME/SMS/$VERSION/RUN/mlog/MLog*

13. Ensure that there is no error or related logs:
    (The log in the below command indicates that the shared memory is exhausted.)
    
    

    grep -i "shared memory capacity exceeded" $EBSHOME/SMS/$VERSION/RUN/mlog/MLog*

14. Ensure that there are no errors:
    
    

    grep -i KILL $EBSHOME/SMS/$VERSION/RUN/mlog/MLog* | wc -l

15. Observe the connection status of the SMSC.
    
    

    netstat -na | grep 6200