MO Messages Not Delivered - Troubleshooting Article

Overview

This article explains how to troubleshoot Mobile Originating Messages not reaching your routers. Some of the common symptoms related to this topic are:

• Approximately half of all MO messages reaching a pair of RTRs set up in a redundant failover configuration are not being delivered with no displayed error.
• You see two MO Messages being received for each, one showing trusted and the other as untrusted.
• Within the Syslog, one of the paired RTRs may show the following errors: 
  • “TNC: unexpected SCTP shutdown indication.”
  • “App: '<your-application>' - Outside 'deliver_sm' response timeout."
    
  • “Unexpected response from App '<your-application>' (trn=77382)”
    

Workflow

 

Instructions

1. Reproduce and Collect Details.

2. Investigate the cause of the App response delay.

3. Run SMPP Trace and Check SCTP connectivity.

4. Check the network Discovery Status.
5. Investigate potential faulty hardware.

 

Reproduce and Collect Details

As an initial step, verify that the behavior is reproducible and not the result of a temporary networking hiccup. If the behavior can be reproduced, collect the following details to assist with getting a better look:

1. As the textpass user, execute the following command and collect the output:
   

   # tp_qcli -s -o=<originator_address>
   

   1. Note: Use the originator address used in your reproduction.
2. Execute the following command on each traffic element:
   

   # tp_walkall

   1. Note: This can help verify MOR and ATOR rule conditions, application settings, storage, and retry schemes involved that might impact the behavior seen.
3. syslog (var/log/messages)

Using the details collected, verify that the SNMP configuration settings you have in place are valid. Similarly, confirm that no explicit errors or alarms involving the affected application. In particular, verify whether or not you see any signs of Application Response Timeouts, as seen below:

MMM dd hh:mm:ss RTR tp_hub: App: 'your_application' - Outside 'deliver_sm' response timeout
MMM dd hh:mm:ss RTR tp_hub: Unexpected response from App 'your_application' (trn=77382)
MMM dd hh:mm:ss RTR tp_hub: App: 'your_application' - Outside 'deliver_sm' response timeout [8 times]
MMM dd hh:mm:ss RTR tp_hub: Unexpected response from App 'your_application' (trn=77403) [8 times]

 

Investigate the cause of the App response delay

If you identify any of the Application Timeouts without the reproduction attempt, this may indicate that the application is experiencing trouble, and may be responsible for the losses MO Messages. Please work with your internal team to confirm the exact cause of the behavior before proceeding.

 

Run SMPP Trace and Check SCTP connectivity

Using your preferred trace capture software (ex. Wireshark) perform a capture of all the SMPP traffic between the affected HUBs. This can help verify the successful SMPP Binds, as well as confirm that the Deliver_SM responses are being sent/received as expected. 

Note: If you have trouble routing to each HUB due to the failover, you can either restart the HUBs one at a time to force the application to connect to the next HUB or alternatively, specify the IP of the Hubs directly in your application.

After your trace capture, verify the SCTP-level connectivity between the affected routers by running netstat|grep sctp from each server. 

 

Check the network Discovery Status

If you find signs that there is a connectivity issue between the affected routers, you should review the network discovery definitions within the common_config.txt file on each router. To list the Network Discovery Status on the routers, run tp_walk networkdiscoverytable on each. 

You can check the status of the Vlans from the server, the switch, and test the IP connectivity between each router. If you continue to see issues with the connectivity:

1. As a first step, execute tp_stop and then tp_start.
2. If the issue persists, you can instead restart the network services or reboot each server.

 

Investigate potential faulty hardware

If after verifying all of the above details you continue to see issues with MO Message delivery, this may indicate that there is an underlying hardware issue, as noted within MO not being delivered after reaching RTR. If you made any recent hardware replacements, verify that all of the connectors are in working order. 

Similarly, if you have not already, verify with your IT department that all of the LAN cables and switches involved in the network conversation are working correctly. Replace any faulty hardware, as needed.