Troubleshooting Intermittent SMS Delivery Delays with High STV “Storage Duration” (AMS Rate Limiting)

How to Recognize the Issue

Symptoms

• STV “Storage Duration” histogram shifts from mostly sub-5 seconds storage to 1–4 minutes, often with a long tail (>10 minutes) during busy hours.
• More severe cases may reach a multi-minute gap between the time a message is stored and the time it is delivered (e.g., OTP and MT delivery delays).

Root Cause

This delay pattern can occur when the AMS reaches its configured delivery-attempt ceiling. When instantaneous demand exceeds the configured limit, the AMS buffers messages (which is the expected behavior), and that buffering is reflected as increased STV “Storage Duration”.

• Runtime ceiling (what the running system is using):
  • amsPropMaxDeliveryRate.0 visible in tp_walkall
  • The default value is 150 messages/second
• Configured value (what you set in configuration):
  • amsmaxdeliveryrate="<value>" in the 

    /usr/TextPass/etc/common_config.txt

Important nuances

• Even if average delivery rate is below the ceiling, short bursts can still clip at the limit and cause short-lived Storage Duration spikes.  Generally this is only observed with short bursts, and is not a constant issue.
• High failed delivery attempts (retries) consume delivery-attempt capacity and can amplify delays by increasing pressure on AMS.

Troubleshooting Methodology

1) Confirm whether AMS is clipping at the rate limiter

On each SMSC, check the runtime ceiling:

tp_walkall | grep amsPropMaxDeliveryRate

If Storage Duration spikes coincide with periods where delivery attempts are pinned near this value, AMS rate limiting is a strong contributor.

Optional: compute delivery-attempt rate from two samples

Take two tp_walkall samples separated by a known interval (Δt seconds):

• Method A (Delivered + Failed):
  • Rate ≈ ((ΔamsCntDeliveredMaster.0 + ΔamsCntFailedMaster.0) / Δt)
• Method B (MXIP counter):
  • Rate ≈ (ΔmxipTagCntRxDeliverSmRequest.0 / Δt)

We have attached a script: amsDeliveryRate_DelayCollectorCalculator.zip that you can run on the AMS nodes, which will collect and calculate this data for you automatically.  Instructions on how to use the script are in the bundled README file.  It has very low overhead, and should have no impact on the performance of the server.  If you believe you are experiencing this issue, please run this script during the busiest traffic of the day, then stop it and collect and share the results with Support for verification.

The data collected may look like this when plotted:

A purple horizontal line was added to indicate the current amsPropMaxDeliveryRate value.  In this case, there were two instances where the Average AMS Delivery Rate was touching the celling, which means that there were instances of clipping (as the average indicates that both higher and lower values of the rate exist.)

2) Ensure configuration changes are applied at runtime (common pitfall)

It is possible for configuration files to show a new value while the running system still uses the old value. Always verify after changes:

tp_walkall | grep amsPropMaxDeliveryRate

Solution (Step-by-Step)

Step 1 — Back up and open the common configuration file

Log in to the OAM/MGR node as the textpass user. The standard common semi-static configuration file is:

/usr/TextPass/etc/common_config.txt

Create a backup before making any changes:

cd /usr/TextPass/etc
cp -p common_config.txt common_config.txt.$(date +%Y%m%d%H%M%S)

Then open the file for editing:

vi /usr/TextPass/etc/common_config.txt

This is the system-wide common configuration file used for common semi-static settings.

Step 2 — Update the AMS delivery rate limit configuration

Locate the amsmaxdeliveryrate setting in the file and change it to the target value.  We recommend increasing gradually (by 15-30 at a time) then testing to see if the issue is resolved.

Example:

amsmaxdeliveryrate="270"

Example incremental tuning pattern (adjust to your environment): 250 → 260 → 270 → 285.

Step 3 — Validate the common configuration (safe, no downtime)

Validate the updated file before applying the change:

tp_config --validatecommonconfig usr/TextPass/etc/common_config.txt

If the command returns no output, the file validated successfully. The validation modes do not restart services.

Step 4 — Apply the change (TextPass restart may be required)

After the file is updated and validated (check the last modified time of the common_config.txt file on each target node), restart TextPass on each affected traffic element node one node at a time during a maintenance window.

Example:

tp_stop --textpass 
tp_start --textpass 
tp_status

Wait until textpass returns to operating before moving to the next node.

Step 5 — Verify the runtime value after restart

On each node:

tp_walkall | grep amsPropMaxDeliveryRate

Confirm it reflects the new target (for example, 270). If it still shows the old value, the change did not apply to the running system.

Step 6 — Monitor and tune incrementally

• Increase in small steps (commonly +15 or +30), then observe over the next high traffic period:
  • STV “Storage Duration”
  • Customer-reported latency
  • System stability (CPU, disk I/O, alarms)
• Revert to the last stable value if raising the rate introduces instability.
• Plan for ongoing tuning/capacity management if intermittent recurrences continue during later peak periods.

Validation (Confirming Improvement)

• After applying the change and restarting TextPass:
  • tp_walkall shows the updated amsPropMaxDeliveryRate.0.
• During the next busy window:
  • STV “Storage Duration” no longer exhibits sustained minute-level spikes, or spikes are reduced.
  • Collected metrics/counters show delivery attempts are no longer clipped flat at or hitting the configured ceiling for extended periods.
• If delays recur, correlate timestamps between STV spikes and the collection window to ensure the captured data overlaps the actual incident period (including any system-time vs local-time offset).