SMSC (TextPass RTR/AMS/HUB) Cluster: Growing AMS Queue Backlog, "load_percent" Near 100%, and SMS Traffic Loss

Solution

Recognizing the issue and reading the queue statistics correctly

Use these symptoms to confirm this article applies:

• A large, growing number of allocated recipient queues per AMS node (for example ~300,000 per node) and a per-node load_percent statistic reading ~100%.
• SMS traffic loss / undelivered messages, often on mass-campaign or notification streams.
• /var/log/messages and the journal are nearly empty or retain only a few minutes of history; monitoring scripts that grep the syslog (such as an amsStoreFull cron) never fire.
• The following appears in syslog — this is the fingerprint of the logging defect:

  imjournal: NNNNN messages lost due to rate-limiting (20000 allowed within 600 seconds)
  systemd-journald[...]: Suppressed NNNN messages from session-NN.scope

The queue statistics are usually misread. The amsCntTotalQueuesAllocated / queueTable counters are recipient buffers (distinct recipients with messages waiting), not a fill level against a limit. A load_percent column computed as each node's total divided by the cluster average simply shows that load is evenly distributed — ~100% on every node means balanced, not full. Likewise, a roughly 50/50 master/replica split shows replication is balanced. Neither figure indicates the store is full. Do not alarm on the allocated-queue count alone.

Part 1 — Confirm whether the store is actually the constraint (read-only)

On an AMS-hosting node, as the textpass user:

tp_status                     # every process 'operating'?
tp_qcli -s -n                 # all nodes 'available'?
tp_walk --tp_ams amsCounters
tp_walk --tp_ams amsProperties

Check the following in the counter output:

• amsCntTotalStored vs amsPropMaxNumberOfMessages — a healthy, non-saturated store sits well below the maximum (for example ~1.19M of 6,000,000, roughly 20%, peaking near 28%).
• All amsCntRejected* capacity counters — amsCntRejectedMaxTotalMessagesExceeded, amsCntRejectedMaxQueueSizeExceeded, amsCntRejectedMaxRecipientMessagesExceeded, database-error and message-store-disabled rejections. If these are zero, the store has never refused a message and storage is not the problem. (Ignore amsCntRejectedNumberOfAttempts, which increments normally when a message arrives already past its attempt limit.)

If storage is not saturated, the loss is a delivery-completion / expiry problem, not a capacity problem. Do not increase amsPropMaxNumberOfMessages or queueSizeMaximum — more storage only holds messages longer before they expire.

Tip: For definitions of every counter referenced here, see the ZMM_AMS_R01.13.01_Operator_Manual_22.12_rev_a.pdf (AMS Counters Reference).

Part 2 — Restore logging (workaround for the debug-trace defect)

Affected versions: AMS R01.13.01.15 and later / HUB R04.12.05.14 and later. The two debug traces (SMPP_parse_msg: src trn=N and The more to send flag is N) are compiled unconditionally with no runtime switch. The fix is in AMS R01.13.01.24 and HUB R04.12.05.18, which are included in Lithium release v24.03.24.19. To get this release, contact your account manager. Until your nodes run these versions, apply the OS-level workaround below.

Note: Lower-traffic sites may not show symptoms even on the affected versions — a site running well below ~33 msg/s stays under the rate limiter by volume alone, not because of different configuration. Both high- and low-volume sites run the same OS defaults.

Back up each file before editing. Apply on one node first, verify, then roll out one node at a time.

1. In /etc/systemd/journald.conf:

   MaxLevelStore=info
   RateLimitBurst=0
   SystemMaxUse=2G

   MaxLevelStore=info discards debug-severity records at ingest, before the rate limiter — this removes ~94.5% of the volume and is the essential step. Debug is never written to /var/log/messages on a healthy system, so no operational information is lost.
2. Enable a persistent journal (survives reboot, extends retention):

   mkdir -p /var/log/journal
   systemd-tmpfiles --create --prefix /var/log/journal

3. In /etc/rsyslog.conf, disable the imjournal limiter on the module line:

   module(load="imjournal" StateFile="imjournal.state" Ratelimit.Interval="0")

   (On older RHEL-7-style configs the equivalent legacy directive is $imjournalRateLimitInterval 0.)
4. Apply and validate:

   systemctl restart systemd-journald
   rsyslogd -N1        # must end "End of config validation run" with no errors
   systemctl restart rsyslog

   If rsyslogd -N1 reports an error, do not restart rsyslog — restore the backup first.

Verify:

journalctl --since "-2min" | grep -cE "SMPP_parse_msg|more to send flag"   # expect 0
grep "lost due to rate-limiting" /var/log/messages | tail -3                # no entries after the restart
grep -c "tp_ams" /var/log/messages                                          # usually more than 0 on a node with traffic. 0 alone is not a failure (see Part 5)
journalctl --disk-usage                                                     # within the 2G cap
df -h /var/log

Warning: Restarting journald while first enabling the persistent journal can briefly stall the syslog socket. On a heavily-logging node this can cause missed SCTP heartbeats and short (~10–20 s) inter-node link drops, producing sctpCongestionStarted traps and MT timeouts. To avoid this, run journalctl --rotate before enabling persistence, or schedule the work in a maintenance window.

Tip: Keep lost due to rate-limiting as a permanent first-order alarm. If it ever reappears, the log pipeline is flooded again and every other alarm is unreliable until it is fixed.

Part 3 — Why an AMS restart is usually the wrong action, and the one test that justifies it

An AMS restart is warranted in exactly one situation: a node delivering from stale replica state. Test this read-only on all nodes (replace the device IPs with your AMS node addresses):

# T1
for i in <node-ip-list>; do tp_walk --tp_ams --device=$i amsCounters > /tmp/amscnt_$i._T1.txt; done
# wait ~10 minutes, then T2
for i in <node-ip-list>; do tp_walk --tp_ams --device=$i amsCounters > /tmp/amscnt_$i._T2.txt; done
# compare amsCntDeliveredReplica and amsCntFailedReplica between T1 and T2

• All deltas zero → no node is delivering from replica state → a restart will not help. Do not restart.
• Non-zero on specific nodes → those nodes are in the stale-replica condition → a targeted tp_ams restart is justified, one node at a time, waiting for each node to show available in tp_qcli -s -n before touching the next.

Warning: Never restart more than one AMS at a time. Starting a second AMS before the first has re-established its store can create duplicate AMS identifiers. A restart also costs a ~5-minute delivery-rate ramp on that node (observed ramping from about 40 up to 1,050 msg/s) plus a ~6–7% MT-timeout spike, with no benefit unless the replica test is positive.

For the full procedure to confirm, stop, and safely restart a replica holder — including how to check for duplicate AMS IDs and verify recovery — see Troubleshooting Concurrent AMS Master and Replica Delivery Causing Subscriber Busy Errors.

Part 4 — Root cause of the loss and corrective/preventive tuning

The loss mechanism: intermittent node isolation → the AMS replica mechanism takes over the isolated node's stored messages → parallel/duplicate MT delivery → the visited MSC/VLR answers duplicates with busy/timeout → failed deliveries are re-stored and retried → the backlog grows → messages are lost when their delivery scheme runs out of attempts (typically 2–6 hours), well before their 72-hour validity.

Structurally, cluster delivery throughput is the ceiling (for example ~5,500 successful MT/s, roughly 64–85% of the per-node amsPropMaxDeliveryRate of 1,050/s at peak), while campaign admission was effectively unthrottled. A large push therefore builds an invisible backlog that later expires. Recommended actions, in priority order:

1. Activate the AMS store-usage thresholds so a backlog is alarmed early. If amsPropMessageStoreThreshold1/2 are 0 (inactive) and the active thresholds sit at 80/90/100%, they are unreachable at normal ~15% occupancy. Set an early-warning threshold (~20%) and an "act" threshold (~25%), routed to your SNMP trap receivers (which are unaffected by the syslog loss). Confirm whether the change is dynamic or requires a per-node AMS restart.
2. Align admission with delivery capacity. Review the AO routing-rule throughput limits on the mass-campaign streams — if they are set far above the deliverable rate they never trip, and surplus simply queues and expires. Bringing them nearer reality converts an invisible backlog into immediate, visible back-pressure (SMPP 0x58 "Throughput exceeded"), provided the submitting application retries on that error. Best of all, pace/schedule campaigns at source so large pushes don't coincide or hit the busy hour.
3. Fix short retry schemes. On 3-attempt schemes, Maximum Attempts counts the first delivery attempt, so only two retries actually run and the third configured interval is never used — causing premature expiry (short schemes can expire 21–32% of their traffic). Raising Maximum Attempts from 3 to 4 in MGR ▸ Storage ▸ Schemes makes the third interval reachable; consider longer spans for mass-campaign queues, since a recipient unreachable now is often reachable hours later. Do this together with admission pacing, not instead of it — more retries consume more of the scarce delivery-attempt budget. For a deeper explanation of this behavior, see NewNet TextPass: Delivery (Retry) Scheme Skips Configured Intervals and Ends in "validityPeriodExpired".
4. Do not increase amsPropMaxNumberOfMessages, queueSizeMaximum, amsPropMaxIngress, or validity — none of these was the constraint. amsPropMaxDeliveryRate may be stepped up conditionally, but only after Network MT-Traffic statistics confirm SS7/MT headroom, with a verification window watching MT timeout %. Every retry is an SRI-SM plus MT-FSM over the same SS7 path as live traffic, and a high absent-subscriber rate (~21% observed) means many retries cannot succeed regardless of rate.
5. Application-side losses to close with the app owners: submissions silently discarded on Rx window-size violation when hubRejectWindowViolations=false; a session not draining its TCP socket; applications not acknowledging deliver_sm; throughput-exceeded rejections; and refused binds from unknown system IDs.

Part 5 — Lightweight 5-minute health monitoring (read-only)

Run a read-only health check as textpass from cron every 5 minutes on each node. Alarm on any of the following:

• Any process not operating.
• Fewer than the expected number of nodes available.
• tp_walk --tp_ams amsCounters returning nothing — the AMS agent is not answering; this is the hang to catch.
• Submissions rising while deliveries are flat.
• Any capacity amsCntRejected* delta.
• Stored total rising for several consecutive intervals, or stored above an absolute level (~20% of maximum).

Common pitfalls when writing this check:

• Do not grep -c 'available' from tp_qcli output — the header line "List of available AMS nodes" also contains the word and inflates the count. Match the status field on numeric node rows instead.
• Do not judge success by grep -c tp_ams /var/log/messages alone — it can legitimately be 0.
• Fix any amsstorefull.sh-style cron that greps only the current minute/decade of the log: because cron runs a second into the interval it sees only ~1 s of log. Grep both the current and previous window.

Note: Post-workaround, a draining backlog is confirmed by the stored count falling steadily (for example ~16,000–18,000 per node per 10-minute window) while /var/log/messages grows normally and the journal stays within its 2G cap.