Troubleshooting Concurrent AMS Master and Replica Delivery Causing Subscriber Busy Errors

Overview

In a healthy AMS cluster, each stored message has a master delivery path and a replica copy on another AMS node. The replica is intended to take over delivery only when the master is unavailable.

This article applies to a fault condition where the master is available again, but a replica holder continues delivering the same stored messages. The master and replica can then attempt delivery to the same recipient at nearly the same time. These duplicate MT delivery attempts can result in subscriberBusyForMtSms responses and an increase in the RTR MT Subscriber Busy counters.

Symptoms

• subscriber_busy_ctr increases in the rtr_traffic_mt statistics table.
• Message traces or logs show repeated or near-simultaneous MT delivery attempts for the same recipient and message.
• amsCntDeliveredReplica.0 continues to increase on one or more AMS nodes after the cluster has recovered.
• queueCntDeliveredReplica.<queue_index> continues to increase for an affected application queue.
• The master AMS node is shown as available in tp_qcli -s -n, but another node is still delivering messages through the replica path.

Why this causes Subscriber Busy errors

When both the master and replica deliver the same stored message, the receiving MSC/SGSN can receive overlapping MT-ForwardSM attempts for the same subscriber. One delivery attempt may proceed while another receives subscriberBusyForMtSms. This is a delivery-path collision, not necessarily a subscriber-capacity, application-throughput, or message-queue-capacity problem.

Step 1: Capture the current cluster state

Run the following commands on the affected replica-delivery candidate and on at least one healthy node. Capture the output before restarting any AMS process.

date
tp_qcli -s -n

snmpget -v 1 -m ALL -c private localhost:11661 amsCntDeliveredReplica.0
snmpget -v 1 -m ALL -c private localhost:11661 queueCntDeliveredReplica.<queue_index>

snmpwalk -v 1 -m ALL -c private localhost:11661 failoverControlTable

grep -i "Duplicate AMS Id" /var/log/messages | tail -20
grep "ams_id from database" /var/log/messages | tail -10

Repeat the two snmpget commands after approximately five minutes. A growing replica-delivery counter identifies a node that is actively delivering replicated messages.

Step 2: Check for duplicate AMS IDs first

A duplicate AMS ID can destabilize cluster registration, failover, and replica behavior. Before restarting a replica holder, confirm that every active AMS node has a unique ID and that the membership view is stable.

1. Run tp_qcli -s -n from two AMS nodes and compare the results.
2. Use the ams_id from database log entries to confirm each node's persisted AMS ID.
3. Check all affected nodes for Duplicate AMS Id log entries.

If duplicate IDs are present, correct the duplicate-ID condition before proceeding. Follow How to Change the AMS ID on an AMS Node to stop the affected AMS process, assign an unused unique ID, restart AMS, and validate availability.

Important: Do not use tp_ams --amsid=... on a node whose AMS ID is already confirmed as correct and unique.

Step 3: Identify persistent replica delivery after master recovery

After all AMS IDs are unique and the master is available, identify the node or nodes whose replica-delivery counters continue increasing.

This condition is present when all of the following are true:

• The relevant master appears as available in tp_qcli -s -n.
• The AMS IDs are unique and stable across the cluster.
• amsCntDeliveredReplica.0 or queueCntDeliveredReplica.<queue_index> continues increasing on another node across two time-separated samples.

The affected node is still acting as an active replica holder. This continued delivery must stop before the Subscriber Busy rate can return to its normal baseline.

Step 4: Restart the affected replica holder

When the replica holder (not the master) AMS ID is already correct and unique, perform a plain tp_ams restart on the replica holder node. Restart only one AMS node at a time.

sudo su - textpass

tp_stop --tp_ams
tp_status

tp_start --tp_ams
tp_status

grep "ams_id from database" /var/log/messages | tail -5

Verify that the node restarted with its expected existing AMS ID. Do not use --amsid unless an actual duplicate-ID condition was confirmed in Step 2.

Do not manually alter the AMS failover-control files or perform broad multi-node restarts while this condition is being investigated.

Step 5: Verify that replica delivery has stopped

After the restarted AMS node is available, wait approximately five minutes and collect the following counters twice.

date
snmpget -v 1 -m ALL -c private localhost:11661 amsCntDeliveredReplica.0
snmpget -v 1 -m ALL -c private localhost:11661 queueCntDeliveredReplica.<queue_index>

Repeat the same checks on all nodes that previously showed replica delivery.

Recovery is confirmed when:

• All expected AMS IDs are unique and available in tp_qcli -s -n.
• No new Duplicate AMS Id messages are written after the repair.
• Replica-delivery counters are flat across consecutive samples after the cluster has synchronized.
• The affected application's replica-delivery counter is no longer increasing.
• The MT Subscriber Busy rate in rtr_traffic_mt returns toward the environment's established baseline.

Important operational guidance

• Restart AMS nodes one at a time and verify recovery before restarting another node.
• Do not use a queue-distribution or load_percent view as the sole reason to restart an AMS node. Queue allocation distribution does not by itself prove a delivery backlog or active replica delivery problem.
• A non-zero replica-delivery counter can occur temporarily during a normal failover. It becomes actionable when it continues increasing after the master is available and the cluster has had time to synchronize.
• If replica-delivery counters continue increasing after the relevant AMS process has been restarted, stop further restarts and collect the commands from Step 1 for investigation.

Prevention and monitoring

• After an AMS rebuild, migration, or first startup of a newly added node, confirm the AMS ID and cluster membership before returning the node to service.
• Monitor amsCntDeliveredReplica and application-level queueCntDeliveredReplica deltas. Sustained growth after the master is available indicates a potential concurrent-delivery condition.
• Monitor subscriber_busy_ctr / total_ctr in rtr_traffic_mt as the customer-facing outcome metric.

Related article

• How to Change the AMS ID on an AMS Node