Fixing an AMS Node Showing <code>NOT AVAILABLE</code> in <code>tp_qcli</code> Due to Duplicate AMS ID (and Understanding Related <code>amsStoreFull</code> / SMPP 0x0014)

Solution: Assign a Unique AMS ID

Error messages / indicators

• In /var/log/messages on the affected RTR/AMS node:
  • Duplicate AMS Id (1) detected. Two nodes have the same id!
  • AMS read 1 as ams_id from database for AMS identifier
• In tp_qcli -s -n output (from another node or any node able to query QCLI):
  • An AMS entry shows NOT AVAILABLE (example: 6 NOT AVAILABLE)
• Optional related symptoms during broader store saturation:
  • Sent SMPP error:0x0014 - Message Queue Full
  • amsStoreFull (0x03000003)

Root cause

After an AMS rebuild or DB synchronization, a node can start with an AMS ID that is already used by another node. Because AMS IDs must be unique within the cluster, the duplicate ID prevents correct registration/communication (often visible as QCLI link up/down behavior) and can result in the node showing NOT AVAILABLE in tp_qcli.

Preconditions / impact

• Run commands as the textpass user (or use sudo su - textpass).
• Stopping and restarting AMS on a node may cause temporary processing impact for traffic handled by that node. Use an appropriate maintenance window if needed.

Step 1 — Confirm the duplicate AMS ID in logs

On the affected node:

sudo su - textpass
grep "Duplicate AMS Id" /var/log/messages | tail -20
grep "AMS read" /var/log/messages | tail -20

Look specifically for:

• Duplicate AMS Id (X) detected. Two nodes have the same id!
• AMS read X as ams_id from database for AMS identifier

Step 2 — Identify an unused AMS ID

From any healthy node (or the same node if it can run tp_qcli), list AMS node availability:

sudo su - textpass
tp_qcli -s -n

Choose a unique AMS ID that is not currently in use in the cluster (for example, if IDs 0–5 exist and 6 is missing, use 6):

• <unique_ams_id>

Step 3 — Stop AMS on the affected node

sudo su - textpass
tp_stop --tp_ams

Step 4 — Set the new AMS ID and restart AMS

Use the following sequence:

sudo su - textpass
tp_ams --amsid=<unique_ams_id> --makemyday=yes
tp_start --tp_ams
sleep 30

Step 5 — Validate

1. Confirm the new AMS ID was applied:

   grep "AMS read" /var/log/messages | tail -10

   Expected pattern:

   • AMS read <unique_ams_id> as ams_id from database for AMS identifier

2. Confirm the node is now available in QCLI:

   sudo su - textpass
   tp_qcli -s -n

   Expected result: the previously missing/NOT AVAILABLE entry now appears as available.

Notes: “Message Queue Full” / amsStoreFull Store Saturation

During the incident, AMS store saturation behavior was observed and correlated with SMPP submission failures:

• SNMP trap evidence indicating store growth to the maximum threshold:
  • TRAP6…messageStoreUsageGrowing … amsCntTotalStored.0 = … 6000000
• Explicit submit_sm failures tied to store-full behavior:
  • submit_sm … caused 'amsStoreFull' (0x03000003)
  • Sent SMPP error 0x0014: Message Queue Full

This store-full behavior is expected behavior when AMS reaches its configured maximum stored message threshold (observed at 6,000,000 stored messages). If delivery throughput is briefly capped, the following parameter may be relevant:

• amsPropMaxDeliveryRate.0 = Gauge32: 150

Definition used during the incident:

• Maximum number of delivery attempts per second (range 1–5000; default 150).
• If not present in host/common configuration, it defaults to 150.
• Increases should be performed gradually. Final sizing/target values and deployment-level design changes (for example, queue balancing/replication/topology considerations) are typically handled through Professional Services.