NewNet Lithium AMS: Repeated/Alternating <code>tp_ams</code> Restarts from the <code>incomplete_msgs</code> Assertion (<code>src/ams_mqs.c</code> line 7270)

Solution

1. Recognizing this issue

You are affected by this specific defect if all of the following hold:

• AMS (tp_ams) processes restart repeatedly, often alternately between two traffic nodes (for example, an owner node and the node holding its replicas), with core files generated.
• /var/log/messages contains Assert fail: (!recip_buff->incomplete_msgs) ... src/ams_mqs.c line 7270.
• Core backtraces show the abort occurring during recipient-buffer deletion, reachable from delivery completion (DELIVERY_OK), scheduled expiry, or an AMS-to-AMS DELETE update.
• Stopping AMS on one node halts the loop, but starting it again — even after a full VM restart — resumes the loop, because recovery reloads the same stored state.

This is an internal accounting inconsistency, not stored-message corruption. The incomplete_msgs counter is an in-memory, per-recipient-buffer count of concatenated segments still part of an incomplete set. The assertion enforces that a buffer being deleted (now empty) must have that counter at zero. A stranded non-zero counter aborts the process.

Note: Because the assertion fires when the buffer becomes empty, the message being deleted at that moment can be an ordinary single-segment message that is entirely healthy — it is simply the last message out of a buffer whose counter was left stale earlier (possibly days earlier) by a concatenated-message event. This is why assertions appear at seemingly unpredictable times on ordinary traffic.

2. Root cause (two accounting leak paths)

• Leak A (primary) — recovery same-slot replacement: When two persisted records claim the same concatenation reference and segment index, startup recovery keeps the newer record for the slot and detaches the other (which becomes a standalone single message) without decrementing incomplete_msgs. Duplicate records accumulate during normal operation (a segment stored, its context detached/timed out, then the segment re-sent/retried and stored again); the collision and the stranded counter are produced at recovery.
• Leak B (secondary) — incomplete-set timeout / delete-incomplete: On incomplete-set timeout, segments not in the STORED state are detached without the matching decrement.

Both leave a recipient buffer whose counter exceeds its actual contents; the assertion then fires when the buffer is finally emptied by delivery, expiry, or DELETE.

Fix status: AMS R01.13.01.20 fixes Leak B (the incomplete-set-timeout path) and makes the assertion non-fatal. A fix for Leak A (the recovery same-slot path) is not confirmed. Treat Leak A as still present.

3. Diagnostic steps

1. Search all node syslogs for the assertion and the MRM drop line:

   grep "Assert fail" /var/log/messages
   grep "MRM: Tried to replicate msg 51 times" /var/log/messages

2. Confirm the crash signal/path from any core file (SIGABRT / status 134 on the asserting path; the watchdog kills a node after roughly 100–101 missed health signals).
3. Capture cluster/state evidence: tp_status --tp_ams, tp_qcli -s -n (from more than one node), tp_walkall, and the relevant /var/log/messages window.
4. Rule out look-alikes:
   • This is not the "invalid AMS instance ID" pattern (different log signature/address). An automated first pass may suggest that known issue and an upgrade to v24.03.24.12 — that path does not apply here.
   • This is not the transaction-ID wraparound (that shows amsPropMaxIngress = 500 with a growing pending-completion gauge and no Assert fail line — see Related condition A).

4. Containment (immediate)

• Keep AMS (Lithium) stopped on the looping node using tp_stop.
• Do not repeatedly restart an asserting node on its existing Message Store — recovery reloads the same disturbed state and the loop returns.
• Divert new traffic away from the affected site if possible.
• Avoid bulk message operations (forced-delivery sweeps with tp_qcli -a, mass cancels/purges) — they concentrate exactly the deletion activity that trips stale buffers.
• Avoid unnecessary topology changes (starts/stops/rejoins), which drive replica-target re-selection and concentrated delete activity.

4a. Decision flow when more than one node is affected

These rules apply whenever more than one node on the cluster needs attention at the same time — for example, an assertion on one node while another is approaching a transaction-ID wraparound:

1. One AMS transition at a time, cluster-wide. Each stop/start re-selects replica targets and re-runs recovery. Do not start a second node's transition until the first has fully settled: wait for Active Replica sync to complete and its role hand-back line (if a role was taken), otherwise wait the failover hold-off (default 60 seconds); then wait the purge-list delay (default 360 seconds) after all nodes are back up. "Operating" alone is not a sufficient gate.
2. Two gates before any further restart: an Assert fail: (!recip_buff->incomplete_msgs) on any node means stop all restart activity and collect evidence; and after restarting a wrapped node, compare its recovered-message count with the store size you expect for that node — a large discrepancy (recovered records far above what the node normally holds) means this node is a rebuild candidate, not a node to restart again.
3. The node with the transaction-ID wraparound is the one that is safe to restart. The node with the incomplete_msgs assertion is the one that must not be restarted on its existing store.
4. Never restart a stopped node to compensate for a node that just asserted. That is a second transition before the first has settled, and it can cause duplicate delivery.
5. If a wraparound has already happened, or a node's consumption is at or above about 90% of the limit: check the receiving site's own headroom, divert new traffic to it, wait for inflow to settle, then recover the affected nodes one at a time with both gates above, keeping traffic diverted until the last node is verified.
6. Plan ahead by consumption level: under 75% of the transaction-ID limit, a routine preventive restart is enough (traffic diversion optional). Above 75% with a maintenance window available, treat it as an urgent planned restart and divert traffic if more than one node is involved. At or above roughly 90%, or after a wraparound has already occurred, treat it as an emergency and always divert traffic first.

5. Permanent resolution

Upgrade to AMS R01.13.01.20 or later, which makes the buffer teardown non-fatal and corrects Leak B (see Fix status in the Root cause section). To get the upgrade, contact your account manager. Verify compatibility against your OS environment first (older deployments may be on an OS generation that requires a validated upgrade path) and follow an approved maintenance procedure.

6. Supported field recovery for an affected node (Message Store rebuild)

There is no supported tool to selectively inspect, repair, or delete the affected records, and no in-service reset. The supported recovery is a controlled Message Store rebuild, one node at a time (never concurrent):

1. Stop AMS on the target node (graceful), as the textpass user:

   tp_stop --tp_ams

   Confirm the graceful stop in syslog:

   Application (pid=N) exited (0), terminating watchdog

   Tip: If a guaranteed-graceful stop is required, issue the drain directly and wait for exit, avoiding the wrapper's short kill-fallback timeout:

   tp_set --tp_ams deviceAdminState.0 i destroy

2. Create per-node, per-attempt backup directories with unique names so nodes are never mixed:

   /data/Backup_<node>_<date>/log_master
   /data/Backup_<node>_<date>/store_master
   /data/Backup_<node>_<date>/log_replica
   /data/Backup_<node>_<date>/store_replica

3. Move (do not delete) the existing store/log contents into the backups:

   mv /dbamslog/master/*   <backup>/log_master/
   mv /dbamsstore/master/* <backup>/store_master/
   mv /dbamslog/replica/*  <backup>/log_replica/
   mv /dbamsstore/replica/* <backup>/store_replica/

   Warning: The moved files are preserved for forensics/evidence only. This is not a supported rollback path — restoring them re-introduces the bad store and the loop.
4. Restore ownership/permissions so textpass owns the (now empty) store/log directories.
5. CRITICAL — leave the .fcl files untouched. The .fcl directory holds only the node's own AMS identifier. Preserving it guarantees the node rejoins as the same AMS ID, with replica relationships re-derived dynamically. Deleting it would cause the node to negotiate a brand-new ID (for example, AMS4). No manual reconfiguration of AMS ID or replica topology is required.
6. Start AMS, as textpass:

   tp_start --tp_ams

7. Verify (see section 7) before starting or soaking the next node.

Suggested order for a paired alternating-restart case: rebuild the owner node first, soak until stable, then rebuild the peer node in a separate window. Keep traffic diverted during the operation.

Message-loss gate: A rebuild clears that node's master and replica stores. Because masters are normally replicated to two peers, the population existing only on the rebuilt node (with no replica anywhere) is bounded/small — but not provably zero. Accept this residual loss as an explicit go/no-go input before each rebuild window.

7. Post-rebuild / post-recovery validation

Confirm on each node before proceeding to the next:

• tp_status --tp_ams reports operating.
• tp_qcli -s -n (from more than one node) shows the node present with its original AMS ID.
• The recovery log shows near-zero recovered messages (a genuinely fresh store) and no Assert fail line. A large recovered-message count (hundreds of thousands or millions, where you expect tens of thousands) is not itself data loss — it means the node's completion bookkeeping was stuck and is now catching up. Compare it against the expected store size before deciding whether this node needs a rebuild instead of a restart (see 4a. Decision flow above).
• Replica target lists settle to the full complement (Replica list re-populated appears once per join, then stops).
• amsCntDelayedReplication / amsCntInprogressReplication drain, with no new MRM: Tried to replicate msg 51 times, dropping it.
• The pending-completion gauge textpass.2.6.20.0 remains near zero on all nodes.
• A successful test SMS, including a concatenated message, with replica creation confirmed.

Abort criterion at any point: any new Assert fail: (!recip_buff->incomplete_msgs) or repeated watchdog restart — keep that node stopped, preserve logs and cores, and escalate before further action.

Related condition A — Transaction-ID wraparound (distinct defect; restart, not rebuild)

Symptoms: amsPropMaxIngress pinned at 500; sustained/repeating amsCntRejectedRateExceeded; peers logging MRM: Tried to replicate msg 51 times, dropping it; linear, unbounded tp_ams memory growth; no Assert fail line; typically after very long uptime (multi-year) on a high-traffic node.

Cause: A per-environment 32-bit DB transaction counter wraps at 2³² operations; the wraparound-handling escape never re-arms, permanently deadlocking replica-store completion. Writes still commit (Berkeley DB is intact); only the in-memory completion bookkeeping is stuck, producing an ever-growing pending-completion list (textpass.2.6.20.0).

Recovery: A controlled tp_ams restart resets the counter and clears the condition — a rebuild is NOT required for the wraparound itself, provided it is caught early (ordinary recovery counts) and the incomplete_msgs assertion is absent.

Permanent fix: AMS R01.13.01.22 corrects the wraparound handling. To get it, contact your account manager. Until you upgrade, the restart-based recovery above is the correct workaround — a Message Store rebuild is still not needed for this condition.

Prevention: Plan preventive one-node-at-a-time restarts before consumption reaches about 75% of 2³². Estimate per-instance consumption from amsLoadCntStoreNonVolatileTotalRequests + amsLoadCntUpdateTotalRequests + amsLoadCntDeleteTotalRequests (take two samples ~1 hour apart for the rate; remaining margin = (2³² − current sum) / rate). The counter is per store environment (master and replica each wrap independently), and the SNMP counters themselves are 32-bit, so cross-check rate × uptime for hidden wraps.

CRITICAL caution: Do not perform sequential multi-node restarts under live traffic. Restarts re-select replica targets and re-run recovery; doing several in quick succession under traffic caused duplicate/split concatenated segments across replica holders and triggered an incomplete_msgs assertion storm that reduced a healthy cluster to two nodes. If wraparound has already occurred or consumption is far above threshold, first divert new traffic to the other site, let inflow settle, then recover one node at a time with full verification (recovery-count check, assertion check on all nodes, and completion of the prior transition's synchronization/handback and purge-list delay) between each.

Related condition B — Replication-path SIGSEGV (distinct defect)

Symptoms: kernel: traps: tp_ams[...] general protection and Application (pid=N) died (status 139), restarting; a core backtrace in MRM_fast_periodic → MQS_allocate_context_with_data → MQS_allocate_context; may be preceded by MRM: removing stale replica element.

Cause: A replication work item referencing a freed/invalid message-data pointer (use-after-free). Present in all current releases with no released fix; the watchdog auto-restart is the designed interim handling. Track and escalate this separately with the full core.