tp_ams Restarts Caused by Invalid AMS Instance ID Requests

Overview

This article describes a specific tp_ams restart pattern where AMS receives a request containing an AMS instance ID that is numerically valid but does not correspond to any AMS node configured on the system at that moment.

When this condition occurs, tp_ams can crash, restart, and briefly interrupt alarm-processing activity.  The issue may be confirmed from /var/log/messages.  A product fix was developed for this issue.

Fixed version: v24.03.24.12

How to Identify the Issue in /var/log/messages

The most useful indicator is the ENGFEA3 lookup sequence in the AMS logs.

Primary log pattern

Look for a sequence like the following:

tp_ams[PID]: ENGFEA3 find_stored_message:
tp_ams[PID]: ENGFEA3 find_stored_message: using message ID
tp_ams[PID]: ENGFEA3 find_msg_by_identifier: msg_id.ams_id: <value>
tp_ams[PID]: ENGFEA3 find_msg_by_identifier: ams_id<=AMS_MAX_NODES: 1, active_ams_found with id <value>: 0
tp_ams[PID]: ENGFEA3 find_msg_by_identifier: Only look for Master or Active Replica messages. MasterAMSId: <value>
kernel: tp_ams[PID]: segfault at 8c ip 00000000004801f2 ...

What matters in this sequence:

• msg_id.ams_id is an AMS instance ID carried in the incoming request.
• active_ams_found with id <value>: 0 means AMS did not find a configured AMS instance for that ID.
• If that sequence is followed by segfault at 8c, you are seeing this crash pattern.

Follow-on confirmation pattern

After the crash, the same incident typically produces one or more of the following lines:

tp_ams[PID]: Missing health signal, missed N signals, allowed 100
tp_ams[PID]: Application (pid=<old_pid>) died (status 139), restarting
tp_ams[PID]: Application terminated, too many restarts within predefined interval

These lines confirm that the watchdog detected the failed AMS process and restarted it. If the condition repeats, AMS can enter a restart cycle.

How to Interpret the msg_id.ams_id Value

Compare the logged msg_id.ams_id value against the AMS IDs that are actually configured in your environment. The exact valid ID set is deployment-specific.

• If the value belongs to a configured AMS node, the request should complete normally.
• If the value does not belong to a configured AMS node and the log shows active_ams_found with id <value>: 0, that request matches the failure condition described in this article.

In the investigated incident, unexpected values such as 100, 101, and 151 appeared in the logs before the crash sequence.

Commands to Check

Run the following commands on the affected node using an account that can read /var/log/messages:

grep 'ENGFEA3 find_msg_by_identifier' /var/log/messages | tail -50
grep 'segfault at 8c' /var/log/messages | tail -20
grep 'Missing health signal' /var/log/messages | tail -20
grep 'Application terminated, too many restarts within predefined interval' /var/log/messages | tail -20

If you want to capture the restart evidence as well, you can also check for status 139 restart lines in the same log file.

Impact

• tp_ams crashes and restarts.
• Alarm-processing activity is briefly interrupted while AMS is down.
• If the same trigger repeats, AMS can restart multiple times in a short period.

Resolution

The fix for this issue is included in v24.03.24.12.  If you have encountered this issue, please speak to your Account Manager about upgrading to this version or later.

After upgrading, the same type of request should no longer crash the AMS. The invalid reference may still be logged, but AMS should handle it gracefully instead of producing a segmentation fault and restart.

Post-Upgrade Validation

After the upgrade, monitor /var/log/messages and confirm the following:

• No new segfault at 8c lines for tp_ams
• No new status 139 restart cycle for the same condition
• No repeated Missing health signal sequence tied to this crash pattern

Workaround

No reliable product-side workaround has been identified short of upgrading to a version that contains the fix. Until the upgrade is applied, the practical action is to monitor /var/log/messages for the pattern described above and collect the relevant log window if the issue recurs.