Overview

You see the applicationHeartBeatMissed SNMP trap and Missing health signal lines in the syslog for a Textpass process, for example the router process (textpass) or the AMS (tp_ams):

May 9 07:56:10 TKLCBU-TPR05 snmptrapd[2679]: 07:56:10 TRAP6.TEXTPASS-GEN-MIB::applicationHeartBeatMissed TEXTPASS-GEN-MIB::deviceType.0 = STRING: "RTR" from TKLCBU-TPR05
May 9 07:56:11 TKLCBU-TPR05 textpass: Missing health signal, missed 2 signals, allowed 5

The process may then be restarted (intermittent Textpass process restarts), or it may carry on without a restart. You may or may not already know of a hardware problem on the server, for example a failed disk.

What the alarm means

• Each Textpass process is monitored by a watchdog process, which expects a health signal from it every second.
• When signals are missing, the watchdog logs Missing health signal, missed N signals, allowed M, where N is the number of signals missed in a row. From the second missed signal in a row, it also sends the applicationHeartBeatMissed trap. The alarm therefore means that the process stopped responding long enough to miss two signals in a row. A pause of under two seconds can be enough.
• The watchdog restarts the process only when N becomes larger than M. It then logs Missed N health signals: trying to cleanly abort process, Application killed and Application restarted. M depends on the process: by default it is 5, as in the router example above, and 100 for the AMS. A router process is therefore restarted after about six seconds without a signal, and an AMS only after nearly two minutes. For shorter AMS pauses, the alarm is the only sign.
• If these restart lines are absent, the process was not restarted: it paused and then resumed by itself.
• The likely reasons for missed health signals are high traffic load or a hardware (disk or CPU) problem. A process can pause while it waits for a slow disk. For example, the AMS can pause while it waits for the disk that holds its database transaction logs. This can happen while the RAID controller still reports all drives as OK.

Solution

1. Check whether the process was restarted

As root, run:

zgrep -hE "Missing health signal|health signals: trying|Aborting on request|Application.*(killed|restarted|aborted|terminated|died)|applicationHeartBeatMissed" /var/log/messages*

This searches the current system log and the rotated ones. The process name (for example textpass or tp_ams) or the device type (for example "RTR" or "AMS") in each line shows which process missed its signals.

If the process was restarted, check whether core dumps were created at the time of the restart, as described in Handling a Process Restarting with Core File/How to Debug a Core Dump File. If there are core dumps, please contact NewNet support and share them.

2. Check the disks and the RAID controller

Do this even if no hardware alarm has been raised. On HPE servers with a Smart Array controller, run as root:

ssacli ctrl all show config

Every logical drive and every physical drive should show OK. A status such as Predictive Failure or Failed is a hardware fault.

ssacli ctrl all diag file=/tmp/ssa_diag_report.zip

This saves HPE's diagnostic report for the controller and all drives in /tmp. The report includes each drive's error counters (read errors, retries and time-outs). A drive can record read errors for weeks before the controller marks it Predictive Failure, and the first command shows only each drive's status, not its error history. If one drive has recorded errors while the others have none, have that drive checked first (see step 3).

zgrep -hE "kernel: .*(hpsa|smartpqi|\[sd[a-z]+\]|I/O error|blk_update_request|EXT4-fs|blocked for more than)" /var/log/messages*

This lists what the operating system has logged about the disks, the controller and the file systems. An empty result does not rule out a failing drive.

On an AMS node, also note which disks hold the message store database and its transaction logs (by default /dbamsstore and /dbamslog):

df -h /dbamsstore /dbamslog

On servers from other vendors, run the same checks with the vendor's RAID or disk tools.

3. If a drive or the controller is faulty, get it fixed first

Getting the hardware fault fixed is also likely to fix the missed health signals.

• Have the drive replaced in a maintenance window, through whoever provides hardware support for the server. If NewNet supplied the server as part of your platform, open a ticket with us for the replacement. Otherwise, open a case with your hardware vendor. Check the bay number in the ssacli output so that the faulty drive is the one that is removed. See How to safely replace a failed HD on a Lithium Server/Node and Hardware alert for degraded disk.
• If the Textpass process on this node keeps restarting, it can be stopped until the hardware is fixed. There should be no loss of service, as the other nodes in the network are available to handle the traffic.
• After the replacement, run ssacli ctrl all show config again until the logical drive and all of its drives show OK. Then keep monitoring the alarm in your NMS. You do not need to send us daily results. If the alarm comes back, send the data listed in step 4.

4. If no hardware fault is found, or the alarm continues

Open a ticket and attach the following from the node that raised the alarm. None of these commands changes the system.

1. The output of the command in step 1, and the /var/log/messages file that covers the time of the alarm (as root).
2. The system activity file for the day of the alarm, /var/log/sa/sa<DD>, where DD is the day of the month (as root). These files are kept only for a limited time, so collect it soon.
3. The diagnostic report from step 2 (/tmp/ssa_diag_report.zip) and the output of the other step 2 commands.
4. The output of tp_status (as textpass).