Kernel Memory Lockdown During Unclean Restarts

Overview

After upgrading a node in a NewNet environment, certain components such as the PBC or the AMS may fail to start. This can be caused by a kernel memory lockdown during an unclean restart, which prevents critical components from accessing /dev/mem.

Symptoms

• Services such as tp_pbc and tp_ams do not come up after a node restart.

• The following error messages are observed in the system logs:

Sep 16 11:54:13 smsr-sm3 tp_pbc[67549]: Cannot retrieve external serial nr, contact support!!
Sep 16 11:54:14 smsr-sm3 tp_pbc[67545]: Application (pid=67549) exited (1), terminating watchdog

Sep 16 15:00:54 smsr-sm3 tp_pbc[78398]: Searching for License, HWSerial = 'e6726420', SS7Serial = '', InstanceSerial = '200', Status EXTserial = reliable
Sep 16 15:00:54 smsr-sm3 tp_pbc[78398]: Cannot retrieve external serial nr, contact support!!
Sep 16 15:00:55 smsr-sm3 tp_pbc[78397]: Application (pid=78398) exited (1), terminating watchdog

• Manual restart attempts with tp_start tp_pbc fail with the following errors:

/dev/mem: Operation not permitted
Table is unreachable, sorry.
/dev/mem: Operation not permitted

Root Cause

The unclean restart triggers a kernel-level memory lockdown. This blocks access to /dev/mem, preventing components from reading required serial information for licensing and hardware checks.

Solution

Reinstalling the affected component packages restores proper memory access permissions and resolves the issue.

Commands to Reinstall Components

For RHEL 7 and earlier:

rpm -ivh --replacepkgs TextPassTools-R04.08.00.02-RHEL6.x86_64.rpm

For RHEL 8 and later:

sudo yum reinstall -y ./TextPassTools-R04.08.00.02-RHEL6.x86_64.rpm

Note: While the example uses TextPassTools, the same procedure applies to other affected NewNet SMSC components.

Verification

1. Restart the services normally using tp_start.

2. Confirm that the affected services (tp_pbc, tp_ams, etc.) start without errors.

3. Monitor logs to ensure no further /dev/mem access errors are reported.