Fix OOM on LGP nodes caused by oversized innodb_buffer_pool_size

Overview

If a NewNet Lithium/TextPass LGP node has MySQL innodb_buffer_pool_size configured to a value larger than the available system RAM, the host can hit Linux OOM conditions that terminate mysqld and may also cause SNMP (snmpd) to go down. In one example , innodb_buffer_pool_size was configured to ~55GB, however the server only had 29GB of RAM, which caused major memory exhaustion and service instability. Reducing the buffer pool to a RAM-appropriate value (for example: 20G) stabilized the servers.

Symptoms 
The syslog may contain memory exhaustion/OOM errors with MySQL and SNMP service failures.

Root cause

- innodb_buffer_pool_size set to a value that cannot fit in physical RAM (oversized relative to host memory), increasing the likelihood of OOM-killer events and process termination.

Solution

Fix (apply on all affected LGP nodes)

1. Choose an appropriate buffer pool size

    - Use a value that fits your server RAM headroom. The innodb_buffer_pool_size should never exceed 75% of the total system memory.

2. Update MySQL configuration

    - Edit /etc/my.cnf and change:
        - innodb_buffer_pool_size = 55G → innodb_buffer_pool_size = 20G (this is an example value; set to the target selected in step 1)

3. Restart in safe order (as used in the ticket)

    - As textpass: tp_stop --tp_lgp
    - As root: systemctl restart mysql
    - As textpass: tp_start --tp_lgp

4. Verify the active value

    - mysql -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"
    - Confirm it matches the intended value (e.g., 20G = 21474836480 bytes).

5. Post-change validation (stability)

    - Check memory: free -hm
    - Check top RSS: ps -eo user,pid,cmd,rss --sort=-rss | head -n 20
    - Confirm no new OOM events: grep -i "out of memory" /var/log/messages | tail -n 50
- Confirm services are running:
        - systemctl status mysql
        - systemctl status snmpd

Optional: Search validation (if users also report 600s UI timeout)

- Re-run the same search as a Background Query (so UI timeout doesn’t mask completion), and if needed split the time range into smaller windows.

References (public KB)

- Troubleshooting High Memory Usage on LGP Node Due to MySQL Database
- Troubleshooting Excessive Memory Usage in LGP
- MySQL query is stuck on MGR