How to create a Syslog server to store netconsole logs from the LoadMaster

How to create a Syslog server to store netconsole logs from the LoadMaster

Product: LoadMaster.

Version: Any.

Platform: Any.

Application: Any.

How to create a Syslog server to store netconsole logs from the LoadMaster and preserve logs caused by reboots, crashes, and lockups.

For the purpose of this article, we will use rsyslog in an Ubuntu server as our Syslog host.

The steps to install and properly configure rsyslog to store netconsole logs from the LoadMaster are as follows:

1. Update our repositories on our server using "sudo apt-get update"

2. Install the rsyslog package using "sudo apt-get install rsyslog"

3. Once the package is installed, start the rsyslog service by issuing the "sudo systemctl enable --now rsyslog" command.

4. Next, we need to open the rsyslog.conf file located in the /etc folder to start making the necessary changes.

5. Enable the relevant protocol for syslog reception (TCP or UDP) by uncommenting the lines under #provides UDP syslog reception or #provides TCP syslog reception. For this article, we will use UDP.

6. Next, we need to create a template. Any output that is generated by rsyslog can be modified and formatted according to your needs with the use of templates. We will add the following to our rsyslog.conf file in the GLOBAL DIRECTORIES section:

#Custom template to generate the log filename dynamically based on the
#client's IP Address and place it in /var/log/<client's IP>/netconsole.log file

$template FILENAME,"/var/log/%fromhost-ip%/netconsole.log"
*.* ?FILENAME

7. Save the changes in our configuration file.

8. Now that the configuration changes are made, we need to make sure that our server will not block traffic destined to UDP port 514. For this, we open a firewall rule if necessary on our server using "sudo ufw allow 514/udp".

9. Restart the rsyslog service to apply the changes using "sudo systemctl restart rsyslog".

Now that rsyslog is installed and configured, we need to configure the LoadMaster to send netconsole logs to this host.

For LoadMasters running firmware version 7.2.55 and below:

Navigate to the LoadMaster's WUI > System Configuration > Logging Options > System Log Files > Debug Options > Populate the server's IP in the Netconsole Host field > Select the relevant Interface > Click Set Netconsole Host.

For LoadMasters running firmware version 7.2.56 and above:

Navigate to the LoadMaster's WUI > System Configuration > Troubleshooting > Populate the server's IP in the Netconsole Host field > Select the relevant Interface > Click Set Netconsole Host.

From a configuration standpoint, this is all we need. 

To confirm that this works, in the event of a unexpected reboot, we should see a file named netconsole.log in the /var/log/<LoadMaster's IP>/ folder on our server and the call traces will be present in this file as seen below:

It is worth noting that this rsyslog configuration will not only work for netconsole logs, but also can be used to store regular logs from the LoadMaster via syslog.

To configure the LoadMaster to export logs via syslog, navigate to the WUI > System Configuration > Logging Options > Syslog Options > Enter the Syslog host and select the severity.

From the server side, we can inspect the same file as before (/var/log/<LoadMaster's IP>/netconsole.log) and we'll see the logs exported from the LoadMaster:

https://support.kemptechnologies.com/hc/en-us/articles/216491943-How-to-configure-the-LoadMaster-to-send-unexpected-reboot-event-logs-to-a-Syslog-Server

https://kifarunix.com/setup-rsyslog-server-on-ubuntu-20-04/