VMware Log Insight Manager

1 Introduction

VMware vCenter Log Insight delivers real-time log management and log analysis with machine learning-based Intelligent Grouping, high performance search and better troubleshooting across physical, virtual and cloud environments.

Deployment_Guide-VMware_vCenter_Log_Insight_Manager_1.png

The flow of traffic in the above diagram is as follows:

1. The syslog clients create logs

2. The syslog clients then send the messages to the Virtual IP address on the LoadMaster

3. The LoadMaster distributes these messages to the Log Insight nodes

Log Insight supports receipt and ingestion of syslog messages that are sent over UDP, TCP, TCP with SSL encryption and using the API. The LoadMaster provides specialized Log Insight-aware services to optimize high availability and scalability of Log Insight deployments. Users can then perform deep analytics, discovery and search of the ingested data to get an enhanced operational view of their environment.

 

An inherent challenge that arises when syslog messages are sent using methods other than UDP, is that clients will often open long-lived connections that are then used for large amounts of messages. With this behavior, even when a scaled out architecture and application load balancer are implemented, traffic is not distributed in a close-to-even fashion across the pool of available nodes. The LoadMaster offers a solution that allows messages to be parsed within a connection to allow a more even distribution across servers in a pool, as well as simplified scalability of Log Insight environments.

1.1 Document Purpose

The purpose of this document is to explain how to configure the LoadMaster to optimize VMware Log Insight traffic flows.

1.2 Intended Audience

This document is intended to be read by anyone who is interested in configuring the LoadMaster to optimize VMware Log Insight deployments

2 Configure the LoadMaster

A number of Virtual Services will need to be created for the LoadMaster to work effectively with Log Insight. The services that is used depends on the methods that are used in the environment to send syslog messages to the Log Insight nodes. Refer to the sections below for detailed, step-by-step instructions.

2.1 Configure Log Insight Message Split Interval

The Log Insight Split Interval value controls how many syslog messages should be sent to each server in the pool before moving to the next server. For example, if there are three Log Insight nodes and the Log Insight Message Split Interval is set to 1 - a single message is sent to server A, and then to server B and then server C before again distributing a message to server A.

To set the Log Insight Split Interval, follow the steps below:

1. In the main menu of the WUI, go to System Configuration > Miscellaneous Options > L7 Configuration.

Configure Log Insight Message.png

2. Set the Log Insight Message Split Interval.

The default value is 10. The range is 1-100.

2.2 Create the TCP Syslog Virtual Service

A TCP syslog Virtual Service must be created if clients will send syslog messages to Log Insight over TCP. To do this, follow the steps below:

1. In the main menu of the LoadMaster WUI, select Virtual Services > Add New.

Create the TCP Syslog Virtual.png

2. Enter a valid Virtual Address.

3. Enter 514 as the Port.

4. Enter a recognizable Service Name, for example Log Insight TCP.

5. Click Add this Virtual Service.

The Service Type (in the Basic Properties section) of the 514 Virtual Service is set to Log Insight by default.

6. Expand the Standard Options section.

Create the TCP Syslog Virtual_1.png

7. Select round robin as the Scheduling Method.

Round robin is typically best to accomplish desired behavior of even traffic distribution. Least connection will result in an uneven distribution for syslog over TCP, especially when there is a low number of connections. If the Scheduling Method is set to least connection and there are a low number of connections, the Log Insight Split Interval (see below) will not behave as expected.

8. Expand the Real Servers section.

Create the TCP Syslog Virtual_2.png

9. Enter 514 as the health check port.

10. Click Add New.

Create the TCP Syslog Virtual_3.png

11. Enter the Real Server Address.

12. Click Add This Real Server.

2.3 Create the UDP Syslog Virtual Service

A UDP Syslog Virtual Service must be created if clients will send syslog messages to Log Insight over UDP. To do this, follow the steps below:

1. In the main menu of the LoadMaster WUI, select Virtual Services > Add New.

Create the UDP Syslog Virtual.png

2. Enter a valid Virtual Address.

3. Enter 514 as the Port.

4. Enter a recognizable Service Name, for example Log Insight UDP.

5. Select udp as the Protocol.

6. Click Add this Virtual Service.

7. Expand the Standard Options section.

Create the UDP Syslog Virtual_1.png

Ensure that the Transparency check box is selected. This allows the client’s IP address to be presented to the Log Insight servers. Depending upon your network topology, transparency may not be supported. If this is the case, you can safely disable this Transparency option and the source IP presented to Log Insight is that of the Virtual Service. The hostname will remain unchanged.

8. Enter a low Idle Connection Timeout value and click Set Idle Timeout.

A value of 1 will typically result in the best performance for the Log Insight UDP service.

9. Expand the Real Servers section.

Create the UDP Syslog Virtual_2.png

10. Ensure that ICMP Ping is selected in the Real Server Check Parameters drop-down list.

11. Click Add New.

Create the UDP Syslog Virtual_3.png

12. Enter the Real Server Address.

13. Click Add This Real Server.

2.4 Create the SSL Syslog Virtual Service

A SSL syslog Virtual Service must be created if clients will send syslog messages to Log Insight over TCP. To do this, follow the steps below:

1. In the main menu of the LoadMaster WUI, select Virtual Services > Add New.

Create the SSL Syslog Virtual.png

2. Enter a valid Virtual Address.

3. Enter 1514 as the Port.

4. Enter a recognizable Service Name, for example Log Insight SSL.

5. Click Add this Virtual Service.

Create the SSL Syslog Virtual_1.png

6. Select Log Insight as the Service Type.

7. Expand the SSL Properties section.

Create the SSL Syslog Virtual_2.png

8. Select Enabled.

9. Click OK.

10. Click Manage Certificates.

Create the SSL Syslog Virtual_3.png

11. Click Import Certificate.

Create the SSL Syslog Virtual_4.png

12. Click the first Choose File button.

13. Browse to and select the relevant certificate file.

14. If needed, upload a Key File and enter the Pass Phrase.

15. Enter a name in the Certificate Identifier text box.

16. Click Save.

Create the SSL Syslog Virtual_5.png

17. Select the relevant Virtual Service from the Available VSs box.

18. Click the right arrow button to assign the Virtual Service to the certificate.

19. Click Save Changes.

20. Expand the Real Servers section.

Create the SSL Syslog Virtual_6.png

21. Ensure that TCP Connection Only is selected in the Real Server Check Parameters drop-down list.

22. Click Add New.

Create the SSL Syslog Virtual_7.png

23. Enter the Real Server Address.

24. Enter 514 as the Port.

25. Click Add This Real Server.

26. Add any other Real Servers as needed.

2.5 Log Insight API Ingest Service

If HTTP POST requests are used to programmatically send log information to the Log Insight cluster, a “Log Split” content rule is required and an accompanying Virtual Service must be created. Content rules interrogate incoming client connections and make decisions as well as header modification based on the contents of the requests. Follow the steps in the two sections below for instructions on how to do this. This rule will ensure even distribution of messages across the cluster of Log Insight nodes when the API Ingest Service is utilized.

2.5.1 Create the Log Split Content Rule

A “Log Split” content rule is required to minimize “lumpiness” and accomplish a more even distribution of messages that are posted.

To create the content rule, follow the steps below:

1. In the main menu of the LoadMaster WUI, select Rules & Checking > Content Rules.

2. Click Create New.

Create the Log Split Content.png

3. Enter a recognizable Rule Name, for example LogInsightAPI.

4. Select Replace Header as the Rule Type.

5. Enter Connection as the Header Field.

6. Enter keep-alive as the Match String.

7. Enter close as the Value of Header Field to be replaced.

8. Click Create Rule.

For more information, refer to the Feature Description, Content Rules document.

2.5.2 Create the API Ingest Virtual Service

Now, an API ingest Virtual Service must be created. To do this, follow the steps below:

1. In the main menu, select Virtual Services > Add New.

Create the API Ingest Virtual.png

2. Enter a valid Virtual Address.

3. Enter 9000 as the Port.

4. Enter a recognizable Service Name, for example Log Insight API.

5. Click Add this Virtual Service.

Create the API Ingest Virtual_1.png

6. Select HTTP/HTTPS as the Service Type.

7. Expand the Standard Options section.

Create the API Ingest Virtual_2.png

8. Ensure that the Transparency check box is selected.

9. This allows the client’s IP address to be presented to the Log Insight servers. Depending upon your network topology, transparency may not be supported. If this is the case, you can safely disable this Transparency option and the source IP presented to Log Insight is that of the Virtual Service. The hostname will remain unchanged.

10. Expand the Real Servers section.

Create the SSL Syslog Virtual_6.png

11. Ensure that TCP Connection Only is selected in the Real Server Check Parameters drop-down list.

12. Click Add New.

Create the API Ingest Virtual_3.png

13. Enter the Real Server Address.

14. Click Add This Real Server.

15. Click OK.

16. Add any other Real Servers as needed.

17. Click Back.

18. Expand the Advanced Properties section.

VSVSAP028.png

19. Click Enable.

20. Click Show Header Rules.

Create the API Ingest Virtual_5.png

21. In the Request Rules section, select the relevant rule and click Add.

References

Unless otherwise specified, the following documents can be found at

http://kemptechnologies.com/documentation.

Feature Description, Content Rules

Web User Interface, Configuration Guide

Document History

Date

Change

Reason for Change

Version

Resp.

Sep 2014

Initial draft

First draft of document

1.0

LB

Nov 2014

Minor update

Defects resolved

1.1

LB

Jan 2015

Release updates

Updates for 7.1-24 release

1.2

LB

June 2015

Release updates

Updates for 7.1-28 release

1.3

LB

Oct 2015

Release updates

Updates for 7.1-30 release

3.0

LB

Dec 2015

Release updates

Updates for 7.1-32 release

4.0

LB

Jan 2016

Minor update

Updated Copyright Notices

5.0

LB

July 2016

Release updates

Updates for 7.1.35

6.0

LB

Jan 2017

Minor update

Enhancements made

7.0

LB

July 2017 Release updates Updates for 7.2.39 8.0 LB

 

Was this article helpful?

0 out of 0 found this helpful

Comments