Progress DataDirect Hybrid Data Pipeline

Updated: March 27, 2023 11:57

1 Introduction

Progress DataDirect® Hybrid Data Pipeline provides simple, secure, and scalable access to universal data connectivity.

The Progress LoadMaster delivers an exceptional, cost effective, and easy to use solution which by employing intelligent server health checking, load balancing, high availability, and security can support an always-on application experience for DataDirect Hybrid Data Pipeline.

1.1 Document Purpose

This document provides the recommended LoadMaster settings used when providing load balancing for DataDirect Hybrid Data Pipeline. The Progress Support team is available to provide solutions for scenarios not explicitly defined. The Progress Kemp Support site can be found at: https://support.kemptechnologies.com.

1.2 Intended Audience

This document is intended to be read by anyone who is interested in configuring the LoadMaster to optimize DataDirect Hybrid Data Pipeline.

2 Template

Progress Kemp has developed a template containing our recommended settings for this workload. You can install this template to help create Virtual Services (VSs) because it automatically populates the settings. You can use the template to easily create the required VSs with the recommended settings. For some workloads, additional manual steps may be required such as assigning a certificate or applying port following. These steps are covered in the document, if needed.

You can remove templates after use and this will not affect deployed services. If needed, you can make changes to any of the VS settings after using the template.

Download released templates from the following page: LoadMaster Templates.

For more information and steps on how to import and use templates, refer to the Virtual Services and Templates, Feature Description.

3 Architecture

Progress DataDirect deployments consist of multiple servers running Hybrid Data Pipeline behind a load balancer.

4 Configure the LoadMaster

Refer to the sections below for details on some recommended global settings.

4.1 Enable Subnet Originating Requests Globally

It is best practice to enable the Subnet Originating Requests option globally.

In a one-armed setup (where the Virtual Service and Real Servers are on the same network/subnet) Subnet Originating Requests is usually not needed. However, enabling Subnet Originating Requests should not affect the routing in a one-armed setup.

In a two-armed setup where the Virtual Service is on network/subnet A, for example, and the Real Servers are on network B, Subnet Originating Requests should be enabled on LoadMasters with firmware version 7.1-16 and above.

When Subnet Originating Requests is enabled, the Real Server sees traffic originating from 10.20.20.21 (LoadMaster eth1 address) and responds correctly in most scenarios.

With Subnet Originating Requests disabled, the Real Server sees traffic originating from 10.0.0.15 (LoadMaster Virtual Service address on eth0) and responds to eth0 which could cause asymmetric routing.

When Subnet Originating Requests is enabled globally, it is automatically enabled on all Virtual Services. If the Subnet Originating Requests option is disabled globally, you can choose whether to enable Subnet Originating Requests on a per-Virtual Service basis.

To enable Subnet Originating Requests globally, follow the steps below:

1. In the main menu of the LoadMaster User Interface (UI), go to System Configuration > Miscellaneous Options > Network Options.

2. Select the Subnet Originating Requests check box.

5 Virtual Services

This step-by-step setup of Virtual Services (VSs) leverages the Progress Kemp application template for DataDirect Hybrid Data Pipeline. This template configures the Virtual Services to publish DataDirect with HTTP, HTTPS with TLS/SSL Offloading or HTTPS with TLS/SSL Reecryption.

The table in each section outlines the settings configured by the application template. You can use this information to manually configure Virtual Services or use the Progress LoadMaster Application Programming Interface (API) and automation tools.

5.1 Create the DataDirect Hybrid Data Pipeline – HTTPS Offloaded Virtual Service

The following are the steps involved and the recommended settings to configure the DataDirect Hybrid Data Pipeline HTTPS Offloaded Virtual Service.

In the main menu of the LoadMaster User Interface (UI), go to Virtual Services > Add New.
Type a valid Virtual Address.
Select the DataDirect HDP HTTPS Offloaded template in the Use Template drop-down list.
Click Add this Virtual Service.

5.1.1 Modify the DataDirect Hybrid Data Pipeline Default Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Offloaded Virtual Service on port 443.
Expand the SSL Properties section.
Select the certificate to use from Available Certificates and click the arrow (>) to move it to Assigned Certificates.
Expand the SubVSs section.
Click Modify for the HDP Offloaded Default Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address.
Enter 8080 for the Port.
Click Add This Real Server.
Repeat these steps to add more Real Servers as needed.

5.1.2 Modify the DataDirect Hybrid Data Pipeline Notification Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Offloaded Virtual Service on port 443.
Expand the SubVSs section.
Click Modify for the HDP Offloaded Notification Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address.
Enter 11280 for the Port.
Click Add This Real Server.
Repeat these steps to add more Real Servers as needed.

5.1.3 Modify the DataDirect Hybrid Data Pipeline OPA – Node 1 Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Offloaded Virtual Service on port 443.
Expand the SubVSs section.
Click Modify for the HDP Offloaded OPA Node 1 Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address.
Enter 40501 for the Port.
Click Add This Real Server.

5.1.4 Modify the DataDirect Hybrid Data Pipeline OPA – Node 2 Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Offloaded Virtual Service on port 443.
Expand the SubVSs section.
Click Modify for the HDP Offloaded OPA Node 2 Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address for the second DatatDirect Hybrid Data Pipeline Server.
Enter 40501 for the Port.
Click Add This Real Server.
Repeat these steps to add more Real Servers as needed.

Additional OPA Sub Virtual Services can be created if more than two servers are in the environment.

5.1.5 Modify the DataDirect Hybrid Data Pipeline OPA Content Rules

Content Rules are created to direct traffic based on the path of the URL. The OPA Content Rules have unique paths for each environment and must be modified.

1. Click Rules and Checking and Content Rules.

2. Click Modify on the opa_Node1_xxxxx Content Rule.

3. Change the ADD_NODE_SERVER_IP to the unique path for the OPA Node 1 server.

4. Click Modify Rule to apply changes.

5. Repeat steps for remaining OPA Content Rules.

5.1.6 DataDirect Hybrid Data Pipeline HTTPS Offloaded Virtual Service Recommended Settings (optional)

This table outlines the recommended settings set using the Progress Kemp application template. You can use the API parameters and values with scripts and automation tools.

API Parameter	API Value	WUI Field Name	WUI Field Value
Root Virtual Service
port	443	Port	443
prot	tcp	Protocol	tcp
VStype	http	Service Type	HTTP-HTTP/2-HTTPS
Schedule	lc	Scheduling Method	least connection
Persist	none	Persistence Options	None
SubnetOriginating	1	Subnet Originating Requests	Enabled
Forcel7	1	Force L4	Disabled
SSLAcceleration	1	SSL Acceleration	Enabled
TLSType	3	Supported Protocols	TLS1.1, TLS1.2, and TLS1.3 (Enabled)
CipherSet	BestPractices	Cipher Set	BestPractices
HDP Offloaded Default - Sub Virtual Service
SubnetOriginating	1	Subnet Originating Requests	Enabled
Schedule	lc	Scheduling Method	least connection
Persist	src	Persistence Options	Source IP Address
PersistTimeOut	300	Timeout	5 Minutes
CheckType	http	Real Server Check Method	HTTPS Protocol
CheckUseGet	0	HTTP Method	HEAD
CheckPort	8080	Checked Port	8080
CheckUrl	/api/healthcheck	URL	/api/healthcheck
HDP Offloaded Notification - Sub Virtual Service
SubnetOriginating	1	Subnet Originating Requests	Enabled
Schedule	lc	Scheduling Method	least connection
Persist	src	Persistence Options	Source IP Address
PersistTimeOut	300	Timeout	5 Minutes
CheckType	http	Real Server Check Method	HTTP Protocol
CheckUseGet	0	HTTP Method	HEAD
CheckPort	11280	Checked Port	11280
HDP Offloaded OPA Node 1 - Sub Virtual Service
SubnetOriginating	1	Subnet Originating Requests	Enabled
Schedule	lc	Scheduling Method	least connection
CheckType	http	Real Server Check Method	HTTP Protocol
CheckUseGet	0	HTTP Method	HEAD
CheckPort	40501	Checked Port	40501
HDP Offloaded OPA Node 2 - Sub Virtual Service
SubnetOriginating	1	Subnet Originating Requests	Enabled
Schedule	lc	Scheduling Method	least connection
CheckType	http	Real Server Check Method	HTTP Protocol
CheckUseGet	0	HTTP Method	HEAD
CheckPort	40501	Checked Port	40501

5.2 Create the DataDirect Hybrid Data Pipeline – HTTPS Reencrypt Virtual Service

The following are the steps involved and the recommended settings to configure the DataDirect Hybrid Data Pipeline HTTPS Reencrypt Virtual Service.

In the main menu of the LoadMaster User Interface (UI), go to Virtual Services > Add New.
Type a valid Virtual Address.
Select the DataDirect HDP HTTPS Reencrypt template in the Use Template drop-down list.
Click Add this Virtual Service.

5.2.1 Modify the DataDirect Hybrid Data Pipeline Default Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Reencrypt Virtual Service on port 443.
Expand the SSL Properties section.
Select the certificate to use from Available Certificates and click the arrow (>) to move it to Assigned Certificates.
Expand the SubVSs section.
Click Modify for the HDP Reencrypt Default Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address.
Enter 8443 for the Port.
Click Add This Real Server.
Repeat these steps to add more Real Servers as needed.

5.2.2 Modify the DataDirect Hybrid Data Pipeline Notification Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Reencrypt Virtual Service on port 443.
Expand the SubVSs section.
Click Modify for the HDP Reencrypt Notification Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address.
Enter 11443 for the Port.
Click Add This Real Server.
Repeat these steps to add more Real Servers as needed.

5.2.3 Modify the DataDirect Hybrid Data Pipeline OPA – Node 1 Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Reencrypt Virtual Service on port 443.
Expand the SubVSs section.
Click Modify for the HDP Reencrypt OPA Node 1 Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address for the first DataDirect Hybrid Data Pipeline Server.
Enter 40501 for the Port.
Click Add This Real Server.

5.2.4 Modify the DataDirect Hybrid Data Pipeline OPA – Node 2 Sub Virtual Service

Click Virtual Services and View/Modify Services.
Click Modify on the DataDirect HDP HTTPS Reencrypt Virtual Service on port 443.
Expand the SubVSs section.
Click Modify for the HDP Reencrypt OPA Node 2 Sub Virtual Service.
Expand the Real Servers section.
Click Add New.
Type the Real Server Address for the second DataDirect Hybrid Data Pipeline Server.
Enter 40501 for the Port.
Click Add This Real Server.
Repeat these steps to add more Real Servers as needed.

Additional OPA Sub Virtual Services can be created if more than two servers are in the environment.

5.2.5 Modify the DataDirect Hybrid Data Pipeline OPA Content Rules