Kubernetes API Server issues behind VLM

0

Hi,

im using the KEMP VLM in front of a kubernetes cluster. Standard kubectl command work just fine, but other tools accessing the kubernetes api drop out with an EOF when querying resources

 

I see these errors in the log

Jun 11 11:53:35 KEMP kernel: L7: ffff888075045c88 Error handling task 0 2
Jun 11 11:53:35 KEMP kernel: L7: ffff88807e8f3368: VS 172.16.1.208:443 (16): Connection Failed to 172.26.1.60:6443 (0)

I have SSL acceleration / reencrypt enabled

I only have a single RS at the moment, which is beeing checked by icmp only... and is/was reachable at all times

If i change my kubeconfig to point to the RS directly, ignoring the invalid certificate, all commands/tools work just fine.

 

Any help/info welcome ;)

 

17 comments

Avatar
0
Nick Smylie

Hi,

It could be a lot of reasons it sound like.  Can you try disabling SSL acceleration and testing again?  This will turn off things like content switching, ESP and most types of persistence so please take note of that.  However it will also make the connection not be offloaded so we can rule some things out then if that works.

Avatar
0
d.buehring

With disabled SSL it works. But that is not really an option i like  :(

The api request is failing with an unexpected EOF if that helps...

My VS is really not that complicated, thats whats confusing me... aside from the fact that kubectl command are all working without problems

 

 

 

 

Avatar
0
Nick Smylie

Hi,

Well now that we have something we can do a few things with SSL enabled.

1.  Under advanced properties, set the additional headers to 'none'.

-If still nothing try number 2 on top of this.

2.  In SSL properties, you will see a 'Reencryption SNI Hostname' field.  For that put in the FQDN you are using to connect.

With that being said, if you are off-loading/reencrypting and not using content rules, L7 persistence types, ESP or WAF, there really is no added benefit of having the LM terminate the SSL and reencrypt it.  The only use case would be that if your server does not have the correct SSL cert installed on it.

Avatar
0
d.buehring

Still not working.

And yes, i just need the lb in front of my kubernetes api servers, because they get generated certificates ( managed by rancher ) and i need an endpoint with a trusted cert in front under my control.

 

 

Avatar
0
Nick Smylie

Are you able/willing give me a screenshot of your advanced properties and standard options?  You can black out anything that is vital.  I just need to see what you do or do not have configured

Avatar
0
d.buehring

Sure,

 

Avatar
0
d.buehring

Btw. i tried a simple haproxy config just to verify im not having any other problems.. and it works just fine

 

#### global #######
global
user root
group root
daemon
maxconn 2048
tune.ssl.default-dh-param 2048
ssl-server-verify none

log /dev/log local0
log /dev/log local1 notice

#### defaults #######
defaults
option httplog
log global
mode http
option http-server-close
timeout connect 10000
timeout client 60000
timeout server 60000

### HTTP(S) frontend ###
frontend k8s-api
mode http
option httplog
bind *:443 ssl crt /etc/haproxy/WILDCARD.crt
default_backend k8s

### Backend ####
backend k8s
mode http
option httplog
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100

server apiserver1 xx.xx.xx.xx:6443 check ssl

Avatar
0
d.buehring

I wanted to try this on our old hardware loadmaster, but i guess that one does not yet offer the reencrypt option :) its still at Vers:5.1-32

 

Avatar
0
Nick Smylie

Hi,

So you tried putting in a reencryption SNI hostname?  If it is still failing after that configured it could be a SSL failed negotiation on the front.  The system message logs will show those.  You can search for your client IP to to see if you are failing.

I would also turn on Subnet originating requests under standard options. 

Avatar
0
d.buehring

Yeah i tried SNI

With SNI still enabled and Subnet originating too

I get these errors in the log... as you suspected

Jun 17 13:07:45 KEMP vsslproxy: (20033) listening for REALSERVER reencrypt on yy.yy.yy.60:6443 (id:41) 
Jun 17 13:07:50 KEMP kernel: L7: ffff888073b9f100 Error handling task 0 2
Jun 17 13:08:06 KEMP sslproxy: Client xx.xx.xx.194 failed SSL negotiation: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
Jun 17 13:08:29 KEMP syslogd: last message [sslproxy: Client xx.xx.xx.194 failed SSL negotiation: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown] repeated 4 times
Jun 17 13:09:02 KEMP kernel: L7: ffff88807e98b838 Error handling task 0 2
Jun 17 13:09:02 KEMP kernel: L7: ffff888072f65778 Error handling task 0 2
Jun 17 13:09:04 KEMP kernel: L7: ffff888072885590 Error handling task 0 2
Jun 17 13:09:40 KEMP sslproxy: Client xx.xx.xx.194 failed SSL negotiation: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
Avatar
0
Nick Smylie

OK so, are using client certs?  And also do those times match up with the times you tested?  I want to make sure this not a red herring.

Avatar
0
d.buehring

Im using a KUBECONFIG which contains the ca-data (abbreviated version below) , which is the same as the cert+chain+key i use in the VS

And yes times match.

As i said earlier, its weird that i had no issues whatsoever using kubectl command until now (at least those i use regularly) and only got problems when using kapp ( a tool from https://k14s.io/)

Using kapp i can deploy to my kubernetes cluster without any errors, i only get them, when i do an "inspect" or "delete" which query lots of resources via the kubernetes api to find everything that has a matching label...

 

apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdNekNDQlJ1Z0F3SUJBZ0lRQ0VqaHpqTStvZWp6TGI5Y3JDVXFIVEFOQmdrcWhraUc5dzBCQVFzRkFEQmcKTVFzd0NRWURWUVFHRXdKVlV6RVZNQk1HQTFVRUNoTU1SR2xuYVVObGNuUWdTVzVqTVJrd0Z3WURWUVFMRXhCMwpkM2N1WkdsbmFXTmxjblF1WTI5dE1SOHdIUVlEVlFRREV4WlNZWEJwWkZOVFRDQlVURk1nVWxOQklFTkJJRWN4Ck1CNFhEVEl3TURJeE1UQXdNREF3TUZvWERUSXlNRFF4TVRFeU1EQXdNRm93SHpFZE1Cc0dBMVVFQXd3VUtpNTYKWlc1MGNtRnNaUzVsZUhCbGNuUXVaR1V3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQgpBUURBVEZnOXdOVUlJMVRVVSsxOTg4ZE1rVGN1MUtzY3VzSnMxQTFnA2RHNFOHJhQnNh........ and so on
server: https://k8s-xx.xx.xx.de
name: zentrale-dev-fqdn
contexts:
- context:
cluster: zentrale-dev-fqdn
user: zentrale-dev
name: zentrale-dev-fqdn
current-context: zentrale-dev-fqdn
kind: Config
preferences: {}
users:
- name: zentrale-dev
user:
token: kubeconfig-user-fw5bh.c-g7hnt:pwh5swxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Avatar
0
Nick Smylie

Hi,

Is 'Inspect' a HTTP request method?  I am not familiar with that one.  Can you set the service type to 'generic' please?  This is near the top under basic properties.

Avatar
0
d.buehring

No nothing HTTP specific.. just one of the commands of that tool.

I'll try setting it to generic

 

Avatar
0
d.buehring

Turns out it was the formatting of the certificate-authority-data in the KUBECONFIG file.

I still dont unterstand why most command work, and only some cause problems, but i got it working now...

 

Thanks for your effort Nick, much appreciated !!!

 

Avatar
0
mbsakho

Hi d.buehring,

I'm trying to use Kemp in front of openshift which is distribution of Kubernetes.

I'm struggling to make it work. Even the basic kubectl commands are not working.

We instructed the kemp administrator to activate the L4 layer instead of the L7 as recommended while using openshift.

It seems like you are using the L7 layer.

Any input regarding that?

Could you share your configuration with me?

Kind regards

Avatar
0
d.buehring

Hey mbsakho,

i don't have any special config, but kubectl is working normally as of now.

kapp, the tool i started this thread for, is still causing problems.. only with kemp. My last comment regarding the formatting seems to be wrong, somehow the tool was using another context from my kubeconfig.

 

anyways my config needs to be L7 because of the way Rancher creates an "authorized cluster endpoint" to access kubernetes directly without going through Rancher. Rancher creates a kubeconfig for a FQDN i define using a certificate i supplied, but that cert is not used by the kubernetes apiserver.

So i terminate SSL on the kemp using the cert i provided in rancher and set kemp to reencrypt to the apiservers

...and i use content rules to select the SubVS by URL regex, but that can be ignored if you have only one cluster

 

And the first of the four SubVS ( all the same settings-wise)