How to troubleshoot Kerberos Constrained Delegation ESP with an SSO Debug trace
Scope
This article demonstrates how you can troubleshoot Kerberos Constrained Delegation by analyzing the data in both the SSOMGR Debug traces and Wireshark.
To collect the data, enable verbose Edge Security Pack (ESP) logging by navigating to:
Logging Options > System Log Files > Debug Options > SSO MGR Logging.
To view the log data navigate to:
Logging Options > System Log Files > System Message File.
For firmware older than 7.2.41, navigate to:
Logging Options > Extended Log Files > SSO MGR Audit Logs > SSO MGR > View.
1. Reverse DNS Lookup
Once a client has successfully completed front-end authentication, for example, Forms Based, NTLM, Client Certificates, or SAML, the LoadMaster requests a Kerberos ticket for the back-end server.
Depending on what destination server is chosen from the Virtual Service pool, the LoadMaster must perform a reverse Domain Name System (DNS) lookup on the server. Depending on the hostname returned from the DNS, this name is used to form the Service Principal Name (SPN).
In the example below, the IP address is 10.110.30.5 and the hostname is sharepoint1.kemptest.com. Therefore, the SPN becomes: http/sharepoint1.kemptest.com@KEMPTEST.COM.
Feb 14 09:36:21 lb100 ssomgr: #32301# baseUserName: basename=|admin|
Feb 14 09:36:21 lb100 ssomgr: #32301# >>> kcd_get_user_ticket
Feb 14 09:36:21 lb100 ssomgr: #32301# >>>resolve_destination_address: Attempt to resolve destination [10.110.30.5][2]
Feb 14 09:36:21 lb100 ssomgr: #32301# <<<resolve_destination_address: Resolved destination host name [sharepoint1.kemptest.com]
Feb 14 09:36:21 lb100 ssomgr: #32301# kcd_get_user_ticket: user=[admin@kemptest.com] [basename=[admin]
Feb 14 09:36:21 lb100 ssomgr: #32301# kcd_get_user_ticket: Destination name=[http/sharepoint1.kemptest@KEMPTEST.com]
Feb 14 09:36:21 lb100 ssomgr: #32301# kcd_get_user_ticket: kcd_ticket:0x7fde4f1f2ef0 [65536/65536]
Feb 14 09:36:21 lb100 ssomgr: #32301# >>> get_impersonator_cred_handle
Feb 14 09:36:21 lb100 ssomgr: #32301# >>> get_impersonator_cred_handle - handle=0x7fde300008c0
Note: If you see Resolved destination hostname followed by [0], it means that the LoadMaster was unable to perform a Reverse DNS Lookup. In this case, check your Reverse DNS Lookup records.
You can also test by adding host file entries to the LoadMaster by navigating to:
System configuration > Host & DNS Configuration.
2. Contact Kerberos Server
In the example below, the LoadMaster attempts to retrieve a ticket on behalf of the user Admin, using the Kerberos Trusted User Name (kempkcd) configured in the Server Side Single Sign On Configuration:
Feb 14 09:36:21 lb100 ssomgr: #32301# kcd_get_user_ticket: Get a ticket on behalf of user admin
Feb 14 09:36:21 lb100 ssomgr: #32290# kcd_get_user_ticket: Credentials aquired
Feb 14 09:36:21 lb100 ssomgr: #32290# init_accept_sec_context(): Target name: [kempkcd@kemptest.com]
Feb 14 09:36:21 lb100 ssomgr: #32290# Target mech: [{ 1 3 6 1 5 5 2 }]
Feb 14 09:36:21 lb100 ssomgr: #32290# init_accept_sec_context(): Source name: [admin@KEMPTEST.COM]
Feb 14 09:36:21 lb100 ssomgr: #32290# init_accept_sec_context(): Source mech: [{ 1 2 840 113554 1 2 2 }]
Feb 14 09:36:21 lb100 ssomgr: #32290# Attribute urn:mspac: Authenticated Complete
Feb 14 09:36:21 lb100 ssomgr: #32290# Attribute urn:mspac:logon-info Authenticated Complete
Feb 14 09:36:21 lb100 ssomgr: #32290# Attribute urn:mspac:client-info Authenticated Complete
Feb 14 09:36:21 lb100 ssomgr: #32290# Attribute urn:mspac:upn-dns-info Authenticated Complete
Feb 14 09:36:21 lb100 ssomgr: #32290# Attribute urn:mspac:server-checksum Authenticated Complete
Feb 14 09:36:21 lb100 ssomgr: #32290# Attribute urn:mspac:privsvr-checksum Authenticated Complete
Feb 14 09:36:21 lb100 ssomgr: #32290# Proxy name: [kempkcd@KEMPTEST.COM]
Feb 14 09:36:21 lb100 ssomgr: #32290# Target name: [http/sharepoint.kemptest.com@KEMPTEST.COM]
Feb 14 09:36:21 lb100 ssomgr: #32290# Delegated name: [admin@kemptest.COM]
Feb 14 09:36:21 lb100 ssomgr: #32290# Delegated mech: [{ 1 2 840 113554 1 2 2 }]
Feb 14 09:36:21 lb100 ssomgr: #32290# encode_ticket: sz=2060 ticket=0x7fde4f255ef0 len=65536 buf_sz=65536
3. Successfully Retrieved Kerberos Ticket
If successful, you should see the Negotiate authorization header with a long random string of characters.
Feb 14 09:36:21 lb100 ssomgr: #32290# <<< kcd_get_user_ticket - ret=0
Feb 14 09:36:21 lb100 ssomgr: #32290# kcd_reauth_thread: status and cookie are both OK
Feb 14 09:36:21 lb100 ssomgr: #32290# kcd_reauth_thread: up->kcd_data[Negotiate YIIF+wYJKoZIhvcSAQICAQBuggXqMIIF5qADAgEFoQMCAQ6iBwMFAAAAAACjggT+YYIE+jCCBPagAw
IBBaEMGwpTVFJFRVQuQ09NoiowKKADAgEBoSEwHxsEaHR0cBsXZXhjaGFuZ2UyMDEzLnN0cmVldC5j
b22jggSzMIIEr6ADAgESoQMCAQSiggShBIIEnag04R/bTzdmz+ZEjyNgYx9hm9gESS2aY3OwbxvT85Nrq
guM31y9HHBLRDehk/mfX13S0rEwtwyHT9EWwD5g7zNVpoeczaHN+waU74GjS7Th+WYrLHCNbaQK5
v269n/L+s1E1M53QCARCSsoTYvfZDZDbLQNFNFDVBWR2JDE0s0iljupa+KBWampg//BPV0xLiUzQdKGYtScpec4buOS2eJmiaI7fcoj+OlnMnL47m2L/5IxjNC8QM5U1GdcLuSUSPdsnIdrinKjHsfXc8msyiXyWIbe2SnVxs18tPVVul0KGnzxAdB8X17RbKb7ViDV
s3IZckF7azNqhwg3C0tBJlAys5ySQ3rXSLHBeF46Wyn8bLiGkLyVMQ6bRmb8fLPnEshakPrtuR+YuaZZ/uM
EaeP7hIBQ2JkppX4xquAszoTwXOREgTjqL6BOdHGM9UKvevRQxa9x9Rod7kW532kUTc+ayB5S8DhW2F
T79zvRhKmKXk8n2dsGLl2csiBC+bl9uMdCykFrWz+nl2DzHS8L+6tM0JjqxetmqNdQvuCMSLaBS1Upg2Z
hR/2vNh641h7NzZSBNPXcfzionp1nRkk2jPYSmq45d+/c6/Jabied5PwvUYXur91ijjJ8+cnTmWZtQjN6I4EDW3Td/g9UnG/PzEzWivQN03lAjQwRa20K+HM4w4kOADw7DcEDErSF2RZsKjHh2cTvf4/sZlC1TY38l5BNHBLWu7bwnvChkjJAkxaC/A4
Feb 14 09:36:21 lb100 ssomgr: #32290# >>> release_sso_conf(0x6b52b0), calling function(kcd_reauth_thread)
If you receive the error message shown in the example below, ensure that the delegated account is configured correctly. Also ensure that you add the server name under the Delegated tab and not, for example, the name of your SharePoint site.
Feb 14 09:36 lb100 ssomgr: #57508# kinit_domain: Getting credentials for kempkcd@kemptest.com Feb 14 09:36 lb100 ssomgr: kinit_domain: Could not initialize credentials: error -1765328360
Feb 14 09:36 lb100 ssomgr: #24663# gss_init_sec_context: Unspecified GSS failure. Minor code may provide more information
Feb 14 09:36 lb100 ssomgr: #24663# gss_init_sec_context: KDC can't fulfill requested option
4. Wireshark
If you think that your configuration is correct, perform a Transmission Control Protocol (TCP) trace and filter on your Kerberos server IP address.
To perform a TCP trace, navigate to:
Logging Options --> System Log files --> Debug Options --> TCP Dump. Select Interface = All and IP Address = <Domain Controller IP>.
When you open the Wireshark PCAP file, filter using the word Kerberos.
Pay attention to the following information:
AS-REQ: This is where the client is authenticated and a ticket-granting ticket (TGT) is retrieved. In this case, your client is the Kerberos Trusted User (kempkcd).
TGS-REQ: Here, you present your TGT to Kerberos where it is verified against its database.
KCD Error Codes and possible causes
For more information on Kerberos Constrained Delegation (KCD) error codes and possible causes, visit: Kerberos Error Codes
If you receive no errors in Wireshark, then try flushing the SSO Cache.
Logging Options > System Log Files > Debug Options > Flush SSO Cache. If it still doesn't work try rebooting the LoadMaster.
Altering Kerberos SSO Settings.
If a change of credentials is required, you will need to flush SSO Cache.
Related KBs
ESP Client Certificate SSO Debug Trace