Kerberos, LDAP, SSH, and NAT/AWS

Kerberos, and therefore LDAP with GSSAPI, has issues with servers behind NAT, or anywhere the forward DNS lookup does not match the reverse DNS lookup.

For instance, in our lab we have an OpenLDAP LDAP server:

$ dig +noall +answer ldap.itlab.stanford.edu
ldap.itlab.stanford.edu. 207    IN  CNAME   idp.itlab.stanford.edu.
idp.itlab.stanford.edu. 200 IN  A   54.189.121.117

However, since it's running on an EC2 instance, the reverse lookup returns a record that matches neither the alias (ldap.itlab.stanford.edu), nor the "real" name (idp.itlab.stanford.edu).

$ dig +noall +answer -x 54.189.121.117
117.121.189.54.in-addr.arpa. 300 IN PTR ec2-54-189-121-117.us-west-2.compute.amazonaws.com.

To further complicate matters, DNS queries return the public IP associated with the EC2 instance; internally, the instance has a different IP address on eth0:

$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 22:00:0a:51:04:32 brd ff:ff:ff:ff:ff:ff
    inet 10.81.4.50/26 brd 10.81.4.63 scope global eth0
    ...

Anything running on the instance will see the system's IP address as 10.81.4.50; anything outside AWS will see the IP address as 54.189.121.117.

Fixing the backend

The first thing we need to fix is how processes on the EC2 instance identify the system. This can be handled with entries in /etc/hosts:

127.0.0.1       localhost
::1             ip6-localhost ip6-loopback
fe00::0         ip6-localnet
ff00::0         ip6-mcastprefix
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
54.189.121.117  idp.itlab.stanford.edu ldap.itlab.stanford.edu
10.81.4.50      ldap.itlab.stanford.edu idp.itlab.stanford.edu

These entires can be added or updated when the system boots, using the EC2 metadata:

$ echo $(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
10.81.4.50
$ echo $(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)
54.189.121.117

Fixing SSH

SSH works if the client acquires address-less tickets (Stanford's default), and if the client is also configured to not use reverse DNS lookups to determine the service name. For both Heimdal and MIT libaries, these settings are in /etc/krb5.conf in the libdefaults section, as noaddresses and rdns, respectively:

[libdefaults]

  default_realm = itlab.stanford.edu
  noaddresses = true
  rdns = false

If you're using MIT libraries and OpenLDAP clients, you also need this:

  dns_canonicalize_hostname = false

These options are all describe in the MIT Kerberos Docs

SSH with GSSAPI now works as expected (using idp.itlab.stanford.edu, since that's the "real" name of the system):

$ssh -v admin@idp.itlab.stanford.edu
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Connecting to idp.itlab.stanford.edu [54.189.121.117] port 22.
debug1: Connection established.
...
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive
...
debug1: Next authentication method: gssapi-with-mic
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to idp.itlab.stanford.edu ([54.189.121.117]:22).
...
Linux idp.itlab.stanford.edu 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64

admin@idp:~$

Fixing LDAP

Without modifications, attempts to query the ldap server result in errors like this:

$ ldapsearch -Q -LLL uid=swl cn
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
    additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
    Minor code may provide more information (Server not found in Kerberos database)

With the Heimdal libraries, the krb5.conf settings do not appear to affect the OpenLDAP ldapsearch client. However, with the MIT libraries, dns_canonicalize_hostname must be set to false.

Additionally, ldapsearch uses a SASL library to use GSSAPI, and that library handles service/server name canonicalization internally. The SASL library's canonicalization can be disabled in ldap.conf with this line:

SASL_NOCANON on

Now GSSAPI LDAP binding works:

$ ldapsearch -Q -LLL uid=swl cn
dn: uid=swl,cn=people,dc=itlab,dc=stanford,dc=edu
cn: Scotty Logan

What about Macs?

MacOS X comes with the Heimdal clients and libraries, and appears to have canonicalization disabled.

MacOS X also includes the OpenLDAP clients and libraries, but does not appear to support SASL_NOCANON in /etc/openldap/ldap.conf. Instead the -N option must be used with ldapsearch (this option also works on Linux).

2 comments

  1. To be clear, fixing LDAP search entails disabling the Kerberos reverse DNS lookup on the *client* side. Is that correct?

    1. Correct – changes are required to both the Kerberos and LDAP client configuration.

Comments are closed.