slow, delayed, hanging ssh connect from one Centos 6 to an other Centos 6

I had this really annoying delay when ssh’ing from one centos 6.4 instance to an other centos 6.4 instance. The ‘usual’

UseDNS no

in /etc/ssh/sshd_config on the server side didn’t resolve the issue. Both the client and the server had proper dns and reverse dns records as well.
The thing that kind of put me off was that I had a situation like this:

$ ssh hostname ( delayed, slow connection)
$ ssh FQDN ( delayed, slow connection) 
$ ssh  IP address ( fast, normal connection ) 

So the symptoms was as if there really _was_ some dns problems …
I found some posts via google pointing out disabling GSSAPIAuthentication could help, which it didn’t in my case when I did that on the server side. But I did notice that disabling GSSAPIAuthentication on the client side did help a bit, but the connection was still delaying a bit compared to using ip only.
To me it still looked a bit like dns problems…
The man ssh_config page gave me a hint regarding AddressFamily:

   AddressFamily
             Specifies which address family to use when connecting.  Valid arguments are “any”, “inet”
             (use IPv4 only), or “inet6” (use IPv6 only).

Since we don’t have any working ipv6 dns servers or network, setting AddressFamily to inet + disabling GSSAPIAuthentication on both the client and server completely solved my problem.
We don’t use GSSAPIAuthentication anyway, so I don’t have any problems with disabling that.

so,
/etc/ssh/ssh_config

[..]
        GSSAPIAuthentication no
# ipv4 only 
        AddressFamily inet
[...]

and /etc/ssh/sshd_config

[...]
        GSSAPIAuthentication no
[...]

It’s with IPv6 as it is with saving the environment:
you know you have to do something about it, it’s just inconvenient to fix it right now, and it still works the old way so…