How does this DNS check actually work?

A zone transfer, also sometimes known by its (most common) opcode mnemonic AXFR, is one of many ways for a DNS system to replicate its data across a set of DNS servers. The client (slave) requests the data from a portion of the database to be transferred to it and the server (master) supplies the data from it's database. The portion of the database that is replicated is called a zone. Note that some sources refer to slave as a "secondary" server and the master as a "primary" server.

The data contained in an entire DNS zone may be sensitive in nature. Because of this, a lot of administrators restrict access so that the master only replies to it's slaves when a AXFR request is received. This way, a request from an outsider will be rejected.

So, where does the magic come in?

There is no magic in what we're doing. However, we don't give up when the master server rejects our request, instead we continue by sending the same request to the slave servers as well. We've noticed that a lot of slave servers allow zone transfers even though the master is configured to reject them.

This could be for several reasons. A lot of networks share their DNS servers with eachother in order to offer a secure and redundant DNS system. However, it seems that example.com doesn't always notice that ns.company.com is wide open and allows anyone to request a zone transfer and therefore exposing example.com's entire zone.

Okey, so you got a zone-file, now what?

First of all, a zone transfer shouldn't be a security problem. It simply contains a list of machines, their addresses, their purpose (mail server, dns server, etc) and it can also contain a lot of other information such as normal text, geographical location, operating system, list of services running on that machine, etc. Depending on how specific your nameserver is, an attacker can gather a lot of useful information about your network without actually connecting to each and every machine.

Also, sometimes people tend to add private/test systems into the nameserver, assuming it's safe since they've used a hard to guess hostname. This is really bad practice and should be avoided. If a system is private, keep it private.

Again, a zone transfer itself shouldn't be a security problem. If a machine in your network is compromised it's most likely because it wasn't updated properly or was poorly configured, not because someone managed to guess the address of it. Seeing that most attacks these days are done by bots running on compromized hosts, probing huge portions of the Internet, it doesn't really matter hostname or address you use.