Ensure printed messages are flushed immediately (fixes #59)

Also send error messages to stderr

.gitignore

@@ -1,3 +1,6 @@
+# virtualenv
+.venv/
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

.gitmodules

@@ -1,7 +0,0 @@
-[submodule "dnspython"]
-	path = dnspython
-	url = https://github.com/farrokhi/dnspython.git
-	branch = python3
-[submodule "cymruwhois"]
-	path = cymruwhois
-	url = https://github.com/JustinAzoff/python-cymruwhois.git

.travis.yml

@@ -1,5 +1,18 @@
 language: python
-python: "3.4"
-install: pip install  --process-dependency-links dnsdiag
+sudo: false
+install: "pip install -r requirements.txt"
 script: nosetests dnstraceroute.py
-
+matrix:
+  fast_finish: true
+  include:
+    - python: "3.3"
+    - python: "3.4"
+    - python: "3.5"
+    - python: "3.6"
+    - python: "3.7"
+      dist: xenial
+      sudo: true
+    - python: "3.8-dev"
+      dist: xenial
+      sudo: true
+    - python: "pypy3"

MANIFEST.in

@@ -1,2 +1 @@
-include LICENSE README.md TODO.md public-servers.txt
-include cymruwhois/*.py
+include LICENSE README.md TODO.md public-servers.txt public-v4.txt rootservers.txt

README.md

@@ -1,4 +1,4 @@
-[![Build Status](https://travis-ci.org/farrokhi/dnsdiag.svg)](https://travis-ci.org/farrokhi/dnsdiag) [![PyPI](https://img.shields.io/pypi/v/dnsdiag.svg?maxAge=8600)](https://pypi.python.org/pypi/dnsdiag/) [![PyPI](https://img.shields.io/pypi/l/dnsdiag.svg?maxAge=8600)]() [![PyPI](https://img.shields.io/pypi/pyversions/dnsdiag.svg?maxAge=8600)]() [![GitHub stars](https://img.shields.io/github/stars/farrokhi/dnsdiag.svg?style=social&label=Star&maxAge=8600)](https://github.com/farrokhi/dnsdiag/stargazers)
+[![Build Status](https://travis-ci.org/farrokhi/dnsdiag.svg)](https://travis-ci.org/farrokhi/dnsdiag) [![PyPI](https://img.shields.io/pypi/v/dnsdiag.svg?maxAge=8600)](https://pypi.python.org/pypi/dnsdiag/) [![PyPI](https://img.shields.io/pypi/l/dnsdiag.svg?maxAge=8600)]() [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Ffarrokhi%2Fdnsdiag.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Ffarrokhi%2Fdnsdiag?ref=badge_shield) [![PyPI](https://img.shields.io/pypi/pyversions/dnsdiag.svg?maxAge=8600)]() [![GitHub stars](https://img.shields.io/github/stars/farrokhi/dnsdiag.svg?style=social&label=Star&maxAge=8600)](https://github.com/farrokhi/dnsdiag/stargazers)
 
 DNS Diagnostics and Performance Measurement Tools
 ==================================================
@@ -28,40 +28,45 @@ of view.
 # prerequisites
 This script requires python3 as well as latest
 [dnspython](http://www.dnspython.org/) and
-[cymruwhois](https://pythonhosted.org/cymruwhois/). Please note that
-"dnstraceroute" requires a modified version of dnspython module. All required
-third-party modules are included as GIT submodules. You just need to run `git
-submodule update --init` and project directory to pull the required code.
+[cymruwhois](https://pythonhosted.org/cymruwhois/).
 
 # installation
 
+There are several ways that you can use this toolset. However using the sourcecode is always recommended.
+
+## From Source Code
+
 1. You can checkout this git repo and its submodules
 
 ```
 git clone https://github.com/farrokhi/dnsdiag.git
 cd dnsdiag
-git submodule update --init
+pip3 install -r requirements.txt
 ```
 
 2. You can alternatively install the package using pip:
 
 ```
-pip3 install --process-dependency-links dnsdiag
+pip3 install dnsdiag
 ```
 
+## From Binary
+
+From time to time, binary version will be released for Windows, Mac OS X and Linux platforms. You can grab the latest release from [releases page](https://github.com/farrokhi/dnsdiag/releases).
+
 # dnsping
 dnsping pings a DNS resolver by sending an arbitrary DNS query for given number
 of times:
 ```
-% ./dnsping.py -c 3 -s 8.8.8.8 -t MX wikipedia.org
-dnsping.py DNS: 8.8.8.8:53, hostname: wikipedia.org, rdatatype: MX
-101 bytes from 8.8.8.8: seq=0   time=262.896 ms
-101 bytes from 8.8.8.8: seq=1   time=305.608 ms
-101 bytes from 8.8.8.8: seq=2   time=307.221 ms
+% ./dnsping.py -c 3 -t AAAA -s 8.8.8.8 dnsdiag.org
+dnsping.py DNS: 8.8.8.8:53, hostname: dnsdiag.org, rdatatype: AAAA
+4 bytes from 8.8.8.8: seq=0   time=123.509 ms
+4 bytes from 8.8.8.8: seq=1   time=115.726 ms
+4 bytes from 8.8.8.8: seq=2   time=117.351 ms
 
 --- 8.8.8.8 dnsping statistics ---
 3 requests transmitted, 3 responses received,   0% lost
-min=262.896 ms, avg=291.908 ms, max=307.221 ms, stddev=25.138 ms
+min=115.726 ms, avg=118.862 ms, max=123.509 ms, stddev=4.105 ms
 ```
 This script calculates minimum, maximum and average response time as well as
 jitter (stddev)
@@ -73,23 +78,16 @@ it to your actual network traceroute and make sure your DNS traffic is not
 routed to any unwanted path.
 
 ```
-% ./dnstraceroute.py --expert -s 8.8.4.4 yahoo.com
-dnstraceroute.py DNS: 8.8.4.4:53, hostname: yahoo.com, rdatatype: A
-1       204.109.58.53 (204.109.58.53) 1 ms
-2       208.79.80.5 (208.79.80.5) 1 ms
-3       162.223.13.177 (162.223.13.177) 1 ms
-4       208.79.80.254 (208.79.80.254) 7 ms
-5       eqixva-google-gige.google.com (206.126.236.21) 7 ms
-6       209.85.242.142 (209.85.242.142) 7 ms
-7       72.14.236.148 (72.14.236.148) 8 ms
-8       209.85.250.70 (209.85.250.70) 16 ms
-9       74.125.37.222 (74.125.37.222) 16 ms
-10       *
-11      google-public-dns-b.google.com (8.8.4.4) 15 ms
+% ./dnstraceroute.py --expert -C -t A -s 8.8.4.4 facebook.com
+dnstraceroute.py DNS: 8.8.4.4:53, hostname: facebook.com, rdatatype: A
+1	192.168.0.1 (192.168.0.1) 1 ms
+2	192.168.28.177 (192.168.28.177) 4 ms
+3	192.168.0.1 (192.168.0.1) 693 ms
+4	172.19.4.17 (172.19.4.17) 3 ms
+5	google-public-dns-b.google.com (8.8.4.4) 8 ms
 
 === Expert Hints ===
- [*] public DNS server is next to an invisible hop (probably a firewall)
-
+ [*] public DNS server is next to a private IP address (possible hijacking)
 ```
 
 Using `--expert` will instruct dnstraceroute to print expert hints (such as warnings of possible DNS traffic hijacking).
@@ -99,17 +97,21 @@ dnseval is a bulk ping utility that sends an arbitrary DNS query to a give list
 of DNS servers. This script is meant for comparing response time of multiple
 DNS servers at once:
 ```
-% ./dnseval.py wikipedia.org
-server             avg(ms)     min(ms)     max(ms)     stddev(ms)  lost(%)
---------------------------------------------------------------------------
-4.2.2.1            151.067     131.270     221.742     28.643      %10
-4.2.2.2            142.175     132.921     178.133     13.348      %0
-64.6.64.6          133.047     109.145     162.938     20.609      %0
-64.6.65.6          377.270     97.669      661.471     172.717     %0
-8.8.4.4            389.048     294.581     511.134     67.953      %0
-8.8.8.8            0.000       0.000       0.000       0.000       %100
-208.67.222.222     179.068     135.975     258.582     50.681      %0
-208.67.220.220     137.817     135.822     140.113     1.504       %0
+% ./dnseval.py -t AAAA -f public-servers.txt -c10 yahoo.com
+server           avg(ms)     min(ms)     max(ms)     stddev(ms)  lost(%)  ttl     flags
+------------------------------------------------------------------------------------------------------
+8.8.8.8          270.791     215.599     307.498     40.630      %0       298     QR -- -- RD RA -- --
+8.8.4.4          222.955     171.753     307.251     60.481      %10      291     QR -- -- RD RA -- --
+ns.ripe.net      174.855     160.949     187.458     10.099      %0       289     QR -- -- RD RA -- --
+4.2.2.1          172.798     163.892     189.918     7.823       %0       287     QR -- -- RD RA -- --
+4.2.2.2          178.594     169.158     184.696     5.067       %0       285     QR -- -- RD RA -- --
+4.2.2.3          153.574     138.509     173.439     12.015      %0       284     QR -- -- RD RA -- --
+4.2.2.4          153.182     141.023     162.323     6.700       %0       282     QR -- -- RD RA -- --
+4.2.2.5          154.840     141.557     163.889     7.195       %0       281     QR -- -- RD RA -- --
+209.244.0.3      156.270     147.320     161.365     3.958       %0       279     QR -- -- RD RA -- --
+209.244.0.4      159.329     151.283     163.726     3.958       %0       278     QR -- -- RD RA -- --
+195.46.39.39     171.098     163.612     181.147     5.067       %0       276     QR -- -- RD RA -- --
+195.46.39.40     175.335     160.920     185.618     8.726       %0       274     QR -- -- RD RA -- --
 ```
 
 ### Author
@@ -117,7 +119,7 @@ server             avg(ms)     min(ms)     max(ms)     stddev(ms)  lost(%)
 Babak Farrokhi 
 
 - twitter: [@farrokhi](https://twitter.com/farrokhi)
-- github: [/farrokhi](https://github.com/farrokhi/)
+- github: [github.com/farrokhi](https://github.com/farrokhi/)
 - website: [farrokhi.net](https://farrokhi.net/)
 
 
@@ -125,8 +127,4 @@ Babak Farrokhi
 
 dnsdiag is released under a 2 clause BSD license.
 
-### Credits
-
-- [@rthalley](https://github.com/rthalley) for invaluable [dnspython](https://github.com/rthalley/dnspython) library
-- [@JustinAzoff](https://github.com/JustinAzoff) for [python-cymruwhois](https://github.com/JustinAzoff/python-cymruwhois) library
-- [@bortzmeyer](https://github.com/bortzmeyer) for his feedback and patches
+[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Ffarrokhi%2Fdnsdiag.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Ffarrokhi%2Fdnsdiag?ref=badge_large)

TODO.md

@@ -1,6 +1,3 @@
 # TODO
 
-- implement basic DNS client functionality into dnstraceroute to eliminate need
-  of modified dnspython module
-- add support for python 2.x for the conservatives and faint-hearted ;-)
 - dnsfingerprint.py tool to fingerprint DNS servers

cymruwhois

@@ -1 +0,0 @@
-Subproject commit a34543335cbef02b1b615e774ce5b6187afb0cc2

dns

@@ -1 +0,0 @@
-dnspython/dns

dnseval.py

@@ -25,35 +25,61 @@
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 
+import os
+
 import getopt
 import ipaddress
-import os
 import signal
 import socket
 import sys
 import time
+import random
+import string
 from statistics import stdev
 
 import dns.rdatatype
 import dns.resolver
 
-__VERSION__ = 1.0
-__PROGNAME__ = os.path.basename(sys.argv[0])
+__author__ = 'Babak Farrokhi (babak@farrokhi.net)'
+__license__ = 'BSD'
+__version__ = "1.6.4"
+__progname__ = os.path.basename(sys.argv[0])
 shutdown = False
 
 resolvers = dns.resolver.get_default_resolver().nameservers
 
 
+class Colors(object):
+    N = '\033[m'  # native
+    R = '\033[31m'  # red
+    G = '\033[32m'  # green
+    O = '\033[33m'  # orange
+    B = '\033[34m'  # blue
+
+    def __init__(self, mode):
+        if not mode:
+            self.N = ''
+            self.R = ''
+            self.G = ''
+            self.O = ''
+            self.B = ''
+
+
 def usage():
-    print("""%s version %1.1f
+    print("""%s version %s
 
 usage: %s [-h] [-f server-list] [-c count] [-t type] [-w wait] hostname
-  -h  --help      show this help
-  -f  --file      dns server list to use (default: system resolvers)
-  -c  --count     number of requests to send (default: 10)
-  -w  --wait      maximum wait time for a reply (default: 5)
+  -h  --help        Show this help
+  -f  --file        DNS server list to use (default: system resolvers)
+  -c  --count       Number of requests to send (default: 10)
+  -m  --cache-miss  Force cache miss measurement by prepending a random hostname
+  -w  --wait        Maximum wait time for a reply (default: 2)
   -t  --type        DNS request record type (default: A)
-""" % (__PROGNAME__, __VERSION__, __PROGNAME__))
+  -T  --tcp         Use TCP instead of UDP
+  -e  --edns        Disable EDNS0 (Default: Enabled)
+  -C  --color       Print colorful output
+  -v  --verbose     Print actual dns response
+""" % (__progname__, __version__, __progname__))
     sys.exit()
 
 
@@ -69,41 +95,108 @@ def maxlen(names):
     return len(sn[-1])
 
 
-def dnsping(host, server, dnsrecord, timeout, count):
+def _order_flags(table):
+    return sorted(table.items(), reverse=True)
+
+
+def flags_to_text(flags):
+    # Standard DNS flags
+
+    QR = 0x8000
+    AA = 0x0400
+    TC = 0x0200
+    RD = 0x0100
+    RA = 0x0080
+    AD = 0x0020
+    CD = 0x0010
+
+    # EDNS flags
+
+    DO = 0x8000
+
+    _by_text = {
+        'QR': QR,
+        'AA': AA,
+        'TC': TC,
+        'RD': RD,
+        'RA': RA,
+        'AD': AD,
+        'CD': CD
+    }
+
+    _by_value = dict([(y, x) for x, y in _by_text.items()])
+    _flags_order = _order_flags(_by_value)
+
+    _by_value = dict([(y, x) for x, y in _by_text.items()])
+
+    order = sorted(_by_value.items(), reverse=True)
+    text_flags = []
+    for k, v in order:
+        if flags & k != 0:
+            text_flags.append(v)
+        else:
+            text_flags.append('--')
+
+    return ' '.join(text_flags)
+
+
+def random_string(min_length=5, max_length=10):
+    char_set = string.ascii_letters + string.digits
+    length = random.randint(min_length, max_length)
+    return ''.join(map(lambda unused: random.choice(char_set), range(length)))
+
+
+def dnsping(host, server, dnsrecord, timeout, count, use_tcp=False, use_edns=False, force_miss=False):
     resolver = dns.resolver.Resolver()
     resolver.nameservers = [server]
     resolver.timeout = timeout
     resolver.lifetime = timeout
     resolver.retry_servfail = 0
+    flags = 0
+    ttl = None
+    answers = None
+    if use_edns:
+        resolver.use_edns(edns=0, payload=8192, ednsflags=dns.flags.edns_from_text('DO'))
 
-    response_time = []
+    response_times = []
     i = 0
 
     for i in range(count):
-        if shutdown:
+        if shutdown:  # user pressed CTRL+C
             break
         try:
-            stime = time.time()
-            answers = resolver.query(host, dnsrecord)  # todo: response validation in future
-            etime = time.time()
+            if force_miss:
+                fqdn = "_dnsdiag_%s_.%s" % (random_string(), host)
+            else:
+                fqdn = host
+
+            stime = time.perf_counter()
+            answers = resolver.query(fqdn, dnsrecord, tcp=use_tcp,
+                                     raise_on_no_answer=False)  # todo: response validation in future
+
         except (dns.resolver.NoNameservers, dns.resolver.NoAnswer):
             break
         except dns.resolver.Timeout:
             pass
-        else:
+        except dns.resolver.NXDOMAIN:
+            etime = time.perf_counter()
+            if force_miss:
                 elapsed = (etime - stime) * 1000  # convert to milliseconds
-            response_time.append(elapsed)
+                response_times.append(elapsed)
+        else:
+            elapsed = answers.response.time * 1000  # convert to milliseconds
+            response_times.append(elapsed)
 
     r_sent = i + 1
-    r_received = len(response_time)
+    r_received = len(response_times)
     r_lost = r_sent - r_received
     r_lost_percent = (100 * r_lost) / r_sent
-    if response_time:
-        r_min = min(response_time)
-        r_max = max(response_time)
-        r_avg = sum(response_time) / r_received
-        if len(response_time) > 1:
-            r_stddev = stdev(response_time)
+    if response_times:
+        r_min = min(response_times)
+        r_max = max(response_times)
+        r_avg = sum(response_times) / r_received
+        if len(response_times) > 1:
+            r_stddev = stdev(response_times)
         else:
             r_stddev = 0
     else:
@@ -112,29 +205,41 @@ def dnsping(host, server, dnsrecord, timeout, count):
         r_avg = 0
         r_stddev = 0
 
-    return (server, r_avg, r_min, r_max, r_stddev, r_lost_percent)
+    if answers is not None:
+        flags = answers.response.flags
+        if len(answers.response.answer) > 0:
+            ttl = answers.response.answer[0].ttl
+
+    return server, r_avg, r_min, r_max, r_stddev, r_lost_percent, flags, ttl, answers
 
 
 def main():
     try:
         signal.signal(signal.SIGTSTP, signal.SIG_IGN)  # ignore CTRL+Z
-        signal.signal(signal.SIGINT, signal_handler)  # ignore CTRL+C
-    except AttributeError:  # Some systems may not support all signals
+        signal.signal(signal.SIGINT, signal_handler)  # catch CTRL+C
+    except AttributeError:  # Some systems (e.g. Windows) may not support all signals
         pass
 
     if len(sys.argv) == 1:
         usage()
 
+    # defaults
     dnsrecord = 'A'
     count = 10
-    waittime = 5
+    waittime = 2
     inputfilename = None
     fromfile = False
+    use_tcp = False
+    use_edns = True
+    force_miss = False
+    verbose = False
+    color_mode = False
     hostname = 'wikipedia.org'
 
     try:
-        opts, args = getopt.getopt(sys.argv[1:], "hf:c:t:w:",
-                                   ["help", "file=", "count=", "type=", "wait="])
+        opts, args = getopt.getopt(sys.argv[1:], "hf:c:t:w:TevCm",
+                                   ["help", "file=", "count=", "type=", "wait=", "tcp", "edns", "verbose", "color",
+                                    "force-miss"])
     except getopt.GetoptError as err:
         print(err)
         usage()
@@ -154,50 +259,111 @@ def main():
             fromfile = True
         elif o in ("-w", "--wait"):
             waittime = int(a)
+        elif o in ("-m", "--cache-miss"):
+            force_miss = True
         elif o in ("-t", "--type"):
             dnsrecord = a
+        elif o in ("-T", "--tcp"):
+            use_tcp = True
+        elif o in ("-e", "--edns"):
+            use_edns = False
+        elif o in ("-C", "--color"):
+            color_mode = True
+        elif o in ("-v", "--verbose"):
+            verbose = True
         else:
             print("Invalid option: %s" % o)
             usage()
 
+    color = Colors(color_mode)
+
     try:
         if fromfile:
+            if inputfilename == '-':
+                # read from stdin
+                with sys.stdin as flist:
+                    f = flist.read().splitlines()
+            else:
+                try:
                     with open(inputfilename, 'rt') as flist:
                         f = flist.read().splitlines()
+                except Exception as e:
+                    print(e)
+                    sys.exit(1)
         else:
             f = resolvers
         if len(f) == 0:
             print("No nameserver specified")
 
-        f = [name.strip() for name in f]
+        f = [name.strip() for name in f]  # remove annoying blanks
+        f = [x for x in f if not x.startswith('#') and len(x)]  # remove comments and empty entries
+
         width = maxlen(f)
         blanks = (width - 5) * ' '
-        print('server ', blanks, ' avg(ms)     min(ms)     max(ms)     stddev(ms)  lost(%)')
-        print((60 + width) * '-')
+        print('server ', blanks, ' avg(ms)     min(ms)     max(ms)     stddev(ms)  lost(%)  ttl        flags')
+        print((93 + width) * '-')
         for server in f:
             # check if we have a valid dns server address
+            if server.lstrip() == '':  # deal with empty lines
+                continue
+            server = server.replace(' ', '')
             try:
                 ipaddress.ip_address(server)
             except ValueError:  # so it is not a valid IPv4 or IPv6 address, so try to resolve host name
                 try:
-                    s = socket.getaddrinfo(server, port=None)[1][4][0]
+                    resolver = socket.getaddrinfo(server, port=None)[1][4][0]
                 except OSError:
                     print('Error: cannot resolve hostname:', server)
-                    s = None
+                    resolver = None
+                except:
+                    pass
             else:
-                s = server
+                resolver = server
 
-            if not s:
+            if not resolver:
                 continue
-            (s, r_avg, r_min, r_max, r_stddev, r_lost_percent) = dnsping(hostname, s, dnsrecord, waittime,
-                                                                         count)
 
-            s = server.ljust(width + 1)
-            print("%s    %-8.3f    %-8.3f    %-8.3f    %-8.3f    %%%d" % (
-                s, r_avg, r_min, r_max, r_stddev, r_lost_percent), flush=True)
+            try:
+                (resolver, r_avg, r_min, r_max, r_stddev, r_lost_percent, flags, ttl, answers) = dnsping(
+                    hostname,
+                    resolver,
+                    dnsrecord,
+                    waittime,
+                    count,
+                    use_tcp=use_tcp,
+                    use_edns=use_edns,
+                    force_miss=force_miss
+                )
+            except dns.resolver.NXDOMAIN:
+                print('%-15s  NXDOMAIN' % server)
+                continue
+            except Exception as e:
+                print('%s: %s' % (server, e))
+                continue
+
+            resolver = server.ljust(width + 1)
+            text_flags = flags_to_text(flags)
+
+            s_ttl = str(ttl)
+            if s_ttl == "None":
+                s_ttl = "N/A"
+
+            if r_lost_percent > 0:
+                l_color = color.O
+            else:
+                l_color = color.N
+            print("%s    %-8.3f    %-8.3f    %-8.3f    %-8.3f    %s%%%-3d%s     %-8s  %21s" % (
+                resolver, r_avg, r_min, r_max, r_stddev, l_color, r_lost_percent, color.N, s_ttl, text_flags),
+                  flush=True)
+            if verbose and hasattr(answers, 'response'):
+                ans_index = 1
+                for answer in answers.response.answer:
+                    print("Answer %d [ %s%s%s ]" % (ans_index, color.G, answer, color.N))
+                    ans_index += 1
+                print("")
 
     except Exception as e:
-        print('error: %s' % e)
+        print('%s: %s' % (server, e))
         sys.exit(1)
 
 

dnsping.py

@@ -34,17 +34,20 @@ import sys
 import time
 from statistics import stdev
 
+import dns.flags
 import dns.rdatatype
 import dns.resolver
 
-__VERSION__ = 1.1
-__PROGNAME__ = os.path.basename(sys.argv[0])
+__author__ = 'Babak Farrokhi (babak@farrokhi.net)'
+__license__ = 'BSD'
+__version__ = "1.6.4"
+__progname__ = os.path.basename(sys.argv[0])
 shutdown = False
 
 
 def usage():
-    print("""%s version %1.1f
-usage: %s [-h] [-q] [-v] [-s server] [-p port] [-P port] [-S address] [-c count] [-t type] [-w wait] hostname
+    print("""%s version %s
+usage: %s [-ehqv] [-s server] [-p port] [-P port] [-S address] [-c count] [-t type] [-w wait] hostname
   -h  --help      Show this help
   -q  --quiet     Quiet
   -v  --verbose   Print actual dns response
@@ -55,10 +58,12 @@ usage: %s [-h] [-q] [-v] [-s server] [-p port] [-P port] [-S address] [-c count]
   -6  --ipv6      Use IPv6 as default network protocol
   -P  --srcport   Query source port number (default: 0)
   -S  --srcip     Query source IP address (default: default interface address)
-  -c  --count     Number of requests to send (default: 10)
-  -w  --wait      Maximum wait time for a reply (default: 5)
+  -c  --count     Number of requests to send (default: 10, 0 for infinity)
+  -w  --wait      Maximum wait time for a reply (default: 2 seconds)
+  -i  --interval  Time between each request (default: 1 seconds)
   -t  --type      DNS request record type (default: A)
-""" % (__PROGNAME__, __VERSION__, __PROGNAME__))
+  -e  --edns      Disable EDNS0 (default: Enabled)
+""" % (__progname__, __version__, __progname__))
     sys.exit(0)
 
 
@@ -82,7 +87,8 @@ def main():
     # defaults
     dnsrecord = 'A'
     count = 10
-    timeout = 5
+    timeout = 2
+    interval = 1
     quiet = False
     verbose = False
     dnsserver = dns.resolver.get_default_resolver().nameservers[0]
@@ -90,16 +96,17 @@ def main():
     src_port = 0
     src_ip = None
     use_tcp = False
+    use_edns = True
     af = socket.AF_INET
     hostname = 'wikipedia.org'
 
     try:
-        opts, args = getopt.getopt(sys.argv[1:], "qhc:s:t:w:vp:P:S:T46",
-                                   ["help", "output=", "count=", "server=", "quiet", "type=", "wait=", "verbose",
-                                    "port", "dstport=", "srcip=", "tcp", "ipv4", "ipv6"])
+        opts, args = getopt.getopt(sys.argv[1:], "qhc:s:t:w:i:vp:P:S:T46e",
+                                   ["help", "count=", "server=", "quiet", "type=", "wait=", "interval=", "verbose",
+                                    "port=", "srcip=", "tcp", "ipv4", "ipv6", "srcport=", "edns"])
     except getopt.GetoptError as err:
         # print help information and exit:
-        print(err)  # will print something like "option -a not recognized"
+        print(err, file=sys.stderr)  # will print something like "option -a not recognized"
         usage()
 
     if args and len(args) == 1:
@@ -111,7 +118,7 @@ def main():
         if o in ("-h", "--help"):
             usage()
         elif o in ("-c", "--count"):
-            count = int(a)
+            count = abs(int(a))
         elif o in ("-v", "--verbose"):
             verbose = True
         elif o in ("-s", "--server"):
@@ -123,6 +130,8 @@ def main():
             verbose = False
         elif o in ("-w", "--wait"):
             timeout = int(a)
+        elif o in ("-i", "--interval"):
+            interval = int(a)
         elif o in ("-t", "--type"):
             dnsrecord = a
         elif o in ("-T", "--tcp"):
@@ -131,10 +140,12 @@ def main():
             af = socket.AF_INET
         elif o in ("-6", "--ipv6"):
             af = socket.AF_INET6
+        elif o in ("-e", "--edns"):
+            use_edns = False
         elif o in ("-P", "--srcport"):
             src_port = int(a)
             if src_port < 1024:
-                print("WARNING: Source ports below 1024 are only available to superuser")
+                print("WARNING: Source ports below 1024 are only available to superuser", flush=True)
         elif o in ("-S", "--srcip"):
             src_ip = a
         else:
@@ -147,7 +158,7 @@ def main():
         try:
             dnsserver = socket.getaddrinfo(dnsserver, port=None, family=af)[1][4][0]
         except OSError:
-            print('Error: cannot resolve hostname:', dnsserver)
+            print('Error: cannot resolve hostname:', dnsserver, file=sys.stderr, flush=True)
             sys.exit(1)
 
     resolver = dns.resolver.Resolver()
@@ -157,47 +168,61 @@ def main():
     resolver.port = dst_port
     resolver.retry_servfail = 0
 
+    if use_edns:
+        resolver.use_edns(edns=0, payload=8192, ednsflags=dns.flags.edns_from_text('DO'))
+
     response_time = []
     i = 0
 
-    print("%s DNS: %s:%d, hostname: %s, rdatatype: %s" % (__PROGNAME__, dnsserver, dst_port, hostname, dnsrecord))
+    print("%s DNS: %s:%d, hostname: %s, rdatatype: %s" % (__progname__, dnsserver, dst_port, hostname, dnsrecord),
+          flush=True)
 
-    for i in range(count):
-        if shutdown:
+    while not shutdown:
+
+        if 0 < count <= i:
             break
+        else:
+            i += 1
+
         try:
-            stime = time.time()
-            answers = resolver.query(hostname, dnsrecord, source_port=src_port, source=src_ip, tcp=use_tcp, af=af)
-            etime = time.time()
+            stime = time.perf_counter()
+            answers = resolver.query(hostname, dnsrecord, source_port=src_port, source=src_ip, tcp=use_tcp,
+                                     raise_on_no_answer=False)
+            etime = time.perf_counter()
         except dns.resolver.NoNameservers as e:
             if not quiet:
-                print("No response to dns request")
+                print("No response to dns request", file=sys.stderr, flush=True)
                 if verbose:
-                    print("error:", e)
+                    print("error:", e, file=sys.stderr, flush=True)
             sys.exit(1)
         except dns.resolver.NXDOMAIN as e:
             if not quiet:
-                print("Hostname does not exist")
+                print("Hostname does not exist", file=sys.stderr, flush=True)
             if verbose:
-                print("Error:", e)
+                print("Error:", e, file=sys.stderr, flush=True)
             sys.exit(1)
         except dns.resolver.Timeout:
             if not quiet:
-                print("Request timeout")
+                print("Request timeout", flush=True)
             pass
         except dns.resolver.NoAnswer:
             if not quiet:
-                print("No answer")
+                print("No answer", flush=True)
             pass
         else:
-            elapsed = (etime - stime) * 1000  # convert to milliseconds
+            elapsed = answers.response.time * 1000  # convert to milliseconds
             response_time.append(elapsed)
             if not quiet:
                 print(
-                    "%d bytes from %s: seq=%-3d time=%3.3f ms" % (
-                        len(str(answers.rrset)), dnsserver, i, elapsed))
+                    "%d bytes from %s: seq=%-3d time=%.3f ms" % (
+                        len(str(answers.rrset)), dnsserver, i, elapsed), flush=True)
             if verbose:
-                print(answers.rrset)
+                print(answers.rrset, flush=True)
+                print("flags:", dns.flags.to_text(answers.response.flags), flush=True)
+
+            time_to_next = (stime + interval) - etime
+            if time_to_next > 0:
+                time.sleep(time_to_next)
 
     r_sent = i + 1
     r_received = len(response_time)
@@ -217,9 +242,10 @@ def main():
         r_avg = 0
         r_stddev = 0
 
-    print('\n--- %s dnsping statistics ---' % dnsserver)
-    print('%d requests transmitted, %d responses received, %3.0f%% lost' % (r_sent, r_received, r_lost_percent))
-    print('min=%3.3f ms, avg=%3.3f ms, max=%3.3f ms, stddev=%3.3f ms' % (r_min, r_avg, r_max, r_stddev))
+    print('\n--- %s dnsping statistics ---' % dnsserver, flush=True)
+    print('%d requests transmitted, %d responses received, %.0f%% lost' % (r_sent, r_received, r_lost_percent),
+          flush=True)
+    print('min=%.3f ms, avg=%.3f ms, max=%.3f ms, stddev=%.3f ms' % (r_min, r_avg, r_max, r_stddev), flush=True)
 
 
 if __name__ == '__main__':

dnspython

@@ -1 +0,0 @@
-Subproject commit bc7445dcd5fe3917f96cac07c7b1f2f645699204

dnstraceroute.py

@@ -35,13 +35,35 @@ import socket
 import sys
 import time
 
+import dns.query
 import dns.rdatatype
 import dns.resolver
-from cymruwhois import cymruwhois
 
+import cymruwhois
+
+# Global Variables
 __author__ = 'Babak Farrokhi (babak@farrokhi.net)'
 __license__ = 'BSD'
-__version__ = 1.3
+__version__ = "1.6.4"
+_ttl = None
+quiet = False
+whois_cache = {}
+shutdown = False
+
+# Constants
+__progname__ = os.path.basename(sys.argv[0])
+WHOIS_CACHE = 'whois.cache'
+
+
+class CustomSocket(socket.socket):
+    def __init__(self, *args, **kwargs):
+        super(CustomSocket, self).__init__(*args, **kwargs)
+
+    def sendto(self, *args, **kwargs):
+        global _ttl
+        if _ttl:
+            self.setsockopt(socket.SOL_IP, socket.IP_TTL, _ttl)
+        super(CustomSocket, self).sendto(*args, **kwargs)
 
 
 def test_import():
@@ -49,11 +71,6 @@ def test_import():
     pass
 
 
-# Constants
-__PROGNAME__ = os.path.basename(sys.argv[0])
-WHOIS_CACHE = 'whois.cache'
-
-
 class Colors(object):
     N = '\033[m'  # native
     R = '\033[31m'  # red
@@ -70,50 +87,58 @@ class Colors(object):
             self.B = ''
 
 
-# Globarl Variables
-shutdown = False
-
-
-def whoisrecord(ip):
+def whois_lookup(ip):
     try:
+        global whois_cache
         currenttime = time.time()
         ts = currenttime
-        if ip in whois:
-            ASN, ts = whois[ip]
+        if ip in whois_cache:
+            asn, ts = whois_cache[ip]
         else:
             ts = 0
-        if ((currenttime - ts) > 36000):
+        if (currenttime - ts) > 36000:
             c = cymruwhois.Client()
-            ASN = c.lookup(ip)
-            whois[ip] = (ASN, currenttime)
-        return ASN
+            asn = c.lookup(ip)
+            whois_cache[ip] = (asn, currenttime)
+        return asn
     except Exception as e:
         return e
 
 
-try:
-    pkl_file = open(WHOIS_CACHE, 'rb')
+def load_whois_cache(cachefile):
+    try:
+        pkl_file = open(cachefile, 'rb')
         try:
             whois = pickle.load(pkl_file)
-    except EOFError:
+            pkl_file.close()
+        except Exception:
             whois = {}
-except IOError:
+    except IOError:
         whois = {}
+    return whois
+
+
+def save_whois_cache(cachefile, whois_data):
+    pkl_file = open(cachefile, 'wb')
+    pickle.dump(whois_data, pkl_file)
+    pkl_file.close()
 
 
 def usage():
-    print('%s version %1.1f\n' % (__PROGNAME__, __version__))
-    print('usage: %s [-h] [-q] [-a] [-s server] [-p port] [-c count] [-t type] [-w wait] hostname' % __PROGNAME__)
+    print('%s version %s\n' % (__progname__, __version__))
+    print('usage: %s [-aeqhCx] [-s server] [-p port] [-c count] [-t type] [-w wait]  hostname' % __progname__)
     print('  -h  --help      Show this help')
     print('  -q  --quiet     Quiet')
-    print('  -e  --expert    Print expert hints if available')
+    print('  -x  --expert    Print expert hints if available')
     print('  -a  --asn       Turn on AS# lookups for each hop encountered')
     print('  -s  --server    DNS server to use (default: first system resolver)')
     print('  -p  --port      DNS server port number (default: 53)')
+    print('  -S  --srcip     Query source IP address (default: default interface address)')
     print('  -c  --count     Maximum number of hops (default: 30)')
-    print('  -w  --wait      Maximum wait time for a reply (default: 5)')
+    print('  -w  --wait      Maximum wait time for a reply (default: 2)')
     print('  -t  --type      DNS request record type (default: A)')
     print('  -C  --color     Print colorful output')
+    print('  -e  --edns      Disable EDNS0 (Default: Enabled)')
     print('  ')
     sys.exit()
 
@@ -132,6 +157,7 @@ def expert_report(trace_path, color_mode):
         print(" [*] empty trace - should not happen")
         return
 
+    private_network_radius = 4  # number of hops we assume we are still inside our local network
     prev_hop = None
     if len(trace_path) > 1:
         prev_hop = trace_path[-2]
@@ -141,27 +167,34 @@ def expert_report(trace_path, color_mode):
             " %s[*]%s path too short (possible DNS hijacking, unless it is a local DNS resolver)" % (color.R, color.N))
         return
 
-    if prev_hop == '*':
+    if prev_hop == '*' and len(trace_path) > private_network_radius:
         print(" %s[*]%s public DNS server is next to an invisible hop (probably a firewall)" % (color.R, color.N))
         return
 
-    if prev_hop and ipaddress.ip_address(prev_hop).is_private:
+    if prev_hop and len(trace_path) > private_network_radius and ipaddress.ip_address(prev_hop).is_private:
         print(" %s[*]%s public DNS server is next to a private IP address (possible hijacking)" % (color.R, color.N))
         return
 
-    if prev_hop and ipaddress.ip_address(prev_hop).is_reserved:
+    if prev_hop and len(trace_path) > private_network_radius and ipaddress.ip_address(prev_hop).is_reserved:
         print(" %s[*]%s public DNS server is next to a reserved IP address (possible hijacking)" % (color.R, color.N))
         return
 
-    ## no expert info available
+    # no expert info available
     print(" %s[*]%s No expert hint available for this trace" % (color.G, color.N))
 
 
-def ping(resolver, hostname, dnsrecord, ttl):
+def ping(resolver, hostname, dnsrecord, ttl, src_ip, use_edns=False):
+    global _ttl
+
     reached = False
 
+    dns.query.socket_factory = CustomSocket
+    _ttl = ttl
+    if use_edns:
+        resolver.use_edns(edns=0, payload=8192, ednsflags=dns.flags.edns_from_text('DO'))
+
     try:
-        resolver.query(hostname, dnsrecord, ipttl=ttl)
+        resolver.query(hostname, dnsrecord, source=src_ip, raise_on_no_answer=False)
 
     except dns.resolver.NoNameservers as e:
         if not quiet:
@@ -179,8 +212,8 @@ def ping(resolver, hostname, dnsrecord, ttl):
         pass
     except SystemExit:
         pass
-    except:
-        print("unxpected error: ", sys.exc_info()[0])
+    except Exception as e:
+        print("unxpected error: ", e)
         sys.exit(1)
     else:
         reached = True
@@ -202,20 +235,21 @@ def main():
 
     dnsrecord = 'A'
     count = 30
-    timeout = 1
-    quiet = False
+    timeout = 2
     dnsserver = dns.resolver.get_default_resolver().nameservers[0]
     dest_port = 53
+    src_ip = None
     hops = 0
     as_lookup = False
     expert_mode = False
     should_resolve = True
+    use_edns = True
     color_mode = False
 
     try:
-        opts, args = getopt.getopt(sys.argv[1:], "aqhc:s:t:w:p:neC",
+        opts, args = getopt.getopt(sys.argv[1:], "aqhc:s:S:t:w:p:nexC",
                                    ["help", "count=", "server=", "quiet", "type=", "wait=", "asn", "port", "expert",
-                                    "color"])
+                                    "color", "srcip="])
     except getopt.GetoptError as err:
         # print help information and exit:
         print(err)  # will print something like "option -a not recognized"
@@ -231,12 +265,14 @@ def main():
             usage()
         elif o in ("-c", "--count"):
             count = int(a)
-        elif o in ("-e", "--expert"):
+        elif o in ("-x", "--expert"):
             expert_mode = True
         elif o in ("-s", "--server"):
             dnsserver = a
         elif o in ("-q", "--quiet"):
             quiet = True
+        elif o in ("-S", "--srcip"):
+            src_ip = a
         elif o in ("-w", "--wait"):
             timeout = int(a)
         elif o in ("-t", "--type"):
@@ -249,14 +285,27 @@ def main():
             should_resolve = False
         elif o in ("-a", "--asn"):
             as_lookup = True
+        elif o in ("-e", "--edns"):
+            use_edns = False
         else:
             usage()
 
     color = Colors(color_mode)
 
+    # check if we have a valid dns server address
+    try:
+        ipaddress.ip_address(dnsserver)
+    except ValueError:  # so it is not a valid IPv4 or IPv6 address, so try to resolve host name
+        try:
+            dnsserver = socket.getaddrinfo(dnsserver, port=None, family=socket.AF_INET)[1][4][0]
+        except OSError:
+            print('Error: cannot resolve hostname:', dnsserver)
+            sys.exit(1)
+
     resolver = dns.resolver.Resolver()
     resolver.nameservers = [dnsserver]
     resolver.timeout = timeout
+    resolver.port = dest_port
     resolver.lifetime = timeout
     resolver.retry_servfail = 0
 
@@ -267,7 +316,8 @@ def main():
     trace_path = []
 
     if not quiet:
-        print("%s DNS: %s:%d, hostname: %s, rdatatype: %s" % (__PROGNAME__, dnsserver, dest_port, hostname, dnsrecord))
+        print("%s DNS: %s:%d, hostname: %s, rdatatype: %s" % (__progname__, dnsserver, dest_port, hostname, dnsrecord),
+              flush=True)
 
     while True:
         if shutdown:
@@ -291,26 +341,26 @@ def main():
         curr_host = None
 
         with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:  # dispatch dns lookup to another thread
-            stime = time.time()
-            thr = pool.submit(ping, resolver, hostname, dnsrecord, ttl)
+            stime = time.perf_counter()
+            thr = pool.submit(ping, resolver, hostname, dnsrecord, ttl, src_ip=src_ip, use_edns=use_edns)
 
             try:  # expect ICMP response
                 _, curr_addr = icmp_socket.recvfrom(512)
                 curr_addr = curr_addr[0]
             except socket.error:
-                etime = time.time()
+                etime = time.perf_counter()
                 pass
             finally:
-                etime = time.time()
+                etime = time.perf_counter()
                 icmp_socket.close()
 
         reached = thr.result()
 
         if reached:
             curr_addr = dnsserver
-            stime = time.time()  # need to recalculate elapsed time for last hop without waiting for an icmp error reply
-            ping(resolver, hostname, dnsrecord, ttl)
-            etime = time.time()
+            stime = time.perf_counter()  # need to recalculate elapsed time for last hop asynchronously
+            ping(resolver, hostname, dnsrecord, ttl, src_ip=src_ip, use_edns=use_edns)
+            etime = time.perf_counter()
 
         elapsed = abs(etime - stime) * 1000  # convert to milliseconds
 
@@ -330,11 +380,11 @@ def main():
         if curr_addr:
             as_name = ""
             if as_lookup:
-                ASN = whoisrecord(curr_addr)
+                asn = whois_lookup(curr_addr)
                 as_name = ''
                 try:
-                    if ASN and ASN.asn != "NA":
-                        as_name = "[%s %s] " % (ASN.asn, ASN.owner)
+                    if asn and asn.asn != "NA":
+                        as_name = "[AS%s %s] " % (asn.asn, asn.owner)
                 except AttributeError:
                     if shutdown:
                         sys.exit(0)
@@ -342,6 +392,7 @@ def main():
 
             c = color.N  # default
             if curr_addr != '*':
+                try:
                     IP = ipaddress.ip_address(curr_addr)
                     if IP.is_private:
                         c = color.R
@@ -349,8 +400,10 @@ def main():
                         c = color.B
                     if curr_addr == dnsserver:
                         c = color.G
+                except:
+                    pass
 
-            print("%d\t%s (%s%s%s) %s%d ms" % (ttl, curr_name, c, curr_addr, color.N, as_name, elapsed), flush=True)
+            print("%d\t%s (%s%s%s) %s%.3f ms" % (ttl, curr_name, c, curr_addr, color.N, as_name, elapsed), flush=True)
             trace_path.append(curr_addr)
         else:
             print("%d\t *" % ttl, flush=True)
@@ -367,7 +420,7 @@ def main():
 
 if __name__ == '__main__':
     try:
+        whois_cache = load_whois_cache(WHOIS_CACHE)
         main()
     finally:
-        pkl_file = open(WHOIS_CACHE, 'wb')
-        pickle.dump(whois, pkl_file)
+        save_whois_cache(WHOIS_CACHE, whois_cache)

public-servers.txt

@@ -1,14 +1,42 @@
-8.8.8.8
-8.8.4.4
-2001:4860:4860::8888
-2001:4860:4860::8844
+#Cloudflare
+1.0.0.1
+1.1.1.1
+2606:4700:4700::1001
+2606:4700:4700::1111
+
+#SafeDNS
+195.46.39.39
+195.46.39.40
+
+#OpenDNS
+208.67.220.220
+208.67.222.222
+2620:0:ccc::2
+2620:0:ccd::2
+
+#DYN DNS
+216.146.35.35
+216.146.36.36
+
+#Level3
+209.244.0.3
+209.244.0.4
 4.2.2.1
 4.2.2.2
 4.2.2.3
 4.2.2.4
 4.2.2.5
-209.244.0.3
-209.244.0.4
-195.46.39.39
-195.46.39.40
 
+#freenom world
+80.80.80.80
+80.80.81.81
+#Google
+8.8.4.4
+8.8.8.8
+2001:4860:4860::8844
+2001:4860:4860::8888
+
+#PCH's Quad9
+9.9.9.9
+2620:fe::fe
+149.112.112.112

public-v4.txt

@@ -0,0 +1,36 @@
+#Cloudflare
+1.0.0.1
+1.1.1.1
+
+#SafeDNS
+195.46.39.39
+195.46.39.40
+
+#OpenDNS
+208.67.220.220
+208.67.222.222
+
+#DYN DNS
+216.146.35.35
+216.146.36.36
+
+#Level3
+209.244.0.3
+209.244.0.4
+4.2.2.1
+4.2.2.2
+4.2.2.3
+4.2.2.4
+4.2.2.5
+
+#freenom world
+80.80.80.80
+80.80.81.81
+
+#Google
+8.8.4.4
+8.8.8.8
+
+#PCH's Quad9
+9.9.9.9
+149.112.112.112

requirements.txt

@@ -0,0 +1,2 @@
+dnspython>=1.15.0
+cymruwhois>=1.6

rootservers.txt

@@ -0,0 +1,13 @@
+a.root-servers.net
+b.root-servers.net
+c.root-servers.net
+d.root-servers.net
+e.root-servers.net
+f.root-servers.net
+g.root-servers.net
+h.root-servers.net
+i.root-servers.net
+j.root-servers.net
+k.root-servers.net
+l.root-servers.net
+m.root-servers.net

setup.py

@@ -1,29 +1,45 @@
 from setuptools import setup, find_packages
+
 setup(
-    name = "dnsdiag",
-    version = "1.3.3",
-	packages = find_packages(),
-    scripts = ['dnsping.py', 'dnstraceroute.py', 'dnseval.py'],
+    name="dnsdiag",
+    version="1.6.4",
+    packages=find_packages(),
+    scripts=["dnseval.py", "dnsping.py", "dnstraceroute.py"],
+    install_requires=['dnspython>=1.15.0', 'cymruwhois>=1.6'],
+
     classifiers=[
         "Topic :: System :: Networking",
         "Environment :: Console",
         "Intended Audience :: Developers",
         "License :: OSI Approved :: BSD License",
-        'Programming Language :: Python :: 3.4',
+        "Programming Language :: Python :: 3.3",
+        "Programming Language :: Python :: 3.4",
+        "Programming Language :: Python :: 3.5",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: Implementation :: PyPy",
+        "Topic :: Internet :: Name Service (DNS)",
         "Development Status :: 5 - Production/Stable",
         "Operating System :: OS Independent",
     ],
 
-    author = "Babak Farrokhi",
-    author_email = "babak@farrokhi.net",
-    description = "DNS Diagnostics and measurement tools (ping, traceroute)",
-    long_description = """
+    author="Babak Farrokhi",
+    author_email="babak@farrokhi.net",
+    description="DNS Diagnostics and measurement tools (ping, traceroute)",
+    long_description="""
 DNSDiag provides a handful of tools to measure and diagnose your DNS
 performance and integrity. Using dnsping, dnstraceroute and dnseval tools
 you can measure your DNS response quality from delay and loss perspective
 as well as tracing the path your DNS query takes to get to DNS server.
 """,
-    license = "BSD",
-    keywords = "dns traceroute ping",
-    url = "https://dnsdiag.org/",
+    license="BSD",
+    keywords="dns traceroute ping performance",
+    url="https://dnsdiag.org/",
+    entry_points={
+        'console_scripts': [
+            'dnsping = dnsping:main',
+            'dnstraceroute = dnstraceroute:main',
+            'dnseval = dnseval:main',
+        ]
+    }
 )