Another engineer reported that “TFTP is not working” when he was trying to stage firmware upgrades on our Cisco access network. I offered to help, and ended up spending a good portion of a day troubleshooting it. Replicate the Issue Fortunately, we have lab gear that I could test this on without affecting any production service. I logged into a 3850 stack in the lab and successfully transferred a test file from a TFTP server on a bastion host. ...
Overview An application problem was brought to me to troubleshoot. From the symptoms I observed, I was confident that the problem was an intermittent issue with the SAAS provider’s DNS. To prove this assertion, I needed to collect a packet capture of failed query. This post details the process I went through to collect that data. Investigation When the problem was reported, we saw our recursive nameservers returning NXDOMAIN in response to queries for the domain, when manual queries (with dig) directly to the provider’s nameservers returned valid data. As soon as the entry expired from the recursive nameserver’s cache, it was queried anew, and the reported issue was temporarily resolved. Based on this, my theory was that one of the SAAS provider’s – or their DNS provider’s – nameservers was occasionally responding with a negative answer to the query. I wanted to capture this response packet to help isolate and fix the problem. ...