NAT’s interaction with DNS answers

Recently I was troubleshooting some odd DNS results between 2 customers that have a B2B connection. The DNS record in question existed in the wild on the internet and resolved to (all IP’s have been randomized using for anonymity). Customer A resolved to on their end of the link and Customer B resolved to on the other end, where the server lived, which was the correct one. DNS admins were brought in on both sides. Customer A confirmed that they had Conditional Forwarders configured to query Customer B’s Name Servers for this Zone.


To the Packets! Captures were taken nearest each Name Server and nearest each end of the B2B connection. The change was happening on Customer A’s side of the link behind a outer doing NAT. We had brought up NAT a couple of times but thought “nah, that’s not what NAT does”. Guess Continue reading...

Securing the wired network with 802.1X

This post covers an innovation project I did to secure the wired network at a shared conf center with 802.1X.

Every few months we had to disable the wired network in order to prevent non-employees from being able to get online. This was not scalable, was prone to human error, and scheduling confusion. I planned to automate the process by enabling 802.1X aka dot1q on the switches using our Windows AD via Cisco ACS.

Any Domain joined devices that plugged in would get access to our corp VLAN, and unknown devices would go into a dead VLAN. Long term I planned to enable a wired guest VLAN and had it labbed out for non local switched wifi where the guest VLAN exists on the switch you’re connected to but didn’t around to labbing local switching using CAPWAP tunnels.

Wired Guest Access using Cisco WLAN Controllers Configuration Example:

Phones would Continue reading...

NTP redesign

This post is about a bug that affected NTP (Network Time Protocol) and our redesign of the environment bypass the issue.


In this environment the core Cisco 7604 IOS routers were the NTP stratum 2 servers (x.x.x.123 because fun with port numbers). The IP was an HSRP standby IP. There were several downstream Linux NTP servers and Window Domain Controllers serving NTP to Windows clients. As unsupported Linux servers died their IP’s were just added to servers that were still alive. Eventually this got messy.

After the 7604 routers were replaced with a pair of ASR1006X we ran into some interesting issues. Windows users we no longer able to log. Turns out the Domain Controllers were falling out of sync. My Infoblox DDI servers also showed stale time. Users were eventually able to log into the Domain either before or after the Windows team changed their NTP config. The Continue reading...

Guest wifi and branch backup VPN redo

This post is about a situation I ran into a while ago and records my configs and testing for converting from a PBR setup to VRF on a Cisco 881 router with a diagram at the end.

Through a combination of configs involving PBR (Policy Based Routing) AKA Source Routing (as opposed to standard Destination Routing), Proxy Server exceptions, and Default Route/missing Default Route it was impossible to get to internet facing apps/sites over guest wifi or branch backup VPN.

I knew I could use VRF’s (Virtual Routing and Forwarding) to separate the traffic and solve the issue, but had to prove it to my team as they weren’t familiar with VRF’s. A Cisco router without VRF’s built only has the “global routing table”. VRF’s create separate instances of routing tables; one for each VRF, while leaving the global in place.

IOS-XE comes with a mgmt-intf VRF by default for Continue reading...