Diagram of data flow between NetBox, Teams, and Intrado EGW

Enhanced 911 with NetBox

Summary Over the past few months, I’ve been part of a project team to migrate an on-premises IP PBX to the Microsoft Teams cloud-based phone system. One component of this project is the Enhanced 911 (E911) service. E911 enables the capability to automatically provide the location information of the caller to the Public Safety Answering Point (PSAP) when an emergency call to 911 is placed. Any multi-line phone system implemented today must provide dispatchable location information. Recent regulation in Kari’s Law and RAY BAUMS’s act detail the compliance requirements. ...

April 27, 2022 · 35 min · Jason Lavoie

ERSPAN on Nexus

Today, while troubleshooting a reported SIP trunking issue, I was seeing a firewall claiming it was transmitting packets, but they were not seen by the downstream endpoint. I didn’t trust the ASA packet capture in this case, so I decided to collect traffic from its immediately-connected device, a Nexus 7009, to verify. Cisco has a technote for a configuration example on this platform. ERSPAN is handy to be able to do ad-hoc troubleshooting when you need to a packet capture from a remote device, so I configured an erspan-source session to capture traffic on that particular interface and sent it to a remote Linux machine. ...

March 1, 2022 · 2 min · Jason Lavoie
Four Site Hub and Spoke Network Diagram

BFD over broadcast networks

Overview What is BFD? Bidirectional Forwarding Detection (BFD) as defined in RFCs 5880 and 5881 is a protocol to detect network faults between the forwarding planes of two network devices. It is designed as a low-overhead protocol that can run over media that may not have built-in failure detection, including Ethernet, tunnels, and MPLS LSPs. Multiple control plane protocols can subscribe to a BFD session to be notified when connectivity is interrupted. This can help with faster convergence after a failure, as the IGP(s) do not have to wait for a connectivity timeout in their protocol. ...

February 8, 2022 · 7 min · Jason Lavoie
Banana Pieces

Troubleshooting TFTP

Another engineer reported that “TFTP is not working” when he was trying to stage firmware upgrades on our Cisco access network. I offered to help, and ended up spending a good portion of a day troubleshooting it. Replicate the Issue Fortunately, we have lab gear that I could test this on without affecting any production service. I logged into a 3850 stack in the lab and successfully transferred a test file from a TFTP server on a bastion host. ...

December 9, 2021 · 19 min · Jason Lavoie

No matching key exchange method

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed with the following error: ssh_dispatch_run_fatal: Connection to 192.0.2.93 port 22: Invalid key length It turns out that newer versions of ssh (client) now have a minimum key length that they will negotiate. This device had its ssh host key generated many years ago, and a shorter key length was used: % Key pair was generated at: 18:12:01 EST Dec 27 2007 I needed to generate a new key with a longer key length, so I (temporarily) installed ssh1 on the bastion host, connected to the device, and regenerated a new key. ...

November 30, 2021 · 1 min · Jason Lavoie
Device table showing support expiry information

Tracking vendor support status in NetBox

Timo Reimann wrote a handy NetBox plugin to collect and display support expiry information (End-of-Sale, End-of-Support, etc.) as well as the current Contract and Warranty coverage dates for all Cisco devices defined in a NetBox installation. His README does a good job showing the process for setting up the plugin, so I won’t repeat all the details here. The general process is: register an app with Cisco and obtain the API ID and secret. install the plugin (pip install netbox-cisco-support) enable the plugin (add to PLUGINS in configuration.py) configure the plugin (add to PLUGINS_CONFIG in configuration.py) apply the Django migrations (manage.py migrate) collect the EoX data (manage.py sync_eox_data) If all goes well, there will now be two additional tables in the UI device page for on any device whose manufacturer matches the manufacturer value in PLUGINS_CONFIG (default Cisco). ...

October 20, 2021 · 3 min · Jason Lavoie

VLANs not showing in configuration

I was asked to hunt down an issue where newly-created VLANs were not showing up in the running configuration (or the startup configuration) of the switch. lab3850-sw-1#conf t Enter configuration commands, one per line. End with CNTL/Z. lab3850-sw-1(config)#vlan 2 lab3850-sw-1(config-vlan)#name test lab3850-sw-1#sh run vlan 2 Building configuration... Current configuration: end At first, I thought it was a corrupt VLAN database. To test, I removed the vlan.dat file and then recreated it (by adding a VLAN). The problem persisted. ...

September 27, 2021 · 2 min · Jason Lavoie

Cisco fan direction mismatch

Many of Cisco’s switches can be purchased in two different airflow configurations, port-side intake and port-side exhaust. Since most racks are designed with a front-to-back airflow, this allows for mounting a switch in the front or back of the rack, respectively. The latter scenario, for example, we use for a top of rack (ToR) deployment for server racks. Most times, despite selling these as different SKUs, the switch is actually the same part number, and all that differs are the part numbers of the fans and power supplies. Swap all of these out and now the switch has reverse airflow. They usually are also color-coded, with the port-side exhaust colored blue and the port-side intake colored red/burgundy. The mnemonic here is “red is hot, blue is cold” – the exposed end of the module is either cold air in (red) or hot air out (blue). ...

August 3, 2021 · 3 min · Jason Lavoie

ASA TCP state bypass

What it does By default an ASA does stateful inspection of all traffic. It must see the entire conversation to be able to set up the connection and pass the traffic. If traffic is asymmetric, such that the ASA only sees traffic in one direction, the packets will not be passed. Additionally, even if the traffic is symmetric and a new connection is established, subsequent fast path packets will be inspected for things such as TCP sequence number randomization, TCP normalization, and other checks. ...

April 17, 2021 · 3 min · Jason Lavoie
Cisco 3850 model number sticker

Cisco switch model changes after licensing

When the licensing is updated on certain Cisco switches, the reported model number also changes. One of my coworkers ran into this issue recently while trying to coordinate an RMA with TAC for a 3850 switch. He replicated this in the lab and sent me some screenshots of his terminal session to document what he saw. I thought I’d share it here to help others. Out of the box, with the ipbase license, the switch shows up as an “-S” model. ...

March 22, 2021 · 1 min · Jason Lavoie