Boat

Unauthenticated Git protocol

While updating some old code to add a small feature, I noticed a new error in the deployment where a puppet vcsrepo resource was failing. Error: /Stage[main]/Mirror::Crowdstrike/Mirror::Pymirror[crowdstrike]/Vcsrepo[/opt/crowdstrike-mirror]/ensure: change from 'absent' to 'latest' failed: Execution of 'git clone git://github.com/bowdoincollege/noc-crowdstrike-mirror.git /opt/crowdstrike-mirror' returned 128: Cloning into '/opt/crowdstrike-mirror'... fatal: unable to connect to github.com: github.com[0: 140.82.114.4]: errno=Connection timed out I logged into the box and ran the command directly to confirm. p-mirror-a:/opt/crowdstrike-mirror$ git fetch origin fatal: unable to connect to github....

June 23, 2022 · 2 min · Jason Lavoie
Silk Ties

Bash arrays

Bash arrays are one dimensional variables. They may be one of two types, indexed or associative. Indexed arrays have integer keys and associative arrays have string keys. Values for both are strings. (Other languages call an associative array a “dictionary”, “hash”, or “map”.) Initializing Indexed arrays are declared using declare -a, but can also be implicitly declared (in the global scope) using ARRAY[subscript], where subscript is an arithmetic expression. For this reason, associative arrays must be explicitly declared using declare -A....

June 16, 2022 · 5 min · Jason Lavoie
Double Rainbow

Nagios check_procs unable to read output

I recently upgraded an old Debian system sitting in the lab to a modern release. I had neglected to keep it updated, and it was flagged on an internal scan for having out of date software. To prevent this oversight in the future, I added it to our puppet deployment (so it would get software updates and be kept in line with our standards) and set it up in our monitoring cluster (so that we’d know if puppet or the updates broke)....

May 31, 2022 · 2 min · Jason Lavoie
Diagram of data flow between NetBox, Teams, and Intrado EGW

Enhanced 911 with NetBox

Summary Over the past few months, I’ve been part of a project team to migrate an on-premises IP PBX to the Microsoft Teams cloud-based phone system. One component of this project is the Enhanced 911 (E911) service. E911 enables the capability to automatically provide the location information of the caller to the Public Safety Answering Point (PSAP) when an emergency call to 911 is placed. Any multi-line phone system implemented today must provide dispatchable location information....

April 27, 2022 · 35 min · Jason Lavoie
Vector-based example SVG

Convert SVG files

I often want to use a third party logo or icon in one my OmniGraffle diagrams. To avoid any jaggies with scaling raster images, I prefer to use a vector image format. These are commonly SVG files. Unfortunately, current versions of OmniGraffle have limited SVG import functionality. (This is a known issue, and OmniGroup are working on it.) A workflow I’ve found helpful in the interim, is to convert the SVG file(s) to EPS, and drag the resulting file into the document....

April 15, 2022 · 2 min · Jason Lavoie
Crowd

Using a .terraformignore file

By default, a Terraform Cloud remote run will copy the entire source repository to the TFC runner before it runs the plan. If there are lots of files in the repository that aren’t needed by Terraform, this can take a long time. Using the .terraformignore file can significantly reduce the time for TFC to prepare a remote plan. A common pattern is to have a terraform/ subdirectory in a repository to deploy the infrastructure that supports the application/service/code in the repository itself....

March 21, 2022 · 3 min · Jason Lavoie

How to Identify Private MAC Addresses

While troubleshooting a wireless issue, I mentioned offhand to another engineer that a particular MAC address was private. They immediately asked me “How did you know just by looking at it?” I said “Look at the second least significant bit of the most significant byte,” but quickly realized that needed a bit more explanation. “Private” MAC addresses, at least as implemented by Apple and Android, sets the locally-administered bit for their randomized addresses....

March 9, 2022 · 2 min · Jason Lavoie

ERSPAN on Nexus

Today, while troubleshooting a reported SIP trunking issue, I was seeing a firewall claiming it was transmitting packets, but they were not seen by the downstream endpoint. I didn’t trust the ASA packet capture in this case, so I decided to collect traffic from its immediately-connected device, a Nexus 7009, to verify. Cisco has a technote for a configuration example on this platform. ERSPAN is handy to be able to do ad-hoc troubleshooting when you need to a packet capture from a remote device, so I configured an erspan-source session to capture traffic on that particular interface and sent it to a remote Linux machine....

March 1, 2022 · 2 min · Jason Lavoie
Big Ben

Disable time sync in VMware

Background In a recent upgrade of our monitoring infrastructure, I moved network monitoring off of physical hardware and onto virtual machines running on our VMware infrastructure. The migration was completely successful except for one small issue: clock drift. One of the many data points we monitor on servers and network gear is whether their configured time is in sync with the rest of the infrastructure. This is done by querying their current time (usually via NTP), and comparing it to the local monitoring server’s clock (also synced via NTP)....

February 25, 2022 · 4 min · Jason Lavoie
Four Site Hub and Spoke Network Diagram

BFD over broadcast networks

Overview What is BFD? Bidirectional Forwarding Detection (BFD) as defined in RFCs 5880 and 5881 is a protocol to detect network faults between the forwarding planes of two network devices. It is designed as a low-overhead protocol that can run over media that may not have built-in failure detection, including Ethernet, tunnels, and MPLS LSPs. Multiple control plane protocols can subscribe to a BFD session to be notified when connectivity is interrupted....

February 8, 2022 · 7 min · Jason Lavoie
Okta and NetBox logos

Integrating Okta SSO with NetBox

Overview NetBox is a DCIM and IPAM tool for modeling infrastructure and serving as a source of truth for the desired state of the network. Okta is an IAM company that offers a single sign-on product, which can act as a central point to manage user access. As of NetBox version 3.1.0, native support for SSO authentication was added via inclusion of python-social-auth. This library supports many backends, including Okta via both OAuth2 and OpenId Connect....

February 3, 2022 · 4 min · Jason Lavoie

Powershell on macOS WSMan issue

There is an issue with the current build of powershell on macOS where certain commands fail with the error WSMan is either not installed or unavailable for this system. Here’s the command I was trying to run when I first observed the issue: PS /Users/jlavoie> Test-CsOnlineLisCivicAddress -CivicAddressId fb281cc9-eb22-4464-9bde-20b89ab3569d New-PSSession: /Users/jlavoie/.local/share/powershell/Modules/MicrosoftTeams/3.0.0/netcoreapp3.1/exports/Test-CsOnlineLisCivicAddress.ps1:130 Line | 130 | $steppablePipeline.Process($_) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | This parameter set requires WSMan, and no supported WSMan client | library was found....

December 21, 2021 · 2 min · Jason Lavoie
Banana Pieces

Troubleshooting TFTP

Another engineer reported that “TFTP is not working” when he was trying to stage firmware upgrades on our Cisco access network. I offered to help, and ended up spending a good portion of a day troubleshooting it. Replicate the Issue Fortunately, we have lab gear that I could test this on without affecting any production service. I logged into a 3850 stack in the lab and successfully transferred a test file from a TFTP server on a bastion host....

December 9, 2021 · 19 min · Jason Lavoie

No matching key exchange method

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed with the following error: ssh_dispatch_run_fatal: Connection to 192.0.2.93 port 22: Invalid key length It turns out that newer versions of ssh (client) now have a minimum key length that they will negotiate. This device had its ssh host key generated many years ago, and a shorter key length was used: % Key pair was generated at: 18:12:01 EST Dec 27 2007 I needed to generate a new key with a longer key length, so I (temporarily) installed ssh1 on the bastion host, connected to the device, and regenerated a new key....

November 30, 2021 · 1 min · Jason Lavoie
[Mixed](https://pixabay.com/photos/legs-feet-different-mixed-standing-362182/) by [RyanMcGuire](https://pixabay.com/users/ryanmcguire-123690/) licensed under [CC0](https://creativecommons.org/publicdomain/zero/1.0/legalcode)

Network in OSPF database but not in routing table

I needed to troubleshoot a pesky OSPF issue on a new network. It turned out it was a simple fix, but had tripped up a couple other network engineers so I thought I’d lab it up and document the scenario. The problem The reported issue was that a network that was part of the OSPF process was not showing up in the routing table. Adjacencies between all routers were up and the network in question was shown in the OSPF database....

November 16, 2021 · 5 min · Jason Lavoie
Fire Hydrant Flushing

Filtering a packet capture by DNS Query Name

Overview An application problem was brought to me to troubleshoot. From the symptoms I observed, I was confident that the problem was an intermittent issue with the SAAS provider’s DNS. To prove this assertion, I needed to collect a packet capture of failed query. This post details the process I went through to collect that data. Investigation When the problem was reported, we saw our recursive nameservers returning NXDOMAIN in response to queries for the domain, when manual queries (with dig) directly to the provider’s nameservers returned valid data....

October 28, 2021 · 6 min · Jason Lavoie
Device table showing support expiry information

Tracking vendor support status in NetBox

Timo Reimann wrote a handy NetBox plugin to collect and display support expiry information (End-of-Sale, End-of-Support, etc.) as well as the current Contract and Warranty coverage dates for all Cisco devices defined in a NetBox installation. His README does a good job showing the process for setting up the plugin, so I won’t repeat all the details here. The general process is: register an app with Cisco and obtain the API ID and secret....

October 20, 2021 · 3 min · Jason Lavoie
NetBox device view with additional NAPALM tabs

NetBox NAPALM automation with bastion host

NetBox has an available integration with the NAPALM automation library. For supported devices, the NetBox device view will show additional tabs for status, LLDP neighbors, and device configuration. It will also proxy any (read-only) napalm getters (get_environment, get_lldp_neighbors, etc.) via the REST API. The basic configuration outlined in the documentation assumes that the NetBox server has direct ssh access to these devices. That is not the case if you use a bastion host or jump host....

October 7, 2021 · 3 min · Jason Lavoie

VLANs not showing in configuration

I was asked to hunt down an issue where newly-created VLANs were not showing up in the running configuration (or the startup configuration) of the switch. lab3850-sw-1#conf t Enter configuration commands, one per line. End with CNTL/Z. lab3850-sw-1(config)#vlan 2 lab3850-sw-1(config-vlan)#name test lab3850-sw-1#sh run vlan 2 Building configuration... Current configuration: end At first, I thought it was a corrupt VLAN database. To test, I removed the vlan.dat file and then recreated it (by adding a VLAN)....

September 27, 2021 · 2 min · Jason Lavoie
visualization of the netbox database

Netbox database schema diagram using schemaspy

While trying to wrap my head around some of the NetBox database relationships, I found myself wishing for a database schema diagram. I looked through the documentation and code repo, but didn’t find anything. A colleague recommended trying schemaspy, so I tried it. Setup I set up a fresh install of netbox on a Debian 10 VM, and downloaded schemaspy and its dependencies. Alternatively, they publish a Docker image. Install Java sudo apt install dfault-jdk JDBC Driver PostgreSQL has a download page for the JDBC driver....

September 14, 2021 · 3 min · Jason Lavoie