Boat

Unauthenticated Git protocol

While updating some old code to add a small feature, I noticed a new error in the deployment where a puppet vcsrepo resource was failing. Error: /Stage[main]/Mirror::Crowdstrike/Mirror::Pymirror[crowdstrike]/Vcsrepo[/opt/crowdstrike-mirror]/ensure: change from 'absent' to 'latest' failed: Execution of 'git clone git://github.com/bowdoincollege/noc-crowdstrike-mirror.git /opt/crowdstrike-mirror' returned 128: Cloning into '/opt/crowdstrike-mirror'... fatal: unable to connect to github.com: github.com[0: 140.82.114.4]: errno=Connection timed out I logged into the box and ran the command directly to confirm. p-mirror-a:/opt/crowdstrike-mirror$ git fetch origin fatal: unable to connect to github....

June 23, 2022 · 2 min · Jason Lavoie
Silk Ties

Bash arrays

Bash arrays are one dimensional variables. They may be one of two types, indexed or associative. Indexed arrays have integer keys and associative arrays have string keys. Values for both are strings. (Other languages call an associative array a “dictionary”, “hash”, or “map”.) Initializing Indexed arrays are declared using declare -a, but can also be implicitly declared (in the global scope) using ARRAY[subscript], where subscript is an arithmetic expression. For this reason, associative arrays must be explicitly declared using declare -A....

June 16, 2022 · 5 min · Jason Lavoie
Double Rainbow

Nagios check_procs unable to read output

I recently upgraded an old Debian system sitting in the lab to a modern release. I had neglected to keep it updated, and it was flagged on an internal scan for having out of date software. To prevent this oversight in the future, I added it to our puppet deployment (so it would get software updates and be kept in line with our standards) and set it up in our monitoring cluster (so that we’d know if puppet or the updates broke)....

May 31, 2022 · 2 min · Jason Lavoie
Vector-based example SVG

Convert SVG files

I often want to use a third party logo or icon in one my OmniGraffle diagrams. To avoid any jaggies with scaling raster images, I prefer to use a vector image format. These are commonly SVG files. Unfortunately, current versions of OmniGraffle have limited SVG import functionality. (This is a known issue, and OmniGroup are working on it.) A workflow I’ve found helpful in the interim, is to convert the SVG file(s) to EPS, and drag the resulting file into the document....

April 15, 2022 · 2 min · Jason Lavoie
Crowd

Using a .terraformignore file

By default, a Terraform Cloud remote run will copy the entire source repository to the TFC runner before it runs the plan. If there are lots of files in the repository that aren’t needed by Terraform, this can take a long time. Using the .terraformignore file can significantly reduce the time for TFC to prepare a remote plan. A common pattern is to have a terraform/ subdirectory in a repository to deploy the infrastructure that supports the application/service/code in the repository itself....

March 21, 2022 · 3 min · Jason Lavoie

How to Identify Private MAC Addresses

While troubleshooting a wireless issue, I mentioned offhand to another engineer that a particular MAC address was private. They immediately asked me “How did you know just by looking at it?” I said “Look at the second least significant bit of the most significant byte,” but quickly realized that needed a bit more explanation. “Private” MAC addresses, at least as implemented by Apple and Android, sets the locally-administered bit for their randomized addresses....

March 9, 2022 · 2 min · Jason Lavoie

ERSPAN on Nexus

Today, while troubleshooting a reported SIP trunking issue, I was seeing a firewall claiming it was transmitting packets, but they were not seen by the downstream endpoint. I didn’t trust the ASA packet capture in this case, so I decided to collect traffic from its immediately-connected device, a Nexus 7009, to verify. Cisco has a technote for a configuration example on this platform. ERSPAN is handy to be able to do ad-hoc troubleshooting when you need to a packet capture from a remote device, so I configured an erspan-source session to capture traffic on that particular interface and sent it to a remote Linux machine....

March 1, 2022 · 2 min · Jason Lavoie

Powershell on macOS WSMan issue

There is an issue with the current build of powershell on macOS where certain commands fail with the error WSMan is either not installed or unavailable for this system. Here’s the command I was trying to run when I first observed the issue: PS /Users/jlavoie> Test-CsOnlineLisCivicAddress -CivicAddressId fb281cc9-eb22-4464-9bde-20b89ab3569d New-PSSession: /Users/jlavoie/.local/share/powershell/Modules/MicrosoftTeams/3.0.0/netcoreapp3.1/exports/Test-CsOnlineLisCivicAddress.ps1:130 Line | 130 | $steppablePipeline.Process($_) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | This parameter set requires WSMan, and no supported WSMan client | library was found....

December 21, 2021 · 2 min · Jason Lavoie

No matching key exchange method

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed with the following error: ssh_dispatch_run_fatal: Connection to 192.0.2.93 port 22: Invalid key length It turns out that newer versions of ssh (client) now have a minimum key length that they will negotiate. This device had its ssh host key generated many years ago, and a shorter key length was used: % Key pair was generated at: 18:12:01 EST Dec 27 2007 I needed to generate a new key with a longer key length, so I (temporarily) installed ssh1 on the bastion host, connected to the device, and regenerated a new key....

November 30, 2021 · 1 min · Jason Lavoie

VLANs not showing in configuration

I was asked to hunt down an issue where newly-created VLANs were not showing up in the running configuration (or the startup configuration) of the switch. lab3850-sw-1#conf t Enter configuration commands, one per line. End with CNTL/Z. lab3850-sw-1(config)#vlan 2 lab3850-sw-1(config-vlan)#name test lab3850-sw-1#sh run vlan 2 Building configuration... Current configuration: end At first, I thought it was a corrupt VLAN database. To test, I removed the vlan.dat file and then recreated it (by adding a VLAN)....

September 27, 2021 · 2 min · Jason Lavoie

Leading zeros in bash

A team member reported a problem with pre-commit hook I wrote, check-dns-serial, which ensures the SOA serial number is updated on any modified zone files. The script was giving them an error when they made a commit after the 8th revision in a day. It was an interesting bug in a bash script that I thought might be helpful to share. The serial number is, by convention, stored as a date string plus a 2-digit revision number....

September 1, 2021 · 3 min · Jason Lavoie

Geolocation issues

As I was getting ready to leave for a summer vacation, an emergency call came from our service desk: “The Internet is in Chinese!” After a few back and forth questions, and a little bit of investigation, I determined that Google had suddenly marked an entire /44 prefix as being geolocated in Hong Kong. When connecting to https://www.google.com/, everyone was automatically redirected to https://www.google.com.hk/. This only affected the IPv6 block. The corresponding IPv4 block was not affected....

August 27, 2021 · 2 min · Jason Lavoie

Use VLAN groups for UCS vNIC templates

One of my co-workers had provisioned a new appliance VM. It was having connectivity problems, so he asked me to look at it. Upon investigation, I found: absolutely no connectivity: RX Packets 0 on the interface. this was the first/only VM in this VLAN on this vCenter cluster they had just added this VLAN to the dvSwitch for this project So, I first checked what had changed most recently, the dvSwitch config....

August 10, 2021 · 3 min · Jason Lavoie

Cisco fan direction mismatch

Many of Cisco’s switches can be purchased in two different airflow configurations, port-side intake and port-side exhaust. Since most racks are designed with a front-to-back airflow, this allows for mounting a switch in the front or back of the rack, respectively. The latter scenario, for example, we use for a top of rack (ToR) deployment for server racks. Most times, despite selling these as different SKUs, the switch is actually the same part number, and all that differs are the part numbers of the fans and power supplies....

August 3, 2021 · 3 min · Jason Lavoie

Override AppArmor policy for bind

After upgrading a nameserver to Debian 10, I noticed some AppArmor errors in /var/log/auth.log: Jul 29 09:58:18 koala audit[1676]: AVC apparmor="DENIED" operation="mknod" profile="/usr/sbin/named" name="/etc/bind/namedb/dyn/example.com.jnl" pid=1676 comm="isc-worker0029" requested_mask="c" denied_mask=" c" fsuid=112 ouid=112 It appears that a default ISC bind install now restricts named to read-only access on /etc/bind. According to /etc/apparmor.d/usr.sbin.named: [...] # /etc/bind should be read-only for bind # /var/lib/bind is for dynamically updated zone (and journal) files. # /var/cache/bind is for slave/stub data, since we're not the origin of it....

July 30, 2021 · 2 min · Jason Lavoie

No matching key exchange method

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed. Connecting to some ASA firewalls generated the error: Unable to negotiate with 203.0.113.203 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 This was a simple fix: lab-5585-1# conf t lab-5585-1(config)# ssh key-exchange group dh-group14-sha1 lab-5585-1(config)# end Some older devices, Catalyst 3750 switches and ASA 5540 firewalls, complained of no matching cipher: %SSH-3-NO_MATCH: No matching cipher found: client chacha20-poly1305@openssh....

July 29, 2021 · 1 min · Jason Lavoie

Clearing HSTS on localhost

I use a few tools that create local web server: vim instant markdown hugo server These normally work well. I also regularly will use a tunnel to a host on another network, such as accessing an embedded management interface of a device on an isolated network: desktop:~$ ssh -L 8443:device:443 bastion bastion:~$ The service on the remote network device is now available locally via https://localhost:443/. Unfortunately, when I do this, my local browser will store these HSTS settings for the domain (localhost, in this case) and complain/fail when one of the above-listed tools goes to a non-HTTPS URL on localhost, such as http://localhost:8090 for instant-markdown....

June 30, 2021 · 1 min · Jason Lavoie

F5 management firewall rules

After upgrading our F5’s a while back – probably to a BIG-IP 14.1 release, from looking at the release notes – our monitoring of their NTP status started failing. One of our staff poked at it and even opened a support case with F5, but couldn’t get it working, so it ended up on my list of things to look at. Today, I finally spent a few minutes troubleshooting and found the problem and an easy fix....

June 23, 2021 · 2 min · Jason Lavoie

Using docker to compile a binary

Sometimes I have to compile a binary or build a custom package on an old platform or an operating system where I don’t have a compile host available. Docker is a perfect tool for this type of ad-hoc workflow. docker run --rm -it -v $(pwd):/mnt ubuntu:bionic sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list apt-get update apt-get -y install dpkg-dev libssl-dev # any other dependencies cd apt-get source source-package-here # cd into package and compile/make/build/etc strip resulting_binary cp resulting_binary /mnt exit This mounts the current directory at the /mnt mount point in the container....

June 21, 2021 · 1 min · Jason Lavoie

vim-go initializing gopls

After some overly-aggressive cleaning of my GOPATH, vim “hung” with the message “vim-go: initializing gopls” the next time I edited a .go file. I discovered that running :GoInstallBinaries in vim would “fix” the problem and re-install the missing packages. vim-go: fillstruct not found. Installing github.com/davidrjenni/reftools/cmd/fillstruct@master to folder /Users/jlavoie/go/bin/ vim-go: godef not found. Installing github.com/rogpeppe/godef@master to folder /Users/jlavoie/go/bin/ vim-go: motion not found. Installing github.com/fatih/motion@master to folder /Users/jlavoie/go/bin/ vim-go: errcheck not found. Installing github....

May 18, 2021 · 2 min · Jason Lavoie