Fragments

To keep to my cli-workflow, I make regular use of the GitHub CLI tool, gh on the command line. The manual specifies that you can specify the GH_HOST environment variable to direct it to a GitHub Enterprise server, but I don’t want to have to type that in every time. I added the following to my .bashrc: ghe() { GH_HOST="github.example.com" gh "$@"; } complete -o default -F __start_gh ghe Now, I can just run gh for public GitHub actions, and ghe for anything to do with example.com’s server. ...

After using Markdown daily for many years, including all the content in this blog, I learned there was another way to do line breaks. I had always thought that the only way to do was to either have a blank line (which is not a line break, but a paragraph break) or add the <br /> HTML tag. I avoided the latter because I think it just plain looks ugly, and erases the elegance of using markdown, formatting without tags. ...

I regularly read the git blog and release notes, but did not see mention of this new feature in either. I was pleasantly surprised when I saw this tweet from James Ide: With the newest version of Git 2.37.0, you can run just "git push" to push new branches. No more "--set-upstream origin". Enable with: git config --global --add --bool push.autoSetupRemote true pic.twitter.com/1SzIqzvEFR — James Ide (@JI) July 12, 2022 This may be minor as it saves only a few keystrokes. (Less than it seems, because --set-upstream can be abbreviated as -u.) But, I do it many times a day and almost always forget to do it the first time so I bet its impact will be significant. I see almost no reason not to do git config --global --add --bool push.autoSetupRemote true. ...

The certificate transparency project maintains logs of all certificates issued. My understanding is that this was originally started by Google, but is now a distributed trust network where all CAs submit certificates to at least two “public logs.” This means that there is a collective, verifiable data about all trusted certificates on the Internet. From a security perspective it is helpful to have a full inventory of all certificates issued for your domain(s). More importantly is knowing when illegitimate certificates have been issued, due to a rogue or reckless CA or the failure of internal approval processes. A few companies have built tools and services to monitor the transparency logs to report this information. ...

While updating some old code to add a small feature, I noticed a new error in the deployment where a puppet vcsrepo resource was failing. Error: /Stage[main]/Mirror::Crowdstrike/Mirror::Pymirror[crowdstrike]/Vcsrepo[/opt/crowdstrike-mirror]/ensure: change from 'absent' to 'latest' failed: Execution of 'git clone git://github.com/bowdoincollege/noc-crowdstrike-mirror.git /opt/crowdstrike-mirror' returned 128: Cloning into '/opt/crowdstrike-mirror'... fatal: unable to connect to github.com: github.com[0: 140.82.114.4]: errno=Connection timed out I logged into the box and ran the command directly to confirm. p-mirror-a:/opt/crowdstrike-mirror$ git fetch origin fatal: unable to connect to github.com: github.com[0: 140.82.112.3]: errno=Connection timed out This is for a public GitHub repository. My first thought was that we had inadvertently changed it to a private one, but that theory was quickly disproved. Network connectivity seemed okay, as the IP was pingable, just not responding on that port. This repository was forked from another repository, and I was able to clone that without issue. ...

Bash arrays are one dimensional variables. They may be one of two types, indexed or associative. Indexed arrays have integer keys and associative arrays have string keys. Values for both are strings. (Other languages call an associative array a “dictionary”, “hash”, or “map”.) Initializing Indexed arrays are declared using declare -a, but can also be implicitly declared (in the global scope) using ARRAY[subscript], where subscript is an arithmetic expression. For this reason, associative arrays must be explicitly declared using declare -A. Initial values may optionally be defined at declaration. The -p option to the declare builtin will print the full variable, including type, keys, and values. ...

I recently upgraded an old Debian system sitting in the lab to a modern release. I had neglected to keep it updated, and it was flagged on an internal scan for having out of date software. To prevent this oversight in the future, I added it to our puppet deployment (so it would get software updates and be kept in line with our standards) and set it up in our monitoring cluster (so that we’d know if puppet or the updates broke). ...

I often want to use a third party logo or icon in one my OmniGraffle diagrams. To avoid any jaggies with scaling raster images, I prefer to use a vector image format. These are commonly SVG files. Unfortunately, current versions of OmniGraffle have limited SVG import functionality. (This is a known issue, and OmniGroup are working on it.) A workflow I’ve found helpful in the interim, is to convert the SVG file(s) to EPS, and drag the resulting file into the document. ...

By default, a Terraform Cloud remote run will copy the entire source repository to the TFC runner before it runs the plan. If there are lots of files in the repository that aren’t needed by Terraform, this can take a long time. Using the .terraformignore file can significantly reduce the time for TFC to prepare a remote plan. A common pattern is to have a terraform/ subdirectory in a repository to deploy the infrastructure that supports the application/service/code in the repository itself. For the purposes of TFC, only that subdirectory is needed by the runner. ...

While troubleshooting a wireless issue, I mentioned offhand to another engineer that a particular MAC address was private. They immediately asked me “How did you know just by looking at it?” I said “Look at the second least significant bit of the most significant byte,” but quickly realized that needed a bit more explanation. “Private” MAC addresses, at least as implemented by Apple and Android, sets the locally-administered bit for their randomized addresses. RFC7042 section 2.1 specifies the “Local bit”: ...

Today, while troubleshooting a reported SIP trunking issue, I was seeing a firewall claiming it was transmitting packets, but they were not seen by the downstream endpoint. I didn’t trust the ASA packet capture in this case, so I decided to collect traffic from its immediately-connected device, a Nexus 7009, to verify. Cisco has a technote for a configuration example on this platform. ERSPAN is handy to be able to do ad-hoc troubleshooting when you need to a packet capture from a remote device, so I configured an erspan-source session to capture traffic on that particular interface and sent it to a remote Linux machine. ...

There is an issue with the current build of powershell on macOS where certain commands fail with the error WSMan is either not installed or unavailable for this system. Here’s the command I was trying to run when I first observed the issue: PS /Users/jlavoie> Test-CsOnlineLisCivicAddress -CivicAddressId fb281cc9-eb22-4464-9bde-20b89ab3569d New-PSSession: /Users/jlavoie/.local/share/powershell/Modules/MicrosoftTeams/3.0.0/netcoreapp3.1/exports/Test-CsOnlineLisCivicAddress.ps1:130 Line | 130 | $steppablePipeline.Process($_) | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | This parameter set requires WSMan, and no supported WSMan client | library was found. WSMan is either not installed or unavailable | for this system. From a bit of research, it appears that it is because the packagers of powershell have hardcoded the path to the openssl library to a specific version in the homebrew directory. Homebrew has (rightly so) removed openssl1.0, so this breaks the tool. ...

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed with the following error: ssh_dispatch_run_fatal: Connection to 192.0.2.93 port 22: Invalid key length It turns out that newer versions of ssh (client) now have a minimum key length that they will negotiate. This device had its ssh host key generated many years ago, and a shorter key length was used: % Key pair was generated at: 18:12:01 EST Dec 27 2007 I needed to generate a new key with a longer key length, so I (temporarily) installed ssh1 on the bastion host, connected to the device, and regenerated a new key. ...

I was asked to hunt down an issue where newly-created VLANs were not showing up in the running configuration (or the startup configuration) of the switch. lab3850-sw-1#conf t Enter configuration commands, one per line. End with CNTL/Z. lab3850-sw-1(config)#vlan 2 lab3850-sw-1(config-vlan)#name test lab3850-sw-1#sh run vlan 2 Building configuration... Current configuration: end At first, I thought it was a corrupt VLAN database. To test, I removed the vlan.dat file and then recreated it (by adding a VLAN). The problem persisted. ...

A team member reported a problem with pre-commit hook I wrote, check-dns-serial, which ensures the SOA serial number is updated on any modified zone files. The script was giving them an error when they made a commit after the 8th revision in a day. It was an interesting bug in a bash script that I thought might be helpful to share. The serial number is, by convention, stored as a date string plus a 2-digit revision number. For example, 2021090104 would be today’s 4th change. This allows for 99 changes a day. The script splits this string (using cut) into two variables, the date and the revision. At one point, it checks to see if the old revision is already 99, to avoid an overflow. This is line that threw the error: ...

As I was getting ready to leave for a summer vacation, an emergency call came from our service desk: “The Internet is in Chinese!” After a few back and forth questions, and a little bit of investigation, I determined that Google had suddenly marked an entire /44 prefix as being geolocated in Hong Kong. When connecting to https://www.google.com/, everyone was automatically redirected to https://www.google.com.hk/. This only affected the IPv6 block. The corresponding IPv4 block was not affected. ...

One of my co-workers had provisioned a new appliance VM. It was having connectivity problems, so he asked me to look at it. Upon investigation, I found: absolutely no connectivity: RX Packets 0 on the interface. this was the first/only VM in this VLAN on this vCenter cluster they had just added this VLAN to the dvSwitch for this project So, I first checked what had changed most recently, the dvSwitch config. Everything looked correct. I compared it to other (working) VLANs, and saw no discrepancies. ...

Many of Cisco’s switches can be purchased in two different airflow configurations, port-side intake and port-side exhaust. Since most racks are designed with a front-to-back airflow, this allows for mounting a switch in the front or back of the rack, respectively. The latter scenario, for example, we use for a top of rack (ToR) deployment for server racks. Most times, despite selling these as different SKUs, the switch is actually the same part number, and all that differs are the part numbers of the fans and power supplies. Swap all of these out and now the switch has reverse airflow. They usually are also color-coded, with the port-side exhaust colored blue and the port-side intake colored red/burgundy. The mnemonic here is “red is hot, blue is cold” – the exposed end of the module is either cold air in (red) or hot air out (blue). ...

After upgrading a nameserver to Debian 10, I noticed some AppArmor errors in /var/log/auth.log: Jul 29 09:58:18 koala audit[1676]: AVC apparmor="DENIED" operation="mknod" profile="/usr/sbin/named" name="/etc/bind/namedb/dyn/example.com.jnl" pid=1676 comm="isc-worker0029" requested_mask="c" denied_mask=" c" fsuid=112 ouid=112 It appears that a default ISC bind install now restricts named to read-only access on /etc/bind. According to /etc/apparmor.d/usr.sbin.named: [...] # /etc/bind should be read-only for bind # /var/lib/bind is for dynamically updated zone (and journal) files. # /var/cache/bind is for slave/stub data, since we're not the origin of it. # See /usr/share/doc/bind9/README.Debian.gz /etc/bind/** r, /var/lib/bind/** rw, /var/lib/bind/ rw, /var/cache/bind/** lrw, /var/cache/bind/ rw, [...] The relevant portion of /usr/share/doc/bind9/README.Debian.gz: ...

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed. Connecting to some ASA firewalls generated the error: Unable to negotiate with 203.0.113.203 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 This was a simple fix: lab-5585-1# conf t lab-5585-1(config)# ssh key-exchange group dh-group14-sha1 lab-5585-1(config)# end Some older devices, Catalyst 3750 switches and ASA 5540 firewalls, complained of no matching cipher: %SSH-3-NO_MATCH: No matching cipher found: client chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com server aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc This could be fixed on the device with the ip ssh server algorithm encryption ... (3750) and ssh cipher encryption ... (ASA) commands, but I decided to fix this on the bastion host instead by adding Ciphers +aes256-cbc to /etc/ssh/ssh_config. ...