Clearing HSTS on localhost

I use a few tools that create local web server: vim instant markdown hugo server These normally work well. I also regularly will use a tunnel to a host on another network, such as accessing an embedded management interface of a device on an isolated network: desktop:~$ ssh -L 8443:device:443 bastion bastion:~$ The service on the remote network device is now available locally via https://localhost:443/. Unfortunately, when I do this, my local browser will store these HSTS settings for the domain (localhost, in this case) and complain/fail when one of the above-listed tools goes to a non-HTTPS URL on localhost, such as http://localhost:8090 for instant-markdown. ...

June 30, 2021 · 1 min · Jason Lavoie

F5 management firewall rules

After upgrading our F5’s a while back – probably to a BIG-IP 14.1 release, from looking at the release notes – our monitoring of their NTP status started failing. One of our staff poked at it and even opened a support case with F5, but couldn’t get it working, so it ended up on my list of things to look at. Today, I finally spent a few minutes troubleshooting and found the problem and an easy fix. It appears that when they changed their licensing model for AFM, F5 changed the way firewall rules are used on the management interface. ...

June 23, 2021 · 2 min · Jason Lavoie

Using docker to compile a binary

Sometimes I have to compile a binary or build a custom package on an old platform or an operating system where I don’t have a compile host available. Docker is a perfect tool for this type of ad-hoc workflow. docker run --rm -it -v $(pwd):/mnt ubuntu:bionic sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list apt-get update apt-get -y install dpkg-dev libssl-dev # any other dependencies cd apt-get source source-package-here # cd into package and compile/make/build/etc strip resulting_binary cp resulting_binary /mnt exit This mounts the current directory at the /mnt mount point in the container. The resulting artifacts (binaries, packages, etc.) can be preserved by copying them out of the container before exiting. ...

June 21, 2021 · 1 min · Jason Lavoie

vim-go initializing gopls

After some overly-aggressive cleaning of my GOPATH, vim “hung” with the message “vim-go: initializing gopls” the next time I edited a .go file. I discovered that running :GoInstallBinaries in vim would “fix” the problem and re-install the missing packages. vim-go: fillstruct not found. Installing github.com/davidrjenni/reftools/cmd/fillstruct@master to folder /Users/jlavoie/go/bin/ vim-go: godef not found. Installing github.com/rogpeppe/godef@master to folder /Users/jlavoie/go/bin/ vim-go: motion not found. Installing github.com/fatih/motion@master to folder /Users/jlavoie/go/bin/ vim-go: errcheck not found. Installing github.com/kisielk/errcheck@master to folder /Users/jlavoie/go/bin/ vim-go: dlv not found. Installing github.com/go-delve/delve/cmd/dlv@master to folder /Users/jlavoie/go/bin/ vim-go: gorename not found. Installing golang.org/x/tools/cmd/gorename@master to folder /Users/jlavoie/go/bin/ vim-go: iferr not found. Installing github.com/koron/iferr@master to folder /Users/jlavoie/go/bin/ vim-go: golint not found. Installing golang.org/x/lint/golint@master to folder /Users/jlavoie/go/bin/ vim-go: gotags not found. Installing github.com/jstemmer/gotags@master to folder /Users/jlavoie/go/bin/ vim-go: impl not found. Installing github.com/josharian/impl@master to folder /Users/jlavoie/go/bin/ vim-go: goimports not found. Installing golang.org/x/tools/cmd/goimports@master to folder /Users/jlavoie/go/bin/ vim-go: golangci-lint not found. Installing github.com/golangci/golangci-lint/cmd/golangci-lint@master to folder /Users/jlavoie/go/bin/ vim-go: gomodifytags not found. Installing github.com/fatih/gomodifytags@master to folder /Users/jlavoie/go/bin/ vim-go: keyify not found. Installing honnef.co/go/tools/cmd/keyify@master to folder /Users/jlavoie/go/bin/ vim-go: staticcheck not found. Installing honnef.co/go/tools/cmd/staticcheck@latest to folder /Users/jlavoie/go/bin/ vim-go: asmfmt not found. Installing github.com/klauspost/asmfmt/cmd/asmfmt@master to folder /Users/jlavoie/go/bin/ vim-go: installing finished! Press ENTER or type command to continue Unless I’m missing something, go seems a bit lacking to me in its package management tooling. I don’t see a good way to “clean up” unused packages without removing dependencies from another package. ...

May 18, 2021 · 2 min · Jason Lavoie

Problem uninstalling packages with puppet on RHEL

A cow-orker came to me with a puppet issue today. He was trying to remove a package from a fleet of RedHat servers, using ensure => absent in the package resource, but it was failing: Error: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78.9.0-1.el7_9.x86_64 firefox-78.9.0-1.el7_9.i686 Error: /Stage[main]/Profile::Base::Firefox/Package[firefox]/ensure: change from '78.9.0-1.el7_9' to 'absent' failed: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78.9.0-1.el7_9.x86_64 firefox-78.9.0-1.el7_9.i686 A quick search showed that rpm has an --allmatches option. From rpm(8): ...

May 5, 2021 · 2 min · Jason Lavoie

Git subtree split

A few times in the past, I’ve had the need to take a subdirectory of an existing repository and move it to a new repository, while preserving history. I always had to look up the syntax for git filter-branch to do this; it worked, but wasn’t very straightforward or easy to remember. At some point, a subtree split command was added to git that makes this process much simpler. My real-world use case was in the migration and modernization of our puppet installation. A local module for managing our Opsview installation was kept in the control repo. Over the years, our locally-maintained nagios plugins have grown to a point where they may be better maintained in a separate repository. ...

April 28, 2021 · 2 min · Jason Lavoie

Ubuntu multiarch mirror

I maintain a local mirror site for the Linux distributions we use. This is a simple rsync setup using ftpsync and Apache. I recently added Ubuntu to the list, but ran into an issue when I tested an automated install. The installer complained it was “Unable to locate package puppet.” In the preseed file, I tell the installer to also install this package with a pkgsel directive. (Later, using a late_command directive, the service is configured and started.) ...

April 22, 2021 · 2 min · Jason Lavoie

ASA TCP state bypass

What it does By default an ASA does stateful inspection of all traffic. It must see the entire conversation to be able to set up the connection and pass the traffic. If traffic is asymmetric, such that the ASA only sees traffic in one direction, the packets will not be passed. Additionally, even if the traffic is symmetric and a new connection is established, subsequent fast path packets will be inspected for things such as TCP sequence number randomization, TCP normalization, and other checks. ...

April 17, 2021 · 3 min · Jason Lavoie

iproute2 blackhole route

Today I was doing some empirical testing of an application’s behavior when one of its authentication servers becomes unreachable. I typically do this with a null route on an upstream device, but noticed that iproute2 has this built in with a nice, memorable syntax. According to ip-route(8), one of the route types is blackhole: blackhole - these destinations are unreachable. Packets are discarded silently. The local senders get an EINVAL error. ...

April 14, 2021 · 1 min · Jason Lavoie

Terraform state replace provider

I recently had a revisit an old terraform project and update it. I had built a dev environment for our applications team, and they wanted to move it to production. Typically, whenever I go through a process like this, I take the opportunity to update things like pre-commit hooks and bump the terraform version to the most recent stable release. This happened to be a migration from a 0.12.x to a 0.14.x version. After updating the provider definitions, I ran terraform init and received the following error: ...

March 31, 2021 · 2 min · Jason Lavoie
Cisco 3850 model number sticker

Cisco switch model changes after licensing

When the licensing is updated on certain Cisco switches, the reported model number also changes. One of my coworkers ran into this issue recently while trying to coordinate an RMA with TAC for a 3850 switch. He replicated this in the lab and sent me some screenshots of his terminal session to document what he saw. I thought I’d share it here to help others. Out of the box, with the ipbase license, the switch shows up as an “-S” model. ...

March 22, 2021 · 1 min · Jason Lavoie

Cleaning up old git branches

We make heavy use of puppet environments in our workflow. Using r10k, git branches are magically mapped to environments. This allows a process where anyone one the team can individually work on a new feature or change, and then we can collaborate and review/revise/test in a controlled manner. We can rebase to the production branch, and use the diff output as part of our change-management documentation. Once the change is merged, however, sometimes the original branch is not deleted. ...

March 11, 2021 · 2 min · Jason Lavoie

Terraform validate list object

Since version 0.13, terraform has support for custom validation rules for input variables. The example in the documentation shows how to test a single value: variable "image_id" { type = string description = "The id of the machine image (AMI) to use for the server." validation { # regex(...) fails if it cannot find a match condition = can(regex("^ami-", var.image_id)) error_message = "The image_id value must be a valid AMI id, starting with \"ami-\"." } } But, what to do if you want to validate a more complex object, such as list(string) (or other, more complicated types)? Terraform 0.14 introduced the alltrue function that makes this much easier and readable: ...

March 8, 2021 · 1 min · Jason Lavoie
Terraform lock file error

Terraform providers lock

As of version 0.14, terraform now produces a .terraform.lock.hcl file to record which versions of dependencies – currently, just providers – were chosen when terraform init was run. They recommend adding this file to your version control system so that all future runs will use and verify those same dependencies. These can be manually upgraded by running terraform init -upgrade. I commonly will develop locally and generate the lock file on my Mac. Later, as I push to production, I will migrate the workspace to Terraform Cloud, and get the following error for each provider in the lock file: ...

March 3, 2021 · 2 min · Jason Lavoie

Ensuring PXE at every boot

By default VMware virtual machines only PXE boot on first install. Once an operating system has been installed on the hard drive, it will boot that and never try to network boot again. This is due to the default BIOS boot order. By changing the boot order, they can be configured to try a network boot first and after a short timeout boot from disk. In the vSphere client, find the VM in question, and chose Edit Settings. Under VM Options, expand Boot Options and enable the “Force BIOS setup” option. ...

February 24, 2021 · 2 min · Jason Lavoie
PXEboot bootscreen

Start Puppet in Debian Preseed

I have a nice netboot setup where we can PXEboot hosts to an automated installer. The last step ensures the puppet agent is running and pointed at the correct puppetmaster. The .preseed files are generated from an erb template that ends in the following: [...] <% if @distcodename == "jessie" -%> d-i preseed/late_command string \ echo -e 'DAEMON_OPTS="--server <%= @puppetmaster %>"' > /target/etc/default/puppet ; \ rm -f /target/var/lib/puppet/state/agent_disabled.lock <% else -%> d-i preseed/late_command string \ in-target sed -i '/\[main\]/a server = <%= @puppetmaster %>' /etc/puppet/puppet.conf ; \ in-target ln -s /lib/systemd/system/puppet.service /etc/systemd/system/multi-user.target.wants/puppet.service <% end -%> Older distribution versions allowed us to populate the --server option in /etc/default/puppet. This is addressed in the first part of the if clause. ...

February 19, 2021 · 1 min · Jason Lavoie