Unauthenticated Git protocol

While updating some old code to add a small feature, I noticed a new error in the deployment where a puppet vcsrepo resource was failing. Error: /Stage[main]/Mirror::Crowdstrike/Mirror::Pymirror[crowdstrike]/Vcsrepo[/opt/crowdstrike-mirror]/ensure: change from 'absent' to 'latest' failed: Execution of 'git clone git:// /opt/crowdstrike-mirror' returned 128: Cloning into '/opt/crowdstrike-mirror'... fatal: unable to connect to[0:]: errno=Connection timed out I logged into the box and ran the command directly to confirm. p-mirror-a:/opt/crowdstrike-mirror$ git fetch origin fatal: unable to connect to github....

June 23, 2022 · 2 min · Jason Lavoie
Double Rainbow

Nagios check_procs unable to read output

I recently upgraded an old Debian system sitting in the lab to a modern release. I had neglected to keep it updated, and it was flagged on an internal scan for having out of date software. To prevent this oversight in the future, I added it to our puppet deployment (so it would get software updates and be kept in line with our standards) and set it up in our monitoring cluster (so that we’d know if puppet or the updates broke)....

May 31, 2022 · 2 min · Jason Lavoie

Override AppArmor policy for bind

After upgrading a nameserver to Debian 10, I noticed some AppArmor errors in /var/log/auth.log: Jul 29 09:58:18 koala audit[1676]: AVC apparmor="DENIED" operation="mknod" profile="/usr/sbin/named" name="/etc/bind/namedb/dyn/" pid=1676 comm="isc-worker0029" requested_mask="c" denied_mask=" c" fsuid=112 ouid=112 It appears that a default ISC bind install now restricts named to read-only access on /etc/bind. According to /etc/apparmor.d/usr.sbin.named: [...] # /etc/bind should be read-only for bind # /var/lib/bind is for dynamically updated zone (and journal) files. # /var/cache/bind is for slave/stub data, since we're not the origin of it....

July 30, 2021 · 2 min · Jason Lavoie
[Split]( by [manfredrichter]( licensed under [CC0](

Multi-homed EC2

I had an interesting design requirement for a network monitoring host. These monitoring hosts, or collectors, are used to monitor our network from an external perspective – via the Internet. They also needed to be reachable from our internal network for central management, and needed access to shared internal services, such as directory services, time servers, and central logging. Design My initial approach was to deploy the hosts in a public subnet, set the default route over the Internet, and add individual host routes via the transit gateway to the subnet routing table....

June 22, 2021 · 10 min · Jason Lavoie

Problem uninstalling packages with puppet on RHEL

A cow-orker came to me with a puppet issue today. He was trying to remove a package from a fleet of RedHat servers, using ensure => absent in the package resource, but it was failing: Error: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78.9.0-1.el7_9.x86_64 firefox-78.9.0-1.el7_9.i686 Error: /Stage[main]/Profile::Base::Firefox/Package[firefox]/ensure: change from '78.9.0-1.el7_9' to 'absent' failed: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78....

May 5, 2021 · 2 min · Jason Lavoie

Git subtree split

A few times in the past, I’ve had the need to take a subdirectory of an existing repository and move it to a new repository, while preserving history. I always had to look up the syntax for git filter-branch to do this; it worked, but wasn’t very straightforward or easy to remember. At some point, a subtree split command was added to git that makes this process much simpler. My real-world use case was in the migration and modernization of our puppet installation....

April 28, 2021 · 2 min · Jason Lavoie

Iteration in Puppet

The problem In the process of modernizing our puppet infrastructure, I’ve been able to remove/delete many of the custom modules I had written many years ago and use community developed and supported modules from the puppet forge. Many of these modules accommodate a pattern of a single include in the manifest, and putting all the site-specific configuration data (if any) in hiera. That said, some don’t allow this, and resources must be explicitly configured....

April 2, 2021 · 3 min · Jason Lavoie

Cleaning up old git branches

We make heavy use of puppet environments in our workflow. Using r10k, git branches are magically mapped to environments. This allows a process where anyone one the team can individually work on a new feature or change, and then we can collaborate and review/revise/test in a controlled manner. We can rebase to the production branch, and use the diff output as part of our change-management documentation. Once the change is merged, however, sometimes the original branch is not deleted....

March 11, 2021 · 2 min · Jason Lavoie
PXEboot bootscreen

Start Puppet in Debian Preseed

I have a nice netboot setup where we can PXEboot hosts to an automated installer. The last step ensures the puppet agent is running and pointed at the correct puppetmaster. The .preseed files are generated from an erb template that ends in the following: [...] <% if @distcodename == "jessie" -%> d-i preseed/late_command string \ echo -e 'DAEMON_OPTS="--server <%= @puppetmaster %>"' > /target/etc/default/puppet ; \ rm -f /target/var/lib/puppet/state/agent_disabled.lock <% else -%> d-i preseed/late_command string \ in-target sed -i '/\[main\]/a server = <%= @puppetmaster %>' /etc/puppet/puppet....

February 19, 2021 · 1 min · Jason Lavoie