Puppet

While updating some old code to add a small feature, I noticed a new error in the deployment where a puppet vcsrepo resource was failing. Error: /Stage[main]/Mirror::Crowdstrike/Mirror::Pymirror[crowdstrike]/Vcsrepo[/opt/crowdstrike-mirror]/ensure: change from 'absent' to 'latest' failed: Execution of 'git clone git://github.com/bowdoincollege/noc-crowdstrike-mirror.git /opt/crowdstrike-mirror' returned 128: Cloning into '/opt/crowdstrike-mirror'... fatal: unable to connect to github.com: github.com[0: 140.82.114.4]: errno=Connection timed out I logged into the box and ran the command directly to confirm. p-mirror-a:/opt/crowdstrike-mirror$ git fetch origin fatal: unable to connect to github.com: github.com[0: 140.82.112.3]: errno=Connection timed out This is for a public GitHub repository. My first thought was that we had inadvertently changed it to a private one, but that theory was quickly disproved. Network connectivity seemed okay, as the IP was pingable, just not responding on that port. This repository was forked from another repository, and I was able to clone that without issue. ...

I recently upgraded an old Debian system sitting in the lab to a modern release. I had neglected to keep it updated, and it was flagged on an internal scan for having out of date software. To prevent this oversight in the future, I added it to our puppet deployment (so it would get software updates and be kept in line with our standards) and set it up in our monitoring cluster (so that we’d know if puppet or the updates broke). ...

After upgrading a nameserver to Debian 10, I noticed some AppArmor errors in /var/log/auth.log: Jul 29 09:58:18 koala audit[1676]: AVC apparmor="DENIED" operation="mknod" profile="/usr/sbin/named" name="/etc/bind/namedb/dyn/example.com.jnl" pid=1676 comm="isc-worker0029" requested_mask="c" denied_mask=" c" fsuid=112 ouid=112 It appears that a default ISC bind install now restricts named to read-only access on /etc/bind. According to /etc/apparmor.d/usr.sbin.named: [...] # /etc/bind should be read-only for bind # /var/lib/bind is for dynamically updated zone (and journal) files. # /var/cache/bind is for slave/stub data, since we're not the origin of it. # See /usr/share/doc/bind9/README.Debian.gz /etc/bind/** r, /var/lib/bind/** rw, /var/lib/bind/ rw, /var/cache/bind/** lrw, /var/cache/bind/ rw, [...] The relevant portion of /usr/share/doc/bind9/README.Debian.gz: ...

I had an interesting design requirement for a network monitoring host. These monitoring hosts, or collectors, are used to monitor our network from an external perspective – via the Internet. They also needed to be reachable from our internal network for central management, and needed access to shared internal services, such as directory services, time servers, and central logging. Design My initial approach was to deploy the hosts in a public subnet, set the default route over the Internet, and add individual host routes via the transit gateway to the subnet routing table. This was not great from an operational perspective and violated the requirements when one of the statically-routed hosts also needed to be monitored externally. ...

A cow-orker came to me with a puppet issue today. He was trying to remove a package from a fleet of RedHat servers, using ensure => absent in the package resource, but it was failing: Error: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78.9.0-1.el7_9.x86_64 firefox-78.9.0-1.el7_9.i686 Error: /Stage[main]/Profile::Base::Firefox/Package[firefox]/ensure: change from '78.9.0-1.el7_9' to 'absent' failed: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78.9.0-1.el7_9.x86_64 firefox-78.9.0-1.el7_9.i686 A quick search showed that rpm has an --allmatches option. From rpm(8): ...

A few times in the past, I’ve had the need to take a subdirectory of an existing repository and move it to a new repository, while preserving history. I always had to look up the syntax for git filter-branch to do this; it worked, but wasn’t very straightforward or easy to remember. At some point, a subtree split command was added to git that makes this process much simpler. My real-world use case was in the migration and modernization of our puppet installation. A local module for managing our Opsview installation was kept in the control repo. Over the years, our locally-maintained nagios plugins have grown to a point where they may be better maintained in a separate repository. ...

The problem In the process of modernizing our puppet infrastructure, I’ve been able to remove/delete many of the custom modules I had written many years ago and use community developed and supported modules from the puppet forge. Many of these modules accommodate a pattern of a single include in the manifest, and putting all the site-specific configuration data (if any) in hiera. That said, some don’t allow this, and resources must be explicitly configured. In the past, we’ve used create_resources in combination with hiera lookups. ...

We make heavy use of puppet environments in our workflow. Using r10k, git branches are magically mapped to environments. This allows a process where anyone one the team can individually work on a new feature or change, and then we can collaborate and review/revise/test in a controlled manner. We can rebase to the production branch, and use the diff output as part of our change-management documentation. Once the change is merged, however, sometimes the original branch is not deleted. ...

I have a nice netboot setup where we can PXEboot hosts to an automated installer. The last step ensures the puppet agent is running and pointed at the correct puppetmaster. The .preseed files are generated from an erb template that ends in the following: [...] <% if @distcodename == "jessie" -%> d-i preseed/late_command string \ echo -e 'DAEMON_OPTS="--server <%= @puppetmaster %>"' > /target/etc/default/puppet ; \ rm -f /target/var/lib/puppet/state/agent_disabled.lock <% else -%> d-i preseed/late_command string \ in-target sed -i '/\[main\]/a server = <%= @puppetmaster %>' /etc/puppet/puppet.conf ; \ in-target ln -s /lib/systemd/system/puppet.service /etc/systemd/system/multi-user.target.wants/puppet.service <% end -%> Older distribution versions allowed us to populate the --server option in /etc/default/puppet. This is addressed in the first part of the if clause. ...