Process flow of a GitHub AWS Connector App connecting to CodePipeline and publishing to an SNS topic

Connecting GitHub to SNS using CodePipeline

Background In the last post, I documented an approach to fan-out GitHub repository updates to AWS services using API Gateway, Lambda, and SNS. In my conclusion, I wrote: The whole time I was building and testing this, I kept thinking to myself, “I must be overlooking a more obvious solution.” I’ve asked around, and it seems that others have also run into this issue, but ended up using a different approach that didn’t involve authorization....

August 20, 2021 · 7 min · Jason Lavoie
Process flow of a webhook through API Gateway using a lambda integration to publish to SNS

Publish to SNS with GitHub webhooks

Motivation and Design I have a bunch of “audit scripts” that run against the network configurations (and other data sources, such as DNS and DHCP) to check for common problems, mistakes, and inconsistencies. They run on a centralized server that periodically fetches the latest data from all these sources, runs the scripts, and emails about any discrepancies. This data sources are kept in git repositories, either updated by operations staff, or automatically....

August 16, 2021 · 10 min · Jason Lavoie

Use VLAN groups for UCS vNIC templates

One of my co-workers had provisioned a new appliance VM. It was having connectivity problems, so he asked me to look at it. Upon investigation, I found: absolutely no connectivity: RX Packets 0 on the interface. this was the first/only VM in this VLAN on this vCenter cluster they had just added this VLAN to the dvSwitch for this project So, I first checked what had changed most recently, the dvSwitch config....

August 10, 2021 · 3 min · Jason Lavoie

Cisco fan direction mismatch

Many of Cisco’s switches can be purchased in two different airflow configurations, port-side intake and port-side exhaust. Since most racks are designed with a front-to-back airflow, this allows for mounting a switch in the front or back of the rack, respectively. The latter scenario, for example, we use for a top of rack (ToR) deployment for server racks. Most times, despite selling these as different SKUs, the switch is actually the same part number, and all that differs are the part numbers of the fans and power supplies....

August 3, 2021 · 3 min · Jason Lavoie

Override AppArmor policy for bind

After upgrading a nameserver to Debian 10, I noticed some AppArmor errors in /var/log/auth.log: Jul 29 09:58:18 koala audit[1676]: AVC apparmor="DENIED" operation="mknod" profile="/usr/sbin/named" name="/etc/bind/namedb/dyn/example.com.jnl" pid=1676 comm="isc-worker0029" requested_mask="c" denied_mask=" c" fsuid=112 ouid=112 It appears that a default ISC bind install now restricts named to read-only access on /etc/bind. According to /etc/apparmor.d/usr.sbin.named: [...] # /etc/bind should be read-only for bind # /var/lib/bind is for dynamically updated zone (and journal) files. # /var/cache/bind is for slave/stub data, since we're not the origin of it....

July 30, 2021 · 2 min · Jason Lavoie

No matching key exchange method

After upgrading some bastion hosts to Debian 10, connections to some older network gear failed. Connecting to some ASA firewalls generated the error: Unable to negotiate with 203.0.113.203 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 This was a simple fix: lab-5585-1# conf t lab-5585-1(config)# ssh key-exchange group dh-group14-sha1 lab-5585-1(config)# end Some older devices, Catalyst 3750 switches and ASA 5540 firewalls, complained of no matching cipher: %SSH-3-NO_MATCH: No matching cipher found: client chacha20-poly1305@openssh....

July 29, 2021 · 1 min · Jason Lavoie

Clearing HSTS on localhost

I use a few tools that create local web server: vim instant markdown hugo server These normally work well. I also regularly will use a tunnel to a host on another network, such as accessing an embedded management interface of a device on an isolated network: desktop:~$ ssh -L 8443:device:443 bastion bastion:~$ The service on the remote network device is now available locally via https://localhost:443/. Unfortunately, when I do this, my local browser will store these HSTS settings for the domain (localhost, in this case) and complain/fail when one of the above-listed tools goes to a non-HTTPS URL on localhost, such as http://localhost:8090 for instant-markdown....

June 30, 2021 · 1 min · Jason Lavoie

F5 management firewall rules

After upgrading our F5’s a while back – probably to a BIG-IP 14.1 release, from looking at the release notes – our monitoring of their NTP status started failing. One of our staff poked at it and even opened a support case with F5, but couldn’t get it working, so it ended up on my list of things to look at. Today, I finally spent a few minutes troubleshooting and found the problem and an easy fix....

June 23, 2021 · 2 min · Jason Lavoie
[Split](https://pixabay.com/photos/log-bark-ball-glass-ball-split-4164303/) by [manfredrichter](https://pixabay.com/users/manfredrichter-4055600/) licensed under [CC0](https://creativecommons.org/publicdomain/zero/1.0/legalcode)

Multi-homed EC2

I had an interesting design requirement for a network monitoring host. These monitoring hosts, or collectors, are used to monitor our network from an external perspective – via the Internet. They also needed to be reachable from our internal network for central management, and needed access to shared internal services, such as directory services, time servers, and central logging. Design My initial approach was to deploy the hosts in a public subnet, set the default route over the Internet, and add individual host routes via the transit gateway to the subnet routing table....

June 22, 2021 · 10 min · Jason Lavoie

Using docker to compile a binary

Sometimes I have to compile a binary or build a custom package on an old platform or an operating system where I don’t have a compile host available. Docker is a perfect tool for this type of ad-hoc workflow. docker run --rm -it -v $(pwd):/mnt ubuntu:bionic sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list apt-get update apt-get -y install dpkg-dev libssl-dev # any other dependencies cd apt-get source source-package-here # cd into package and compile/make/build/etc strip resulting_binary cp resulting_binary /mnt exit This mounts the current directory at the /mnt mount point in the container....

June 21, 2021 · 1 min · Jason Lavoie
Diagram of SQL MI creation flow

Updating AzureRM templates from Terraform

Summary I have deployed some Azure SQL Managed Instances using Terraform. Since there are no native resources for this service in the Azure provider, I used an Azure Resource Manager deployment template. Recently, I had to add an output to that template (so that another workspace could set up remote logging), and wanted to note my experience with updating deployment templates from Terraform. Here, I’ll detail the original design and then walk through the update process....

May 19, 2021 · 10 min · Jason Lavoie

vim-go initializing gopls

After some overly-aggressive cleaning of my GOPATH, vim “hung” with the message “vim-go: initializing gopls” the next time I edited a .go file. I discovered that running :GoInstallBinaries in vim would “fix” the problem and re-install the missing packages. vim-go: fillstruct not found. Installing github.com/davidrjenni/reftools/cmd/fillstruct@master to folder /Users/jlavoie/go/bin/ vim-go: godef not found. Installing github.com/rogpeppe/godef@master to folder /Users/jlavoie/go/bin/ vim-go: motion not found. Installing github.com/fatih/motion@master to folder /Users/jlavoie/go/bin/ vim-go: errcheck not found. Installing github....

May 18, 2021 · 2 min · Jason Lavoie

Problem uninstalling packages with puppet on RHEL

A cow-orker came to me with a puppet issue today. He was trying to remove a package from a fleet of RedHat servers, using ensure => absent in the package resource, but it was failing: Error: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78.9.0-1.el7_9.x86_64 firefox-78.9.0-1.el7_9.i686 Error: /Stage[main]/Profile::Base::Firefox/Package[firefox]/ensure: change from '78.9.0-1.el7_9' to 'absent' failed: Execution of '/bin/rpm -e firefox' returned 1: error: "firefox" specifies multiple packages: firefox-78....

May 5, 2021 · 2 min · Jason Lavoie
Sonus SBC 2000

Ribbon SBC interface redundancy

Single-homed SBC In planning to migrate phone traffic from PRI to SIP, we decided to use an existing pair of session border controllers (SBCs) that were already in production for another (smaller) deployment. Before cutting over the whole organization’s voice traffic, I revisited the (3-year old) network design. While the two SBCs are in separate datacenters in separate buildings, each SBC is only single-homed. This means that there is SBC high-availability in terms of new calls, but existing calls will be dropped if there is a failure or maintenance on the switch....

April 29, 2021 · 8 min · Jason Lavoie

Git subtree split

A few times in the past, I’ve had the need to take a subdirectory of an existing repository and move it to a new repository, while preserving history. I always had to look up the syntax for git filter-branch to do this; it worked, but wasn’t very straightforward or easy to remember. At some point, a subtree split command was added to git that makes this process much simpler. My real-world use case was in the migration and modernization of our puppet installation....

April 28, 2021 · 2 min · Jason Lavoie
Crossroads

Direct Connect with VPN backup

The problem A common AWS connectivity design is to have a direct connect (DX) connection with a VPN backup. There are some routing concerns to consider when implementing this design to make sure that traffic prefers the DX circuit and only uses the backup VPN path if the DX is unavailable. Traffic from AWS transit gateway (TGW) will always prefer the direct connect gateway (DXGW) path, but traffic in the other direction (to AWS) is dependent on the customer gateway (CGW) routing policy....

April 27, 2021 · 3 min · Jason Lavoie

Ubuntu multiarch mirror

I maintain a local mirror site for the Linux distributions we use. This is a simple rsync setup using ftpsync and Apache. I recently added Ubuntu to the list, but ran into an issue when I tested an automated install. The installer complained it was “Unable to locate package puppet.” In the preseed file, I tell the installer to also install this package with a pkgsel directive. (Later, using a late_command directive, the service is configured and started....

April 22, 2021 · 2 min · Jason Lavoie

ASA TCP state bypass

What it does By default an ASA does stateful inspection of all traffic. It must see the entire conversation to be able to set up the connection and pass the traffic. If traffic is asymmetric, such that the ASA only sees traffic in one direction, the packets will not be passed. Additionally, even if the traffic is symmetric and a new connection is established, subsequent fast path packets will be inspected for things such as TCP sequence number randomization, TCP normalization, and other checks....

April 17, 2021 · 3 min · Jason Lavoie
Multi-region dual-stack TGW/DXGW design

Where AWS IPv6 networking fails

Introduction AWS has made much progress over the years with IPv6 support. From S3, EC2, Cloudfront, Route53, and EC2 support back in 2016, to more recent updates to NLB and the EC2 API, I’ve appreciated every advancement and patiently waited for the next. Unfortunately, there are still pieces missing that prevent me from making full use of IPv6 in my employer’s current environment. Existing architecture The architecture is modeled after one of AWS’s recommended connectivity designs....

April 15, 2021 · 4 min · Jason Lavoie

iproute2 blackhole route

Today I was doing some empirical testing of an application’s behavior when one of its authentication servers becomes unreachable. I typically do this with a null route on an upstream device, but noticed that iproute2 has this built in with a nice, memorable syntax. According to ip-route(8), one of the route types is blackhole: blackhole - these destinations are unreachable. Packets are discarded silently. The local senders get an EINVAL error....

April 14, 2021 · 1 min · Jason Lavoie