Motivation and Design
I have a bunch of “audit scripts” that run against the network configurations (and other data sources, such as DNS and DHCP) to check for common problems, mistakes, and inconsistencies. They run on a centralized server that periodically fetches the latest data from all these sources, runs the scripts, and emails about any discrepancies. This data sources are kept in git repositories, either updated by operations staff, or automatically. In the case of networking gear, by a tool called RANCID that collects the text configuration and output of many useful “show” commands and pushes any changes a git repository for the role/group of the device.
In modernizing our stack to a more event-driven approach, I wanted to re-architect a bit so that any commit to these repositories would trigger a run of the appropriate audit scripts. Since each check is relatively independent, this seemed like a good use of Amazon Simple Notification Service (SNS). I could configure a webhook to publish to an SNS topic whenever there was an update pushed to a GitHub repository.
THIS WILL NOT WORK
I expected this to be a simple configuration, but it wasn’t all that straightforward. For what I think would be such a common task, it took me quite a bit of research and reading of documentation to get a satisfactory solution. Here I detail the approach(es) that I’ve come up with to help assist anyone else that needs this functionality.
I will share two workable solutions. A simple one without authentication, and a more complex one that allows you to specify a “secret” token with the GitHub webhook.
Alternatives
GitHub Services
There is a lot old information out there, including this AWS blog post, that recommends using “GitHub Services” to integrate with AWS services such as SNS. This feature has been deprecated and they recommend using webhooks instead.
GitHub Actions
Another approach would be to use GitHub Actions to publish to SNS. Here is an example action which allows you to do this with just a few lines in a workflow file. This may be a simpler solution for some, but I did not choose – or test – this path, as I did not want to modify the repositories themselves and wanted to manage this entirely within Terraform.
API Gateway Service Proxy
This is the simplest solution I’ve found: using API Gateway and a service proxy to build a HTTP REST API with a resource that will publish to an SNS topic. The integration allows for manipulation of the input data into the format needed by the SNS API by using the flexible Velocity Templating Language (VTL). There is no additional coding needed.
The biggest drawback of this solution is that there is no authorization on the endpoint; anyone can POST to this URL. If you have other means of limiting access, such as limiting IP access with a resource policy, or using a VPC endpoint for the REST API with an endpoint policy, then this may be your best bet. Later, I explore another solution that works with the native signature authorization available with GitHub webhooks.
SNS
First, create a topic that we will publish to.
resource "aws_sns_topic" "this" {
name = "sns-topic-${var.environment}-demo"
}
For testing, manually create (and confirm) an email subscription to this topic.
IAM
Next, we need a IAM role that will permit API Gateway to publish to the topic.
data "aws_iam_policy_document" "sns-publish" {
statement {
actions = ["sns:Publish"]
resources = [aws_sns_topic.this.arn]
effect = "Allow"
}
}
data "aws_iam_policy_document" "apigw-proxy-sns" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["apigateway.amazonaws.com"]
}
effect = "Allow"
}
}
resource "aws_iam_role" "sns-publish" {
name = "sns-publish"
assume_role_policy = data.aws_iam_policy_document.apigw-proxy-sns.json
inline_policy {
name = "sns-publish"
policy = data.aws_iam_policy_document.sns-publish.json
}
}
API Gateway
API Gateway resource and method
Configure a REST API with a single resource, /ingest, which has a
POST method configured for the integration with SNS.
I found that the response (and the response_integration) also need
to be configured or the integration will not work.
resource "aws_api_gateway_rest_api" "sns-proxy" {
name = "sns-proxy"
}
resource "aws_api_gateway_resource" "sns-ingest" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
parent_id = aws_api_gateway_rest_api.sns-proxy.root_resource_id
path_part = "ingest"
}
resource "aws_api_gateway_method" "sns-ingest" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
resource_id = aws_api_gateway_resource.sns-ingest.id
authorization = "NONE"
http_method = "POST"
}
resource "aws_api_gateway_method_response" "sns-ingest" {
http_method = aws_api_gateway_method.sns-ingest.http_method
resource_id = aws_api_gateway_resource.sns-ingest.id
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
status_code = "200"
response_models = { "application/json" = "Empty" }
}
API Gateway service proxy configuration
SNS expects to receive the publish request as a POST with all parameters
passed as url-encoded form data in the body of the request. The API
Documentation lists both the required (Message and
Topic/TargetArn) and optional parameters. Using a request template
allows us to map the data to the format expected by SNS. I found that
using the AWS CLI in debug mode was a helpful way to see what was
actually being sent to the SNS API.
The credentials parameter reference the IAM role we defined earlier to
permit API Gateway to publish to the the topic.
Setting passthrough_behavior to NEVER rejects any requests that do
not match a request template (application/json).
resource "aws_api_gateway_integration" "sns-publish" {
http_method = aws_api_gateway_method.sns-ingest.http_method
integration_http_method = "POST"
resource_id = aws_api_gateway_resource.sns-ingest.id
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
type = "AWS"
uri = "arn:aws:apigateway:${var.region}:sns:path//"
credentials = aws_iam_role.sns-publish.arn
request_parameters = {
"integration.request.header.Content-Type" = "'application/x-www-form-urlencoded'"
}
request_templates = {
"application/json" = join("&", [
"Action=Publish&TopicArn=$util.urlEncode('${aws_sns_topic.this.arn}')",
"Message=$util.urlEncode($input.body)",
"Subject=$util.urlEncode('webhook')"
])
}
passthrough_behavior = "NEVER"
}
resource "aws_api_gateway_integration_response" "sns-publish" {
resource_id = aws_api_gateway_resource.sns-ingest.id
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
http_method = aws_api_gateway_method.sns-ingest.http_method
status_code = "200"
response_templates = {
"application/json" = jsonencode({ body = "Message received." })
}
}
API Gateway deployment and stage
A deployment and stage need to be created to deploy the REST API. I copied the trigger from the terraform documentation; note the caveats on types of changes that might not trigger a redeployment.
resource "aws_api_gateway_deployment" "sns-proxy" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
# will re-deploy when resource id changes (not all configuration changes)
triggers = {
redeployment = sha1(jsonencode([
aws_api_gateway_resource.sns-ingest.id,
aws_api_gateway_method.sns-ingest.id,
aws_api_gateway_integration.sns-publish.id,
aws_api_gateway_method_response.sns-ingest.id,
aws_api_gateway_integration_response.sns-publish.id,
]))
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_api_gateway_stage" "sns-proxy" {
deployment_id = aws_api_gateway_deployment.sns-proxy.id
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
stage_name = var.environment
}
GitHub Webhook
Lastly, we configure the webhook on a GitHub repository, specifying the URL and the content-type.
data "github_repository" "repo" {
full_name = "exampleorg/reponame"
}
resource "github_repository_webhook" "webhook" {
repository = data.github_repository.repo.name
events = ["push"]
configuration {
url = "https://${aws_api_gateway_rest_api.sns-proxy.id}.execute-api.${var.region}.amazonaws.com/${var.environment}${aws_api_gateway_resource.sns-ingest.path}"
content_type = "json"
}
}
API Gateway Lambda Integration
Once I had the basics working, I went to add authentication. I had hoped to use an API Gateway Lambda authorizer to validate the signature GitHub generates against the webhook payload. This shared secret would allow me to restrict access to the API to only valid senders, but writing a small lambda function to validate the signature in the request header.
THIS WILL NOT WORK
Unfortunately, this is not possible. As far as I can tell, authorizers do not have access to the request payload. Since validating the signature requires running the same hash algorithm on the received payload and comparing the received and calculated digests, a lambda authorizer is not the tool for the job.
Instead, I removed the SNS proxy and replaced it with a lambda integration which first verifies the signature and then publishes to the SNS topic.
There’s some significant overlap, and some subtle differences, between this solution and the previous “no-authorization” solution. For the sake of clarity, I will include all the relevant code for this solution, despite the repetition.
SNS
As before, create a topic.
resource "aws_sns_topic" "topic" {
name = "sns-topic-${var.environment}-demo"
}
IAM
Define an IAM role that will be assumed by the lambda function, permitting it to publish to the topic.
data "aws_iam_policy_document" "sns-publish" {
statement {
actions = ["sns:Publish"]
resources = [aws_sns_topic.topic.arn]
effect = "Allow"
}
}
data "aws_iam_policy_document" "apigw-proxy-lambda" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["lambda.amazonaws.com"]
}
effect = "Allow"
}
}
resource "aws_iam_role" "assume-lambda" {
name = "assume-lambda"
assume_role_policy = data.aws_iam_policy_document.apigw-proxy-lambda.json
inline_policy {
name = "sns-publish"
policy = data.aws_iam_policy_document.sns-publish.json
}
}
API Gateway
API Gateway integration
This part is a bit simpler than before. Define a single /ingest
resource, and map its POST method to invoke the (to-be-defined) lambda
function. In this case no response configuration is needed.
resource "aws_api_gateway_rest_api" "sns-proxy" {
name = "sns-proxy"
}
resource "aws_api_gateway_resource" "sns-ingest" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
parent_id = aws_api_gateway_rest_api.sns-proxy.root_resource_id
path_part = "ingest"
}
resource "aws_api_gateway_method" "sns-ingest" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
resource_id = aws_api_gateway_resource.sns-ingest.id
authorization = "NONE"
http_method = "POST"
}
resource "aws_api_gateway_integration" "sns-publish" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
resource_id = aws_api_gateway_resource.sns-ingest.id
http_method = aws_api_gateway_method.sns-ingest.hrtp_method
integration_http_method = "POST"
type = "AWS_PROXY"
uri = aws_lambda_function.lambda.invoke_arn
}
API Gateway deployment and stage
Same as before – except a couple fewer resources on the trigger – define a deployment and stage for the REST API.
resource "aws_api_gateway_deployment" "sns-proxy" {
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
triggers = {
redeployment = sha1(jsonencode([
aws_api_gateway_resource.sns-ingest.id,
aws_api_gateway_method.sns-ingest.id,
aws_api_gateway_integration.sns-publish.id,
]))
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_api_gateway_stage" "sns-proxy" {
deployment_id = aws_api_gateway_deployment.sns-proxy.id
rest_api_id = aws_api_gateway_rest_api.sns-proxy.id
stage_name = var.environment
}
Lambda
Upload the code, passing the ARN of the SNS topic and the webhook secret to the function as environment variables. Use the IAM role defined earlier as the function’s execution role, so that it has permissions to publish to the SNS topic.
Give API Gateway permission to run the function, by
specifying the principal and source_arn. The API Gateway
Documentation was helpful in determining its ARN.
data "aws_caller_identity" "current" {}
resource "aws_lambda_permission" "apigw_lambda" {
statement_id = "AllowExecutionFromAPIGateway"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.lambda.function_name
principal = "apigateway.amazonaws.com"
source_arn = "arn:aws:execute-api:${var.region}:${data.aws_caller_identity.current.account_id}:${aws_api_gateway_rest_api.sns-proxy.id}/*/${aws_api_gateway_method.sns-ingest.http_method}${aws_api_gateway_resource.sns-ingest.path}"
}
data "archive_file" "lambda" {
type = "zip"
output_path = "lambda.zip"
source_file = "lambda.py"
}
resource "random_password" "github-secret" {
length = 32
}
resource "aws_lambda_function" "lambda" {
filename = "lambda.zip"
function_name = "webhook-sns-publish"
handler = "lambda.handler"
runtime = "python3.8"
role = aws_iam_role.assume-lambda.arn
environment {
variables = {
"GITHUB_SECRET" = random_password.github-secret.result
"TOPIC_ARN" = aws_sns_topic.topic.arn
}
}
source_code_hash = data.archive_file.lambda.output_base64sha256
}
Lambda Function
This is a minimal to check the signature and, if valid, publish the payload to the SNS topic.
According to their developer documentation,
GitHub generates a signature by “using a HMAC hex digest to compute the
hash” using the secret token and the payload of the webhook it is sending.
This hash is sent in the X-Hub-Signature-256 header. To validate it,
we do the same computation with the secret token and the received payload
and check to see if the resulting hash is the same.
Publishing to the topic uses the boto3 library to submit the necessary parameters to the SNS API.
import os
import boto3
import botocore
from json import dumps
from hashlib import sha256
from hmac import HMAC, compare_digest
def handler(event, context):
if verify_signature(event["headers"], event["body"]):
if publish_sns(event["body"]):
return respond("Success")
else:
return respond("Failed", 500)
return respond("Forbidden", 403)
def respond(message, code=200):
return {"statusCode": code, "body": dumps({"message": message})}
def publish_sns(message):
try:
arn = os.environ.get("TOPIC_ARN")
client = boto3.client("sns")
response = client.publish(
TargetArn=arn,
Message=dumps({"default": dumps(message)}),
MessageStructure="json",
)
except botocore.exceptions.ClientError as e:
print(f"ClientError: {e}")
return False
else:
return True
def verify_signature(headers, body):
try:
secret = os.environ.get("GITHUB_SECRET").encode("utf-8")
received = headers["X-Hub-Signature-256"].split("sha256=")[-1].strip()
expected = HMAC(secret, body.encode("utf-8"), sha256).hexdigest()
except (KeyError, TypeError):
return False
else:
return compare_digest(received, expected)
GitHub Webhook
Finally, manage a webhook on the specified GitHub repository, setting
the url to the API endpoint and supplying the shared key as secret.
data "github_repository" "repo" {
full_name = "exampleorg/reponame"
}
resource "github_repository_webhook" "webhook" {
repository = data.github_repository.repo.name
events = ["push"]
configuration {
url = "https://${aws_api_gateway_rest_api.sns-proxy.id}.execute-api.${var.region}.amazonaws.com/${var.environment}${aws_api_gateway_resource.sns-ingest.path}"
secret = random_password.github-secret.result
content_type = "json"
}
}
Conclusion
The whole time I was building and testing this, I kept thinking to myself, “I must be overlooking a more obvious solution.” I’ve asked around, and it seems that others have also run into this issue, but ended up using a different approach that didn’t involve authorization. If you know of a better/different solution, please reach out!
I plan to flesh this out and create a terraform module out of it, as I expect I will want to use this pattern in other places.