Month: November 2016

Easy-peasy VPC Reference Configuration for Scenario 2 Deployments

A very popular VPC configuration is the multi-AZ public/private layout that AWS describes as “Scenario 2”:

“The configuration for this scenario includes a virtual private cloud (VPC) with a public subnet and a private subnet. We recommend this scenario if you want to run a public-facing web application, while maintaining back-end servers that aren’t publicly accessible.”

Historically, AWS has provided a NAT instance AMI to use for Scenario 2 VPC’s, along with a HA-heartbeat configuration script that runs on each NAT instance. They’ve even published a CloudFormation template to build out a VPC according to this design. Recently however, with the advent of the NAT Gateway service, AWS now promotes that solution as preferable to NAT instance configurations for Scenario 2 deployments.

So Why Make a New Scenario 2 CloudFormation Template?

Given that AWS has published a CF template for Scenario 2 deployments, you may wonder why I chose to create my own set of templates. Let’s talk about why…

First, I realized that I wanted to be able to deploy a Scenario 2 VPC with *either* a NAT instance configuration or a NAT Gateway configuration. This new template reference allows me to do that. It also allowed me to discover why I might not want to use NAT Gateways, but I’ll get to that a little later.

Secondly, the published Scenario 2 VPC template does not include any perimeter security configuration a la network ACLs. Given that there are publically accessible subnets in a Scenario 2 deployment, I wanted to have the extra layer of security that network ACLs can provide.

Note: The default VPC configuration in your AWS account includes network ACLs that are wide-open, and when you create a new custom VPC like a Scenario 2 deployment, you must configure network ACLs from scratch.

Lastly, I wanted to integrate a VPC endpoint for S3 access to give that design a whirl. VPC endpoints are very useful in that they allow public service access inside a VPC directly without crossing an Internet gateway. They also isolate a substantial stream of network traffic from affecting either your NAT or Internet gateway flows. There are some caveats to using a S3 VPC endpoint, more on those later in this post.

A New Template-based Reference Configuration for Scenario 2 VPC Deployments

I’ve added my new Scenario 2 VPC reference configuration templates to my aws-mojo repository on GitHub. Feel free to pull those up in another window while we review them in more detail.

I initially started by creating a typical Scenario 2 VPC template with NAT instances. This template provides:

  • a VPC with four subnets (2 public, 2 private) in two availability zones
  • Network ACLs for both public and private subnets
  • one NAT instance for each availability zone, each with its own Elastic IP (EIP)
  • a NAT instance IAM role/policy configuration (with slight modification)
  • cargo-cult porting of AWS’s nat_monitor.sh HA-heartbeat scripts (with slight modification) for the NAT instances (parameter defaults from AWS)
  • a RDS subnet group in the private zones

The one change to the nat_monitor.sh script I made was to add some code to associate the newly created EIP with the NAT instances during instance first-boot. I found that this decreases the wait time required for the NAT instances to become operational via their EIP’s. Otherwise, there is some additional delay time for the automatic association of the EIP’s to the instances that normally occurs.

Here’s the relevant bit of code that I added to the UserData section of the NAT instance resource definition:

"UserData": {
  "Fn::Base64": {
    "Fn::Join": [
       "",
       [
         "#!/bin/bash -v\n",
         "yum update -y aws*\n",
         ". /etc/profile.d/aws-apitools-common.sh\n",
         "# Associate EIP to ENI on instance launch\n",
         "INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`\n",
         "EIPALLOC_ID=$(aws ec2 describe-addresses --region ",
         {
           "Ref": "AWS::Region"
         },
         " --filters Name=instance-id,Values=${INSTANCE_ID} --output text | cut -f2)\n",
         "aws ec2 associate-address --region ",
         {
           "Ref": "AWS::Region"
         },
         " --instance-id $INSTANCE_ID --allocation-id $EIPALLOC_ID\n",

Note: For this to work, I also had to modify the IAM policy for the NAT instance role to include the actions ec2:DescribeAddresses and ec2:AssociateAddress.

With the addition of the Network ACL configuration, I eventually surpassed the template body size limit for validating CloudFormation templates via the AWSCLI. I also knew that I wanted to create a couple of sample EC2 security groups for private and public instances, in addition to the S3 VPC endpoint. So, at this point, I opted to created a nested NAT instance template, which contains resource definitions for three additional CloudFormation child stacks:

  • aws-vpc-network-acls [ json | yaml ]
  • aws-vpc-instance-securitygroups [ json | yaml ]
  • aws-vpc-s3endpoint [ json | yaml ]

I followed general recommendations from AWS for the network ACLs and instance security groups. I also modified the configurations to suit my own needs as well, so you should review them and decide if they are secure enough for your own deployments.

For a NAT Gateway version of this Scenario 2 deployment, just use the aws-vpc-nat-gateway template (json|yml) instead of the aws-vpc-nat-instances template (json|yml). It also is a nested template and references the three child templates listed above.

Here are diagrams showing the high-level architecture of each reference stack:

vpc-reference-nat-instances

vpc-reference-nat-gateways

So How Do I Use This New VPC Mojo?

Download the templates from my aws-mojo repo and using the CloudFormation console, load the parent template of your choice (NAT instance or NAT Gateway). You should store the templates in the S3 bucket location of your choice prior to launching in the console (Duh!). However, you should make note of the S3Bucket and TemplateURL parameters in the parent template as you will need to input those values during the template launch.

Other parameters that will require either your input or consideration:

  • Environment – used for tag and object naming
  • ConfigS3Endpoint – defaults to no, see caveats below
  • NatKeyPair – an existing EC2 keypair used to create NAT instances
  • NatInstanceType – defaults to t2.micro which is fine for PoC deployment, not production
  • RemoteCIDR – the network block from which you are accessing your AWS account

Note: The NAT Gateway version of the template does not require either NatKeyPair or NatInstanceType parameters.

The NAT instances template will render a useable VPC in about 15 minutes when I deploy into the busy us-east-1 region; the NAT gateway template renders in about 5-10 minutes. YMMV.

After reviewing the VPC endpoint caveat below, you can try using the S3 endpoint configuration by simply updating the parent CF stack and selecting “yes” for the S3Endpoint parameter.

Deploy instances into the public and private subnets using the security groups provided to test out your VPC networking and operational limits.

Caveats and Alligators

Caveat #1 – Don’t Use This As-Is For Production

I’ve designed this reference configuration for free-tier exploration of VPC buildouts. The NAT instance template defaults to using t2.micro instances, which is clearly insufficient for any real-world production usage. Feel free to use this configuration as a foundation for building your own real-world template-based deployments.

Caveat #2 – With my mind on my money and my money on my mind

I discovered the hard way about using NAT Gateways for my lab work. NAT Gateways are billed on an hourly basis along with usage fees. After deploying a VPC with NAT Gateways instead of NAT instances and letting it hang out for a while, I noticed my monthly bill jumped by quite a bit. Keep this in mind. In addition, you will need to maintain at least one bastion instance in one of the public subnets so you can get to your private zone instances. All things said, NAT Gateways are much preferred for production deployment vs. instances as they are simpler to manage and avoid the whole heartbeat/failover false-positive and/or split-brain problem associated with NAT instance configurations. However, for PoC work, you will accrue costs quickly with a NAT Gateway solution. I like to use NAT instances and then turn them off when I’m not actively working on a project.

Caveat #3 – S3 VPC Endpoint Gotchas

Offloading S3 traffic from your NATs and Internet gateways is a good thing. However, there are known issues with using VPC Endpoints. The endpoint policy I use in this reference stack deals with the issue of allowing access to AWS repos for AMZN Linux package and repo content, but there are other issues that you will need to address should you go down the path of using S3 Endpoints.

 

CloudFormation Templates in YAML

AWS recently announced support for authoring CloudFormation templates in YAML instead of JSON. This is a big deal for one simple reason: YAML supports the use of comments, which has been a major gap in JSON templating.

YAML is a ubiquitous data serialization language and is used a lot for configuration file syntax as well as an alternative to JSON and XML. It has a smallish learning curve because of non-intuitive features like the syntactical importance of indentation. Nevertheless, it offers a strong alternative to authoring files in JSON because of its readability and relative lack of delimiter collision.

If you have existing JSON CloudFormation templates, you can convert them to YAML via the most excellent Python package “json2yaml“. Installing the package is as simple as:

pip install json2yaml

Once installed, you can try converting a template as follows:

cd /path/to/templates
json2yaml ./mytemplate.json ./mytemplate.yml

If you do not specify the 2nd parameter for the YAML output file, json2yaml will stream the converted file content to STDOUT.

I used json2yaml to convert a relatively sophisticated JSON-based CloudFormation template for deploying a CodeCommit repository and then used the YAML output version to create a new CF stack and it worked flawlessly.

To learn more about YAML, I recommend reading the Wikipedia page about it along with using this handy reference sheet from yaml.org.

Now, go forth and create stacks with all the comments you have ever wanted to include!

 

Confusing syntax error with AWS CLI validation of CF templates

When using awscli to create CloudFormation stacks, there’s a pre-create step of template validation that checks for template syntax issues:

aws cloudformation validate-template --template-body file:///path/to/template.json

The URI prefix file:// indicates that we are using local templates while templates at web-accessible locations like your S3 bucket are accessed using the –template-url option. For more information see the awscli docs or CLI help:

aws cloudformation validate-template help

For local templates, make sure you don’t forget the file:// URI and try to refer to the template via normal filesystem paths, otherwise you’ll get a confusing syntax error.

Without file:// URI:

aws cloudformation validate-template --template-body /Users/foo/git-repos/aws-mojo/cloudformation/aws-deploy-codecommit-repo.yml

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: unsupported structure.

That’s a rather unhelpful error message that makes me think I’ve got some sort of template content error.

When we run the same command with file:// URI, we get the expected output:

aws cloudformation validate-template --template-body file:///Users/foo/git-repos/aws-mojo/cloudformation/aws-deploy-codecommit-repo.yml
{
   "Description": "CloudFormation template for creating a CodeCommit repository along with a SNS topic for repo activity trigger notifications",
   "Parameters": [
 {
   "NoEcho": false,
   "Description": "Email address for SNS notifications on repo events",
   "ParameterKey": "Email"
 },
 {
   "NoEcho": false,
   "Description": "A unique name for the CodeCommit repository",
   "ParameterKey": "RepoName"
 },
 {
   "DefaultValue": "Dev",
   "NoEcho": false,
   "Description": "Environment type (can be used for tagging)",
   "ParameterKey": "Environment"
 },
 {
   "NoEcho": false,
   "Description": "A description of the CodeCommit repository",
   "ParameterKey": "RepoDescription"
 }
 ]
}

Git Smart with CodeCommit!

AWS recently announced that CodeCommit repositories can now be created via CloudFormation, which spurred me finally to take the opportunity to create my own home lab git repo. While I do have public GitHub repos, I have wanted a private repo for my experimental coding and other bits that aren’t ready or destined for public release. I could build my own VM at home to host a git repo (I recently tinkered with GitLab Community Edition), but then I have to worry about backups, accessibility from remote locations, etc.  As it turns out, you can build and use a CodeCommit repo for free in your AWS account, which made it even more compelling. So, I decided to give CodeCommit a try.

CodeCommit is a fully managed Git-based source control hosting service in AWS. Being fully managed, you can focus on using the repo rather than installing one, then maintaining, securing, backing it up, etc. And, it’s accessible from anywhere just like your other AWS services. The first 5 active users are free, which includes unlimited repo creation, 50 GB of storage, and  10,000 Git requests per month. Other benefits include integration paths with CodeDeploy and CodePipeline for a full CD/CI configuration. For a developer looking for a quick and easy way to manage non-public code, AWS offers a very attractive proposition to build your Git repo in CodeCommit.

QuickStart: Deploying Your Own CodeCommit Repo

  1. Download my CodeCommit CloudFormation template (json|yaml) and use to create your new repo.
  2. Add your SSH public key to your IAM user account and configure your SSH config to add a CodeCommit profile.
  3. Clone your new repo down to your workstation/laptop (be sure to use the correct AWS::Region and repository name): 
    git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/yournewrepo

DeeperDive: Deploying Your Own CodeCommit Repo

Step 1: Building the CodeCommit Repository

I’ve created a CloudFormation template that creates a stack for deploying a CodeCommit repository. There are two versions, one in JSON and one in YAML, which is now supported for CF templating. Take your pick and deploy using either the console or via the AWS CLI.

You need to specify four stack parameters:

  • Environment (not used, but could be used in Ref’s for tagging)
  • RepoName (100-character string limit)
  • RepoDescription (1000-character string limit)
  • Email (for SNS notifications on repo events)

Here are the awscli commands required with sample parameters:

# modify the template if needed for your account particulars then validate:
$ aws cloudformation validate-template --template-body file:///path/to/template/aws-deploy-codecommit-repo.yml

$ aws cloudformation create-stack --stack-name CodeCommitRepo --template-body file:///path/to/template/aws-deploy-codecommit-repo.yml  --parameters ParameterKey=Environment,ParameterValue=Dev ParameterKey=RepoName,ParameterValue=myrepo ParameterKey=RepoDescription,ParameterValue='My code' ParameterKey=Email,ParameterValue=youremail@someplace.com

In a few minutes, you should have a brand new CloudFormation stack along with your own CodeCommit repository. You will receive a SNS notification email if you use my stock template, so be sure to confirm your topic subscription to receive updates when the repository event trigger runs (e.g., after commits to the master branch).

Step 2: Configure Your IAM Account With a SSH Key

Assuming that you, like myself, prefer to use SSH for git transactions, you will need to add your public SSH key to your IAM user in your AWS account. This is pretty straightforward and the steps are spelled out in the CodeCommit documentation.

Step 3: Clone Your New Repo

Once you’ve configured your SSH key in your IAM account profile, you can verify CodeCommit access like so:

ssh git-codecommit.us-east-1.amazonaws.com

Once you are able to talk to CodeCommit via git over SSH, you should be able to clone down your new repo:

git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/yournewrepo

You will want to specify a repo-specific git config if you don’t use the global settings for your other repos:

git config user.name "Your Name"
git config user.email youremail@someplace.com

Now you are ready to add files to your new CodeCommit repository. Wasn’t that simple?