git

Configuring git credentials for CodeCommit and other repositories

Working with AWS CodeCommit repositories in addition to other repositories from the same git configuration can be a challenge depending on your git configuration. I’d like to share an approach that works for me when using HTTPS (instead of SSH keys) and hopefully it will be helpful for you, too. I will be describing a solution that works for macOS and MSFT VisualStudio Code.

The solution I’m going to suggest works with using static CodeCommit credentials from the Credentials section of your AWS IAM user object. An alternative solution is to use IAM keys with CodeCommit, but it requires credential helper configuration with the AWS CLI in order to handle the dynamics of IAM session management. With static git credentials, there is no need for AWS CLI integration. Either solution also requires adding a section to your $HOME/.gitconfig file for the credential being used.

I have found that it helps to break out different repositories (CodeCommit, GitLab, GitHub, etc.) into separate .git-credentials files for HTTPS access. This is because the git-credential helper logic is sensitive to credential sorting by top-level domain. For example, I’ve tried using a single .git-credentials file for multiple repos at the same top level domain and it works when a single set of credentials is used for access to all the repositories. However, if I have different credentials (e.g., different personal access tokens in GitLab) for different repositories within the same top-level domain, problems arise.

After you’ve generated your CodeCommit credentials in the IAM console for your IAM user, you will need to configure a .git-credentials file in your home directory for the repository(ies) you want access to via git commands. Let’s say we are working with a repo called “awesome-microservice” in us-east-1. Here’s what the HTTPS git credential string looks like:

https://jsmith-at-012345678912:somesuperdupersecretstring@git-codecommit.us-east-1.amazonaws.com/v1/repos/awesome-microservice

Next, store this string in a file, I might name it something like:

$HOME/.git-credentials.awesome-microservice

Now that you have your credential file, we need to tell the git binaries where to find the credential. Create/update your $HOME/.gitconfig file with a new credential section for your CodeCommit credentials:

[credential "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/awesome-microservice"]
  helper = store --file /Users/jsmith/.git-credentials.awesome-microservice

 
At this point, you should be able to clone down your repo from CodeCommit without having to input a username/password. If you have other repos to access, in CodeCommit or someplace else, you can repeat these steps if you use HTTPS and static git credentials to connect to those repos.

Followup: Visual Studio Code

So, it’s been almost a couple of weeks now of hardcore Visual Studio Code (VSC) usage on my part. I have to say, it’s fantastic. Not a single crash after a solid two weeks of varied development (CloudFormation, Python, shell, and HCL (Terraform)) and at some points, intense Git activity with different repositories and different SCM endpoints. It’s easily 30% more performant than my Atom environment ever was.

The rock solid Git integration is the one feature I appreciate the most. It really works well with everything I do on a regular basis. I did install the GitEasy package just to see if it added anything beyond the built-in support. So far, I only use one GitEasy command reliably, and that’s GitEasy:PushCurrentBranchToOrigin.

I’ve also been able to increase my normal productivity after I installed a marketplace extension called “macros“. I use macros to automate combinations of git commands I often chain together manually (as well as any other keybindings I see fit to construct).

Nice job, Microsoft. I think that’s the first time I’ve said those words after nearly 30 years in technology.

 

A more helpful git log

The git log command is useful in viewing history of changed repository content, but the default output leaves a lot to be desired:

gitlog1

An easy enhancement to the default is to add the “–oneline” parameter which makes it easier to see commit history in a linear fashion:

gitlog2

The colors here are part of my .gitconfig settings and are helpful for parsing commit SHA’s from commit log messages. But, we can do better than this…

Try adding this git “hist” alias to your own .gitconfig file to produce an even more helpful git log output:

[alias]
 fa = fetch --all
 far = fetch --all --recurse-submodules 
 hist = log --pretty=format:'%Cred%h%Creset - %s %Cgreen(%cr) %C(bold blue)<%an>%Creset %C(yellow)%d%Creset' --abbrev-commit

Now, running “git hist” will produce this more easily parseable version of git log output, one that can be quite useful in finding exact commits by relative date:

gitlog3

Much better, don’t you think?

 

Managing pre-commit hooks in Git

Git comes with support for action sequences based on repository activity. Pushes, pulls, merges, and commits can all be configured to trigger specific custom actions responsively. Often, the custom actions are geared towards promoting intra-team communication, process-gating like enforcing commit log standards and best practices for code content/syntax. Of course, CI workflows already are a popular frontline of defense for bad code pushes by using linters and syntax checks as part of an initial test stage in a pipeline. However, those linting processes have a cost in terms of compute resources they utilize. In a large development environment, this can translate into real money costs pretty quickly. Why spend compute cycles on pipeline jobs that can be just as easily run on a developer’s workstation or laptop? So, let’s take a closer look at git hooks…

Git hooks are configured in a given repo via files located in /.git/hooks. A new repository will be automagically populated with these handy tools, which are inactive by default thanks to the file suffix “.sample”:

[rcrelia@fuji hooks (GIT_DIR!)]$ ls -latr
total 40
-rwxr-xr-x 1 rcrelia staff 3611 Dec 22 2014 update.sample
-rwxr-xr-x 1 rcrelia staff 1239 Dec 22 2014 prepare-commit-msg.sample
-rwxr-xr-x 1 rcrelia staff 4951 Dec 22 2014 pre-rebase.sample
-rwxr-xr-x 1 rcrelia staff 1356 Dec 22 2014 pre-push.sample
-rwxr-xr-x 1 rcrelia staff 1642 Dec 22 2014 pre-commit.sample
-rwxr-xr-x 1 rcrelia staff  398 Dec 22 2014 pre-applypatch.sample
-rwxr-xr-x 1 rcrelia staff  189 Dec 22 2014 post-update.sample
-rwxr-xr-x 1 rcrelia staff  896 Dec 22 2014 commit-msg.sample
-rwxr-xr-x 1 rcrelia staff  452 Dec 22 2014 applypatch-msg.sample
drwxr-xr-x 11 rcrelia staff 374 Dec 22 2014 .
drwxr-xr-x 15 rcrelia staff 510 Jan 4 2015 ..

Having individual hooks like these provides a powerful framework for customizing your repository usage to your specific needs and workflows. Each one is triggered at the stage of git workflow described by the filename (pre-commit, post-update, etc.)

Tools like linters fit nicely into the pre-commit action sequence. By configuring the pre-commit hook with a linter, you are delivering higher quality code to your pipelines which makes for a more efficient use of your compute resource budget.

Hook Management: Yelp’s pre-commit

I recently started using an open source utility released by Yelp’s engineers called simply, “pre-commit”. Essentially, it is a framework for managing the pre-commit hook in a git repository, using a single configuration file with multiple action sequences. This allows for a single pre-commit hook to do many different sorts of actions. It includes some basic linter capabilities as well as other code quality control features, but also is integrated with other projects (e.g., ansible-lint has a pre-commit hook available).

Setup is straightforward, as is the usage. Here’s how I did it:

pip install pre-commit
cd repodir
pre-commit install
# edit a new file called .pre-commit-config.yaml in the root of your repo
git add .pre-commit-config.yaml
git commit -m "turn on pre-commit hook"

Immediately you should see the pre-commit utility do its thing when you commit this or any other change to your repository.

Here is the current working version of my pre-commit config file (one per repo):

- repo: git://github.com/pre-commit/pre-commit-hooks
  sha: v0.7.1
  hooks:
   - id: trailing-whitespace
     files: \.(js|rb|md|py|sh|txt|yaml|yml)$
   - id: check-json
     files: \.(json|template)$
   - id: check-yaml
     files: \.(yml|yaml)$
   - id: detect-private-key
   - id: detect-aws-credentials
- repo: git://github.com/detailyang/pre-commit-shell
  sha: 6f03a87e054d25f8a229cef9005f39dd053a9fcb
  hooks:
   - id: shell-lint

So, I’m using some of pre-commit’s built-in handlers for whitespace cleanup, JSON linting, YAML linting, and checking to make sure I don’t include any private keys or AWS credentials in my commits. Also, I’ve integrated a third-party tool, pre-commit-shell, that is a wrapper to shellcheck for syntax checking and enforcing best practices in any shell scripts I might add to the repo.

And here is an example of a code commit that triggers pre-commit’s operation:

[rcrelia@fuji aws-mojo (master +=)]$ git commit -m "pre-commit"
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Trim Trailing Whitespace.................................................Passed
Check JSON...........................................(no files to check)Skipped
Check Yaml...............................................................Passed
Detect Private Key.......................................................Passed
Detect AWS Credentials...................................................Passed
Shell Syntax Check...................................(no files to check)Skipped
[master 7d837e7] pre-commit
 1 file changed, 15 insertions(+)
 create mode 100644 .pre-commit-config.yaml

While pre-commit doesn’t handle management of the other available Git hooks, it does a very good job with what it does control, with a robust plugin interface and the ability to write custom hooks.

If you find yourself in need of some automated linting of your code before you push to your remote repositories, I highly recommend the use of pre-commit for its ease of use and operational flexibility.

Happy coding!

Git Smart with CodeCommit!

AWS recently announced that CodeCommit repositories can now be created via CloudFormation, which spurred me finally to take the opportunity to create my own home lab git repo. While I do have public GitHub repos, I have wanted a private repo for my experimental coding and other bits that aren’t ready or destined for public release. I could build my own VM at home to host a git repo (I recently tinkered with GitLab Community Edition), but then I have to worry about backups, accessibility from remote locations, etc.  As it turns out, you can build and use a CodeCommit repo for free in your AWS account, which made it even more compelling. So, I decided to give CodeCommit a try.

CodeCommit is a fully managed Git-based source control hosting service in AWS. Being fully managed, you can focus on using the repo rather than installing one, then maintaining, securing, backing it up, etc. And, it’s accessible from anywhere just like your other AWS services. The first 5 active users are free, which includes unlimited repo creation, 50 GB of storage, and  10,000 Git requests per month. Other benefits include integration paths with CodeDeploy and CodePipeline for a full CD/CI configuration. For a developer looking for a quick and easy way to manage non-public code, AWS offers a very attractive proposition to build your Git repo in CodeCommit.

QuickStart: Deploying Your Own CodeCommit Repo

  1. Download my CodeCommit CloudFormation template (json|yaml) and use to create your new repo.
  2. Add your SSH public key to your IAM user account and configure your SSH config to add a CodeCommit profile.
  3. Clone your new repo down to your workstation/laptop (be sure to use the correct AWS::Region and repository name): 
    git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/yournewrepo

DeeperDive: Deploying Your Own CodeCommit Repo

Step 1: Building the CodeCommit Repository

I’ve created a CloudFormation template that creates a stack for deploying a CodeCommit repository. There are two versions, one in JSON and one in YAML, which is now supported for CF templating. Take your pick and deploy using either the console or via the AWS CLI.

You need to specify four stack parameters:

  • Environment (not used, but could be used in Ref’s for tagging)
  • RepoName (100-character string limit)
  • RepoDescription (1000-character string limit)
  • Email (for SNS notifications on repo events)

Here are the awscli commands required with sample parameters:

# modify the template if needed for your account particulars then validate:
$ aws cloudformation validate-template --template-body file:///path/to/template/aws-deploy-codecommit-repo.yml

$ aws cloudformation create-stack --stack-name CodeCommitRepo --template-body file:///path/to/template/aws-deploy-codecommit-repo.yml  --parameters ParameterKey=Environment,ParameterValue=Dev ParameterKey=RepoName,ParameterValue=myrepo ParameterKey=RepoDescription,ParameterValue='My code' ParameterKey=Email,ParameterValue=youremail@someplace.com

In a few minutes, you should have a brand new CloudFormation stack along with your own CodeCommit repository. You will receive a SNS notification email if you use my stock template, so be sure to confirm your topic subscription to receive updates when the repository event trigger runs (e.g., after commits to the master branch).

Step 2: Configure Your IAM Account With a SSH Key

Assuming that you, like myself, prefer to use SSH for git transactions, you will need to add your public SSH key to your IAM user in your AWS account. This is pretty straightforward and the steps are spelled out in the CodeCommit documentation.

Step 3: Clone Your New Repo

Once you’ve configured your SSH key in your IAM account profile, you can verify CodeCommit access like so:

ssh git-codecommit.us-east-1.amazonaws.com

Once you are able to talk to CodeCommit via git over SSH, you should be able to clone down your new repo:

git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/yournewrepo

You will want to specify a repo-specific git config if you don’t use the global settings for your other repos:

git config user.name "Your Name"
git config user.email youremail@someplace.com

Now you are ready to add files to your new CodeCommit repository. Wasn’t that simple?