GitLab Workflow and VSC

I work almost exclusively these days in Visual Studio Code, for reasons blogged about previously. I also primarily use GitLab for SCM and today I discovered a nifty VSC extension, GitLab Workflow.

To use GitLab Workflow, simply add configuration to your user settings for your GitLab instance URL (if self-hosted) and a GitLab personal access token. This configuration is done automatically when you launch VSC the first time after installing the extension.

Some of the immediate benefits I’m realizing with the extension are:

  • ability to create a merge request (MR) for your current working branch,
  • ability to quickly inspect a CI pipeline via status bar indicator and then a simple click and select to open the pipeline in your browser
  • sidebar navigation for GitLab including issues and MRs

I love working in GitLab and VSC and now, especially with some custom keybindings in VSC, I am able to work even more productively within the VSC editor window without needing to switch over to my browser.

Empathy In The Last Place You’d Expect It

For *nix tech friends, who’d have thought we’d ever see this day?

“I am going to take time off and get some assistance on how to
understand people’s emotions and respond appropriately.”

               — Linus Torvalds, LKML, Sun, 16 Sep 2018 12:22:43 -0700

I’ve always chalked up Linus’ poor empathy skills to the demands of maintaining this amazing juggernaut he created and gave to the world freely. This was especially so in the early days when he was regularly on defense from others trying to do the same thing as he was, others who were older, more experienced, more respected, etc. (e.g., Tanenbaum, rms, the FreeBSD crew). He was but a young grad student, albeit an extremely industrious and clever one at that. But, he always has had a problem with empathy, and it was always just part of the landscape, perhaps even a “rite of passage” if you will, for becoming a kernel developer (I never was a kernel dev, but as a well-rounded system administrator and free software zealot, I followed the LKML closely for its first decade).

This was, of course, long before the concept of empathy ever entered into the fabric of professional technologists. Some of Linus’ infamous attacks on others who were helping him as volunteers were mind-blowingly abusive and the stuff of hushed conversation at conferences and banter at Linux User Group functions all over. In the late 1990’s, I was doing pro bono work for rms and the GNU Project, first as a volunteer evaluator then as the first coordinator of GNU software evaluators (all volunteers). We worked directly with Richard via email/IRC to review software submitted by developers to be considered for membership in the GNU operating system.

And, while rms could be highly challenging to work with/for, and he was somewhat regularly, he was never abusive. He pushed us hard and could be as stubborn as a mule, but he was always fair and, ultimately, respectful if you had a logical position that was different from his. I wouldn’t say, however, that he was ever “empathic”, because again… it wasn’t a concept that was part of the fabric of FOSS development at the time. It was a very different period in the history of Internet technology. You had to have fire-proof pants to be involved with a lot of what was going on at the time (remember flame wars?).

So, his latest LKML posting reveals a different Linus now, one whom I don’t think I ever expected to see, and it’s promising to see this evolution (if that’s what it is) and what it may mean for the future of his interface with the kernel development community. I’m glad to see this day come and hope it makes Linux even more successful than it already has been. Good luck, Linus!

Pythonic Moments

I’ve been heads-down with development work for months it seems, a lot of it in Python, and found a couple of great resources recently that I wanted to share.

If you code in Python and haven’t already heard of Dan Bader or his book, “Python Tricks: A Buffet of Awesome Python Features“, check both of them out. Dan has a series of helpful YouTube videos as well, some of which cover Python Tricks material.

If you listen to podcasts, check out Michael Kennedy‘s “Talk Python To Me” series. He also does a more concise, headline-oriented podcast called “Python Bytes

Huzzah, I’m AWS Certified!

I passed the AWS Certified Solutions Architect – Associate exam on 12/11 and the AWS Certified Developer – Associate exam on 12/15. Woohoo!

awsCertSolArchAssocawsCertDeveloperAssoc

AWS re:Invent 2017 Download

I had a good time at AWS re:Invent 2017 last week, despite being sick as a dog for most of it. Though I caught fewer sessions than I would have liked, the ones I did attend on serverless topics were top notch. Here are some links to my favorites:

ARC401 – Serverless Architectural Patterns and Best Practices

Highlights:

  • Serverless foundations
  • Web applications
  • Data Lakes
  • Stream processing
  • Operations automation (e.g., Tailor, for automating AWS account creation)
  • Excellent review of best practices and new features in Lambda

SRV401 – Become a Serverless Black Belt: Optimizing Your Serverless Applications

Highlights:

  • Optimization Katas
    • Lean Functions
    • Eventful Invocations
    • Coordinated Calls
    • Serviceful Operations
  • Cold start issues in Lambda
  • Instrumenting Lambda with XRay
  • Resource allocation
  • Concurrency vs. latency
  • Compelling customer story from ACloudGuru’s VP of Engineering on going 100% serverless

SRV305 – What’s New in Serverless

Highlights:

  • Announced Serverless Application Repository
  • Reviewed new Lambda console
  • Reviewed new Lambda features
  • Reviewed Cloud9 IDE
  • Reviewed XRay tracing for Lambda
  • New API Gateway features
  • Compelling customer story from FICO’s VP of Engineering

Integration Gotcha: SNS and Lambda

When using SNS pub/sub components, a common integration pattern is to use Lambda to process SNS messages. This can include the use of data blobs as the SNS payload for doing file processing, data transformations, and archiving data in S3 among other things. SNS messages have a large payload limit of 256KB per message, but I recently ran into a situation where I could not reliably deliver messages that were sized well under that limit.

As it turns out, when Lambda is consuming your SNS large payloads in events, you hit a limit within Lambda that is exactly half of the SNS payload limit. For Event (asynchronous) invocations in Lambda, there is a 128KB payload limit. So, if your SNS messages are not being processed by Lambda, check the size of the messages and verify that they are below 128KB. This was a confusing problem until I looked at the CloudWatch console for SNS message deliveries and noticed the errors there.

 

 

Patterns for Kinesis Streams

I’ve recently been working on a streaming component in a project and have been spending a lot of time with both Kinesis Streams and Kinesis Firehose. I tend to think of the two as event queue frameworks, with Firehose also having the ability to forward events to other AWS services like ElasticSearch (for Kibana-style dashboarding) and backup the same data to a S3 bucket. If you don’t need either of those destinations, then most likely you will get plenty of mileage out of working with Streams alone.

Potential uses abound, but one powerful pattern is making Kinesis a destination for CloudWatch Logs streams via subscription filters. By creating a Kinesis stream and making it a CloudWatch log destination in one account, you can readily add CloudWatch subscription filters in other accounts to create a cross-account log sink. Once your CloudWatch Logs are in one or more Kinesis Streams shards, you can process that log data via Lambda and/or possibly forward to Kinesis Firehose for ES/S3 delivery. There’s a great blog post over at Blend about this exact sort of usage, including a link to their GitHub repo for the CloudFormation templates they use to build and deploy the solution.

One of the best overviews I’ve read recently about design and scale-out issues around event queue processing and how Kinesis resolves, by design, a lot of the challenges therein (e.g., data duplication, ABA problems) is by the fine folks over at Instrumental, entitled “Amazon Kinesis: the best event queue you’re not using“. If you are considering using Kinesis at scale, or are already designing/deploying a consumer/producer pattern to be used with Kinesis, I highly recommend you check out the Instrumental blog post.