Something Wicked This Way Comes?

I’ve been through multiple ‘reboots’ in my tech career. I started out initially doing networking on PCs (DOS/Win/Mac/Linux): installing network interface cards (NICs), configuring them, and then moving on to the next machine. It was a very dynamic and transformational period in the tech world. Then, I became a “systems administrator”, which at the time in the mid-90’s connoted a high degree of skill with UNIX and UNIX-like operating systems used for distributed applications like the brand new “World Wide Web” among other things. At the time, I was as enthusiastic as ever, but also now a bit seasoned, which gave me just enough bravado to advance my career into the private sector (after starting out exclusively in academia and higher-ed).

That was the biggest jump I’d experienced in my nascent tech career path, at a time when companies were hiring ALL THE PEOPLE who had remotely even heard something about this thing called the Internet. I was lucky, I had both knowledge and experience. As a grad student, before deciding to become a computer professional, in addition to reading journal articles for my doctoral program of study, I’d print out and read Internet RFCs. This was back when there were less than a few hundred or so of them, and you recognized a lot of the authors’ names because they were writing them all. This Internet thing was undeniably super cool and, like many of my peers, I could already sense that it was going to the world dramatically.

A few years before, I recall having a conversation with my major professor at one point (before leaving the academic world for the computing world), and describing to him this thing called The Well and proposing that we do some research into the influence of computer networks on human social behavior. Mind you, this was around 1990, still several years before the average person owned a computer or means of connecting to the Internet from home. I was envisioning an advanced network of social communication not unlike today’s social media platforms (my chosen field of study was called social cognition, and I focused on impression formation and attitude formation).

I think he laughed at me, because neither one of us at the time had any real grasp of the concept at that time. I had been on the Well, and using BBS’es a lot, was starting to explore FTP and Archie and Gopher. And, I had heard of HCI research (Human Computer Interaction) but was imagining something else as a field of study that didn’t exist, with an added social element that I thought was completely intriguing to consider.  He didn’t really know where to go with my idea, so we dropped it and decided that it was really outside of what we were studying and trying to accomplish with our respective lines of research.

After making the jump to the private sector and privatization of Internet “sites”, most of my work was still in the sysadmin realm, but soon expanded to include both storage and networking on an enterprise scale. I wore whatever hat needed wearing on a given day in a given workplace, and grateful that my self-taught tech skills seem to allow me to progress in my career without many obstacles. As time marched on, problems with staffing became more of a scale issue: hire more and more engineers to get the infrastructure substrate solid and support application development faster.

Around 2005 or so, I began noticing a trend of people applying for systems and software engineering jobs with little to no experience and deep understanding, but plenty of technical training and certifications. Positions were starting to go unfilled but, also many new hires were simply not up to speed and often caused more problems than they solved. It didn’t matter what industry or the company, no workplace was immune to this problem it seemed.

Now, it seems this problem has become both chronic and more acute. I believe that’s part of the rise of DevOps. DevOps itself has resisted hard definitions, but I tend to think of it as basically adopting a more agile methodology for software development projects and infrastructure operations, by tapping people to do both development and operations work. It’s been embraced by organizations large and small, but it still remains early. And finding a good devops engineer, one that can effectively traverse the trees as well as the forest, so to speak, is even more difficult. Positions are still going unfilled and orgs are turning to trying to coach and train existing staff, to get them “up to speed” on devops practices.

And, lo and behold, guess what? It’s not working,  except in the rarest of occasions in my experience. Yes, devops is expanding, but it’s dragging a lot of legacy cruft with it and dilutes the gains of the paradigm shift. I just recently read this article by James Beswick in which he describes almost exactly the same story I’ve been sharing with you here, but with the backdrop of cloud adoption and the role that automation is playing in eliminating human labor. It’s a real thing, I see it happening in many places.

To the extent that the cost-reduction process is the prime driver for cloud adoption, it’s natural to expect some sort of fallout like eliminating staff. However, and this is where my concern spikes a bit, companies seem to be risking the gains offered by devops in the cloud by an attitude of “Well, we can’t find the right person, so let’s just have our existing staff to get training and certifications and all will be well”. In all but the rarest of situations, that isn’t working. Many people are not able to make the jump, and it drags down the entire org which is still managing to operate in the field of the devops/cloud paradigm without realizing the essential benefits (streamlining of workflows, automation and code testing, rapid cut/release cycles, etc.).

It remains to be seen if we are simply at the bottom of the S-curve as Beswick suggests; it’s a compelling analogy, one that’s worked to describe technological bifurcation in the last 40 years or so. But, we clearly have a definite problem in terms of a lack of skills and capabilities in the existing labor supply that is being magnified with an insatiable demand by orgs to move faster and better daily. Is it simply a waiting game, whereby more automation will eventually fill in the skilled labor gaps? Machine learning and state machine processing can effectively replace many humans. Data mining is occurring on a level that is mind-boggling and, to be honest, downright frightening.

For example, Michael Bloomberg’s recent massive investment in a data mining approach to controlling the electorate does not bode well for you and I. While I applaud the goal, I am deeply afraid of the implications for what is left of consumer (i.e., voter) privacy.  Where are the conversations about ethics and individual freedom and privacy? Where is the EFF or other orgs that normally care about such things? Have we all just given up any and all pretense of having privacy in this new world of data mining and individual profiling? We know where this can go, look at China’s recent ratcheting up of control over its citizens via massive investments in data-mining and facial recognition.

From a purely technological perspective, it’s nothing short of amazing how advances in cloud computing are removing barriers, sometimes daily, to processing what has always seemed like an infinite sea of data. Yet, at the same time, I am starting to feel a bit like how I think Bill Joy must have felt in the early 2000’s when he expressed regret over contributing indirectly to the rise of nanotechnology and what that meant for the future of humankind. My fearfulness is very real and legitimate, but at the end of the day, I don’t feel like I can stop the machine. I hope we survive this new period of even more rapid technological advancements in our culture and learn from it.

GitLab Workflow and VSC

I work almost exclusively these days in Visual Studio Code, for reasons blogged about previously. I also primarily use GitLab for SCM and today I discovered a nifty VSC extension, GitLab Workflow.

To use GitLab Workflow, simply add configuration to your user settings for your GitLab instance URL (if self-hosted) and a GitLab personal access token. This configuration is done automatically when you launch VSC the first time after installing the extension.

Some of the immediate benefits I’m realizing with the extension are:

  • ability to create a merge request (MR) for your current working branch,
  • ability to quickly inspect a CI pipeline via status bar indicator and then a simple click and select to open the pipeline in your browser
  • sidebar navigation for GitLab including issues and MRs

I love working in GitLab and VSC and now, especially with some custom keybindings in VSC, I am able to work even more productively within the VSC editor window without needing to switch over to my browser.

Empathy In The Last Place You’d Expect It

For *nix tech friends, who’d have thought we’d ever see this day?

“I am going to take time off and get some assistance on how to
understand people’s emotions and respond appropriately.”

               — Linus Torvalds, LKML, Sun, 16 Sep 2018 12:22:43 -0700

I’ve always chalked up Linus’ poor empathy skills to the demands of maintaining this amazing juggernaut he created and gave to the world freely. This was especially so in the early days when he was regularly on defense from others trying to do the same thing as he was, others who were older, more experienced, more respected, etc. (e.g., Tanenbaum, rms, the FreeBSD crew). He was but a young grad student, albeit an extremely industrious and clever one at that. But, he always has had a problem with empathy, and it was always just part of the landscape, perhaps even a “rite of passage” if you will, for becoming a kernel developer (I never was a kernel dev, but as a well-rounded system administrator and free software zealot, I followed the LKML closely for its first decade).

This was, of course, long before the concept of empathy ever entered into the fabric of professional technologists. Some of Linus’ infamous attacks on others who were helping him as volunteers were mind-blowingly abusive and the stuff of hushed conversation at conferences and banter at Linux User Group functions all over. In the late 1990’s, I was doing pro bono work for rms and the GNU Project, first as a volunteer evaluator then as the first coordinator of GNU software evaluators (all volunteers). We worked directly with Richard via email/IRC to review software submitted by developers to be considered for membership in the GNU operating system.

And, while rms could be highly challenging to work with/for, and he was somewhat regularly, he was never abusive. He pushed us hard and could be as stubborn as a mule, but he was always fair and, ultimately, respectful if you had a logical position that was different from his. I wouldn’t say, however, that he was ever “empathic”, because again… it wasn’t a concept that was part of the fabric of FOSS development at the time. It was a very different period in the history of Internet technology. You had to have fire-proof pants to be involved with a lot of what was going on at the time (remember flame wars?).

So, his latest LKML posting reveals a different Linus now, one whom I don’t think I ever expected to see, and it’s promising to see this evolution (if that’s what it is) and what it may mean for the future of his interface with the kernel development community. I’m glad to see this day come and hope it makes Linux even more successful than it already has been. Good luck, Linus!

Pythonic Moments

I’ve been heads-down with development work for months it seems, a lot of it in Python, and found a couple of great resources recently that I wanted to share.

If you code in Python and haven’t already heard of Dan Bader or his book, “Python Tricks: A Buffet of Awesome Python Features“, check both of them out. Dan has a series of helpful YouTube videos as well, some of which cover Python Tricks material.

If you listen to podcasts, check out Michael Kennedy‘s “Talk Python To Me” series. He also does a more concise, headline-oriented podcast called “Python Bytes

Huzzah, I’m AWS Certified!

I passed the AWS Certified Solutions Architect – Associate exam on 12/11 and the AWS Certified Developer – Associate exam on 12/15. Woohoo!

awsCertSolArchAssocawsCertDeveloperAssoc

AWS re:Invent 2017 Download

I had a good time at AWS re:Invent 2017 last week, despite being sick as a dog for most of it. Though I caught fewer sessions than I would have liked, the ones I did attend on serverless topics were top notch. Here are some links to my favorites:

ARC401 – Serverless Architectural Patterns and Best Practices

Highlights:

  • Serverless foundations
  • Web applications
  • Data Lakes
  • Stream processing
  • Operations automation (e.g., Tailor, for automating AWS account creation)
  • Excellent review of best practices and new features in Lambda

SRV401 – Become a Serverless Black Belt: Optimizing Your Serverless Applications

Highlights:

  • Optimization Katas
    • Lean Functions
    • Eventful Invocations
    • Coordinated Calls
    • Serviceful Operations
  • Cold start issues in Lambda
  • Instrumenting Lambda with XRay
  • Resource allocation
  • Concurrency vs. latency
  • Compelling customer story from ACloudGuru’s VP of Engineering on going 100% serverless

SRV305 – What’s New in Serverless

Highlights:

  • Announced Serverless Application Repository
  • Reviewed new Lambda console
  • Reviewed new Lambda features
  • Reviewed Cloud9 IDE
  • Reviewed XRay tracing for Lambda
  • New API Gateway features
  • Compelling customer story from FICO’s VP of Engineering

Integration Gotcha: SNS and Lambda

When using SNS pub/sub components, a common integration pattern is to use Lambda to process SNS messages. This can include the use of data blobs as the SNS payload for doing file processing, data transformations, and archiving data in S3 among other things. SNS messages have a large payload limit of 256KB per message, but I recently ran into a situation where I could not reliably deliver messages that were sized well under that limit.

As it turns out, when Lambda is consuming your SNS large payloads in events, you hit a limit within Lambda that is exactly half of the SNS payload limit. For Event (asynchronous) invocations in Lambda, there is a 128KB payload limit. So, if your SNS messages are not being processed by Lambda, check the size of the messages and verify that they are below 128KB. This was a confusing problem until I looked at the CloudWatch console for SNS message deliveries and noticed the errors there.