Category: Quality Engineering

Unless you’ve been living under a rock, if you work in IT you’ve heard of Spectre and Meltdown. As you’ll see in a moment, Meltdown is a specific example and exploit of a class of weaknesses called Spectre that affects basically all computer hardware made in the past decade or more. On systems that are affected, the exploit allows an unpriveleged user to peek at anything in the system’s memory, regardless of permissions.

The suggested fix is to replace a systems CPU with new hardware… that hasn’t yet been created, or at least not released publicly. There are a few ways to shore up the operating system’s defenses and security, but it will definitely be available as a backdoor to anyone capable of misusing them.

One of my favorite youtube channels, Computerphile, posted a video several weeks ago that helps clarify what’s going on (comp:

I’m usually a bit annoyed with any company that has serious security flaws, and realize it’s not always justified. I tested software errata with security fixes long enough to have a fairly quick understanding of how a software project goes through the steps that lead to releasing specific fixes.

First of all, I don’t blame anyone in Quality Assurance for not finding the issue. It’s a design flaw, and in some environments the common reaction is to ignore QA when they point those out. I’ve been told several times that a flaw which I found in Quality Engineering was not serious, because it was in the last release and no customers have yet complained about it. Also, it’s not easy to find security flaws in software quickly, even for those trained in the dark arts. Quality Assurance and the tests they devise are designed to prove that software can work as intended without problem. Unfortunately, you can never test bugs out of existence, and the sheer volume of code required to try would dwarf the size of the software intended for release. The best you can do is say “it didn’t work like claimed” if the actual tests don’t pass.

By the way, in the vast majority of cases that I’ve seen, QA spends the most time working on regression testing, preferably automated. Meaning that the tests they devise passed for the previous release, and QA is checking to see if any relevant bugs occur again, after they’ve already been fixed once. Really, no one thinks QA is sexy.

Spectre is a cross-platform problem that is independent of operating system, and affects all of the major CPUs created to run more than one program. Basically everything except embedded systems like Arduino (I really need to check whether it’s affected). If it runs anything like a desktop or server or laptop, it’s vulnerable. So it’s not something that would easily be picked up by operating system creators.

The bug has been around for most of the careers of most hardware developers. It’s going to be a while before we really work out the best solution.

I started working with Red Hat Linux, Fedora’s predecessor, in roughly 1996. At the time, friends and I at JMU were establishing the school’s Linux Users Group. Red Hat was kind enough to offer some limited sponsorship, in the way of software for our group to share. We set up a mailing list and quickly gathered a large group that surprised everyone in the beginning.

My junior year, I went to LinuxCon, the conference in Durham, NC that Red Hat sponsored. It was small, only about 100 people. But there were a few very top kernel developers associated with Red Hat, even then. The friend who I went with scored a hard drive as a door prize. I scored a copy of Red Hat Linux that Red Hat didn’t even make themselves yet. I also picked up my Red Red Hat Hat, with the “alpo” logo, sold to me by none other than Bob Young. Even then, he was one of the most humble business people I’ve met, convinced he was almost out of place working with fledgling industry giants.

A couple of years later, just before I interviewed at Red Hat, I had the good fortune of seeing what that looked like behind the scenes. It was clear to me then, that Red Hatters were a special breed of committed, highly competent geeks devoted to revolutionizing the industry. It wasn’t just t-shirts and free beer, although there was some of that.

For most of college, I worked summers at a small dial-up internet service provider (ISP). We made our real money selling web development to realtors getting in on the internet at the ground floor. I designed a few sites, which in hindsight was a fairly scary assignment for a newly-minted engineer. By the time I left that ISP, I was their system administrator. That’s right, I was a team of 1. In the end, they were just too small to provide the benefits I needed, so I entered the corporate world. I had a linux system in our quasi-colo facility, and knew that many of our competitors exclusively ran linux.

I took a brief hiatus from doing the Linux thing professionally for a job with a telco, where I honed my Perl skills. About 14k lines worth, in one project. They weren’t yet ready for Linux. In those days, the first wave of Linux developers and sysadmins were leaving college and turning their hobby into something that earned a paycheck. Linux was still sneaking in the back door.

By the time I reached NASA a couple of years later for another sysadmin gig, Red Hat Linux was the base desktop for my users. We used Solaris servers, one or two of which I actually EOL’ed for Y2K compliance. While I was there, it was easy to recognize that every bit of NASA was populated with highly talented people. It was an honor to work among them.

Then I started working in tech support, developer support, and into engineering for quality assurance. I worked in that role for a little more than 15 years, during which time my employer rode the Linux wave from 200 people to 10,000. From the time I walked in the door to my employer, I needed to constantly learn just to keep up. I’d often hear about new technologies as someone talked about implementing them, or as I began testing them. Working inside a company like that presents its own challenges, that I’ll save for a later time.

Looking forward, I’m going to stay involved with Linux. My expertise is with Red Hat’s offerings, but of course I’ll branch out a bit into other distributions. I’d like to
continue my career trajectory through virtualization technologies, I feel it’s still a field that’s maturing. To be honest, I want to stay at the forefront of Linux development. It’s addictive in a fun way, after all of these years.