bbrock

Spectre and Meltdown

Unless you’ve been living under a rock, if you work in IT you’ve heard of Spectre and Meltdown. As you’ll see in a moment, Meltdown is a specific example and exploit of a class of weaknesses called Spectre that affects basically all computer hardware made in the past decade or more. On systems that are affected, the exploit allows an unpriveleged user to peek at anything in the system’s memory, regardless of permissions.

The suggested fix is to replace a systems CPU with new hardware… that hasn’t yet been created, or at least not released publicly. There are a few ways to shore up the operating system’s defenses and security, but it will definitely be available as a backdoor to anyone capable of misusing them.

One of my favorite youtube channels, Computerphile, posted a video several weeks ago that helps clarify what’s going on (comp:

I’m usually a bit annoyed with any company that has serious security flaws, and realize it’s not always justified. I tested software errata with security fixes long enough to have a fairly quick understanding of how a software project goes through the steps that lead to releasing specific fixes.

First of all, I don’t blame anyone in Quality Assurance for not finding the issue. It’s a design flaw, and in some environments the common reaction is to ignore QA when they point those out. I’ve been told several times that a flaw which I found in Quality Engineering was not serious, because it was in the last release and no customers have yet complained about it. Also, it’s not easy to find security flaws in software quickly, even for those trained in the dark arts. Quality Assurance and the tests they devise are designed to prove that software can work as intended without problem. Unfortunately, you can never test bugs out of existence, and the sheer volume of code required to try would dwarf the size of the software intended for release. The best you can do is say “it didn’t work like claimed” if the actual tests don’t pass.

By the way, in the vast majority of cases that I’ve seen, QA spends the most time working on regression testing, preferably automated. Meaning that the tests they devise passed for the previous release, and QA is checking to see if any relevant bugs occur again, after they’ve already been fixed once. Really, no one thinks QA is sexy.

Spectre is a cross-platform problem that is independent of operating system, and affects all of the major CPUs created to run more than one program. Basically everything except embedded systems like Arduino (I really need to check whether it’s affected). If it runs anything like a desktop or server or laptop, it’s vulnerable. So it’s not something that would easily be picked up by operating system creators.

The bug has been around for most of the careers of most hardware developers. It’s going to be a while before we really work out the best solution.

a Linux career

I started working with Red Hat Linux, Fedora’s predecessor, in roughly 1996. At the time, friends and I at JMU were establishing the school’s Linux Users Group. Red Hat was kind enough to offer some limited sponsorship, in the way of software for our group to share. We set up a mailing list and quickly gathered a large group that surprised everyone in the beginning.

My junior year, I went to LinuxCon, the conference in Durham, NC that Red Hat sponsored. It was small, only about 100 people. But there were a few very top kernel developers associated with Red Hat, even then. The friend who I went with scored a hard drive as a door prize. I scored a copy of Red Hat Linux that Red Hat didn’t even make themselves yet. I also picked up my Red Red Hat Hat, with the “alpo” logo, sold to me by none other than Bob Young. Even then, he was one of the most humble business people I’ve met, convinced he was almost out of place working with fledgling industry giants.

A couple of years later, just before I interviewed at Red Hat, I had the good fortune of seeing what that looked like behind the scenes. It was clear to me then, that Red Hatters were a special breed of committed, highly competent geeks devoted to revolutionizing the industry. It wasn’t just t-shirts and free beer, although there was some of that.

For most of college, I worked summers at a small dial-up internet service provider (ISP). We made our real money selling web development to realtors getting in on the internet at the ground floor. I designed a few sites, which in hindsight was a fairly scary assignment for a newly-minted engineer. By the time I left that ISP, I was their system administrator. That’s right, I was a team of 1. In the end, they were just too small to provide the benefits I needed, so I entered the corporate world. I had a linux system in our quasi-colo facility, and knew that many of our competitors exclusively ran linux.

I took a brief hiatus from doing the Linux thing professionally for a job with a telco, where I honed my Perl skills. About 14k lines worth, in one project. They weren’t yet ready for Linux. In those days, the first wave of Linux developers and sysadmins were leaving college and turning their hobby into something that earned a paycheck. Linux was still sneaking in the back door.

By the time I reached NASA a couple of years later for another sysadmin gig, Red Hat Linux was the base desktop for my users. We used Solaris servers, one or two of which I actually EOL’ed for Y2K compliance. While I was there, it was easy to recognize that every bit of NASA was populated with highly talented people. It was an honor to work among them.

Then I started working in tech support, developer support, and into engineering for quality assurance. I worked in that role for a little more than 15 years, during which time my employer rode the Linux wave from 200 people to 10,000. From the time I walked in the door to my employer, I needed to constantly learn just to keep up. I’d often hear about new technologies as someone talked about implementing them, or as I began testing them. Working inside a company like that presents its own challenges, that I’ll save for a later time.

Looking forward, I’m going to stay involved with Linux. My expertise is with Red Hat’s offerings, but of course I’ll branch out a bit into other distributions. I’d like to
continue my career trajectory through virtualization technologies, I feel it’s still a field that’s maturing. To be honest, I want to stay at the forefront of Linux development. It’s addictive in a fun way, after all of these years.

post-work

I’ve run Fedora on my desktops and home server since it was Red Hat Linux, back in 1996. When I started work in tech support, the first task of any support engineer was to install your desktop system. Cakewalk. It was the first time I’d ever run a kernel directly from a software organization, rather than building my own. I was skeptical. That skepticism served me very well in QE.

Now that I’ve got some time to spare, I’ve got some home projects that I might work on.

I’ve got a network server in need of an upgrade, waiting for time that I can take down the computer network for a couple of hours. I’ve got a hosting provider with several sites that needs to be organized. Including ever getting email, and assigning an IP from one of my domains to my home system.

I’ve got a Raspberry Pi in the garage. I’ve been using it to burn arduino images, but it might find another purpose. After all, who can’t use a server or desktop in their garage?

I haven’t gamed much in the past few months, and I miss that. I tend to play Blizzard games, and have been on WoW in particular for… 7 years? 9 years? I’ll settle for a half-dozen guilds and one vanity guild with my wife. I can’t remember the last time I ran an instance.

First Post

After 17 years at Red Hat, I’m moving on.

Looking back, I worked on the first iterations of several teams when Red Hat was much smaller.  Much of that time, the general feeling was that everyone was taking on a world of responsibilities.  People still describe Red Hat the same way, but back in the day we were always concerned that one mistake could tank the entire company.  I’m convinced that raw willpower and some incredible support staff allowed the company enough extra power to survive the dotcom bubble.

I’ve done phone and email tech-support, developer support, and QA. I tested the installer, kernel, LAMP stack, and much of the glue holding that together. I also worked with virtualization, which I parlayed into work in documentation. Before that, I was initially a systems administrator, and I’ve tried to stay grounded with that perspective.

In college, the powers that be told us that 80% of the software life cycle is in maintenance. So, repeatedly testing the same piece of software as it slowly advances. Fortunately I worked on a variety of projects I was already curious about. I also like to solve puzzles and building things, and QA includes a lot writing and maintaining programs and scripts to do the job. There’s also some exploratory system administration to shake things up.

Writing has been a passion of mine since I was a kid.  Most of the time, I’ve flexed those creative muscles writing fiction.  But my first salaried job was basically technical writing. When I took a break from engineering near the end of last year I started writing again, this time for software documentation instead of IT consulting. It’s gratifying and helped round out my perspective and better understand what engineers look like to the rest of the world.

Now I’m nearing the end of a break from the tech world, aside from a hobby or two in electronics. I wrote half of a science fiction novel. Remember how I said 80% of software engineering is maintenance? With writing, 80% of the work is editing. I like editing, so I could do that as well as writing. But more importantly at the moment, I can write blog articles and might make more use of those skills here.