[RndTbl] Attempted murder of sysadmin

[Adam Thompson]

> I've been thinking about this a lot lately, as each new buzzword layer
> enters the scene (SaaS, IaaS, MS, VZ, cloud, containers, ...).  Each time
> one of these technologies comes along I look at it and say "boy, is that
> stupid / inefficient / costly".  Of course, I'm thinking as a sysadmin.

More specifically, you're thinking as a sysadmin who isn't running on zero energy reserves and isn't half burned out already.

> A couple of years ago it dawned on me that the whole point of these
> techs was to eliminate sysadmins.  

That's no surprise.  
Well, virtualization itself isn't about eliminating sysadmins, it's about eliminating CapEx, power, space and cooling, all of which it did & does very, very, well.
But a lot of the other technologies you mention explicitly SAY they're about eliminating system administrators and/or network administrators and/or specialists of various stripes.

> that they aren't as cheap as I can do it myself (i.e. cloud vs dedicated
> server).  But that's me, computer expert, who can do everything myself
> (from hw to os to sw) very quickly.  Not CEO who doesn't even know

Disagree with cloud being cheaper than dedicated server.  At your scale, yes, at larger scales, no, not until you get to massive scale.
The tiny end of the scale (<10 servers) can afford to spin up a new server, because their single server closet still has enough power, space and cooling to do so.
The middle-market (i.e. companies that have a few dozens of servers) is crippled by the enormous cost of spinning up another physical server wherein they have to find space, power and cooling.
The high-end (hundreds of servers) has already figured out how to solve that and how to build that into the cost of spinning up a new server.
 
> I mean, look at docker or flatpacks or even VMs.  These things are the
> opposite of what we spent 30 years working against... duplication.  So

Don't forget that Docker images can't be updated.  Ever.  (You have to replace the entire image with a new image that contains updated components.)
So for any (i.e. most) orgs that create a docker image, think "Cool!" and deploy it into production - never to be updated again - I foresee massive swarms of 'bots in the future, that basically can't be fixed.

> now we have the same/similar 100-thousand OS files duplicated
> everywhere.  Now they want to undo the entire concept of shared
> libraries and essentially make libs static again (well, shared but distribute
> specific lib versions with the apps, duplicating out the wazoo).  For an
> efficiency purist it seems insane.

Yup.
There's deduplicating storage, which helps somewhat...
But yeah, Docker takes that downside to VMs and magnifies it 100-fold.
(In the case of VMs, the files would have been duplicated anyway - just on another physical server & physical disk.)
 
> My only question is one of curiosity, will they succeed in murdering most
> sysadmins?  Will they save their $100kUS$/yr/sysadmin by giving it all to
> AWS?  

I don't know about you, but I've never actually met any of these mythical $100k rainbow unicorns.  I get paid less than some of the developers at my company.


> Will people accept the obvious inefficiencies of flatpacks in order
> to delete the cost of people?  

Absolutely.  Without even a split-second's thought.
Because they don't require full-stack knowledge & expertise, it's easier to find replacement people.
At anything past about 5 people in a company, you have to serious worry about the "bus factor".
I work for a 13-person (as of today) company - I represent one of the biggest business-continuity risks the company has!  Bigger than total physical loss of the building & infrastructure, even.  We added a 2nd sysadmin both to relieve some of the pressure off me, and to alleviate the bus factor problem.  I can't blame my employer for looking at ways to reduce that risk; I *help* my employer look for ways to reduce that risk!

> Actually, I'm dubious, as I've read these
> buzzwords in trade pubs ever since I started in tech ~1997, every single
> one claiming to be the admin-killing panacea, and they came and went,
> and most turned out to be waaay overblown.  I'll be shocked if they
> actually succeed in killing sysadmins this time.
> 
> Anyone else thinking about these things?
> 
> Bonus question: What happens the day AWS (or whatever) screws up /
> has a major data loss / some other calamity?  You sure are putting a lot of
> trust in *their* sysadmins, who may not be as smart as you are!

It's happened more than once already.  And it's OK, because there's a 3rd-party to blame.  (Well, it's not "OK", but no-one gets fired for depending on Amazon.)  It's also OK because when one AWS Zone goes offline, it takes out dozens if not hundreds (if not thousands) of companies - and the public outrage can be spread so thin across so many companies that no single org really feels the burn very badly.  Amazon feels it, but ultimately only 5% of their customers are affected in any given outage, so...
If you don't remember, Twitter, Pinterest, Netflix, Imgur, etc... can't remember the others - all went down for almost 24hrs because of an AWS outage two years ago(?).  I don't see any lasting damage to them OR to Amazon.  (There is a growing recognition that you need to architect systems differently to protect against cloud failure, though.)

Other sysadmins are thinking about the issue, naturally.  Not all, but some.

But all that happens is that your specific technical skills change.  The ultimate skill set, of being able to visualize the full-stack and orchestrate dozens of technologies to work together to deliver a service, remains a valuable, fairly rare, skill.  Whether my job is called "system administrator" or "cloud administrator" in 5 years' time doesn't really change the fundamental nature of what I do all that much.

Systems Administration has never been about knowing the UNIX toolkit inside out and backwards.  Or understanding the technical details of how disk accesses are faster at the start of the disk.  Or anything like that.  It's always been about know *enough* about enough different parts of the system that you can put together a functional service.  All those specific technical skills are just tools to manage technical complexity.  A machinist today doesn't need to know how to fold and hammer steel to make a sword - they have newer & better technologies at their disposal, but they can still create a tool that lets the customer cut stuff.

As technology evolves, an org might not need a full-blown sysadmin to spin up a single Docker instance, or even an orchestrated group of VMs/Lambdas/Containers/whatevers, because the hard work of doing so has been pre-scripted and automated.  That's fine.  Guess who built that automation?  Guess who gets called in to troubleshoot it?  Guess who gets tapped on the shoulder to re-architect the whole structure because performance sucks?  People like you and me.

So I don't really care if the "system administrator" dies off.  In many ways, that should be a goal for our industry, not something to be feared.  It's pathetic that we still can't produce reasonably bug-free systems that interoperate with each other in a plug-and-play (sorry) fashion.  It's pathetic that the average 10-person shop *needs* a system administrator, at least on call, if not full-time.  It's beyond absurd that my bookkeeper's 2-person office would be dead in the water without someone like me to help them.  In an ideal world, our job should not exist.

Since we're human, however, and the technology field doesn’t look like it's going to stop sucking hard (from the mid-sized consumer's standpoint) within my lifetime, there's always going to be some job that really just amounts to "Complexity Management".  And you and I will probably still be doing it.

(Oh, and it's not just sysadmins.  Remember "visual programming"?  Each time it gets reinvented, it's going to let business users do their own programming and get rid of all those overpaid programmers.  Yeah, right.  I still see lots of programmers.  Just, hardly any of them write in assembler or COBOL anymore.  Although 4GLs haven't exactly taken over the world...)

-Adam