This is my response to this blog entry: http://gibsonandlily.com/blogs/48

This is an open letter from us, that is, the system administrators, to you, management, users, and everybody that has to deal with us and our sometimes very bizarre behavior.

I'm 41 years old and I've been a linux/unix system administrator for about 11 years. Before that I was a unix programmer for about 5 years. One day when I was only a few years out of my C.S. degree, I shared an office with our solaris sysadmin. We became good friends and got along great, except for when I asked him to install something on a system for me, he'd give me this look that said to me, "Dude, we're great friends, why do you want to spoil that and have me do some work for you?". I never really understood that look. I figured that it was his job to support me and to do what I ask him to do. If I need something done to do my job, he should do it without any attitude and just let me know when it's done. We continued to be great friends throughout the company's existence and still contact him from time to time, but I could never get over or understand that look ... until now.

Many people treat me and my fellow system administrators as an angry god when they want us to do some work for them. They offer a peace offering and apologize 5-10 times for interrupting us and then explain the problem hoping that we won't get angry. Or perhaps they see the same look that I used to get from my friend when I asked him to do work.

Being a system administrator for 11 years taught me a few things. I now understand the sysadmin attitude. It's not that we're assholes. It's not that we are lazy and don't like to work. It's not that we hate or think less of a certain type of employee. It's very simple.

The cardinal rule: Don't make us do any unnecessary work

That's it. Keep that in mind and you'll never feel weird when approaching your local sysadmin and asking him/her to do something for you.

Nobody likes to do unnecessary work. Not programmers, not managers, not the office secretary, nobody. Programmers fully question whether some work that they were asked to do is truly necessary and can't be solved better using existing technology (don't re-invent the wheel) or done using hardware or other solutions. It's normal - everyone in the office has a TO DO list as long as their arm already and doesn't need someone coming up to them asking them to do something that hasn't been thought through and will generate a lot of unnecessary or duplicate work. An example of this is when someone approaches a sysadmin and requests to have them install some known-buggy or known-insecure software on a system. The sysadmin knows that one day the software will crash or be compromised and can cause bigger problems down the road.

There is no System Administrator college or degree. You can take an operating systems class, or get a degree in I.S., but there isn't a major in I.T. that I know of. Most really good sysadmins that I have worked with in the past have a programming degree or have some sort of a programming background. I'm not talking about java or perl or other interpreted language programmers. I'm talking about programmers to write code for a specific architecture and know their hardware. Programmers that know about memory and performance issues and think about those issues before writing one line of code. Most system administrators that I know that evolved from being a programmer were working for a smallish company and needed someone to do some administration of some software or system when none was available and just morphed into the role.

Now don't think that I think that every other type of sysadmin sucks, I don't think that at all. I work with many people who didn't evolve into being System Administrators through the programming path, and they're very good administrators, but I feel that they don't have the same understanding of a system as an ex-programmer does. I've worked with many younger "hot shot" sysadmins that are in their mid-20's and don't have a degree of any kind but can out-admin me any time of the day and I look up to and respect their skills. However, this is mostly a personal 'fight' with me now that I'm in my 40's. I sometimes find myself in a decision where a younger admin has an idea that conflicts with my own personal beliefs. I used to stand up for myself and fight the fight for the better solution, but I recently have lost the will to fight the fight and let the young guy take the reins and run the show. I simply don't have the energy to fight for the right way with some of my co-workers anymore, and more times than not, it leads to problems down the road.

I remember when I was fresh out of school and started my programming career. I came into a company and tried to convert the engineers to the latest and greatest technologies. I converted our project from C to C++ and from TCL/TK to motif on Solaris. I thought that I was doing the right thing because I was using the things that I learned in school and through all of my reading about the latest technologies, I figured that using a newer, better technology could only be better. I managed to get the company to switch gears and go the way of C++/Motif, but it was wedging different technology into a working system, making it more broken than fixed. The one thing that I lacked was knowledge of the second rule (good for sysadmin and programmers alike):

Use the proper tool for the task

The whole key to this rule is that "proper" doesn't mean 'latest flavor of the month' or even 'best practice'. Sometimes TCL/TK is better than Motif. Sometimes C is better than C++. It depends on the situation, the project, the scope of the whole task, if rapid-prototyping is more important than having a polished application, ... To me, the way to decide which tool to use in most of these decisions is to follow the third rule:

Keep It Simple, Stupid

The K.I.S.S. rule works for most situations. If you can keep it simple (keeping in mind future maintenance), it's probably the right way to go.

How I see the role of a system administrator: The system administrator is the construction man and the fireman. We try to build a structure that will stand up to the weather and sit around waiting for the alarms to go off (sometimes in the middle of the night) from a possible storm and when the structure falls or burns down, we rescue what we can and rebuild the structure.

The key to all of this is experience. I cannot stress enough the value of experience in the role of a sysadmin. Best practice is not always the best way to go in every situation. Not every circumstance can be solved by just stating the 'best practice' and going with it. Just because a tool is good doesn't mean that it has to be used to solve every situation (rule 2). You need to pick the tool that fits the situation - not the tool that your flavor-of-the-day favorite blog or instructor decides for you is the best.

What does all of this mean? Is Steve on a rant with no direction?

When I asked my friend the Solaris admin to install some software, he wasn't thinking about me asking him to do work. He was snarling at the software that I was requesting him to install. In the back of his mind, he was thinking to himself, "Oh god, that's going to snowball into a shitstorm of a mess and I'm going to have to clean it all up!". He didn't want to tell me that, but he was thinking it. I was a hotshot kid and wanted to bring positive change to the office and the project but the office elders knew better than I did. The old "Wayne's World" line, "We fear change" in this instance is well-founded. Experienced engineers and system administrators work their best and are the most productive when they are in a familiar environment using familiar tools. When someone comes along and throws a wrench in the works with some newfangled concept or tool and says that it'll be better than the existing framework, there are two ways that the elders can go. They can reject the new guys' new ideas and continue to push steadily down the road as they were before possibly missing out on a better path, or they can shift ownership to the new guy with the new ideas. The new idea may be better, but it puts the control in the hands of the new guy and the elders have to stop, back-pedal to learn the new framework and then try to catch up to where they were before the change if indeed the new framework fits into the old project well without too much 'massaging'.

Sysadmins are no exception. When approaching a sysadmin and asking for work or a new tool installed, a good system administrator will ask you, "WHY?". Don't feel put-off by this. We're just checking to see if there might be a better option. It's our house, we don't need some punk asking to knock down a load-bearing wall in our carefully-designed structure (of course if a boss demands to knock down the load-bearing wall, we kind of have to do it even though it may be a stupid decision and everyone knows it - after all we all need our jobs, right? If you have a boss or manager that hands down decisions like this, you may find that your work environment is less than pleasant and you constantly find yourself fighting fires that you wouldn't have had to do if the decision hadn't been handed down from above). So, if a sysadmin asks you "why", just rationally defend your decision and you'll probably get your task done in a speedy manor, assuming the task is sound. If there is a better option, please feel free to listen to the sysadmin's options and be open about it. It may not be what you expected to hear, but the sysadmin is probably thinking holistically about the whole system when making his/her decision about an alternative option and may be much easier to work with in the long-run.

How to manage a sysadmin: Pretty simple. My rule is: give your sysadmin the tools and time to do their job and leave them alone. When it comes to layoffs, do what any good manager does, stand up for them. When it comes to raises and promotions, it's simple. Promote from within (don't hire from the outside) and give appropriate raises based on performance and completeion of milestones. It's not hard. Basic management skills always work with system administrators. Micro-management, implementing the latest thing from wired magazine because it sounds cool, jumping in bed with the vendor that offers you the most bells and whistles ... ALL BAD DECISIONS! When in doubt, K.I.S.S. Trust me, I have experience in these things. :)