Everything Sysadmin

See you at MacWorld!

What should you write your LISA paper about?

Status: draft

The Usenix LISA 2010 "Call for Participation" is out. I encourage everyone to think about what they're doing to improve system administration, what innovation they've brought to their network, and write a paper about it.

People often ask me for a definition of "system administrator". TPOSANA/2ed has a great definition in the preface (read it and see).

But lately I've been thinking that one way to define it is in terms of lifecycle. I think of sysadmins as being the people that are responsible for technology from cradle-to-grave. Developers might create it. Executives might pay for it. Customers might request it. But we facilitate the cradle-to-grave process:

• acquisition
• deployment
• maintenance
• repair
• scaling
• decommission

Each of these phases can be a big deal and can be done well or badly.

Most sysadmins are mired in the operational aspects (keeping it running) and don't even realize the other phases exist. Consider the (usually Windows) desktop support person at most companies: the OS comes loaded from the vendor, is never changed, just maintained until the machine is thrown away. We (they) get a reputation for being janitors, not engineers.

The biggest innovations come from focusing on the other parts: getting customer requirements as part of acquisition; deploying things very well (especially desktops or things that have opportunity for mass-production cloning, etc.), scaling (growing from 100 machines , 10,000 machines, 1 million machines; or from hundreds to thousands to millions of users), decommissioning: securely destroying information, and so on.

You can optimize these phases individually or look for cross-phase improvements. If you have optimized all of these phases the only thing left is release engineering and scaling.

Surrounding these issues is all the soft skills that relate to "professionalism": how you deal with people, manage your time, etc.

So what are you going to write your LISA paper about? Which one of these phases have you improved? Did you write a new tool? Develop a new technique? Start an open-source project? Or maybe you have perfected a cross-phase methodology?

Alternatively, maybe you've noticed that there is a thing that sysadmins do, and you can study many groups of sysadmins doing that thing, and can draw comparisons and conclusions about what worked best.

Or maybe you're facing a problem that nobody else has faced, and it is worthwhile to publish your results. Maybe you've scaled Apache to more users than anyone else has, and learned something useful. Maybe you've broken from tradition "on a hunch" and your new disk backup software works better for a particularly common edge-case (Just kidding... I'm sick of papers about backup systems.) Heck, maybe you're just the first sysadmin to deploy a new technology (10G ethernet to the desktop?) and learned something about managing it that the engineers that invented it would have never expected.

What is your sysadmin team most proud of? What thing are you doing that others think is "futuristic" or "cutting edge"?

William Gibson famously wrote, "The future is here. It's just not evenly distributed yet." I suspect that most people reading this blog live in the future and by writing or speaking about what we're doing, we can spread it to others.

That's the goal of LISA, isn't it?

Read the "Call for Participation" here.

What I want in a mobile task management software

The biggest impediment to recording a todo item is that it is inconvenient. I use that excuse to tell myself, "oh, i'll write it down later". Later never comes.

The fewer clicks to the "add a task" prompt, the less likely I can give myself that excuse.

90% of time management is mental.

That's why I recommend paper (no boot-up time), and PDAs like the original Palm that make it very fast (minimal clicks) to write down an idea.

A related excuse happens when I'm in the NYC subway. With no internet connectivity (2G, 3G, or WiFi), any great idea I have on the subway is destined to be not recorded, and often forgotten, if the app I'm using requires the network.

What would be optimal? A "record a task" button right on the phone. You would press-and-hold the button, it would wake up and say "Recording". You would then say your task and use speech-to-text technology to transcribe the idea. If the speech-to-text server isn't reachable, it should hold the audio clip until it can be reached; possibly doing the translation in the background.

The on-screen or physical keyboard should be available too, of course, but what I really want is a super smart, voice activated, task recorder.

Making enterprise ShortName service shorter

Previously I wrote about the Google Apps shortname service which lets you set up a tinyurl service for your enterprise.

The article implies that the service can be used without using the FQDN. This is not true. In other words, I had said that "go.example.com/lunch" could be shortened to "go/lunch".

There is a workaround that makes it work. It is difficult to configure, but I've set up a Community Wiki on ServerFault.com that explains all the steps. As a wiki, I hope people can fill in the items I left blank, particularly specific configuration snippets for ISC BIND, Windows DHCP server, Linux DHCP clients, and so on.

The new article is here: How to set up Google ShortName service for my domain, so that the FQDN isn't needed

Tom @ BayLISA, Thu, Jan 21, Mountain View, CA

Tom will be speaking about the Ganeti open-source project (manages virtual server clusters). More info at www.baylisa.org.

Two talks in one:

Talk 1: Tom Limoncelli's Top 5 Tips: Time Management for System Administrators: Based on his O'Reilly book, Tom will present his top 5 tips for system administrators. Tom will also have copies of the new 2nd edition of The Practice of System and Network Administration.

Talk 2: Ganeti: Open Source Virtual Cluster Manager: Why should virtualization clusters be difficult to manage? Why do they require expensive SANs and management software? Ganeti is a management layer that sits on top of the virtualization stack of your choosing: Virtualization using Xen, KVM or Containers plus storage using DRBD, flat files, or SANs. Ganeti is open source, written in Python, and is under active development. The largest known Ganeti cluster is 40 physical machines with 240 virtual machines. It supports virtual Linux and Windows instances. The management layer lets you create, manage, and migrate virtual machines with very simple commands. Live migration is supported. Removing the requirement of an expensive SAN permits users to do things such as install one tiny cluster in each of 100 offices around the world, or setting up dozens of large clusters in a major data center. The open source homepage is here: http://code.google.com/p/ganeti/

Checklists in Finance and Airlines

At the risk of being a total fan-boy for Atul Gawande's 'The Checklist Manifesto: How to Get Things Right' (book and ebook), I want to point readers to this extract published in The Financial Times.

It covers a VC that uses checklists to get better results when selecting investments, and a dramatic description of the checklist use during the US Airways flight 1549 flight where Captain Chesley B. "Sully" Sullenberger III did an emergency landing in the Hudson river.

Three favorite quotes:

On resistance to checklists:

The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism needs updating.

People fear rigidity if they adhere to protocol. They imagine mindless automatons, heads down in a checklist, incapable of looking out of their windscreen and coping with the real world in front of them. But what you find, when a checklist is well made, is exactly the opposite. The checklist gets the dumb stuff out of the way.

People have been intensely interested in what he's been buying and how, but the minute the word "checklist" comes out of his mouth, they disappear.

How to kill innovation

This BusinessWeek article spells out the kind of behavior that kills innovation which I saw at Bell Labs. His example is at a bank, where a new product is killed because nobody else is doing this product.

At Bell Labs in the 1990s I felt that the president spent most of his time alternating between two activities: Canceling projects because "if it was a good idea, why don't we hear that the competition is doing it?" and complaining that the competition had just released a product that we hadn't thought to create. The truth was that we had, but he canceled it for the former reason.

The article recommends a better way to encourage innovation: "we can turn to a third form of logic: abductive logic, the logic of what could be. To use abduction, we need to creatively assemble the disparate experiences and bits of data that seem relevant in order to make an inference--a logical leap--to the best possible conclusion."

As system administrators we often put down extremely new ideas. Centralized file servers were a bad idea, until everyone else was doing it. The web was "too much bandwidth and should be blocked." WiFi can't be made secure. Cloud computing is "untested."

Sometimes I am concerned that we get burnt out and forget that while it is our job to measure risk, we do this to find creative was to mitigate it; lest we find ourselves using it to justify stopping innovation.

Here's a good New Years Resolution: Make an effort to take the logical leap to see what could be.

Usenix LISA 2010 Call For Participation

Just moments ago Usenix has published their "Call for Participation" for the Usenix LISA 2010 conference. This is a conference that I attend every year because the value I get out of it is undeniable. The speakers are excellent and the topics make me feel like I have access to a crystal ball that lets me see into the future.

The CfP gives a more detailed explanation of the conference and the kinds of talks, papers, and presentations that they are looking for. This is a community conference; talks come from people in the community, not "top down" vendor presentations.

This year adds a new "Practice and Experience" section where people can give a 20 minute talk where people can explain "substantial system administration project that has been completed." Sounds like a great way to learn from other people's mistakes [After I typed that I thought people might think I was kidding or being cynical. Actually, hearing what speedbumps to watch out is pretty darn important!.]

Usenix LISA is unique in that they have a track of refereed papers. These high-quality papers are where some of the biggest system administration innovations have first been published. This year the committee is not requiring full papers, but instead requests 500-1500 word summary. If your paper is accepted you will be expected to produce the entire paper in time for publication. This lowers the barrier to entry and I hope to see a big increase in paper submissions this year (I'm on the panel that votes on papers). If you have done something fantastic, invented a new technique, or written new software that improves the state of the art for system administration, please submit! (Private email to me is fine if you want to ask for advice). Details about submitting papers is here.

Whether you are planning on submitting a proposal or not, reading the full CfP is a great way to understand how a conference like LISA works. When you are presenting or not, I hope to see you there!

Read the entire call for participation here: http://www.usenix.org/events/lisa10/cfp/

System Downtime

The Checklist Manifesto

Atul appeared on WNYC's The Leanard Lopate Show today. You can listen to the entire show here.

He's speaking at 8:15pm at the 92nd Street Y in NYC tomorrow:

In his most recent book, The Checklist Manifesto, Atul Gawande reveals the surprising power of an ordinary checklist that can save lives and improve the way in which we behave. Gawande is a MacArthur Fellow, a general surgeon at Brigham and Women's Hospital in Boston, a staff writer for The New Yorker and an assistant professor at Harvard Medical School and the Harvard School of Public Health. His other best-selling books include Better and Complications.