Everything Sysadmin

PICC '12 planning committee needs YOU!

Interested in helping make PICC '12 happen?

The committee is the most fun group of people I've planned a conference with. If you live within 500 miles of New Brunswick, NJ we'd love for you to help out.

Commitment is about an hour a week plus a short phone conference call every other Monday at 8pm.

Here's some typical volunteer tasks: (we'll ask you to pick one)

• Forward our mailings to user groups' mailing lists (The hard part is making sure it actually went out!)
• Someone to maintain our Facebook/LinkedIn/Twitter presences.
• Invent new ways to get the word out about the conference.
• Take meeting minutes so the chair can focus on running the meetings.
• Email famous people (or semi-famous computer geeks) and ask them to submit talk proposals.
• Pick a potential sponsor, reach out to them about being a sponsor.
• Website updates (we have a website, we just need occasional edits)
• Coordinate the people reaching out to sponsors.
• Design flyers, posters, etc. (Photoshop or The Gimp experience?)
• Sysadmin-like things like website account management.

As you can see, none of these tasks are difficult but it takes a lot of people all doing a little bit to make a conference. (Kind of like how a bit open source project works)

I like working on PICC because of all the people I meet. That's what makes it fun.

The Jan 16 meeting is our "Kick off" for the new year. It's a really good time to get involved.

Interested? If you are, send email to volunteer@picconf.org

Thanks! Tom

http://www.picconf.org/

Message from the Program Co-Chairs

USENIX Association LISA '10: 24th Large Installation System Administration Conference

(This "welcome" letter appeared on the USB stick given to all attendees. Since most people probably missed it I thought I'd repost it here.)

Message from the Program Co-Chairs

Dear LISA '11 Attendee,

There are two kinds of LISA attendees: those who read this letter at the conference and those who read it after they've returned home. To the first group, get ready for six days of brain-filling, technology-packed, geek-centric tutorials, speakers, papers, and more! To those that are reading this after the conference, we ask, "What's it like living in the future? How was the conference? What cool tips and tools did you take home with you to make your job easier?"

Being a sysadmin is kind of like living in the future. You work with technology every day that would make Buck Rogers jealous. Most of our friends are jealous, too. When LISA started 25 years ago, a "large site" had 10 computers, each the size of a dishwasher, with a few gigabytes of combined storage. Today our cell phones have 32GB of "compact flash," which is often more than the NFS quota we give our users.

Attending LISA is kind of like spending a week living in the future. We learn technologies that are cutting-edge-- little known now, but next year everyone will be talking about them. When we return from LISA we sound like time travelers visiting from the future talking about new and futuristic stuff. LISA makes us look good.

LISA rarely has a cohesive conference theme, but this year we thought it was important to highlight DevOps, as it is a significant cultural change. Although DevOps is often thought of as "something big Web sites do," the lessons learned transfer well to enterprise computing.

LISA has always been assembled using the sweat of many dedicated volunteers. It takes a lot of effort to put a conference like this together, and this year is no different. Most prominent are the Invited Talks committee (Æleen Frisch and Kent Skaar) and the Program Committee (Narayan Desai, Andrew Hume, Duncan Hutty, Dinah McNutt, Tim Nelson, Mario Obejas, Mark Roth, Carolyn Rowland, Federico D. Sacerdoti, Marc Stavely, Nicole Forsgren Velasquez, Avleen Vig, and David Williamson), but also important are the Workshops Coordinator (Cory Lueninghoener), the Guru Is In Coordinator (Chris St. Pierre), the Poster Session Coordinator (Matt Disney), and the Work-in-Progress Reports Coordinator (William Bilancio). We couldn't have done it without every one of them. Of course, nothing would happen without the leadership of the USENIX staff. We are indebted to you all!

Of the 63 papers submitted, we accepted 28. These papers represent the best "deep thought" research, as well as Practice and Experience Reports that tell the stories from people "in the trenches." We encourage you to read them all. However, the power of LISA is the personal interaction: introduce yourself to the attendees standing in line near you, strike up a conversation with the person sitting next to you. And remember to have fun!

Sincerely,

Thomas A. Limoncelli, Google, Inc.
Doug Hughes, D. E. Shaw Research, LLC
Program Co-Chairs

Impromptu talk about Google SRE tonight at 9pm.

I've booked a BoF room at 9pm to give my talk "SRE@Google: Thousands of DevOps Since 2004".

Tuesday, December 6, 9:00 p.m.-10:00 p.m., Fairfax B

"Tom will describe technologies and policies that Google uses to do what is (now) called DevOps. Google doesn't just empower developers and operations to work together, we have a system that empowers all groups to be their own devops team. (This is based on my opening keynote at the Pittsburgh Perl Workshop.)"

LISA11 Diary: Tue, Dec 6

(In an effort to get these out sooner rather than later I'm not spending a lot of time editing and proofreading. You've been warned.)

Daytime: Today I spent the day in the Advanced Technology Workshop.

What is a workshop? People need a space to spend an entire day (or half-day) to talk about a topic. There are workshops for people researching certain areas and their workshop at LISA is a once-a-year touchstone to meet in person, give presentations, share ideas, and so on. The Configuration Management workshop is in its 11th year. In fact, Puppet was inspired by a debate (argument?) at CMW a number of years ago. Other workshops are less research-y, like the one for sysadmins at government and military sites. The full list is here.

The Workshop I attended today is called The Advanced Technology Workshop. It is intended for very senior administrators, provides an informal roundtable discussion of the problems facing system administrators today. It is part support group and part talking about hot-topics. The discussion is relatively confidential so that people can speak freely. However notes are distributed after the fact to attendees.

Dinner: As the conference co-chairs, Doug and I have a lot of preparation to do for the Wednesday morning plenary. We decided to have dinner together and then finish what we needed to do. Most of it was finalizing the slides and rehearsing the dialog for the introductions and opening at the Wednesday plenary. We also met up with his old boss from many years ago and I got to hear some stories about "the old days".

BoFs:

LGBT: 7pm: I went to the "Birds of a Feather" (BoF) session called "Lesbian, Gay, Bisexual, Transgender, and Friends BoF". This was the 11th year (we believe) of this BoF. The room was packed. The first half was people introducing themselves. Everyone said their name, company and described their employer's HR policies related to LGBT people: does their non-discrimination policy include LGB or LBGT status, do healthcare benefits apply to same-sex partners, and so on. The second half we talked about industry news, conference insider tips, and what we can do to increase the number of women that attend Usenix.

SRE@Google: 9pm: Not to be confused with the Google Vendor BoF on Thursday night (where there will be plenty of beer and ice cream for everyone), this BoF was a talk by some guy from Google... oh wait, me! "SRE@Google: Thousands of DevOps Since 2004". First the talk compared software development in the 80s, 90s, and 2000s and how this changes operations (system administration). I then described Google's system administration practices that (1) help developers and operations work collaboratively, (2) drive both reliability and high change rates, (3) make it fun. Turn-out was huge, and the questions and comments were excellent. I'm glad I got to do this talk.

I got to bed by 11pm which was pretty important because tomorrow is a big day. The "technical conference" portion of the conference begins. This is 3 days of invited talks, papers, guru sessions and keynotes.

LISA11 Diary: Mon, Dec 5

(In an effort to get these out sooner rather than later I'm not spending a lot of time editing and proofreading. You've been warned.)

Again woke up around 6am. Rehearse parts of the tutorial, got breakfast at the Sheraton Club on the 29th floor.

Tutorial: The Limoncelli Test: My first new tutorial in years! Based on this blog post, the tutorial lists 32 "best practices" that sysadmin teams should do. I had enough time to discuss half of them. At the start of the class I had everyone take the test, and then focused on discussing the ones that had a lot of "no" answers (by show of hands). An attendee wrote a very complimentary review of the tutorial. The "surprise" I had prepared was that for the entire last hour we talked about nothing but specific techniques for "creating organizational change" (which is a fancy was of saying "how to convince your manager and coworkers agree to these fantastic ideas you have). We talked about why people push back (mostly because they're authority is being challenged or they don't want the discomfort that comes from doing new things). The techniques for working on these issues involve various psychology tips to help you understand how people think and how to work from there.

Lunch: I had the lunch that comes with the tutorial sessions. It was mostly sandwiches: I had the roast beef.

Afternoon: I had free time in the afternoon. I spent some time at the CHIMIT workshop which seeks to help link researchers that study system administrators and the system administrators that are available to be studied.

Dinner: A random group of people that were standing around getting hungry decided to go to dinner. We split into two groups, one that went to PF Chang and another that went "somewhere that doesn't put peppers in everything". It was fun being at dinner in a group where I didn't know everyone. We talked about everything from networking, Puppet, politics and rock and roll.

I didn't go to any BoFs but there were BoFs for small sites, AFS users, software patents and other things.

I hung out and did the unofficial "hallway track". I got some one-on-one time with Philip Kizer (president of LOPSA) to talk about some ideas I had and ask about what the future plans are. I'm glad LOPSA is getting some focus and look forward to hearing more at their LOPSA Annual Meeting and Town Hall.

Later at night I hung out at the bar. Another event was in the hotel and it was a big event with a band, speakers and so on. When it emptied out they came to the bar too. You could tell who was who because LISA attendees were all in tshirts and the other group were in formalwear (suits and dresses). One of the speakers for their event was this famous actor and he was sitting right by us in the bar. I wasn't sure it was him, and William kept asking people to look and see if they thought it was him. About half the people we asked hadn't heard of his movies when we mentioned them. Eventually someone pulled up his picture on IMDB and we decided it had to be him. William finally went up to him and got his autograph. Win!

Later my brother showed up. Yup, my brother Ed works in IT and is at LISA this year! W00t!

At that point it was late so I went to bed.

LISA11 Diary: Sun, Dec 4

Sunday I woke up around 6am, had breakfast at the hotel "club" on the 29th floor (great view!)

Tutorial: Time Management for System Administrators: In the morning I taught a half-day class on Time Management. This is the "personal" time management side of things: making your life more sane. I've taught this class at LISA every year since 2005-ish and this year the turn-out was HUGE (80+ people). No matter how many times I teach this I get new and interesting questions each time. After the tutorial I autographed books and answered questions.

Lunch: I had the lunch that comes with the tutorial sessions. Chicken, salad, etc.

Tutorial: Time Management: Team Efficiency: In the afternoon I taught my new(ish) tutorial on tools for helping your team work together. This is all material that is new since Time Management for System Administrators was published. (1) Making meetings not suck. Meetings that waste your time are evil and there are many good ways to reform bad meetings and escape the unfixable ones. (2) Tools that let the team delegate amongst themselves. You may be the only person that understands the guts of the backup system, but everyone should know how to do routine work like adding a new server, doing a restore, changing tapes, and so on. This is a matter of writing "service documentation" and "procedure documentation"... I know everyone hates writing docs, so we also talk about how to make it painless (hint: put a checklist onto a wiki is better than nothing; a little structure and you are almost entirely there. (3) tools for sharing information, and tearing down power structures that are based on information hiding, (4) tips for creating a more shared, collaborative, oncall/pager experience, (5) templates to use for things like design docs, department web site, and so on.

Dinner: Met two local friends and had dinner at Cheesecake Factory. More calories than is ethical to put on a plate.

Evening: Hung out in the "hallway track"... social spaces around the conference venue where people hang out and chat. I got the "inside scoop" about what is happening with the people in certain open source projects; this will help me make some decisions I've been needing to make.

LISA11 Diary: Sat, Dec 3

Saturday, Dec 3:

Getting There: Since the conference is in Boston, I decided to take the train rather than fly. Amtrak costs about the same but is faster due to the lack of 2-hour wait for TSA and other airport things. I arrived in Boston at about 1pm, checked in at the hotel, changed, and went to the lobby to hang out.

Registration: Registration wouldn't open until 5pm so I hung out, talked with people, got some status updates from the Usenix staff about registration numbers and so on. Registration opened at 5pm spot on and I was 2nd in line :-) so I got registered fast.

Reunion: It was great to see so many familiar faces as people arrive. LISA is kind of one huge family. Kind of a "once a year family reunion" for people that know each other through technical mailing lists and other forums. It was particularly good to see people that haven't made it to LISA in years (hi Kurt!). It's also great to see so many new faces. I think everyone goes out of their way to make new people feel welcome. For example, around dinner time people form groups to go out and new people are recruited. Which brings me to...

Advice for new people: It's easy to be shy at a conference like this. Here's a tip to break out of that: It is always polite to turn to a person you don't know, stick your hand out (to shake their hand) and say, "Hi! I'm [your name]. What's your name?" This works great whether you are standing on line waiting for lunch, or sitting next to a stranger in waiting for a talk to begin. Some of the biggest opportunities to learn at LISA are from the other attendees. Strike up a conversation!

Dinner: Walked with a friend to an Indian restaurant called Kashmir on Newbury Street.

I went to sleep early because tomorrow was going to be a big day.

Google booth during LISA, Dec 7-8, 2011

Google's tech blog posted info about the many things that Google is presenting or doing at Usenix LISA. Beer and ice cream on Thursday night. A "ask an SRE anything" booth in the vendor show. Presenting papers, talks and tutorials and much more! Check it out!

http://goo.gl/XXkpK

P.S. I'll be at the Google Vendor Booth Wed, noon-1pm.

Usenix LISA hotel discount extended to Nov 28!

That's right, folks! Book your room today!

CHIMIT Workshop

I am fascinated by the fact that there are researchers that study system administrators and how to make their work easier/better/etc. The #1 thing they tell me is "we need more interaction with more sysadmins to help guide our research!"

The "CHIMIT workshop" at Usenix LISA 2011 is an opportunity to interact with these researchers.

Read about it here and register to attend!

Thursday keynote at Usenix LISA: Andy Palmer

We have three keynotes this year: Wednesday morning, Thursday morning, and the closing keynote Friday afternoon.

Our Thursday keynote speaker is Andy Palmer, Global Head of Software and Data Engineering, Novartis Institute for Biomedical Research

We invited Andy because he deals with peta-scale data warehousing, big databases and all that fun stuff. I love hearing talks about big big big data. There are always plenty of surprises when things get that big. As conference co-chair, I can't wait to meet him in person!

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

Usenix LISA early-bird discount ends today!

You can save big $$$ by registering for LISA on or by midnight tonight! (California time)

Usenix LISA 2011 is Dec 4-9 in Boston.

I look forward to seeing you there!

Usenix LISA early-bird discount ends tomorrow!

You can save big $$$ by registering for LISA on or by Nov 14th.

Usenix LISA 2011 is Dec 4-9 in Boston.

I look forward to seeing you there!

Usenix LISA early-bird discount ends Nov 14! 4 days left!

You can save big $$$ by registering for LISA on or by Nov 14th.

Usenix LISA 2011 is Dec 4-9 in Boston.

I look forward to seeing you there!

Call for Usenix LISA stories and pictures!

To help us celebrate Usenix LISA's 25th conference, we are looking for stories!

Do you have a favorite LISA story?

• Something you learned at the conference
• A contact you made that really had an impact on your career
• A funny event that still makes you laugh?
• Nostalgic (remember facesaver?) and contemporary too!

We are also looking for photographs! (Even if you don't know someone in the picture.) If you have an old Facesaver picture - even better!

Please send your story or photos by Sunday, November 20 to dinah@usenix.org.

Usenix LISA early-bird discount ends Nov 14! 1 week!

You can save big $$$ by registering for LISA on or by Nov 14th.

Usenix LISA 2011 is Dec 4-9 in Boston.

I look forward to seeing you there!

CHIMIT: REGISTRATIONS NOW OPEN!

If you are coming to Usenix LISA, why not come a day early and go to CHIMIT? CHIMIT is for people that study system administrators (how they work, how they communicate, how to make tools better for them) but they can't exist without sysadmins also attending their conference!

Call For Papers: 5th ACM Symposium on Computer Human Interaction for Management of IT
December 4-5, 2011 - Boston, MA
chimit.acm.org

Information Technology (IT) is central to modern life. We are surrounded by software and hardware systems that support our work and personal lives. The size and complexity of modern infrastructures is increasing rapidly; and we are now at a turning point where we need new approaches to IT system design,management, and services. Clearly, successful systems management involves a complex blend of technical and human issues.

Since 2007, the ACM CHIMIT symposium has provided a unique opportunity for researchers, in fields such as human-computer interaction, human factors, and management and service sciences, and for practitioners in the management of large IT systems to meet, share, and explore new approaches to IT management.

The symposium program will include one day of technical presentations and papers followed by a one-day workshop in conjunction with the LISA 2011 conference, which will provide an opportunity for in-depth discussions with highly experienced system administrators.

Submission topics include, but are not limited to:

• User studies of IT infrastructure management
• Design of human-centered IT systems
• Experience reports by IT practitioners and researchers
• Case studies on specific aspects of IT management
• Experimental studies on the usage of new or existing IT systems
• Tools and techniques incl. system visualizations, collaborative interfaces
• Automation approaches to reduce administration workload
• Collaboration within an organization, community of practice
• Organizational knowledge - knowledge management for IT
• Processes and practices - best practices and processes in IT management
• IT beyond the enterprise - at home, on the road, etc.
• New technologies - incl. cloud computing, mobile devices

CHIMIT is sponsored by ACM SIGCHI in cooperation with USENIX.

See chimit.acm.org for more information.

DevOps keynote at Usenix LISA: Ben Rockwood

We have three keynotes this year: Wednesday morning, Thursday morning, and the closing keynote Friday afternoon.

The Wednesday keynote speaker is Ben Rockwood from Joyent who's talk is titled "The DevOps Transformation".

DevOps has a lot of buzz, but Ben will separate the hype from the reality. DevOps may be a new term, but it's not a new idea. He will deconstruct DevOps into its three transformation phases, look back at the often referenced but rarely explained history that influences it, and see how it is a catalyst that is changing the craft of system administration.

I'm really excited we were able to book Ben for the conference and can't wait to see the talk!

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

Futuristic System Administration

Looking at the Usenix LISA 2011 conference program I'm excited we could book some really powerful "what will I need to know next year" kind of talks. This is what first brought me to LISA many years ago... the fact that by attending I'd be one step ahead of my co-workers as far as what's new in system administration.

Some example talks:

• "What Will Be Hot Next Year?" with moderator: Narayan Desai, Argonne National Lab. Panelists: Kris Buytaert, Inuits; John D'Ambrosia, Force10 Networks; Jacob Farmer, Cambridge Computer
• "Ethernet's Future Trajectory" with John D'Ambrosia, Force10 Networks
• "IPv6, DNSSEC, RPKI, etc.: What's the Holdup and How Can We Help?" with Richard Jimmerson, IETF ISOC
• "DevOps: The past and future are here. It's just not evenly distributed (yet)." with Kris Buytaert, Inuits

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

Config Management? yeah, we got that.

Looking at the Usenix LISA 2011 conference program I'm proud that we have the three major configuration management systems covered: CFEngine3, Chef and Puppet:

Full-day classes:

• "Puppet" with Nan Liu
• "Configuration Management Solutions with CFEngine 3" with Mark Burgess

Invited Talks:

• "3 Myths and 3 Challenges to Bring System Administration out of the Dark Ages" with Mark Burgess (CFEngine Inc)
• "Building IronMan, Not Programming" with Luke Kanies, Founder, Puppet and Puppet Labs
• "Converting the Ad-Hoc Configuration of a Heterogeneous Environment to a CFM, or, How I Learned to Stop Worrying and Love the Chef" with Dimitri Aivaliotis
• "Choose Your Own Adventure" with Adam Jacob

Experience Reports:

• "Getting to Elastic: Adapting a Legacy Vertical Application Environment for Scalability" with Eric Shamow (Puppet Labs)

The Guru is In: (Q&A sessions)

• "Chef" with Aaron Peterson

Refereed Papers:

• Fine-grained Access-control for the Puppet Configuration Language Bart Vanbrabant, Joris Peeraer, and Wouter Joosen, DistriNet, K.U. Leuven

Pretty awesome, eh?

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

LISA LISA LISA LISA

No, not this. I mean the Usenix LISA conference. It is only 5 weeks away. Have you registered yet?

The first few days are half-day tutorials with industry leaders teaching topics like Puppet, CFEngine (we don't take sides... both get a half day!), Time Management (that's me!), IPv6 (real deployments are happening!), and many many more topics.

The last half of the conference is a mixture of invited speakers, refereed papers, and other good stuff. What I like about the refereed papers this year is that we hit the perfect balance: half are "oh, I can use that right now!" and half are "OMG! Far out!"

The invited speakers are a mixture of hot topics (storage, security, virtualization, interesting new tools) and amazing people.

At night there are "Birds of the Feather" sessions, where YOU choose the topic.

Doug Huges and I are co-chairs this year and we're both proud of the work that the program committee has done. We look forward to seeing you there!

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

LOPSA PICC 2012 Call For Participation

http://www.picconf.org/cfp/

Call for Participation:

LOPSA-NJ Professional IT Community Conference 2012

This year's Theme:

"System Administration: Scaling, Security, and Saving Money"

PICC '12: 3rd Annual Professional IT Community Conference

May 11- 12, 2012 New Brunswick, NJ Hyatt Regency New Brunswick

http://www.picconf.org

The organizers of the LOPSA-NJ Professional IT Community Conference (PICC) invite you to submit proposals for papers and talks to be presented at PICC '12.

PICC12 is a gathering of professionals from the diverse IT (computer and network administration) community in New Jersey to learn, share ideas, and network. The conference includes invited speakers and keynotes, training by top-notch experts that is relevant, useful, and recession-friendly; plus an "unconference" track where attendees propose and host their own topics during the event. We expect attendance of 100 to 150 IT professionals from mid to large sized companies and academia from New Jersey/New York/Pennsylvania. We go by many titles but everyone is invited: system administrators, network administrators, network engineers, Windows, Linux, Unix, DBAs, etc

Presentation Topics

We strongly welcome topics on best practices, new developments in systems administration, and cutting edge techniques to better manage Linux, Unix, or Windows hosts and environments.

Papers should be of a technical nature and speakers should assume that members of the audience have at least a few years' experience in general IT, Linux/Unix, and/or Windows administration. The audience will primarily hail from businesses and academic institutions in the New Jersey/New York/Pennsylvania area.

Topics may include (but are not limited to):

• System Administration
• Backup
• Security
• Troubleshooting
• Buying
• Decision Making
• Virtualization
• Cloud computing
• Enterprise Monitoring and Management
• Identity Management
• Web and Email Management
• Spam and Virus Filtering
• Networking
• Wikis
• Clustering and High Availability
• Log Management
• VoIP
• Ticketing systems
• Bootstrapping and automated installation
• Configuration Management and packaging

Topics explicitly NOT requested:

• Sales presentations
• Vendor product demonstrations
• Proposals or vaporware

Here are some topics from our previous two conferences and some ideas for new talks:

• Internal Documentation for System Administrators.
• The Path to Senior SysAdmin
• A senior system administrator describes the 'lessons learned' from converting from one email system to another.
• Someone with recent experience in particular technology (cloud computing, backups, Windows 7, etc.) presents "10 things I wish I knew before I started with [name of product]"
• A Windows engineer describes how they manage their fleet of of desktops/laptops.

Presentation Format

We are actively seeking proposals for presentations at PICC'12. We have openings for:

• Papers: 5-10 page paper, published electronically to attendees at the conference, and publicly after the conference. Presentation at conference will be 30 minutes including Q&A.
• Talks: 20 minute presentation + 10 minute Q&A.
• Posters: One physical poster of a topic or work in progress, to be discussed with conference goers during a specified poster session
• Panels: 45-minute panel discussion

Submit your Presentation

If your presentation is selected you will get a complimentary registration for Saturday May 12. If you want to attend the training on Friday or Saturday you will only pay the difference for the classes. Submissions and questions should be sent to: submissions@picconf.org

Dates and Deadlines

To encourage early submissions, priority (both for inclusion and scheduling) will be given to presentations submitted before the 1st of March.

• January 30, 2012 - Deadline for submissions
• February 14, 2012 - Final program confirmation
• May 11, 2012- Start of conference

Contact and Questions

Please see our website at http://www.picconf.org/ for more information on PICC '12 and presenting at this great event.

If you have any questions, please feel free to email the organizers at: submissions@picconf.org

My new LISA Tutorial: 32 Questions

I'm teaching 3 tutorials at Usenix LISA this year. Two are on time management ("intro" and "team efficiency") but the third is brand new: The Limoncelli Test.

I've identified 32 qualities of well-functioning system administration teams. You've seen them before as "The Limoncelli Test". In this tutorial, I'll be going into more detail about the important ones and leaving plenty of room for Q&A. It is a 3-hour class and I hope to keep it interesting by making it very interactive.

The hardest part of adopting these practices is often your own co-workers resistance to change. Therefore, I'm adding a big section on influencing others and "selling" big changes within an organization. That is new material that hasn't appeared in any of my past tutorials or books. I think it will be the most valuable part of the session.

Register for this tutorial today!.

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

Who's coming to Usenix LISA?

Two things I like about Usenix LISA conference: (1) The speakers are (usually) the inventor. (2) They're accessible, not roped off into a VIP room. You can talk with them, hang out with them. (Boston, Dec 4-9; early registration discount ends soon!)

There are a lot of big names this year: What's your interest?

• Linux? We got Jon "maddog" Hall himself!
• Solaris? (I mean... Illumos) We got Bryan Cantrill!
• Security? We got Susan Landau!
• DevOps? we got Ben Rockwood, Avleen Vig, Adam Jacob, Kris Buytaert, Ian McFarland, Kastner and Goulah from Etsy talking about Deployinator and more!
• Puppet? We got the puppet master himself, Luke Kanies!
• CFEngine3? We got Mark Burgess!
• Chef? We got Adam Jacob and Dimitri Aivaliotis!
• Storage? Who knows more than Jacob Farmer?
• IPv6? We got Owen Delong and Shumon Huque!
• Databases? Andy Palmer and Michael Stonebraker
• RRDTool? Perl6? Tobias Oetiker
• Networks? John D'Ambrosia (IEEE "Ethernet Futures Working Group")
• Linux system administration? Æleen Frisch
• Linux performance tuning? Theodore Ts'o
• Time Management? ME!!! (2 half-day classes!)
• And industry celebrities such as Gerald Carter, David N. Blank-Edelman, Larissa Shapiro, and many, many, more!

The conference is in Boston, Dec 4-9, 2011.

Save money by registering on or before Monday, November 14, 2011.

For more information go to http://www.usenix.org/lisa11/

More... How to ask your boss to pay for you to go to Usenix LISA 2011

In reply to my last blog post a reader sent me email to say: "I've never gone to a conference. What do employers typically pay for?"

Your company probably has a written travel policy and a travel budget.

Most employers pay travel, the conference registration, hotel, and food (T&E); IMHO they should do this since you need to get there, attend, and while there sleeping and eating are kind of important too.

Other companies pay part of the cost or a fixed amount. For example, they might have a $1000/year training budget per employee and that would consume the registration (most people would therefore only go to local conferences that do not require travel). Some have a "pool" of money that permits 1-2 members of a team to attend each year; in that case team mates take turns going to conferences and you might only attend every 2 or 3 years.

When budgets are crunched, you are more likely to get approval for something specific: you need to learn Python and there is a python training on (for example) Tuesday, so they fund that only. Economically speaking, however, once you've paid for the travel it is best to amortize it over the most number of days. In other words, you've paid for the travel, get as much out of the conference while you are there.

If a company has zero budget for training sometimes people pay their own way but their employer gives them the time off without requiring them to take vacation days. That's a last resort. I've also taken vacation time and paid my own way, but I'm that dedicated to attending LISA ;-)

Talk about cost last:

Certainly don't bring up the cost until after your boss sees the value of going. Convince him/her that (for example) you need to learn Puppet and LISA would be a good way to do that. Show the brochure or web site. Once you have agreement, they'll ask about cost and you can broach that subject. If you start by talking about the cost, it will be an up-hill battle from there.

General negotiating advice:

I've noticed that geeks (like myself) are bad negotiators (myself included) but I learned to overcome this with a few techniques. The problem (my problem) is that geeks love information and want to share all information. In negotiations you want to hold back information until the right time.

When negotiating always start by asking for what you want and wait for the rejection before suggesting other lesser options. It is tempting to say, "I'd like the company to pay for the entire trip but if that's too much how about just the registration?" because that is the truth. However, you are over-sharing. If you said that the easy reply for you boss is, "Ok, just the registration! Thanks for that great suggestion you made there!" You've just "negotiated against yourself".

It's best to ask for what you want and then be silent. SIlence is uncomfortable and 1 second of silence feels like 10, but keep your lips sealed. Car salespeople say that when they use silence as a negotiating tool, "The next person to talk is the loser." I don't think this is a winner/loser situation, but you'll see what they mean in this next example.

I typically see geeks negotiate like this:

Geek: I'd like XYZ. Boss: (Silence) Geek: Or just XY. Boss thinks to himself, "wow! i was going to give him everything but I needed some time to think! He just negotiated away Z. Awesome!"

So, ask for what you want. Be silent. Wait for the reply. If you are tempted to speak, say to yourself over and over: "Mary had a little lamb." Really.

If the reply is negative be ready with a second suggestion.

If the reply is, "We can't afford it", ask, "What can we afford?" If they give a number, work within it. Some ways to bring the cost down:

• Split the room cost with someone else (the sage-members list is a good place to find room shares)
• Travel cheaply... car or train.
• Couch surf at the home of a local friend or relative
• Work pays for the tutorial days, you cover the other days.
• Split the cost. [Again, don't offer these all at once. They're just suggestions.]

If they don't give a number, return to talking about the "value". (more tips about that yesterday)

Hotel advice:

As a co-chair of the conference I would be remiss if I didn't point out that it is important to stay at the conference hotel if at all possible: We need to fill a certain number of rooms to fulfill our contract. If you can afford to stay at the main hotel, please do. There are a lot of night-time activities at LISA so your best bet is to say in the conference hotel so you can stay for the late-night BoFs and just take the elevator to your room. That said, if there are budget constraints, staying with a local friend is very economical.

Hope that helps,

Tom

Usenix LISA early-bird discount ends Nov 14

You can save big $$$ by registering for LISA on or by Nov 14th.

Usenix LISA 2011 is Dec 4-9 in Boston.

I look forward to seeing you there!

How to ask your boss to pay for you to go to Usenix LISA 2011.

1. Ask early. Sometimes approval takes a long time. He/she may have to ask higher-ups.

2. Warm them up. One day mention how you wish you had better tools to do something or wish you knew more about something ("time management" maybe?). A few days later say that you found a class on the topic at LISA. (Intro to TM and Advanced TM are both being offered)

3. Talk about end-results, not technologies. "There's a class that will teach me how to automate installations" is much more understandable than "there's a class on Puppet". Find 3 classes or talks you want to attend. Explain them in terms of the results the boss will see. 2 is good, a 3rd is good as a backup. More than 3 sounds like you are trying too hard.

4. If they have a favorite sysadmin book, look to see if the author is speaking at the conference. If they respect the author, they'll be more open to sending you to the conference. If they like any of my books, tell them about my new class!

5. Tell them they can come too. With classes like "A Sysadmin's Guide to Navigating the Business World", "Workplace Presentations 101 for System Administrators" and "Team Efficiency" there is a lot for them to learn too!

Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14. I look forward to seeing you there!

Usenix LISA registration is open...

http://www.usenix.org/events/lisa11/ (as of a few minutes ago)

The 9-11 disaster investigation website

National Institute of Standards and Technology (NIST)'s division of building and fire safety performed the scientific investigation of the World Trade Center (WTC) disaster. Much of the related video, audio and photographic evidence was released under FOIA. Just in time for the 10th anniversary of the disaster the FOIA'd data was released on their website:

http://wtcdata.nist.gov

Since FOIA requires the raw, unaltered, data to be released, many of these videos are at very high resolution. (Lower res versions are available for easier viewing, of course).

If you go to the website, you can watch all the material.

If you go to Usenix LISA 2011, you can see a presentation by the sysadmins that built the site, and learn the technical and non-technical challenges that threatened the project along the way.

This year's Usenix LISA is in Boston from December 4-9, 2011. See you there!

ServerFault Scalability Conference (and DevDays 2011) Cancelled

DevDays 2011 is Cancelled

Q: What about the ServerFault Scalability Conference?

A: That has been canceled, also.

Sigh.

The full story here.

If you had registered hoping to see me speak, my apologies. Please refer to http://everythingsysadmin.com for a list of my other appearances.

People in the Princeton, North Carolina and Pittsburgh area should be particularly interested in that list.

Also... soon I'll be announcing 3 half-day tutorials that I'll be teaching at the Usenix LISA conference in December in Boston. Start warming your boss up to the idea of sending you to a conference right after Thanksgiving. I'm really psyched about the new material. I hope to see you there!

New sponsor: Scalability from serverfault.com

Notice a new advertisement on the right-hand side of this blog from ServerFault.com's "Scalability" 1-day conference. Use the discount code "everythingsysadmin" and get $100 off registration.

I'll be speaking that day. I'm working on my slides right now!

Please put the LISA Button on your web site or blog

Would you like to help Usenix LISA? Please put the LISA11 button on your website or blog. HTML code can be found here: http://www.usenix.org/events/lisa11/

I just put it up on www.TomOnTime.com and changed the position of where it is on EverythingSysadmin.com

The graphic will change over the next 7 months so you don't have to do anything as we move from "collecting submissions" mode to "registration is open" mode to "thanks for attending" mode. Isn't distributed computing awesome?

Tom

MIETSPTUL Part 2: Tip for writing Practice and Experience Reports

As a quick follow-up to Making it easier to submit papers to Usenix LISA, here's a tip for writing a good Practice and Experience Report: Work backwards from the "lessons learned".

Step 1. Write down the 4-5 things that you wish you knew before you started the project. That is, things that readers will feel they've learned after reading your paper.

Step 2. Work backwards from those 4-5 things to figure out what people need to know to understand them (for example, the story of how the project got started, the problems you faced, and how you solved them). You really don't need to include much more. The audience is technical and doesn't need that much hand-holding.

If you work out those things (preferably on a whiteboard or index cards) the paper will write itself. I prefer to start on a whiteboard because it prevents me from starting to write paragraphs too soon.

Making it easier to submit papers to Usenix LISA

[This was originally published on The Usenix Update Blog]

PICC "early bird" discount almost over (plus two interviews)

The generous early-bird discount for LOPSA PICC (http://picconf.org) goes away on April 4th. This is a good time to talk with your boss about sending you while he/she can save money.

Registration information is here: http://www.picconf.org/registration

This is the absolute least expensive way to get my time management training. There is a heck of a lot of other great training and talks planned. I hope to see you there!

Speaking of the conference...

Read this (brand new!) interview with William Bilancio. He is one of the PICC organizers.

-Tom

P.S. I'll be interviewed on Episode 151 of the (Mind Of Root Podcast](http://www.mindofroot.com/). I'll let you know as soon as it comes out!

Free AA_Console for LOPSA_PICC Attendees

Great news from LOPSA!

"Every person who comes to PICC11 will receive [AA Console](http://www.adminarsenal.com/admin-arsenal/main/) (which used to be named Admin Arsenal) PLUS they're also throwing in a copy of [PDQ Deploy Pro](http://www.adminarsenal.com/pdq-deploy-pro/main/). Together, this is a $500 value for absolutely nothing except showing up to a conference you wanted to attend anyway."

http://www.standalone-sysadmin.com/blog/2011/03/admin-arsenal-ups-the-ante-for-picc/

Very impressive!

Time to register, eh?

LOPSA PICC Full Program Announced

What is LOPSA PICC? http://picconf.org

• Presentations, education, and fun.
• IT and syadmin (Linux/Unix, Windows, Networking & storage).
• 2 days, 1 night, conference.
• Low price/high value. Community-based, non-profit.
• April 29-30, 2011 @ Hyatt Regency New Brunswick, New Jersey.

Where else can you find a regional conference with national speakers, hot topics that will help you advance your career, all meals included, and not have to travel 3,000 miles to get there?

Find out more and register: http://www.picconf.org

• Twitter: http://twitter.com/picconf
• Facebook: http://picconf.org/facebook
• Email: http://lists.picconf.org/mailman/listinfo/picc11-announce

HALF-DAY TRAINING SESSIONS:

• "PowerShell Fundamentals", Steven Murawski
• "Grokking Python", Brian K. Jones
• "Over the Edge System Administration, Volume 1", David N. Blank-Edelman
• "Internal documentation for SysAdmins", Chris St' Pierre
• "Windows Enterprise Security", Troy Mckee
• "Advanced Time Management: Team Efficiency", Thomas A. Limoncelli
• "Security Best Practices and Tools for Linux", Matt Disney
• "Workplace Presentations 101 for IT Professionals", Adam Moskowitz
• "Backups, Archiving, and Life Cycle Management: Riding the Wave of Data Proliferation", Jacob Farmer
• "Using and Migrating to IPv6", Robert Harker
• "Blitzkrieg Branding", Jesse Trucks
• "Non-Obvious Nagios", John Sellens

45-MINUTE TALKS AND PRESENTATIONS:

• Effectively Monitoring MySQL
• The Path to Senior SysAdmin
• Change Management and the IT Infrastructure Library (ITIL)
• Using http Response Time Histories to Detect Problems
• Leadership and Troubleshooting from the Trenches
• MongoDB
• Stack Overflow Infrastructure
• Leveraging an Enormous Technology Community
• Thoughts on a University-level Major in System Administration
• Continuous Integration via Hudson
• Lightning Talks

PLUS all attendees receive:

• a 12-month LOPSA membership/renewal
• a licence for PDQ Deploy Pro from Admin Arsenal(a $1000 value)
• the awesome conference bag

FUN STUFF TOO: - Friday night banquet and movie festival! - All meals! (Friday lunch only for people attending training classes)

• $324 without half-day tutorials (all meals except Friday lunch)
• $524 with half-day tutorials and all meals ($449 until April 2! Register now!)

Cascadia IT Conference (Seattle)

LOPSA's Cascadia IT Conference starts today (the training portion started yesterday). Reviewing the program grid it looks like it is going to be a fantastic day. I wish I could be there! Congrats to the committee that put the conference together, especially Lee Damon the conference chair. I love to see community-based, volunteer-only conferences springing up.

You can follow the conference on Twitter hashtag #casitconf

(If you are on the east-coast and jealous of Cascadia, you'll be happy to know that LOPSA's PICC conference in NJ is just 6 weeks away! Register today!)

Call For Papers: 5th ACM Symposium on Computer Human Interaction for Management of IT December 4-5, 2011 - Boston, MA

If you notice there is a new conference listed on the "Awesome Conferences" listing on EverythingSysadmin.com: CHIMIT 2011

CHMIT has published their "Call for Contributions" on their new website http://chimit.acm.org

CHIMIT is a conference for people that study IT people, how we work, and how we can work better. It is a small conference and I encourage people (sysadmins and researchers) to participate. We need more things like this!

I've mentioned CHIMIT a number of times including a longer explanation of why this is an important area of study and reported about a panel I was on too.

This year CHIMIT is in Boston, Dec 4-5, 2011. It is concurrent with LISA so you can attend both very easily.

If you know a researcher that studies Human Factors in IT, please make sure they know about the Call For Contributions: http://chimit.acm.org/cfp

Seattle folks! Still time to register for Cascadia!

It is just days away. Seattle-folks, don't miss out! The training program is top notch and the talks on Saturday look very interesting! (I wish I could be there!)

http://www.casitconf.org

If your boss won't pay for it, tell him I said he/she should send you. (Forge email from me saying so. If he/she asks, I'll back your story.)

Awesome Sysadmin Conferences!

Awesome Sysadmin Conferences

2011 National: Dec 4-9: Usenix LISA '11, Boston, MA 2012 Regional: March 23-24: LOPSA Cascadia IT Conference, Seattle, WA Regional: May 11-12: LOPSA PICC 2012, New Brunswick, NJ

Training classes announced for PICC (NJ-area)

You can attend my new class "Advanced Time Management: Team Efficiency" there.

http://www.picconf.org/training-program#f6

The more of the schedule PICC announces the better it looks. Save April 29-30 on your calendar. Better yet, register today!

Casitconf'11 early-bird registration savings extended to the 23rd

Cascadia IT Conference has moved the early-bird deadline to Wednesday, the 23rd of February. Call it a President's day sale.

http://www.casitconf.org/

Some talks I'm particularly excited about:

• Incident Command for IT: What We Can Learn from the Fire Department, Brent Chapman
• Chose your own Adventure, Adam Jacob
• Talks on Chef, Puppet and Cfengine
• ...and the entire training schedule looks top notch!

http://www.casitconf.org/

Register now!

Cascadia IT Conference - Early Bird Registration ends in 1 week (Feb 16)

The Cascadia IT Conference wants to remind you that this is the last week you'll be able to save money on registration by getting the Early Bird discount!

http://bit.ly/CasITConfReg

Each half day tutorial is $125, and each half day tech session is $100, but with the Early Bird discount, they're only $105 and $80, respectively. You can save even more money by purchasing a bulk pack of 2 days of tutorials for $399 (or one day of training, one day of tech sessions for only $359!).

The Early Bird discount ends on February 16th, which is this coming Wednesday. Register now so you don't forget!

http://bit.ly/CasITConfReg

This conference is the premier opportunity to meet Information Technology experts and professionals and get the inside track on one of the hottest regions for IT in the world. Talks, presentations, poster sessions, lightning talks, and DIY unconference sessions will be available. Exchange ideas, learn, and enjoy the all-important "Hallway Track" networking with other conference attendees and world-renown speakers.

Get more information at http://casitconf.org

Calling All Windows Trainers

The PICC Conference is looking for Windows Trainers. If you have experience training people on PowerShell, ActiveDirectory, or Windows 7 (or know someone who does) please read Matt's blog post.

LISA papers about robots that correct broken configuration files.

Ok, not actual robots but hear me out. I'm sitting in a session at LISA 2010 right now (Wednesday, 2pm Papers session) where all 3 papers are about systems that analyse some kind of configuration file and, given some tests, can find a problem and fix it. The three papers cover (in order): router configuration, Role Based Access Control database, and firewall rule set.

The third paper (which is the one about firewall rules, and happens to have won "Best Student Paper") they have a number of ways to manipulate rules in an effort to fix a broken configuration: swap two rules, delete a rule, change an IP addr in a rule, etc. They've invented a novel way to search all the possible combinations of these things so as to quickly find their way to a modified version of the ruleset that works. Pretty cool.

But wait, how do you know if a ruleset "works"? Well, you generate a bunch of tests and run them through the system. If they all pass, the ruleset "works". A test might be "a packet with src address 1.2.3.4, port 80 should NOT get to 1.4.4.4 port ANY". You can have a human write them. But they went further. They invented an algorithm to generate a nearly-optimal number of tests patterns (for example, if you have a /24, you can deduce cases where you don't need to test all 256 addresses in the /24). A human needs to say whether these tests should permit or reject the packet, but at least the generation is automated.

And how do they test this? They invented an algorithm to efficiently apply all the tests to a ruleset at the same time. The ruleset is turned into a decision tree, and the tests can now be processes very quickly.

You'll have to read the paper for all the details.

...but...

But the point of this blog post is not to explain these papers in detail. What I really want to say is that I find it very interesting that people are applying AI techniques to finding and fixing configuration problems. 3 papers in one conference? It could be a trend.

And lastly, if we can write a lot of "unit tests" and automatically fix a firewall ruleset so that it comes into compliance with them, then my questions: at what point will we just write a heck of a lot of tests and let some AI robot write out firewall rules from scratch?

I, for one, welcome our new AI firewall-rule-writing overlords!

OpsCamp Silicon Valley

Tuesday, November 9, 9:00 a.m.-5:00 p.m., Room C3/4, San Jose Convention Center OpsCamp is an event for the open exchange of ideas around next-gen technologies and strategies for IT Operations. With the rapid change occurring in our industry, we need a place we can meet to share our experiences, challenges, and solutions. OpsCamp is organized in an unconference format.

http://www.opscamp.org/siliconvalley

(this is part of Usenix LISA, but doesn't require registration to attend)

Calling all Pacific NorthWest Sysadmins! Cascadia IT Conference

The Cascadia IT Conference will be Friday and Saturday, March 11th and 12th, 2011 in Seattle, WA.

They've published their "Call For Participation". If you are looking to present a paper, talk, or a tutorial, this is a great venue.

I'm impressed by the makeup of the committee. WIth the likes of Lee Daemon, this is going to be a "can't miss" conference for anyone in the Pacific North West. It is being produced by The League of Professional System Administrators and the Seattle Area System Administrators Guild, both organizations have excellent track records.

The full CfP is here:
http://www.casitconf.org/casitconf11/Call_for_Proposals.html

For more information, follow #CITC11 on Twitter!

Episode 13: Why I'm excited about Usenix LISA: The Usenix Staff

I could probably a lot more episodes but I think that if you haven't registered by now, there isn't much chance you are going to register. Though, if you are local to San Jose, it is pretty easy to stop by for a single day if there is something of interest.

Thus, I'm going to end this series with a little tribute to the Usenix Staff. The LISA conference would not be the success it is without the Usenix staff. They are friendly, detail-oriented, and really get things done. I'm ashamed to admit that sometimes I take for granted how well-run Usenix conferences are until I go to a non-Usenix conference. The difference is, well, sometimes shocking. My expectations have been set very high.

Thank you, Usenix Staff!

Tom

Episode 12: Why I'm excited about Usenix LISA: Wednesday is Storage Day!

Things are really changing in the world of storage. Heck, I can't even keep up. SSD technology is making all the old assumptions about how to design a service completely different. Object storage puts different pressure on our systems. De-duplication is becoming more widely available.

That's why I'm excited about "Data Storage Day" at LISA. Sign up now for a full day of talks by experts in the field about storage, storage, storage.

Registration is free (thanks to sponsorship by Cambridge Computer) and includes lunch plus admission to the LISA '10 Vendor Exhibition and Exhibit Hall Happy Hour and Wednesday BoFs.

If you are near San Jose and your boss isn't sending you to LISA, I bet you can get permission to attend a free one-day technical event that is both educational and, well, they're giving you lunch. What else do you want?

More information here.

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

Episode 11: Why I'm excited about Usenix LISA: David N. Blank-Edelman

[Note: Live interview with David, today at 11am Pacific, http://www.ustream.tv/channel/usenix ]

The closing event of the conference is always something fun. This year is no different.

This year's closing event will be a talk by David N. Blank-Edelman titled, "Look! Up in the Sky! It's a Bird! It's a Plane! It's a Sysadmin!"

For almost eighty years, American comic books have been mulling over just the sorts of questions you and I face each day we live as sysadmins, so why not look at what they have to say on this subject? Complete description here.

David is famous for giving talks that are both hilarious and educational at the same time. Many of his talks have compared system administrators to various social entities or occupations, for example his presentation on how Hollywood portays sysadmins, or how system administration is similar to veterinarians, or how system administration is similar to the work of sex therapists.

David closed the conference in 2007 with his talk about how system administration is similar to cooking, Listen to that talk here.

You may know David N. Blank-Edelman as the author of incredible books like Automating System Administration with Perl (known as "The Otter Book").

Can't get enough David? No worries! Sign up for one or both of his excellent tutorials.

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

Join USENIX today (2010-10-21) at 11am (Pacific) (2pm US/Eastern) for the next installment of the LISA '10 sysadmin chat. The episode will feature an interview with David N. Blank-Edelman. David is the Director of Technology at the Northeastern University College of Computer and Information Science and the author of the O'Reilly book "Automating Systems Administration with Perl". He has been a system administrator for over 25 years. http://www.ustream.tv/channel/usenix

Episode 10: Why I'm excited about Usenix LISA: Cloud Computing

So is Cloud Computing bullshit or what? You can't get a real answer from vendors because they are biased: they sell cloud-related products and will only tell you it is great, or they don't sell cloud-related products and tell you it's crap.

The truth is that specific situations are perfect for cloud computing and others it just doesn't make sense.

What makes things worse is that the "list of specific situations" was different last year, is different this year, and will be different next year.

So how can you find out the truth?

The best way to find out the reality vs. hype of any trend or product is to network with people at LISA. Chances are you'll meet people that have tried products you haven't gotten around to and you will benefit from their experience. And you can share what you've learned too.

What do I think? I think "cloud computing" is a new name for something people at LISA have been doing for decades. The reason it sounds so odd is that most Unix/Linux people have been doing this kind of thing for so long, we didn't realize that the other 95% of the world hasn't. We have a great opportunity to help others by sharing our experience in this area, and a great responsibility to do so with honesty and integrity.

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

Episode 9: Why I'm excited about Usenix LISA: Puppet vs. CFEngine3

There were a couple years where Cfengine didn't have many new releases. During that period Puppet was created. But now Cfengine has had a big, big new release: Cfengine 3. CF3 was "hot off the presses" during last year's LISA and now it has been around for a full year. With a full year under it's belt, I look forward to seeing how it has stacked up.

What are these programs I'm talking about? They are utilities that do "configuration management" or "CM". CM tools are for maintaining many machines from a single information base. For example, you define "a web server has these files" and "a database server has these files". Now you can define certain machines as web servers and others as database servers; run the CM tool and machines get properly configured; machines that someone messed up gets fixed, and machines that didn't need any changes are verified to be good. CM systems are often used for all kinds of configuration: from making sure netmasks are correct, to installing security patches, to deploying massive numbers of complicated servers. This isn't just for servers, by the way, I know one company that uses Puppet to make sure thousands of Mac laptops stay properly configured and stay patched.

If you think CM will be useful in your...

wait a sec....

"IF"??? "IF"?? There is no "if" about CM. All sites need some kind of CM. In fact, all sites DO some form of CM, you just might be doing it manually or... or badly. So, SINCE your site needs CM, what does LISA have for you:

A mini-conference: On Sunday there is a full-day workshop (think of it as a "mini-conference") called "Real-World Configuration Management". This is a yearly event and is a "meeting of the minds" about CM. Many major innovations have been announced and/or birthed here.

Two ways to learn Cfengine 3: Tuesday there is a half-day tutorial: "Configuration Management Solutions with Cfengine 3" by the creator of Cfengine 3 himself and all around amazing guy: Mark Burgess Wednesday there is a half-day tutorial: "Cfengine 3 for Cfengine 2 Users" by the prolific and amazing Æleen Frisch. So if you've used Cfengine 2 and want to upgrade, this is the place for you.

Both Cfengine and Puppet communities have gatherings to meet and hang out during "BoF" sessions at night: Cfengine 3: Tuesday, November 9, 7:00 p.m.-8:00 p.m, hosted by Aleksey Tsalolikhin of VerticalSysadmin. Puppet: Wednesday, November 10, 7:00 p.m.-8:00 p.m., hosted by Nan Liu of Puppet Labs.

During the technical sessions there is a paper co-authored by Mark Burgess: "Troubleshooting with Human-readable Automated Reasoning". I haven't read the paper yet, but I bet it is of interest to CM fans of all stripes.

And last but not least, Usenix veteran D. Brent Chapman has started a company that does CM for your network. Now you can be sure that your firewalls, routers, and switches are all orchestrated as well as your Puppetized or Cfengined servers thanks to his open source project "Netomata". There is a half-day tutorial about it and other similar tools by Brent himself on Friday entitled "Automating Network Configuration and Management"

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

Episode 8: Why I'm excited about Usenix LISA: Free stuff!

I'll be giving away 2 books at each of my tutorials. Maybe more. One book will go to whoever asks the best question, and another will be by random drawing (unless someone has a better suggestion). Depending on how much space I have in my luggage, I might bring more.

• 2 copies: Time Management for System Administrators
• 1 copy: The Practice of System and Network Administration (2nd edition) and one
• 1 copy: The Complete April Fools RFCs
• and more depending on how many fit in my luggage

I'm working on a couple other surprise gifts. You'll have to wait and see.

Other authors often give away books when they speak at LISA. Have you registered yet?

Note: Today is the last day to register and receive the "early bird discount".

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

One day left for Usenix LISA Early Bird Registration!

Tomorrow (Monday) is the last day to get the Early Bird Registration discount. Register soon! http://goo.gl/J0e6

Episode 7: Why I'm excited about Usenix LISA: The Hallway Track

LISA has a couple different "tracks": Invited Talks, Refereed Papers, Guru Sessions, and so on. Then there is the unofficial track: The people that hang out in the hallway and talk.

This is where you learn the dirt.

Which products are /actually/ more reliable than others? Which open source products have the best track record? Which bar in the local area has the best beer?

The hallway track knows all these things. Just hang out and join the conversation.

Usenix LISA is in San Jose, CA from November 7-12, 2010. There are many discounts available. Early registration (for non-students) ends Monday, October 18.

See you there!

Episode 5: Why I'm excited about Usenix LISA: Wicked cool security papers

One perk of being on the committee is you get to read the papers before anyone else. The security papers this year are really exciting.

One being presented on Wednesday (previously I wrote Thursday; that was a typo) applies AI techniques to fixing firewall rulesets. Imagine if a firewall noticed a new set of ACLs made certain unit-tests fail. This system can actually modify the rules and, with pretty good likelihood, fix the problem. The solution might not be optimal, but it is good enough until a human can come in and fix the problem optimally. The abstract is online, and if you are a Usenix member, you can read the entire paper now. The entire paper will be readable by all after the conference. The paper is called "First Step Towards Automatic Correction of Firewall Policy Faults" by Fei Chen and Alex X. Liu, Michigan State University; JeeHyun Hwang and Tao Xie, North Carolina State University.

Maybe you think this kind of self-correcting firewall system is crazy. If you attend the conference, you can be there for the Q&A session and participate.

A list of all the papers and technical presentations are here: http://www.usenix.org/events/lisa10/tech/tech.html

Usenix LISA is in San Jose, CA from November 7-12, 2010. There are many discounts available, but the best one is to simply register by Monday, October 18.

See you there!

Episode 4: Why I'm excited about Usenix LISA: My talks!

A little self-promotion never hurt anyone, right? :-)

I'm giving a few talks this year but there are two I'm particularly proud of.

One is my new time management tutorial, "Time Management: Team Efficiency". I'll be explaining things that I've never put into books or articles before. These are the things we do that save other people time, and therefore save us time too. For example, a well-written blast email message saves the readers time; a badly written blast email message doesn't get read, so it was a waste of your time to write it. There's a big section with specific tips on how to turn wasteful, useless meetings into useful, brief meetings. It's a half-day tutorial that will save you weeks of time. You're entire team doesn't have to attend, but it wouldn't hurt.

The full workshop description is here. And this link lists all my speaking engagements at LISA.

Two tips:

• Register soon! Seating in my tutorials is limited and it may run out.
• Come early. This tutorial is on Monday. If you usually skip the first day or two of tutorials you may miss it.

I look forward to seeing you there!

For more information follow the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10.
You can find a long list of discounts available here. Register soon!

Episode 3: Why I'm excited about Usenix LISA: It doesn't conflict with my birthday.

LISA has often conflicted with my birthday but not this year. That doesn't mean I can't be tempted into throwing a party, but I think I'd rather just attend the BoF (Birds of a Feather) sessions at night.

What is a BoF? Suppose you are really interested in a particular topic: CFEngine, Ubuntu, TCP/IP on 8-bit computers, system administration at companies with non-English speaking users, etc. You can sign up for a 1-hour timeslot and gather people to talk about it.

BoFs are one of the absolute best ways to make connections at LISA. This is a very valuable service.

There are many timeslots and many rooms available. Some people pre-register their topic, others wait until the last minute. The list of pre-registered BoFs is online here. (Oh, and some BoFs are officially sponsored by vendors. You can guess which one I'll be involved with.)

This year's conference is in San Jose, so all my silly-cone valley friends should have no excuse. See you there!

For more information follow the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10.
You can find a long list of discounts available here. Register soon!

Episode 2: Why I'm excited about Usenix LISA: The Chuck Yerkes Award

The Chuck Yerkes Award is presented annually to the person (or people) judged to have most significantly mentored others on the various SAGE member forums. It was created after Chuck Yerkes' untimely death in 2004. Beginning in 2009, the Chuck Yerkes Award is determined and presented by LOPSA.

It was my honor this year to help suggest and gather nominations and aide in the selection process. So, I know who the winner (or winners) are, but you'll have to attend to be there to applaud in person.

By attending the conference you get to make them blush by congratulating them every time you run into them in the hallway.

For more information follow the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10.
You can find a long list of discounts available here. Register soon!

How to handle cross-training and redundancy in system administration teams?

[ This is the kind of topic I'll be covering in detail at my training class at Usenix LISA 2010: Time Management: Team Efficiency.]

I saw this question in email the other day...

In a small team managing [what I think to be] a fairly diverse environment like ours how do you handle [cross-]training/redundancy? I know there are a large number of things right now that no one other than me knows how to do. I think I have the everyday issues well documented, but non-routine issues may cause issues.

First, let's define "cross training". A department with many sub-teams wants everyone to be able to handle tasks from the other sub-teams. For example, you have an IT department with three sub-teams: a Linux sub-team, a Networking sub-team, and a Storage sub-team. In an ideal world, the Storage sub-team members should be able to handle 80% of the requests of the Linux sub-team, and vice versa. Being able to handle 80% of the Storage-related requests probably means knowing about 20% of what someone on the Storage sub-team knows. That's ok. 80% of the requests are probably things like add/delete/change requests (add a new virtual partition, increase the size of an existing one, etc.), and common problems (what to do with a NFS stale file handle, etc.). If everyone in the department could handle those tasks, the individual teams could focus on higher-order issues like scaling, monitoring, and optimization.

The #1 thing you need to be able to do is document those "sharable tasks". But everyone hates writing documentation, right? So don't write documentation: write checklists. You only have to document the steps, and you can use language that assumes the person has basic knowledge. If you keep the checklists on a system that permits anyone to edit the documents (i.e. a wiki or a source code repository), then they can fine-tune your docs as they use them.

Some standard checklists to write are: 1. things we do for each newhire. 2. things we do when an employee is terminated. 3. how to: allocate space in the machine room, set up a new server, deploy a workstation, add to the puppet configuration, etc. etc.

If you do cross-training right, rather than a pager rotation for each sub-team, you can have one globally pager rotation. That means rather than being oncall once every 3 weeks, you might be on call once every 12 weeks!

To set up cross-training for a pager rotation, I also suggest checklists.

For each page that you might receive, have a checklist of what actions to take. Check list, try rebooting that, look at the logs for messages that say such-and-such. The last step should always be "if all that failed to fix the problem, escalate to so-and-so." If so-and-so feels he/she is getting called in the middle of the night too much, ask them to improve the checklist.

Encourage people to write the checklist when they add the alert rule to the monitoring system. If someone won't or doesn't write the checklist for a particular alert it just means they have agreed to be called in the middle of the night every time.

These checklists will grow and improve over time. Every time you have an outage, augment the checklists that would prevent that problem in the future.

In my tutorial at Usenix LISA, I'll expand on this and show how this system can be used to coordinate training in general, especially in ways that help bring newhires up to speed. This is material I've never written about or included in any other tutorial, and you can only see it at Usenix LISA in San Jose, Nov 7-12, 2010. Seating is limited. Register today!

Episode 1: Why I'm excited about Usenix LISA: Keynote about IT at the LHC

The keynote speaker is from the IT department at the Large Hadron Collider (LHC)! This team was building their IT infrastructure since 2000 in prep for what has become one of the biggest data gathering and processing systems in the world.

Tony Cass, CERN
CERN's IT department, in collaboration with particle physics computing institutes around the world, has been preparing for the arrival of Large Hadron Collider (LHC) data since early 2000. Now that the accelerator is in operation--with routine operation at record colliding energies since March--I will report on how the computing infrastructure has stood up to the test of real data, compare today's reality to both recent tests and original predictions, and then risk some predictions about the future evolution of the Worldwide LHC Computing Grid.

All the technical sessions are described here and you can find a long list of discounts available here.

For more information as new speakers get announced, following the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10

Multiple-employee discount for attending LISA 2010 -- San Jose, CA, November 7-12, 2010

Since I'm teaching "Time Management: Team Efficiency" at Usenix LISA (San Jose, CA, November 7-12, 2010) I thought it might be a good idea to point out that there is a discounts for organizations sending 5 or more employees. This and other discounts are listed on the conference web site.

People often ask me "but how do I get my co-workers to also do [whatever]" one way is to have them all experience the same training.

(Please remember that my tutorials are on Monday of the conference. This is rather early in the week, so if you normally skip the first few days you'll need to arrive Sunday night if you want to catch all my tutorials.)

The only place to attend my all-new "Time Management for Teams" class is LISA 2010

The tutorial I'm teaching at Usenix LISA (San Jose, CA, November 7-12, 2010) Monday afternoon is totally new material: "Advanced Time Management: Team Efficiency"

When I say "totally new", I mean it.

1. New material. This is material I've never put in books, articles, blog posts or given talks about at other conferences.
2. New topic. The last 3-4 years I've been repeating tutorials that I've taught before with one exception that was 40% a rehash of my older stuff.
3. New format. Thanks to the pervasiveness of wifi and laptops at LISA, this tutorial will have in-class exercises. It will be a mix of lectures, demos, and in-class exercises.

This is a LISA-2010 exclusive. I'm reducing my conference attendance for a few year so this may be the only chance you have to attend.

The class is on Monday afternoon. Monday morning I'm teaching an upgraded version of "Time Management for System Administrators" class which is a good precursor to the afternoon class.

LISA runs from Sunday to Friday but a lot of people skip the first day or two. If you are making plans to come to LISA (and September is a good month to start talking it up with your manager), make sure your travel arrangements include being at the conference Monday.

Read the complete tutorial description.

The sooner you register, the more discounts are available!

Remember that LOPSA members save $45 off Usenix LISA registration!

LOPSA members save $45 when they register at to attend Usenix LISA, in San Jose, CA, November 7-12, 2010. I'm a member of LOPSA and I hope you join too. To find out about this discount you have to have to be a LOPSA member, and to become a LOPSA member you need to set up a free account first.

Step 1: Register for a free account on the LOPSA website: http://lopsa.org/user/register

Step 2: Become a LOPSA member: http://lopsa.org/joinup

Step 3: Paid members will see the discount code on http://lopsa.org/MemberDiscounts (Non-members see different text. Sneaky, eh?)

Ohio Linux Fest: you don't have to be from Ohio!

What surprised me when I attended Ohio Linux Fest was that it is a national conference; it draws people from all over.

One of the little-known gems at OLF is their training sessions called "OLF University." It is excellent training that is at a very nice price. Considering the high-caliber trainers that they've recruited, I think (and I've told OLFU coordinators) that training like this should be priced 2x or 3x higher. The productivity boost from just one class will pay for itself in a month or two. I recommend people sign up before the organizers start listening to me.

Beth Lynn Eicher wrote a great article about OLF and the training. Check it out.

If you aren't going to OLF University, but are going to OLF, be sure to stop by the LOPSA table!

Exciting news about LISA 2010... soon

I'm going to have a completely new tutorial at LISA 2010. I'm developing the material right now and the more I see, the more I like it.

I'll announce the topic soon :-)

Ohio LinuxFest 2010 - September 10-12, 2010

Registration for OLF opened today. This conference draws people from all over the country, not just Ohio.

The Ohio LinuxFest is proud to announce that registration is now open for Ohio LinuxFest. The schedule has also been announced, and this year will feature a fantastic line-up of talks for new and experienced Linux users. The 2010 Ohio LinuxFest takes place in Columbus, Ohio at the Greater Columbus Convention Center from September 10 through September 12, 2010.

The keynote is Christopher “Monty” Montgomery of Xiph.org will be talking about next generation open source media formats.

Admission is free but space is limited. Supporters can opt for the $65 package that includes lunch and an OLF t-shirt, or the $350 package that includes admission to the “OLF University” training program (which is so good I really think they should charge twice as much. Sign up before they start listening to crazy people like me!)

Sign up today at http://ohiolinux.org/register.html

More info at http://ohiolinux.org

Usenix starts YouTube channel

Usenix now has a YouTube channel!

Whether you want to watch a short, funny talk about Electronic Voting, the "Best Paper" presentation about a new security model for safe web browsing, or a technical and thought-provoking talk about how concurrency is changing everything we do by a computer science legend, this channel has everything a geek could want. (well, it's 8 videos so far, so maybe not everything.)

[If you don't have time for those videos, maybe the videos on www.TomOnTime.com will help you find more free time.]

Certification? Not yet.

At PICC I may have sounded like I thought there was an urgent need to create a sysadmin certification program. While I did talk about what I thought it would/could/should look like, I don't think this is a good time to create such a thing. A long-winded version of this paragraph is below.

An open letter:

I wish to clarify a statement I made at the PICC conference and point those of us that think about the future of system administration in a particular direction.

It has become apparent to me that a certification program cannot exist until the educational standards that it measures are generally accepted. That is, a certification should measure conformance to an pre-existing educational standard.

At the PICC conference, part of my keynote made the case for another attempt at creating a certification for system administrators. In the last few months I've thought a lot about the issue of certification. I've also had the chance to talk with with people that are familiar with how the AMA created its certifications for doctors. While I was not advocating for the immediate creation of a certification program, I may have given that impression. Let me be clear that I do not think that the industry has reached sufficient maturity to warrant a certification program as I described. The AMA's now pervasive certification program came after they worked with universities to develop curricula and other educational programs.

It would be prudent to focus on creating educational standards for the profession of system administration. We, the wider professional system administration community, need to work with academic institutions to create curriculum standards for system administration programs. While there have been attempts in the past, I do not feel this has gotten traction because the profession is not taken seriously in academia. This is changing. A number of factors are leading academia to take notice of the importance of operational excellence in IT. I would be glad to discuss strategy and opportunities with interested parties.

Every movement needs to be, at its heart, an attempt to save the world. It is trite to say that society is more and more dependent on computers. Yet our dependence is staggering even to me. From the logistics of getting food from farms to tables, to providing services related to healthcare, governance, media, security and defense; all of these things are reliant on IT such that they can no longer exists without it. And yet I feel that the digitization of society is still in its earliest of stages.

What could be more a more important way to save the world than making sure that society's underlying IT infrastructures are professionally designed, maintained, secured, and operated? We can not leave these things to amateurs and hobbyists, nor bureaucrats and lobbyists.

Sincerely, Thomas Limoncelli

Australia sysadmins! I'm coming to you!

I'll be keynoting SAGE-AU's conference in Hobart, Taz, AU the August 9-13, 2010.  I'll also be teaching 3 half-day classes.  Be there or be square!  

I'll be speaking at MacTech conference, Nov 3-5, 2010

I don't have a lot of time to post today, so this will be short.

Configuration Management Summit, Boston, June 24, 2010

(Reposting this announcement from Dan)

Fellow SysAds etc.-

First, I'd like to make sure you are all aware of the Configuration Management Summit next week in Boston on June 24 (details are at http://www.usenix.org/events/config10/). The first Configuration Management Summit aims to bring together developers, power users, and new adopters of open source configuration management tools for automating system administration. Configuration management is a growth area in the IT industry, and open source solutions, with cost savings and an active user community, are presenting a serious challenge to today's "big vendor" products. Representatives from Bcfg2, Cfengine, Chef, and Puppet will all be participating in the summit - this will be a valuable opportunity if you have been contemplating a configuration management solution for your systems.

There is also a special one-day training on Cfengine being taught by Mark Burgess on June 25 (details are at http://www.usenix.org/events/config10/#tut_cfengine). This class might be a review session for anyone on this mailing list, but it will also offer useful insights for people who are not new to Cfengine. Additionally, If you have colleagues who need to come up to speed on Cfengine quickly, this class will be an excellent opportunity for them to learn Cfengine directly from the author.

If you are interested in either event, you can register at http://www.usenix.org/events/confweek10/registration/ (and if you have questions, you can email me directly). I hope to see you in Boston!

Daniel Klein
Education Director
USENIX

DebConf coming soon!

Debian Conference is the annual Debian developers meeting, an event filled with coding parties, discussions and workshops - all of them highly technical in nature. It will be held in New York City, USA, August 1-7, 2010.

It is right in my neighborhood too! (so to speak, I live across the river).

More info here: http://debconf10.debconf.org/

CHIMIT '10 paper deadline soon!

If you study system administrators and are interested in presenting a paper about it, don't forget that the deadline for CHIMIT '10 is July 3, 2010. (The conference is Nov 12-13, 2010)

Thanks for a great conference!

LOPSA PICC 2010 was a big success. Thanks to everyone that attended.

LOPSA PICC is this weekend!

I'm spending a lot of time refining my keynote, updating slides for my Time Management and other tutorials.  It isn't too late to register.

LOPSA PICC: Final days before "last minute" price

It costs more to attend PICC if you register after May 1st or "at the door".  Due to higher administrative costs, registration price goes up shortly before the conference.

More thoughts about submitting papers to Usenix LISA

I've been a sysadmin for 2 decades. When I think back about my career I realize that the first time I presented a paper to LISA was a major turning point for me.

It wasn't my idea to submit a paper. My boss at the time put a lot of effort into career development and he suggested that a good bridge from being a junior sysadmin to a senior sysadmin would be to start submitting papers to conferences. I wrote about some projects we had been doing and submitted. I couldn't believe it when I got the acceptance letter!

Being published lead to many important things for me. It got my name around; other people wanted to collaborate with me. It helped me in job hunting; having a paper published gave me a new level of credibility. Most importantly it got me noticed by Addison-Wesley and that lead to co-authoring writing my first book (the second most important turning point in my career; which wouldn't have happened without those early papers). That lead to some other milestones, such as being honored with the SAGE Award.

Maybe you haven't considered writing a paper for LISA. Maybe you think your projects aren't that amazing. My first paper was about how we renumbered the IP address of 1,000 machines (this was before DHCP was popular). Doesn't seem to exciting, does it? We had interesting problems that needed to be worked through: http://bit.ly/9C8ykH

The last few years some papers (not by me) have included topics like: Migrating thousands of users to a new email server and why it became a disaster; the method someone uses for stress-testing their web server to find performance bottlenecks; data mining Cisco network configs stored in a source-code repository for many years; virtualizing networks; using a dependency graph to determine security risk. All of these are interesting because they solve real problems.

So...

The deadline for submitting papers is May 17th. If you want a mentor, ask the chair and a helpful committee member will be assigned to you.

Unlike past years, submitting papers is a bit easier this year:

• This year you don't have to write the entire paper! Submit a 1500-word abstract. If it gets accepted, then you'll have to write the paper (of course!).
• We are now accepting "experience" papers. Do an massive email migration? Deploy a new thingamabob? Survive an interesting attack, management change, or technology ? Tell us all about it!

If you have never submitted a paper to LISA, this is a good time to give it a shot.

It could be a turning point for you too.

Sincerely,
Tom Limoncelli

Submission guidelines:
http://www.usenix.org/events/lisa10/cfp/

Writing advice here:
http://engineerwriting.jottit.com/
...and...
http://everythingsysadmin.com/2010/03/writing-papers-for-usenix-lisa.html

Another reason to come to PICC

For more info about the conference visit picconf.org. Register today!

Writing papers for Usenix LISA (and other) conferences

Update: Someone else said it very well here

When I last mentioned LISA, I forgot to mention the big news! This year submitting papers is a lot easier! Less work for the authors!

Rather than having to submit the entire, nearly finished, draft in advance, you can submit a briefer summary. If it gets accepted, then you have to write the entire thing. This saves a lot of time in case your submission is not accepted (how would that happen?). It also lowers the bar to submitting, which is important. I think more submissions is better. If this is your first time submitting a paper, this is a good opportunity to go for it.

There are three things you might consider proposing:

• Refereed papers: Did you invent something? Prove a new theory? Create a new tool or software system? Submit a paper. Submissions are simply extended abstracts, 500-1500 words plus an outline of what the final paper will look like. (Details here.)
• Practice and Experiences Reports: NEW! This is a new category. It's a bit different. This is a story telling category. Have you completed a major project and would like to share what experience they gained? I think of it as "Here's what we wish we had known before we started." Very useful. (Details here.)
• Invited Talks: A lot of people don't realize this, but some (not all) invited talks are proposed by the people that give them. Hey, the Invited Talk chairs don't have ESP nor are the omnipotent. So if you have a hot topic that you are an expert at, or would like to put together a panel of debating debutants, propose it as an I.T. or a "Guru Session". (Details here.)
• (Other things you can submit)

The deadline is May 17, 2010 (The 2011 deadline is June 9, 2011.). Less than 2 months away!

This year I'm on the committee that will be judging the papers. I thought it would be useful to tell people my personal process for evaluating papers.

I've been on the Usenix LISA program committee a few times. People ask me for advice about submitting papers a lot. Usually I tell them to read the CfP, pay attention to the deadlines, etc. But the real important advice is what I'm about reveal below.

Don't give me a "surprise ending". Please don't force me to read 4 pages before you tell me what your point is. I want to know right away. It saves me time.

A "surprise ending" paper is where the abstract says, "We've developed a new way to do X", the next 10 pages explains the history of X, then their experience with X and what problems they've seen with other systems that do X, and then at the end they finally present their surprise ending: they do X differently.

Or worse, the last paragraph says that they do it with a secret that they'll reveal in the final version of the paper, only to be revealed if the paper is accepted to the conference. I kid you not. People have done that.

The problem with the "surprise ending" is that now I have to go back and re-read the paper because now I finally know what I needed to know to understand the paper.

A surprise ending is great if you are writing a film, not if you are writing a paper.

When submitting papers to conferences, give away the surprise ending in the first sentence:

We present a backup system that is more appropriate for backing up thousands of laptops because it uses file hashes to avoid backing up files that are common to multiple laptops. Our architecture scales to 1,000 laptops.

If this is truly a "first", then I know right away you will have gotten me exceited about reading the rest of the paper. Otherwise, at least I know what I'm in for and can judge it on other merits.

As I read the opening of a paper I construct a mental rubrick that I will use to judge the paper: There better be a section of the paper that validates each of these claims: statistics that demonstrate that there is a problem (how often the same file exists on laptops), proof that demonstrate that this solves the problem (metrics about the reduction in backup time, bandwidth used), proof that shows this doesn't cause new problems (performance impacts), other problems that come to mind (laptops in "sleep mode"), and validation of their claim that it scales.

I will create this list of checkpoints and produce a grade for each one of them. If they wrote the paper well, I can do this in one reading.

By telling the audience the conclusion in the first sentence you save the audience time. This is true for the judges deciding whether or not to accept the paper, or after publication when people are deciding whether or not to read the paper.

Don't think that by holding back the good bit until the very end it will entice people to read the paper. If they are reading the entire book of proceedings they're more likely to skip a paper entirely if the relevence isn't made clear at the start. Sysadmins tend to specialize on storage, networking, operating systems, or so on. People usually read the papers that have to do with their specialization. If they can't tell your paper is about their specialization in the first paragraph, they're going to skip it.

The essence of good writing is brevity. Tell me what you are trying to say, then stop. Just be complete.

Why are you coming to the conference?

We asked the first people that registered for PICC why they are coming and got some surprises! Read them here.

LOPSA PICC: Complete program announced

(Registration price goes up $75 on Monday night.  Sign up now!)

What is LOPSA PICC?  http://picconf.org

    Presentations, education, and fun.

    IT and syadmin (Linux/Unix, Windows, Networking & storage).

    2 days, 1 night, conference.

    Low price/high value.  Community-based, non-profit.

    May 7-8, 2010 @ Hyatt Regency New Brunswick, New Jersey.

KEYNOTES:

* David Blank-Edelman, "How SysAdmins Are Portrayed in Pop

    Culture"

* Thomas A. Limoncelli, "Smooth Operations: Stopping the

    spiral of Emergency System Administration"

* Eben Haber, IBM, "System Administrators in the Wild: What

    we've learned from watching you!"

HALF-DAY TRAINING SESSIONS:

* "Automating System Administration with Perl",

    David N. Blank-Edelman

* "Essential IPv6 for Linux Administrators",

    Owen DeLong

* "Help! Everyone hates our IT department",

    Thomas A. Limoncelli

* "IT Policies: Why IT Policies are needed and how to develop them",

    David Parter

* "In Search of "Senior"",

    Brian Jones

* "Intro to Powershell: Automate like a Wizard",

    Joseph Kern

* "Introduction to Virtualized Storage Management",

    Jesse Trucks

* "Next Generation Storage Networking: Beyond Conventional SAN and NAS",

    Jacob Farmer

* "Time Management for System Administrators: A New Approach",

    Thomas A. Limoncelli

* "What's New in IIS 7.0/7.5?",

    Steve Heckler

45-MINUTE TALKS AND PRESENTATIONS:

* An overview of Google's technologies: GFS, MapReduce, etc.

* Budgeting for System Administrators 

* Drupal On-Demand

* High Performance Computing across the WAN at NOAA

* How to stop hating MySQL: Fixing common MySQL myths

    and mistakes

* Job-Hunting Skills for System Administrators 

* Keeping Nagios Sane

* Mentoring: It's for everyone!

* Panel: Tech Women Rule! Creative Solutions for being

    a (or working with a) female technologist

* Technical Community Response for the Haitian Earthquake

* Using Hierarchical Protection Domains for Network Security

THE "UNCONFERENCE":

* 12 timeslots where YOU pick the topic! ("unconference")

PLUS all attendees receive:

* a 12-month LOPSA membership/renewal

* a licence for Admin Arsenal (a $1000 value)

* the awesome conference bag 

FUN STUFF TOO:

* Friday night banquet and movie festival!

* All meals!  (Friday lunch only for people attending training classes)

$249 without half-day tutorials (all meals except Friday lunch)

$475 with half-day tutorials and all meals ($399 until March 22!  Register now!)

Where else can you find a regional conference with national speakers,

hot topics that will help you advance your career, all meals included,

and not have to travel 3,000 miles to get there?

Find out more and register:

        http://picconf.org

Twitter: @picconf

Facebook: http://picconf.org/facebook

Email: http://lopsanj.org/mailman/listinfo/picc2010-announce

RSS: http://lopsanj.org/events/picc10/feed

LOPSA Conference schedule published!

If you were waiting to register until the complete schedule was revealed, get that credit card out!

LOPSA PICC last night published the final slate of papers and speakers (if you didn't get your accept/sorry email, please let us know). http://picconf.org now contains the complete schedule.

You can attend for as little as $249, or $99 for students. The training program is extra.

If you aren't sure how to ask your boss for permission, we have some advice.

Tom

Countdown to LOPSA PICC!

Click the cartoon for more information!

Submit your talk proposals to PICC today!

(PICC is a regional sysadmin conference to be held in central NJ on May
7-8, 2010. I'm on the planning committee. http://picconf.org)

Today is the deadline for proposals for papers, talks, and such.

We're a little low on submissions so I'd like to make one more "beg". We'd love to have a talk about PHP for sysadmins, something fun you've done with Arduino, your favorite JS
library, a walk-through on setting up Google Apps. Demo your favorite open source project, or propose a panel of people to talk about something you find interesting (I can help find others for your panel). It is an excellent way to spread the word about a project you are involved with.

We've tried to make the proposal process really easy. Just send your
contact info and topic plus a 1-2 paragraph description to
submissions@lopsanj.org

For more info IM me and/or view:
http://lopsanj.org/events/picc10/cfp

Tom

P.S. Today is the deadline but we can grant extensions to anyone that writes and asks.

LOPSA Regional Sysadmin conference announced!

Registration is open! Keynotes announced! Register today!

NJ/NY/PA/DE/CT/RI/MA Regional Sysadmin Conference

I've mentioned the Professional IT Community Conference (PICC) before, but now the fun has really started.

Registration is open and the speakers have been announced!

The cost is low, and the benefits are huge. I know from Google Analytics that this site receives hits from over 500 unique users in the region of this conference, every month. We don't have that kind of space at the conference. It's going to sell out at some point, so make sure that you talk to your boss now about attending. We've even drafted a letter to help convince them that it's worth your time and their money.

It isn't often that you get a local conference with internationally known speakers like David Blank-Edelman (O'Reilly's "Automating System Administration with Perl") as well as Eben M Haber from the IBM research lab in Almaden, CA! This conference is going to get you the biggest bang for your buck out there.

(Oh, and I'll be speaking too. I've reworked 2 of my half-day tutorials ("Time Management" and "Help! Everyone hates our IT department!") plus, I've been asked to give the Saturday morning keynote, which I'll be using to premiere material from my yet-unannounced new book! But the book is a secret still, so hush!)

What I'd like you to do is to help me get the word out. Please. Not everyone reading this is in the NJ/NY/PA/CT/DE/RI/MA area. For those of you who aren't, please tell other people. Follow us on twitter at @picconf, email the site (http://www.picconf.org) to anyone you know in the area who might be interested, tell user groups about it, heck, we've even got a facebook page that you can become a fan of.

This is absolutely a grass-roots kind of effort. We have a very small advertising budget, so I want to use that as intelligently as possible. That means getting your help for the initial waves, and to spread it by word of mouth, by email, link, tweet, IM, and whatever else you've got.

A very big thank you to every one of you out there who reads this blog and supports me. I appreciate all of you.

LOPSA New Jersey conference registration is open!

Come one! Come all! Training schedule and speakers are now listed on the web site. Oh, and most importantly... registration!

picconf.org

I have a lot more to write about this but that will come soon. For now, check out the site!

What should I talk about at a conference?

I'm in the middle of writing proposals for Invited Talks and Tutorials at Usenix LISA 2010 and I thought I'd throw this question out to the readers of this blog:

What would you like to hear me talk about?

I speak at LISA a lot. I talk about Time Management, tips for running an IT department, and a few other things. But what would you LIKE to hear me teach or talk about?

To all my New Jersey friends:

Do you work in New Jersey? Let your "IT guy/gal" know how much you appreciate them this Valentines day!

Send this to them, or better yet, open a ticket at your helpdesk with this text!

(And if you really like them, CC: their boss!)

8< ---------- cut here ---------- >8

Happy Valentines Day to my favorite computer system administrator:

You only hear from me
when my computer is blue.
So this Valentine's Day
I'm saying "Thank you!"

I admit my computer problems,
like it's a reality-show confessional.
But you hide your frown,
and act very professional.

I think that you're great!
I know I'm a pest!
But I bring my troubles to you,
because you're the best!

Some roses are red,
some roses are pink.
No candy this year,
but my card's at this link:
http://picconf.org/vday

Thank you for everything you do! Happy Valentine's Day!

Sincerely

(your name here)

8< ---------- cut here ---------- >8

(Please pass this on to all your friends in New Jersey!)

This campaign is brought to you by EverythingSysadmin.com and LOPSA-NJ (picconf.org).

New Jersey Sysadmin conference announced!

New Jersey (and nearby) sysadmins, network engineers, DBAs, and anyone that considers themselves part of the "IT industry" should check out the LOPSA New Jersey Professional IT Community Conference.

The conference will be Fri/Sat, May 7-8, 2010 in sunny New Brunswick, NJ. I'll be speaking both days.

LOPSA-NJ is excited to announce a conference for the IT and system administration community in New Jersey and surrounding area!

Two days of speakers, panels, training, and "unconference". For technical people, by technical people.

LOPSA-New Jersey Professional IT Community Conference (PICC) Fri/Sat May 7-8, 2010, Hyatt Regency, in sunny New Brunswick, NJ.

http://www.picconf.org

The long version:

PICC10 is a gathering of professionals from the diverse IT (computer and network administration) community in New Jersey to learn, share ideas, and network. The conference includes invited speakers and keynotes, training by top-notch experts that is relevant, useful, and recession-friendly; plus an "unconference" track where attendees propose and host their own topics during the event. We expect attendance of 100 to 150 IT professionals from mid- to large-sized companies and academia from New Jersey/New York/Pennsylvania. We go by many titles but everyone is invited: system administrators, network administrators, network engineers, Windows, Linux, Unix, DBAs, and so on. YOU are invited!

Awesome! Keep me informed!

• Twitters: http://www.twitter.com/picconf
• Facebook: http://tinyurl.com/fb-picc
• Emails: http://lopsanj.org/mailman/listinfo/picc2010-announce

Read the full "Call for Participation":

http://lopsanj.org/events/picc10/cfp (Yes, we need you to propose conference topics)

Sponsors needed! Reach a motivated, highly-targeted demographic:

http://lopsanj.org/events/picc10/sponsors

LOPSA is vendor-independant and non-profit. This conference is too.

See you at MacWorld!

What should you write your LISA paper about?

Status: draft

The Usenix LISA 2010 "Call for Participation" is out. I encourage everyone to think about what they're doing to improve system administration, what innovation they've brought to their network, and write a paper about it.

People often ask me for a definition of "system administrator". TPOSANA/2ed has a great definition in the preface (read it and see).

But lately I've been thinking that one way to define it is in terms of lifecycle. I think of sysadmins as being the people that are responsible for technology from cradle-to-grave. Developers might create it. Executives might pay for it. Customers might request it. But we facilitate the cradle-to-grave process:

• acquisition
• deployment
• maintenance
• repair
• scaling
• decommission

(Hmm... am I missing a phase?)

Each of these phases can be a big deal and can be done well or badly.

Most sysadmins are mired in the operational aspects (keeping it running) and don't even realize the other phases exist. Consider the (usually Windows) desktop support person at most companies: the OS comes loaded from the vendor, is never changed, just maintained until the machine is thrown away. We (they) get a reputation for being janitors, not engineers.

The biggest innovations come from focusing on the other parts: getting customer requirements as part of acquisition; deploying things very well (especially desktops or things that have opportunity for mass-production cloning, etc.), scaling (growing from 100 machines , 10,000 machines, 1 million machines; or from hundreds to thousands to millions of users), decommissioning: securely destroying information, and so on.

You can optimize these phases individually or look for cross-phase improvements. If you have optimized all of these phases the only thing left is release engineering and scaling.

Surrounding these issues is all the soft skills that relate to "professionalism": how you deal with people, manage your time, etc.

So what are you going to write your LISA paper about? Which one of these phases have you improved? Did you write a new tool? Develop a new technique? Start an open-source project? Or maybe you have perfected a cross-phase methodology?

Alternatively, maybe you've noticed that there is a thing that sysadmins do, and you can study many groups of sysadmins doing that thing, and can draw comparisons and conclusions about what worked best.

Or maybe you're facing a problem that nobody else has faced, and it is worthwhile to publish your results. Maybe you've scaled Apache to more users than anyone else has, and learned something useful. Maybe you've broken from tradition "on a hunch" and your new disk backup software works better for a particularly common edge-case (Just kidding... I'm sick of papers about backup systems.) Heck, maybe you're just the first sysadmin to deploy a new technology (10G ethernet to the desktop?) and learned something about managing it that the engineers that invented it would have never expected.

What is your sysadmin team most proud of? What thing are you doing that others think is "futuristic" or "cutting edge"?

William Gibson famously wrote, "The future is here. It's just not evenly distributed yet." I suspect that most people reading this blog live in the future and by writing or speaking about what we're doing, we can spread it to others.

That's the goal of LISA, isn't it?

Read the "Call for Participation" here.

Usenix LISA 2010 Call For Participation

Just moments ago Usenix has published their "Call for Participation" for the Usenix LISA 2010 conference. This is a conference that I attend every year because the value I get out of it is undeniable. The speakers are excellent and the topics make me feel like I have access to a crystal ball that lets me see into the future.

The CfP gives a more detailed explanation of the conference and the kinds of talks, papers, and presentations that they are looking for. This is a community conference; talks come from people in the community, not "top down" vendor presentations.

This year adds a new "Practice and Experience" section where people can give a 20 minute talk where people can explain "substantial system administration project that has been completed." Sounds like a great way to learn from other people's mistakes [After I typed that I thought people might think I was kidding or being cynical. Actually, hearing what speedbumps to watch out is pretty darn important!.]

Usenix LISA is unique in that they have a track of refereed papers. These high-quality papers are where some of the biggest system administration innovations have first been published. This year the committee is not requiring full papers, but instead requests 500-1500 word summary. If your paper is accepted you will be expected to produce the entire paper in time for publication. This lowers the barrier to entry and I hope to see a big increase in paper submissions this year (I'm on the panel that votes on papers). If you have done something fantastic, invented a new technique, or written new software that improves the state of the art for system administration, please submit! (Private email to me is fine if you want to ask for advice). Details about submitting papers is here.

Whether you are planning on submitting a proposal or not, reading the full CfP is a great way to understand how a conference like LISA works. When you are presenting or not, I hope to see you there!

Read the entire call for participation here: http://www.usenix.org/events/lisa10/cfp/

Thanks to everyone that attended my talks at LISA 2009!

Thanks to everyone that attended my tutorials.  They were the #1 and #3 more attended tutorials, topping out at nearly 80 people each. The BOFs I held had packed rooms too. Thanks for your attention, I hope you walked away smarter and feeling better about your time management and sysadmin skills.

Tom @ LISA '09, Nov 1-6, 2009, Baltimore, MD

Tom will be very busy at LISA 2009 both teaching and hosting various events:

• Half-day Tutorial: "Time Management for System Administrators: A New Approach"
• Half-day Tutorial: "Design Patterns for System Administrators"
• Guru Session: Job Hunting (with Andy Lester from http://theworkinggeek.com, author of the new book Land The Tech Job You Love
• BoFs: I'll be hosting 3 BoFs. A 2-hour BoF on "Time Management", a BoF for the open-source project called Ganeti. And I'll be MC of this year's LGBT and Allies BoF.
• I'll be at the Google Vendor BoF on Thursday night to answer questions about what it is like to work at Google.
• Book signing: I hope to arrange a Book Signing during the vendor show.

With all that happening, I've also set a goal for myself to read all the papers before the conference. I promise myself that I'll do this every year, but this time I'm actually setting aside time to do the reading. Let's see how that goes. LISA is rarely in the Washington D.C. area. This gives locals an opportunity to attend LISA without having to pay for travel (I'll be taking Amtrak!). Hopefully this means there will be a lot of new faces. Hope to see you there! Registration is open.

Laptop safety: Disable sshd on your laptop before conferences

I know you don't have sshd enabled on your laptop. Heck, I bet you have everything disabled just as the corporate security policy (or just your general security paranoia) dictates.

My roadmap for LISA

I've used Google Calendar to make a list of what I plan on attending at LISA.  In past years I would spend the breaks trying to decide what to do next. This year I've precompiled my decisions so I can spend the break socializing. You can view my decisions here.

My general plan:

• Attend the sessions I'm involved with
• Attend papers more than Invited Talks
• Go to the LOPSA hospitality party at night when I'm not busy
• Go to CHIMIT 2009 in Baltimore when LISA is over

What's your plan?

Google Vendor BoF Thursday night

I am MC of the "Google Vendor BoF" at LISA09.

Here's how the planning meeting went:

"Ice cream?"
"Check."
"Raffle prizes?"
"Check."
"Beer?"
"Check."
"Ok, we're ready!"

See you there! Thursday, November 5, 9:00 p.m.-11:00 p.m., Essex A, B, & C.

NJ and VA: Remember to vote!

A message to the US readers of this blog.

This year's Usenix LISA conference conflicts with election day. I want to encourage everyone attending to vote absentee, unless of course, you can be home to vote on Tuesday.

There are no federal contests this year. Thus, the conflict does not affect most people. Of course, there are many local contests, and voting in those is very important. However, two states elect governors "out of cycle" with the rest of the nation. Both New Jersey and Virginia's governor races are neck-and-neck. Your vote is very important.

Both states permit registered votes to vote "absentee" (i.e. by mail) if you will be out of state that day. Actually, they permit you to vote absentee for just about any reason. Here are the forms you need:

• Vote Absentee in Virgnia
• Vote Absentee in New Jersey
• Apply to vote Absentee in any state

Applications must be sent by postal mail. Deadlines are soon. Act today!

Interview: Design Patterns for System Administrators Training at LISA 2009

Matt Simmons interviews me about "Design Patterns for System Adminsitrators".

This is a tutorial that I've never taught before. You can see it first at LISA 2009 in November.

In case you missed it, Matthew Sacks interviewed me about my other LISA tutorial. That tutorial also has a lot of new material.

Interview: Time Management for System Administrators Training at LISA 2009

Usenix interviewed me about my Time Management tutorial at the upcoming LISA 2009 conference. It isn't too late to sign up for this class!

Interview with Thomas Limoncelli on TM4SA at LISA2009

I can't wait for LISA!

Hanging out with geeks talking about sysadmin stuff at last night's LOPSA-NJ Chapter meeting last night made me impatient for the LISA 2009 conference to start.  Why do I have to wait a month???

Ohio LinuxFest Registration extended!

Due to overwhelming demand, the committee has extended the Ohio LinuxFest registration until midnight Tuesday, September 22. Please register today if you have not done so already and are planning on coming to Ohio LinuxFest 2009. Walk-in registrations at the day of the show may be possible for the enthusiast and professional packages (OLFU), subject to space availability.

OLF is a Free and Open Source Software Conference and Expo in Columbus, Ohio, September 25-27, 2009

Conference Planning Tips

[ This is a draft, but it's good enough to publish. Please post feedback. ]

Someone recently asked me for my advice about planning a conference.

First, I want to say that conferences are really important to the open source community. Conferences build community. Conferences build community. Repeat that over and over. It is so true it can't be underestimated. From the early Usenix conferences which bolstered Unix, to MacWorld which gave the Mac developer community "a home", to modern Linux and other conferences: If you want to build a community, have an annual conference. A well-run conference has the side effect of building your local community. First, the conference is great PR for your organization. Every planning meeting is an excuse to tell the world that you exist. Second, a conference is "special" and that brings out people that might normally ignore a monthly meeting. Third, if you do the planning right and spread the work around, you will find volunteers come out of the woodwork. The person that was intimidated to run an entire conference is certainly willing to do a small task like reaching out to a potential speaker or coordinating the catering vendor that provides lunch. That grooms people for bigger jobs. By the end of your conference you will have found the next generation of leadership that your organization needs.

So here's my advice.

A successful conference is created by having a logical plan that will carry you from the start all the way to the end.

1. Work from a timeline. I have run or been involved in about a dozen conferences. The most important thing that I learned was to build a timeline. Do this in person. Get everyone in a room for a kick-off meeting. Put huge sheets of paper on the walls marked with the months or weeks leading up to the conference. Mark 'today' and 'conference' as end-points. Now discuss the various aspects of the conference and mark them on the timeline.

Suppose you start with the program: Mark the day you'll announce the "call for papers", mark when the deadline will be for submissions, mark when the program committee will have their selections done, confirmations will go out, when replies are accepted, and so on. Next let's hear from the registration committee. Mark the last day people can pre-register, mark the last day you offer a discount for early registration (which is a lie. Mark 1 week early, but use that day to announce "the discount is extended 1 week!" and have the real deadline 1 week later). PR is important. How often will you send out press releases? Monthly? Mark the first of each month. Magazines have a 4-month lead time, and people make travel plans 6 months in advance: Mark 10 months early that you will have contacted magazines. Keep doing this with every committee... even if you don't have your committees set up. Most of it will be guesses. That's ok. A lot of it will need to be coordinated: The moment you have the keynote speaker, send a press release. People don't register for a conference until they know the entire (draft) program, so be fore that your have a (draft) program early, and that registration deadlines are coordinated around that. etc. etc. Mark the days of the planning conference calls (once per month, then once per week when you are closer to the event).

If you can get all this into a timeline in a single day-long meeting the rest of the story "just writes itself". Each week/month have a conference call where you figure out who is going to do the tasks in the next week/month of the timeline. It lets volunteers feel like no task is too intimidating (they only look a week/month ahead), and leaders don't micromange because they are setting goals. It keeps everyone "on the same page" and removes a lot of the chaos. (Note: If you want to encourage new volunteers, have the meetings in person and announce them publicly. Newbies are intimidated by phone calls.)

2. End the "timeline meeting" with a commitment. The next thing I recommend also takes place at the "timeline" meeting. At the end of the meeting, gather everyone in a circle and ask them all to commit to making sure the "timeline" happens. Go around the circle and have each person saying that they are agreeing to this commitment. I know it sounds really hippy-dippy, but the conferences that I've done this have been the ones where all the volunteers stayed on to the end. The ones were I chickened out and didn't do this were the ones that everyone was fighting, volunteers dropped out, and by the conference date 1-2 people were doing all the work (and hated it). I call this my "good luck ritual". There is magic in this ritual.

3. Get a signed contract for the location as soon as you can. Most volunteers won't volunteer until the date/place are certain. They don't want to put effort into something that they can't attend.

The few volunteers that you do get before you've booked a place... use them to find and book a place.

Once you've booked a place, have the timeline meeting.

4. One more thing... Here are some things to add to the timeline because they are often forgotten.

• Before the conference starts, write "thank you" notes to all the speakers. (Send them 1 day after the event).
• Send rejection letters AND confirmation letters. Many times conferences forget to send rejection letters. Don't leave people hanging.
• Have the "post-conference" party immediately after the conference is done. Don't wait a day... people won't be around for it. Use the party to thank everyone, give them space to relax and celebrate, and make notes for next year. Rather than a formal "what will we improve next year" meeting, just put a big sheet of paper on the wall and let people write. Have another sheet for people to list "Great things about this year's conference!"

Setting up a timeline early on gives the entire process structure. Structure is comforting to volunteers that may be otherwise unsure and, quite possibly, scared. Scared, confused, people are more likely to be stressed, get into arguments, and drop out. Giving people some structure, but not so much as to be micro-managing them, helps build their confidence, makes sure that everyone understands their job and the jobs of everyone else, and that all helps people work better together.

Have a great conference!

My tutorials at LISA 2009

As I mentioned previously, I'll be presenting two tutorials at LISA 2009. Both are new.

The one on Time Management is a total re-write. That's why it is subtitled "a new approach". I've been teaching time management to system administrators for long enough that I've discovered that what people really need is a new way to think about their entire day. By thinking about their day ahead of schedule we can make adjustments to how we operate that day. The result is more satisfaction at the end of the day. People that have taken my class before should find it interesting and new; plus a good refresher on things they may have forgotten, or wasn't relevant until the more basic stuff had "sunk in".

The other class is totally new: Design Patterns for System Administrators . A design pattern is "a general reusable solution to a commonly occurring problem." That is, this class is going to be all the rules of thumb and tips that I find I get asked about, plus a lot of tips I wish people would ask me about! (Yes, there will be rants!)

I'm taking a break from working on my slides to post this. I should get back to work!

'Gorillas in the Mist' or 'Sysadmins at the Keyboard'?

Dear fellow sysadmins,

The surest sign that sysadmins are mis-understood is how difficult it is to install, debug, or maintain various products.  Any sysadmin can tell if the installation process was designed as an afterthought. Any sysadmin can point to a variety of... I'll be polite and say... "design decisions" that make a product completely and utterly impossible to debug.

I've talked with product managers about why their product is the speedbump that slows me down when debugging a problem that is buried in a network of 150 devices from 15 different companies.  In the old days I was told, "that's why you should buy everything from one vendor... us!" and in today's multi-platform arena I'm told, "but our goal is to make our product so easy to use it you don't need to debug it."

I'm sure that last sentence made you cringe.   You get it.

I've explained how GUIs are bad when they prevent the basic principles of system administration: change management, automated auditing, backups, and unfettered debugging.  We have practices and methodologies we need to implement!  Don't get in our way!

The more enlightened product managers understand that the easier it is to automate the installation of their product, the easier it is for me to buy a lot of their product.  The more enlightened product managers understand that an ASCII configuration file can be checked in to SubVersion, audited by a Perl script, or even generated automagically from a Makefile.  Sadly, those product managers are rare.

One would think that companies would be investing millions of dollars in research to make sure their products are beloved by sysadmins.

I like to think that somewhere out there is a group of researchers studying this kind of thing. I imagine that they find sysadmins that volunteer to be videotaped as they do their job.  I imagine the researchers (or their graduate students) pouring over those tapes as they try to understand our strange ways. I imagine Dian Fossey studying not Gorillas in the Mist but Sysadmins at the Keyboard.

These researchers do exist.

I've seen them.

For the last two years they've met and exchanged ideas at a conference called CHMIT.

Some of them actually video tape sysadmins and examine what is it about products that make our lives good and !good.

My favorite moment was watching a researcher describing their observation of a sysadmin the heat of a real outage.  The sysadmin closed the firewall's GUI and connected to the command line interface in two different windows.  In one they kept repeating a command to output some debugging information. In the other they typed commands to fix the problems. This was something the GUI would never had let him do without risking carpel tunnel syndrome.  The researcher beamed as he explained the paradigm we were witnessing.  He sounded like he had been lucky enough to catch the Loch Ness Monster on film but what he had captured was something more valuable: photographic evidence of why sysadmins love command lines!

The person sitting next to me sighed and said, "Oh my god.  Is that why nobody uses the GUI we spend millions to develop?"
 
I love this conference.

These researchers study people like me and it makes the world a better place.

More than researchers attend.  Sysadmins make up a large part of the audience.
 
This year CHMIT 2009 will be in Baltimore, MD the days following LISA 2009 which by amazing coincidence is also in Baltimore, MD.

Will you be there?  I know I will.

Mark November 7-9, 2009 on your calendar.  Registration opens soon.  Papers can be submitted now. www.chimit09.org

Tom Limoncelli

Final slide from my LCA 2009 talk

Off to linux.conf.au 2009 soon

I'm spending this weekend packing for linux.conf.au 2009. I'll be the opening speaker. Hope to see you there!

LOPSA classes at SCALE 7x

The Southern California Linux Expo (SCALE) will include classes by LOPSA instructors.  For more information, check out the LOPSA press release, or the SCALE page.

SCALE 7x is the 7th SCALE conference.

Tom @ Ohio LinuxFest 2008, Columbus, Ohio, October 10-11, 2008

Tom will be teaching two half-day tutorials: "Time Management for System Administrators" and "Interviewing and Hiring System Administrators". This is a rare opportunity to see these talks presented in the Ohio area. Register soon!

With the economy in a down-turn, Time Management is key to being efficient at what you do. With people's hiring budgets being slashed, it is important that the people you do hire are top notch. Both of these tutorials are intended for both the new and experienced system administrator or IT manager.

The sixth annual Ohio LinuxFest will be held on October 10-11, 2008 at the Greater Columbus Convention Center in downtown Columbus, Ohio. Hosting authoritative speakers and a large expo, the Ohio LinuxFest welcomes Free and Open Source Software professionals, enthusiasts, and anyone who wants to take part in the event. The Ohio LinuxFest is a free, grassroots conference for the Linux/Open Source Software/Free Software community

Ready for LISA 2008 in San Diego!

I've registered, I've booked my hotel. Are you going to LISA 2008? On Thursday I will be doing a 90-minute open Q&A session about Time Management. Feel free to stop by and ask me anything. On Friday I will be presenting my newest talk titled, "System Administration and The Economics of Plenty". When we start to see how plentiful the world is, we think about our roles as system administrators differently. It affects everything from how we set policy to how we do our jobs. Register online today! I hope to see you there!

Looking forward to SAGE-AU (Australia)

As I start to pack for my trip to Australia, I'm getting really excited about attending the SAGE-AU 2008 conference.  I just got email with the preliminary registration numbers for my workshops (note: more people will sign up "day of"):

• Interviewing/Hiring Technical People: 8 registrations
• Time Management for System Administrators: 31 registrations
• "Help!  Everyone hates our IT Department!": 24 registrations

I interpret this to mean a few things: 1) not a lot of people plan on hiring in the next year or not a lot of people are coming that early to the conference. 2) a lot of people want help with their time management.  3) there is no shortage of people concerned about fixing their IT group.

By the way... I've completely revamped the Time Management slides.  Previously I've tweaked them between conferences.  This time I took a hard look at what people found useful, didn't need, and the way I was presenting the information.  I refocused the slides around the more streamlined version of "the cycle", dropped the parts that got the most yawns or "why would anyone want that?"-looks.  I also made sure that the more entertaining parts were retained and are spread evenly throughout the presentation.  As I learned from mjd's "Presentation Judo" talk says, "Your primary goal should be to entertain".  Each slide should educate and entertain, but if you have to pick just one, choose to entertain: people are sitting in the same seat for 3 hours, they deserve nothing less.

Now I have to get back to packing.  For the last few months any time I've thought of something to bring I wrote it on my todo list for August 6th.  Packing is less stressful when you have confidence that you won't forget anything.  Now I just have to fit it all in my luggage!

It's not too late to register for SAGE-AU 2008.  I look forward to seeing you there!

Tom + Strata @ LISA '06 in Wash D.C., Dec 3-8, 2006

Tom and Strata be teaching and speaking at LISA 2006 in Washington D.C., Dec 3-9, 2006. This is one of our favorite conferences of the year because it is so dam useful. Get your boss to send ya. This year it is in Washington D.C., which makes it easy to get to for all the east-coasters that usually don't get around.

Tom will be speaking/teaching:

Mon	9am-5pm	Workshop	Managing Sysadmins (co-facilitator)
Wed	2pm-3:30	Invited Talk	Site Reliability at Google/My First Year at Google
Thu	AM	Tutorial	Time Management: Getting It All Done and Not Going (More) Crazy!
Thu	12:30pm-1:30pm	Exhibition	"Meet the Authors" at Reiter's Conference Bookstore
Thu	2pm-3:30	Guru Talk	How to Get Your Paper Accepted at LISA
Thu	4pm-5:40	Guru Talk	Time Management for System Administrators
Fri	11am-12:30	Hit The Ground Running	Mac OS X

Strata Rose Chalup will be speaking/teaching:

Mon	PM	Tutorial	Project Troubleshooting
Wed	PM	Tutorial	Problem-Solving for IT Professionals
Thu	AM	Tutorial	Practical Project Management for Sysadmins and IT Professionals
Wed	9pm-10pm	BOF	Sysadmin Education

In addition, we will be hanging out in what is known as "the hallway track". In fact, if you haven't attended LISA before, you should know that a lot of the educational value is the people you meet. Tom says, "Early in my career a lot of what I learned was from the conversations in the hallway."

• More info about LISA 2006, Washington D.C., Dec 3-8, 2006
• Register for LISA 2006

  Coming soon: Book signing and other events.

  Posted by Tom Limoncelli at December 3, 2006 10:15 AM | Comments (0) | TrackBack

Why I attend the LISA conference

One-day workshops and training programs serve a different purpose from week-long conferences. One-day seminars tend to be tactical: focused on a particular technology or skill (TCP/IP, Storage, Backups, etc). Week-long conferences are strategic: offering opportunities to discuss broader topics, to network, to build community, and to further the craft of SA as a respected profession. Week-long conferences give you "vision".

LISA is one of those week-long conferences.

My favorite part of LISA is what I call "the moment." There's always this moment when I realize that someone has just said something that makes me want to shout, "Oh Damn! That just paid for the entire conference!" There are many little "ah-ha!" moments too, but there's always one big one. Some years there are two or three.

LISA is very cutting-edge. Many times I've seen a new tool at LISA that only became popular years later. I had been using it all along. People wonder where I find out about these things, the answer is usually "LISA"! That's really helped me stay ahead of the pack.

Week-long conferences have a powerful effect, providing a much-needed opportunity to relax, and they provide a supportive environment where you can take a step back from your day-to-day work and consider the big picture. Attendees return to their job brimming with new ideas and vision; refreshed, motivated, and with a new outlook.

This year's conference is on the east-coast in lovely Washington D.C. I hope to see you there!

---

LISA '06, Dec 3-8, in Washington D.C., download this year's flyer (PDF) or go to the LISA'06 home page

Happy Birthday, LOPSA!

Happy Birthday, LOPSA!

You are one year old and look how far you've come! Like most births you were born amid a lot of shouting and confusion, but look how far you've grown! You've formed the organization, build a web site, and had your first regional conference. Congrats! Now you are truly defining yourself, growing up, and becoming your own person.

For those of you that don't know, LOPSA is the League of Professional System Administrators. The goal is to become like the AMA is to doctors, or the APA is to shrinks. That is, work on building the professionalism of our community. If you aren't a member, I highly recommend that you join. Heck, it's free to just register.

Two weeks ago I attended the first LOPSA regional conference in Phoenix, Arizona. I taught a full-day version of my Time Management for System Administrators class. What impressed me about this event was how different it was. Because it was regional most of the speakers were local. There are experts everywhere (not just in California) and seeing them get some spotlight really made me happy. The fact that it was small also meant that it could be at a less expensive hotel, who was more hungry for LOPSA's business. They had a lot of creative ideas that I haven't seen at big hotels. For example, one of the snack-breaks had cookies and milk! I was psyched!

At night we had a lot of deep discussions about the future of system administration, professionalism, and the future of LOPSA. I consulted with some board members about how to get to the next milestone now that the organization is running. I hope to see more regional conferences announced soon. I also brainstormed on ways to reach out to the segments of the IT world that are currently unaddressed.

Why not celebrate the 1st birthday by buying a gift for yourself? The LOPSA CafePress store is ready to fulfill your need for swag, and raises money for a good cause. And if you haven't registered, do that too. They have some extremely useful mailing lists.

LOPSA announces "SysAdmin Days" Conference

LOPSA is proud to present our first in-person training workshop: SysAdmin Days this November in Phoenix, AZ.

SysAdmin Days is a two-day training workshop for system administrators to focus on all the aspects of their professional development, both technical and otherwise. Workshops are available for both specific platforms and technologies and general system administration practices, including Mac OS X Administration, PHP, and Perl. We'll also have sessions about topics important for any sysadmin, including Drafting Policy Documents, Communications Skills, and Time Management.

Instructors include Tom Limoncelli, co-author of The Practice of System and Network Administration and Time Management for System Administrators.

More info is here: http://lopsa.org/SysAdminDays-Pheonix

LISA '06 Call For Papers

Usenix/SAGE has announced their call-for-papers for LISA 2006.

This is the 20th Large Installation System Administration Conference. Has it really been 20 years? Wow, how time flies. This year's conference will be December 3-8, 2006 in Washington, D.C. (The deadline for paper submission is May 23, 2006).

I wrote Bill LeFebvre and told him, "Ah heck, don't do the whole 'call for papers' thing this year! You're a smart guy! Write all the papers yourself! You gots tons of ideas."

But no, he wouldn't listen to me.

He said, "Tom, that's not how the conference works. We collect papers from all over the world that real sysadmins submit. These papers describe solutions and innovations from everything from video processing to new security issues to better ways to run helpdesks. So we collect them and read them all and pick the absolute very best. Just those papers are accepted and presented at the conference." (Footnote: By now I hope you realize this conversation is fictional.)

"Bah!" I replied. "That sounds like a lot of work! The conference has been around for 20 years! Why not just pick 20 past papers and reprint them. Nobody will know the difference."

"Tom, that'd be looking at the past." Bill rebuffed. "While it's always good to remember our history, LISA is about innovation. We look for papers that are forward-looking. Attendees come back from our conference saying, 'Wow! I just saw the future! I'm going to look like a freakin' genius with the new 'vision' I have for our little IT group." (Footnote: Bill would never use the term "freakin'")

"Oh well, I guess you're right." I said. That's a much better idea!

(insert "ABC After School Special" theme song)

And that's how I learned all about the importance of submitting papers to LISA.

(Disclaimer: Bill has no idea I wrote this. Click below to read the actual "Call for Papers")

--------------------------------------------------------------------
Call for Papers
LISA '06: 20th Large Installation System Administration Conference
December 3-8, 2006, Washington, D.C., USA
http://www.usenix.org/lisa06/cfpa
Extended Abstract and Paper Submissions Deadline: May 23, 2006
Sponsored by USENIX and SAGE
--------------------------------------------------------------------

Dear Colleague

The LISA '06 organizers invite you to contribute proposals for refereed papers, invited talks, and workshops, plus any ideas you have for Guru Is In sessions, Work-in-Progress reports, and training sessions.

The Call for Participation with submission guidelines and sample topics can be found on the USENIX Web site at http://www.usenix.org/lisa06/cfpa

The annual LISA conference is the meeting place of choice for system, network, security, and other computing administrators. Administrators of all specialties and levels of expertise meet at LISA to exchange ideas, sharpen skills, learn new techniques, debate current issues, and meet colleagues and friends.

People representing every work assignment from the full-time position at a large site to the part-time one at a small shop come to LISA from over 30 countries, bringing a variety of backgrounds and experience levels to the conference dedicated to them. System and network administrators from environments as diverse as academia, large corporations, small businesses, government organizations, and research sites find LISA to be the place to go for training, education, networking, and interacting with their peers.

The conference's diverse group of participants is matched by an equally broad spectrum of activities:

* Training sessions for both beginners and experienced attendees cover many administrative topics ranging from basic administrative procedures to using cutting-edge technologies.

* Refereed papers present the latest developments and ideas related to system and network administration.

* Invited talks and panels discuss important and timely topics and often spark lively debates and conversation.

* Work-in-progress reports (WiPs) provide brief peeks at next year's innovations.

GET INVOLVED!
* Submit a draft paper or extended abstract proposal for a refereed paper.
* Suggest an invited talk speaker.
* Share your experience by leading a Guru Is In session.
* Propose a training session topic.
* Organize or suggest a Birds-of-a-Feather (BoF) session.
* Email an idea to the chair: lisa06ideas@usenix.org

------------------------------------------------------------
IMPORTANT DATES
Extended Abstract and Paper Submissions Deadline: May 23, 2006
Invited Talks proposals due: June 1, 2006
Notification to authors: July 12 2006
Final papers due: September 12, 2006
Submission guidelines and more information can be found at
http://www.usenix.org/lisa06/cfpa
Sponsored by USENIX and SAGE
-------------------------------------------------------------

We look forward to hearing from you!

On behalf of the LISA '06 Program Committee,

William LeFebvre
lisa06chair@usenix.org

LISA Blog

David Blank-Edelman, chair of LISA2005 has created a blog so you can follow what it takes to create a conference. Check out LISA Conference Blog

LISA 2004 Report: Tuesday

Today I attended the Advanced Topics Workshop. This is a full day workshop for old timers to talk about system administration from a very high level. More and more it is becoming a management thing, because as the old timers get older, we get moved into management.

Advanced Topics Workshop -- This one-day workshop, intended for very senior administrators, provides an informal roundtable discussion of the problems facing system administrators today. Attendance is limited to 30 and based on acceptance of a position paper.
Started by John Schimmel in 1995 (?) as a "formalized hallway track" for "all the really experienced folks who show up on Tuesday before the conference and just hang around because there are no suitable tutorials," this workshop gives senior administrators a chance to meet and talk with their peers. Topics can range from difficulties hiring sysadmins to current technical problems, from managing sysadmins to exchanging notes on the latest new and interesting tools.

"An imperfect, light-weight system that people use is better than a heavy-weight system that nobody uses."
Examples:

• TWiki vs. SharePoint
• RT vs. Remedy
• SubVersion+Bugzilla vs. ClearCase
• ...

STOP THE PRESSES! I passed by two people talking about LiveJournal and when I looked at their badges I realized it was Brad (founder of LiveJournal) and Lisa (sysadmin of LiveJournal). They were prepping for Wednesday's presentation about the database/web/etc. technology of LiveJournal. I ended up talking with them for an hour or so (until about 1am). Steven walked by and joined in and we both told them a bunch of stories about how LJ has changed our lives. This was their first LISA, so we also talked about what they could expect.

That's my favorite thing about LISA and all the Usenix conferences. You meet really famous people. Well, famous in the Unix and Internet sense.

Wednesday's talk about LiveJournal was very impressive and an interesting discussion was posted on LiveJournal while they were talking about Livejournal

And the rest of the Advanced Topics Workshop? Who cares! I met Brad and Lisa!

LISA 2004 Report: Monday

Today I spent the morning working on a presentation I'm making on Friday.

In the afternoon I attended Over the Edge System Administration, Volume 1 taught by David N. Blank-Edelman. The basis of the class was to each how to abuse the sysadmin tools that you already know. His first example was setting up a printqueue in lpr (the Unix print queue system) so that the files don't get sent to a printer, but get sent to your audio device. Now you can send MP3 files to this queue and they start playing one after the next. You can even use the usual lp* commands to re-arrange the play order. If you think that was an interesting abuse of a sysadmin tool, the examples only got better (or worse, depending on how you view it). It was a great tutorial. I learned a lot and was entertained. It added a lot of new tools to my mental toolbox. I hope they repeat this tutorial next year. David N. Blank-Edelman's tutorials always seem to be winners, watch this guy in the future.

Towards the end of the talk my boss contacted me by IM and asked if I could do a quick project for her while I was at the conference. I bring this up because a lot of people can't attend week-long conferences because their boss "can't live without them for a week." It is very comforting that LISA (and most conferences) have very good connectivity to the internet (I'm told four T1s this year) and excellent WiFi coverage. Not only is WiFi available in the workshop/presentation spaces, but also the "hang out" spaces where people congregate during the day and in the evenings. Yes, there is internet access in the hotel rooms ($9.99/night) but having it ubiquitously throughout the conference really makes a big difference. Tell your boss that!

LISA2004 Report: Sunday

[ I'm going to try to blog one entry for every day at LISA 2004. ]

I arrived on Sunday. Though I'm not signed up for anything until Monday, I arrived early (2pm) on Sunday.

So if I'm not signed up for anything on Sunday, why did I arrive so early? There is a lot of useful and interesting stuff going on even if you aren't attending a tutorial or workshop. The benefit of a multi-day conference is the contacts that you make and the interaction with other attendees. There were plenty of ad hoc groups of people hanging out and chatting. I've been attending LISA for years, so there were also a lot of friends that I wanted to catch up with.

Dinner is an excellent way to spend "quality time" with the people you meet. At dinner last night we discussed everything from the new Cisco security features, to Solaris ZFS, to the finer points of various other system administration topics. I learned a lot, and many new ideas sprouted in my mind (or at least there were a lot of URLs mentioned that I found I need to check out.)

How to find a group of people for dinner if you are new to LISA? Usually people gather at the registration area around 6pm or 6:30pm to figure out what they're going to do for the evening. If you see a big group gathering, feel free to ask if you can join them. The worst that can happen is they'll say "no". Don't be offended, they might have a private party already arranged. However typically people will invite you to join along.

One of my goals this year is to try to meet a lot of new people. To that end, I'm introducing myself to random people, people I usually might not have met for one reason or another. If you are a regular attendee, I encourage you to do the same. If you are a first-time attendee introduce yourself to the people at tutorials, lunches, and so on. You'll get a lot more out of the conference.

Tomorrow: Review of DBE's tutorial on "Over The Edge System Administration".

Best. Conference. Evah: LISA

Conference Chair Lee Demon (and his committee) has done a bang-up job planning this year's LISA conference. This year's conference is November 14–19, 2004 in Atlanta,GA.

If you are a system administrator, this is the place to be. This is the kind of conference that you come home from with a million great ideas. Other conferences teach, this conference gives you vision. Early Bird Registration Deadline: October 22.