Super Wi-Fi is better than just "super"
[This is still at 'first draft' quality, but I thought I'd post it sooner rather than later. Please ignore the typos for now.]
I recently twittered my delight that the FCC approval of "super Wi-Fi" is going to be regarded as a historic moment five years from now. I mean it.
Here's why:
In geek terms: This gives permission to treat the airwaves like Ethernet networking, not like Teleco networking. More modern and more flexible.
In non-geek terms, this decision by the FCC makes it easier to innovate. It makes it safe and easy to try new things With the possibility of experimentation comes new applications and ideas. It will be a game-changer.
Let me explain...
Let's first look at how spectrum is allocated today: In blocks. You want to do something "on the air", request a license, go through tons of approvals, put together a consortium of like-minded folks, wait months or years, and get a block of spectrum from frequency X to frequency Y in a particular geographic area. The process is so long that by now I've forgotten what the original idea was. Sigh.
Of course, when the FCC was created in the 1930s this made sense. We didn't know any other way and we didn't have the technology to do it any other way. Electronics were imprecise and stupid (and analog) so the best thing to do was to allocate big blocks and waste some space by putting gaps between those blocks to take into account "drift". It was centrally-controlled, graceless, but it worked. To manage a precious, rare, resource, it made sense. Did I mention this was the 1930s?
This is comparable to how telecoms traditionally have handled bandwidth. You may recall that a T1 line has 24 channels (DS0's) that are 64 kbit/s each. The total bandwidth of a T1 is 1.5 M but it is divided into 24 timeslots. If you are DS0 number 13, you know that your bits are transmitted 13 time units after each clock sync. If you don't have anything to transmit, zeros will be transmitted for you. It is wasteful and graceless but thats the best you can do in 1961 (which meant most of the design was done in the 1950s). That's nearly a decade before we landed a man on the moon and 20 years before the Commodore 64 first shipped.
Compare that kind of resource allocation scheme to Ethernet. On Ethernet every device that wants to transmit "listens" to see if anyone else it talking and as soon as there is silence "you just start sending". If two devices send at the same time it is considered a "collision" and both parties back off and re-try a random amount of time later. No central authority deciding who should talk when. Everyone just has to agree to use the same rules for how to back off when there is a problem. No central authority, just "benevolent self-interest" that requires everyone to follow the same rules: You follow the rules out of your own self-interest because if everyone does that, everyone can win: Talk when nobody else is and politely back off if you find you are interrupting someone.
It works because data protocols are done by computers that can think, look around, and retry when there is a problem. Analog electronics couldn't do that.
Also compare to how TCP/IP allocates bandwidth on the internet. Send all the data you want, ramping up to the max rate you can send. If there is congestion, the network drops your packets and you respond by slowing down. There is no attempt to allocate the perfect amount of bandwidth for you so that you don't have to deal with congestion. Your protocol follows specific rules on how to back off; and it works because everyone is following the same rules. Your protocol can try to cheat, but it is in your own self-interest to follow the rules and the rules are simple: Talk and and politely back off when there is an indication the system is overloaded.
Imagine if the telecom world or other old-school thinkers had tried to invent a data-networking system based on their old antiquated values? Every time you wanted to SSH to a host, first your protocol would contact some great authority in the sky, beg for an allocation of bandwidth, promise not to go outside that allocation, receive that allocation. With that allocation you would then start transmitting, always careful not to send more than you had promised. Once you were done you would notify the great authority in the sky and the bandwidth would be freed for use by others. But what if there wasn't bandwidth available? During the allocation request you would be given a busy signal (or sent to voicemail?) so you knew to try again later. At the end of the month you'd get a "phone bill" that would list every connection you've made with a dollar amount to be paid.
Now, obviously that's a silly way to run a network. Nobody would create a network like that... oh wait... someone did!
Do you think the telecom industry learned from experienced data network inventors? Heck no. In fact, the telecom industry's response to the internet and TCP/IP was ATM (the confusingly named "Asynchronous Transfer Mode") which was based on sending data in timeslots or "cells" that are 53 bytes each. (That's not a typo... yes, the packet size is a prime number!). The first stage of each session was an allocation process. You (your software) would talk to your nearest router and explain how much bandwidth you needed and for how long. It would negotiate on your behalf with every router between you and the destination to allocate your specified amount of bandwidth. That bandwidth would be allocated just to you. (more on that later) You then have this "virtual circuit" that you transmit your data on until you are done and then the routers de-allocate that bandwidth allotment.
Laughable? Yes. But from 1988 to 1995ish the telecom industry tried to "take over" the internet and force it to be replaced with ATM. Imagine every SSH session taking 0.5 to 2 seconds to start up as bandwidth was allocated to you. Of course, that didn't work well therefore the ATM Forum proposed that you allocate yourself a big chunk and use it for all your TCP/IP needs, doing your own suballocations from that larger block. In other words, they were going to give you a T1. In fact, "T1 Emulation" was a big feature of ATM equipment.
And... oh dear. You would you get a phone bill at the end of the month, listing all the connections you made and a sum total for you to pay. Insane. In fact, one of the jokes about ATM was that the acronym was "A Tariffing Mechanism".
ATM did have one concession to the way real-world data networks operate. The channels didn't have to be fixed sizes. They could be "variable rate". Also, if you weren't using your entire allocation the network could use the "spare" bandwidth for "best effort" protocols. In fact, a large part of the research around ATM was how to oversubscribe allocations and still assure that all bandwidth guarantees would be satisfied.
Here's the part I think is the most funny. The most complex part of an ATM system is all these mechanisms serve the purpose of assuring that the endpoints see a perfect network with perfect bandwidth allocation and perfect reliability and perfect fidelity. However, at the top of the protocol stack data you (your protocol) still has to do end-to-end error checking. Even if you are promised the network can not possibly drop or corrupt a packet, the top level protocols (the applications) still check for problems because the error may have been somewhere else: the cable between the computer and the perfect network, for example. Thus ATM went through all this hand wringing on behalf of upper level protocols that didn't need it, or find much utility in it. Here is a list of things that data protocols can handle on their own: missing packets, dropping packets, corrupted packets, data being sent too fast, data not being sent fast enough. Did I say "can"? They have to. Thus, ATM's generous offer to handle all of that for you is a waste of effort on ATM's behalf. It does, however, justify the ability for the ATM provider to send you a bill. What a great business model.
The fact that ATM didn't replace the internet was no accident. It was a huge effort to "push back" against some very big heavy weights. If you recall, these were the same years that small ISPs were being bought up by Telcos. Eventually all the major ISPs were entirely owned by Telecos. The equipment companies were entering the telecom space and didn't want to piss off their new telco customers. Thus, the people that needed to fight back against ATM were now all owned by megacorps that wanted ATM to win. Wired Magazine wrote the definitive history of this battle and I encourage everyone interest in internet history and governance to read it. The people in this story are heros.
This brings us back to the recent FCC decision.
The airwaves are allocated in blocks. It is wasteful, graceless and ham-fisted but it works. And most of all, it worked given the technology of the 1930s that created it.
The new regulations permit radio transmitters to share spectrum. As long as everyone plays by the same rules it all "just works". As long as everyone has an incentive to play by the rules, it will continue to work. The rules are both "the carrot and the stick". "The stick" is FCC penalties. "The carrot" is that if everyone plays by the rules, everyone will continue to be able to play.
So here's how it works. If you want to broadcast on frequency X, you listen to see if anyone else is broadcasting. If nobody is, you start broadcasting until you detect that someone else is broadcast at which point you have to stop broadcasting. It's a lot more technical than that, but that's the premise. It is like Ethernet and TCP/IP: Talk when nobody else is and politely back off if you find you are interrupting someone.
Of course, you probably are going to listen to many frequencies: scanning up and down for free frequencies so you always have enough available to send the data you have. One of the FCC concessions is that there will be a database of frequencies that are allocated "old school style" and devices will have to stay away from those. Devices will download updates from that database periodically. The database is geographic. The entries are not "don't use channel 9" but "In New York, Channel 9 is in use".
The frequencies that are now available include the "whitespace" airwaves (channels that are unused in the TV frequency range) as well as the gaps between channels that used to be needed due to "analog drift". Now that transmitters are digital they are more precise (they can stay within a more narrow frequency band) and self-correcting (no drift). Being able to use those gaps alone is a big innovation.
At last! Instead of going through tons of work to use any airwaves at all, we can simply build devices that know how to "talk when nobody else is", scan frequencies for available bandwidth, and sync up to a central database.
These are things that modern computers do very well.
None of this was possible until recently. In the 1970s a transistor radio might cost $10 and be so simple it might have come in a kit. Imagine if it had to scan frequencies and so on. With 1970s semiconductor technology it would be a million dollar product. Not something anyone could afford. Oh, and your hand-held radio would only fit in your hand if your hands were as big as the Statue of Liberty's.
Moore's law predicts the "march of progress" in semiconductors. It was easy to predict when such compute power would be affordable and therefore making it economically possible for such devices.
While Moore's law may be hitting the limits of physics, we are still benefitting from it. Ironically there are entire industries that have tried to deny its existence. The economic justification for creating ATM was based on the notion that silicon chips would never be sophisticated or powerful enough to be able to process variably-size, large packets; network speed would hit a limit if we didn't change everything to 53-byte packets. This is entirely true to anyone that is ignorant of, or denies, Moore's law. People that lobbied against "super Wi-Fi" and the use of whitespace also were ignorant of, or in denial about, Moore's law. Of course electronics could do this, it was a matter of time. The music industry was told, based on Moore's law, which year MP3 decoders would be inexpensive enough to put music on a PC, and what year it could fit on a portable player, and what year being able to download an MP3 would be economically feasible; their surprise when these things happened were either due to ignorance of, or denial about, Moore's law. The term "feigning surprise" is one way to describe how someone acts when predictions they've ignored all come true.
But I digress...
This new FCC regulation is a major step forward. It is a modernization of how we allocate wireless frequencies. It is an acknowledgement of Moore's law and the improvements digital electronics bring to the field. It is the gateway for new experimentation which will lead to new wireless applications and services.
Mark my words! Five years from now we'll look back at all the progress that has happened and point to this day as the historic moment that started it all, even though the announcement was mostly ignored at the time.
Well, ignored by everyone except you, dear reader.
Tom Limoncelli
(See you Dec 22, 2016!)
P.S. The only coverage of this FCC decision that I've been able to find has been in the foreign press. What's up with that? It's as if the U.S. incumbents are in cahoots to make sure it will be easy to feign surprise about this some day.
AT&T
AT&T's De la Vega is getting in trouble for saying that they want to find ways to discourage people from using their data plans. It turns out that AT&T's data network is overloaded and rather than fix the problem, they think punishing their users will help.
As an AT&T customer, it makes me sick.
As an ex-AT&T employee, it just reminds me of why I was so happy to leave.
This is what you get for having salespeople run the company instead of engineers. Engineers would have budgeted for appropriate growth to match customer growth.
AT&T's mindset is that bandwidth is scarce. Every bit is so impossibly costly that it must be measured, counted, monitored, and charged for. On my first day as an employee I had to watch a 30 minute video that did nothing but explain that I can't make a single personal phone call from the office; it looked like it has been made when phone calls were still $3/minute. Don't waste their precious, precious bandwidth.
Bandwidth was expensive for the first 100 years of their history, but it certainly isn't true now. What made the internet great was thinking in terms of plenty, not scarcity.
I remember when "the web" (HTTP) was new. A friend at a different division of AT&T told me their engineers were fearful of HTTP and didn't want it to catch on because their network could never handle such a graphic-rich system (this was 1992 or 1993). I couldn't figure out why they weren't thinking, "Yeah! An opportunity to sell more bandwidth!" If you sell apples, don't you want to freely distribute apple pie recipes? If you sell paint don't you want to encourage everyone to repair their house? Ugh. If AT&T was selling bacon they'd be encouraging everyone to become a vegan.
At the time UUNET (the first commercial ISP) was giving away free Usenet feeds (at this time this was a HUGE amount of bandwidth) and paying people to develop open source Usenet software: all to make it easier for people to need more bandwidth. I thought UUNET's way was much smarter.
It also annoyed me, as an employee, that AT&T kept acting as if Moore's Law didn't exist. This is odd because the Moore revealed this observation during a presentation at AT&T's Bell Labs. Maybe they have to remember that Nielsen's Law makes similar claims about bandwidth. Pushed on by cheaper electronics, bandwidth gets cheaper too.
The biggest innovations in computing have come from brashly using more resources, usually slightly ahead of the supply curve. Textual user interfaces were a "waste of CPU" when first seen by batch computing people. Graphical user interfaces were a "waste of CPU" at first, but now it is what enables billions of people to use computers. RAID was a "waste of disk" but now I would never build a server without it.
The other attitude that I saw at AT&T was sheer shock and surprise that anything changes. "What? We built this thing for our customer base and... there are more customers a year later? They want new features? How could anyone have expected that?" Combine that with an intentional ignorance of Moore's Law and you have a disaster.
A disaster called AT&T.
Yes, AT&T, you have the best selling phone. People use it for data more than voice. The data apps are what make it such a success. Why do I get the feeling that when you negotiated with Apple you thought, "Sure, we'll throw in flat-rate data plans... it isn't like anyone is going to use that stuff!"
Are you still thinking that the internet is a "fad" like CEO Robert Allen?
My AT&T/iPhone contract is over in a few months. Maybe when it ends I should help De la Vega's bandwidth problem by not using his network at all.
P.S. I have a lot of pent up anger bout my AT&T service because twice a day as I take the train from Bloomfield, NJ to New York City and back I am faced with dead-spots at key locations such as the Secaucus transfer station, Watsessing Ave, and others locations along the way. It is frustrating to be on the train and see other passengers using Verizon and T-Mobile able to talk on their phone (and I presume surf the web) at all the points that I can't. It is my twice-a-day reminder to leave AT&T that I could be doing better with a different vendor.
UK Government officially apologizes for "appalling" treatment of Alan Turing
I've long been a fan of Alan Turing, even writing a big paper about his mistreatment my freshman year of college (talking about gay stuff was much more radical in 1987. I nearly cried while giving the oral report portion of the project). For those of you that don't know, Alan Turing not only invented what we now call computer science, but broke the German code which directly led to The Allies winning World War II. One man can really change the world.Nominations open for "Sysadmin of the year"!
Read all about it! Spread the word so all your friends and co-workers know to nominate that great person that runs their systems! :-)
Follow the event on on Twitter as @SysAdRockstar09, Facebook group, and LinkedIn group. For full information on the contest visit www.bigfix.com/rockstar.
Interviews about IPv6 migration
Randy Bush makes an interesting financial point that might help you explain IPv6 to the finance people: Pay a little now or pay a lot in the future. Plus a very good point: Do a single service like making your DNS dual-stacked. You'll be more focused and you'll find where the problems are going to be.
Netflix Streaming over IPv6
Netflix has announced their streaming service is now accessible over IPv6. This means that their CDN provider, Limelight, is now the first CDN to provide IPv6 service. Netflix says it took two months of engineering (from initial idea to completion) and Limelight says they only had to allocate two engineers to the project. IPv6 is easy. Forget all your old misconceptions.At my house we have Comcast for our internet access. Now I just need them to provide it and I'm ready! If Comcast needs a beta tester, please reach me! tal at everything sysadmin dot com, folks!
Warehouse-Scale Machines: The Datacenter as a Computer
The term "Warehouse-Scale" Machines has been coined. The term describes the specific design that sites like Google use. The data centers that Google runs aren't like other data centers where each rack has a mish-mosh of machines that result as various people request and fill rack space. It's more like a single huge machine running many processes. A machine has memory, CPUs, and storage and buses that connect them all. A warehouse-scale machine has thousands of machines all with a few, specific, configurations. You treat the machines as CPUs and/or storage; the network is the bus that connects them all.There is a new on-line book (108 pages!) by the people at Google that are in charge of the Google data center operations (disclaimer: Urz is my boss's boss's boss's boss's boss)
by Luiz André Barroso and Urs Hölzle, Google Inc.
Abstract
As computation continues to move into the cloud, the computing platform of interest no longer resembles a pizza box or a refrigerator, but a warehouse full of computers. These new large datacenters are quite different from traditional hosting facilities of earlier times and cannot be viewed simply as a collection of co-located servers. Large portions of the hardware and software resources in these facilities must work in concert to efficiently deliver good levels of Internet service performance, something that can only be achieved by a holistic approach to their design and deployment. In other words, we must treat the datacenter itself as one massive warehouse-scale computer (WSC). We describe the architecture of WSCs, the main factors influencing their design, operation, and cost structure, and the characteristics of their software base. We hope it will be useful to architects and programmers of today's WSCs, as well as those of future many-core platforms which may one day implement the equivalent of today's WSCs on a single board.
http://www.morganclaypool.com/toc/cac/4/1
Oracle buys Sun. Good idea for everything except databases
According to Merrill R. (Rick) Chapman's book, there is an oft repeated pattern in the computer industry where a company suddenly finds itself with two products in the same market space, and ends up not being able to sell either. They spend all their time trying to explain to customers why they should buy one or the other, when really the truth is that they are too similar to differentiate. Meanwhile a competitor (usually Microsoft) comes in with one product, a clear message ("it's the best!") and puts the other company out of business. If the other company had sold off or canceled one of its two similar products the disaster would have been avoided.
I consider that book the best book on how the major players in the software industry got to where they are today. When it came out it got hardly any press. Hardly anyone has heard of it. I think that's sad. It is a "best kept secret" book. It is written by a person that was "there when it happened" and he tells the stories in excellent detail. Each chapter teaches you something important. Oh, and most of his case studies involve companies that were beaten by Microsoft. If you don't want history to repeat itself, read this book.
If I was Oracle, I'd sell off MySQL and PostgreSQL right away.
Google enables IPv6 for most services (but there is a catch!)
Google has enabled IPv6 for most services but ISPs have to contact them and verify that their IPv6 is working properly before their users can take advantage of this.I'm writing about this to spread the word. Many readers of this blog work at ISPs and hopefully many of them have IPv6 rolled out, or are in the process of doing so.
Technically here's what happens: Currently DNS lookups of www.google.com return A records (IPv4), and no AAAA records (IPv6). If you run an ISP that has rolled out IPv6, Google will add you (your DNS servers, actually) to a white-list used to control Google's DNS servers. After that, DNS queries of www.google.com will return both an A and AAAA record(s).
What's the catch? The catch is that they are enabling it on a per-ISP basis. So, you need to badger your ISP about this.
Why not just enable it for all ISPs? There are some OSs that have default configurations that get confused if they see an AAAA record yet don't have full IPv6 connectivity. In particular, if you have IPv6 enabled at your house, but your ISP doesn't support IPv6, there is a good chance that your computer isn't smart enough to know that having local IPv6 isn't the same as IPv6 connectivity all the way across the internet. Thus, it will send out requests over IPv6 which will stall as the packets get dropped by the first non-IPv6 router (your ISP).
Thus, it is safer to just send AAAA records if you are on an ISP that really supports IPv6. Eventually this kind of thing won't be needed, but for now it is a "better safe than sorry" measure. Hopefully if a few big sites do this then the internet will become "safe" for IPv6 and everyone else won't need to take such measures.
If none of this makes sense to you, don't worry. It is really more important that your ISP understands. Though, as a system administrator it is a good idea to get up to speed on the issues. I can recommend 2 great books:
- My friend Karl Siil recently released a book specifically on the topic of IPv6 transition issues for government enterprises, yet the on-line preview is brilliantly useful for enterprise and other sysadmins looking to get an introduction to the protocol, the history, and other issues: IPv6 Mandates: Choosing a Transition Strategy, Preparing Transition Plans, and Executing the Migration of a Network to IPv6, by Karl Siil. (Buy Online)
- The best practical sysadmin-oriented book on IPv6 is "IPv6 Network Administration", By Niall Richard Murphy, David Malone.(Buy Online)
Programming competition for East African students
Google (my employer) has announced a "Google Gadget" competition for students in Tanzania, Uganda, Kenya, Rwanda, Burundi and Ethiopia. The designer of the best gadget will a $600 USD stipend, five runners-up will receive a $350 USD stipend. Prize categories include Best Gadget UI, Best Local Content Gadget (Most Locally Useful Gadget), Best Education Specific Gadget, Best Procrastination Gadget, Most Technically Sophisticated Gadget, Gadget Most Likely to Get International Traffic, and Best Social Gadget.
Complete details are available on the East Africa Google Gadget Competition website. A PDF suitable for your university bulletin board is available here.
Women and Latin Americans in Open Source
A co-worker of mine, Fernanda Weiden, was interviewed on the FLOSS Weekly podcast.
Fernanda Weiden of Google in Zurich gives her perspectives on women and Latin Americans in the open source community, the Brazilian Women in Free Software, Debian Women and the Free Software Foundation of Latin America
True story about Fernanda: She taught herself English by reading Linux "man" pages.
I appreciate you!
Today is the 8th Annual System Administrator Appreciation Day. I know this sounds kind of funny, but I really appreciate all the system administrators out there. I meet a lot of system administrators. I visit a lot of sites. I hear stories about heroics, and I hear stories of people who persist even though they are working with terrible management, unappreciative users, and CEOs that treat IT as a "cost center" instead of an investment in future corporate growth.
Last week the 2nd edition of The Practice of System and Network Administration started shipping. The new edition includes a lot of new anecdotes, many from the fan mail we've received over the years. Some of the fan mail is fun, like when we were told that something we suggested helped recover from an outage a few hours faster, which saved his company $100,000. Often we are pleased to receive email from someone who's received a promotion and wanted to thank us for writing a book that was instrumental to their career. But most of all I want to say that I am humbled by the messages we've received from the lonely system administrators: The under-appreciated person struggling to fix a big mess they inherited, with all the responsibility when it fails but none of the authority to fix the larger problems. We received email from one person who, when reading the book, burst into sobs after realizing she wasn't "the only one".
This will be the second year that I'm volunteering to judge SysAdmin Of The Year. Nominations are open, so email the URL (http://www.sysadminoftheyear.com/) to all your friends. The first 2500 nominated sysadmins get a free tshirt, which is pretty cool in itself.
Tom
P.S. If you are in the Philly/NJ/DE/NY area (or aren't, but like last-minute travel), don't forget that I'l be doing my time-management training classes during the tutorial part of LOPSA's SysadminDays local conference, August 6-7, 2007, in Cherry Hill, NJ (just outside Philadelphia).
Book Recommendation: What All Network Administrators Know
I just came across Douglas Chick's book, "What All Network Administrators Know". I immediately rushed to add it to our web page of recommended titles (scroll to the bottom).
One of the problems with TPOSANA is that it really focuses on big sites. This book is perfect for sysadmins that are just getting started or are at a small site. It is down to earth, very practical, and contains tons of excellent advice. (If you want proof, preview it on Amazon by clicking on the "random page" button.)
Sysadmin Appreciation Day and Sysadmin Of The Year
Don't forget (or don't forget to remind your boss) that Friday, July 28th is System Administrator Appreciation Day. www.sysadminday.com
However the new hotness is the 2006 Sysadmin Of The Year contest. Sponsored by Splunk, LOPSA, and many other organizations. One Grand Prize winner will receive a $2,500 Splunk Professional license and an all-expense paid trip to Washington, D.C. to attend the Large Installation System Administration (LISA) Conference December 3-7, 2006. More than 2,500 other prizes will be awarded. Nominate someone today!
