Found: One Troll

I’m at a class for BGP routing this week (yes, you can fill a 50-hour week with just BGP information) and one of the other students happened to mention an RFC I was unfamiliar with. “Request for Comments: 2795″ describes The Infinite Monkey Protocol Suite (IMPS).

Abstract

This memo describes a protocol suite which supports an infinite
number of monkeys that sit at an infinite number of typewriters in
order to determine when they have either produced the entire works of
William Shakespeare or a good television show.  The suite includes
communications and control protocols for monkeys and the
organizations that interact with them.

Read the whole thing.

ETA: There’s a list of more hilarious RFCs here.

I know I said I’d do this during the weekend, but the release of Fedora 11 got pushed back a week to this Tuesday. I’m a big fan of the Red Hat Linux OSs, so even though there’s a distro just for EeePCs, I decided to be difficult and go with Fedora. Over the next few days I’ll probably spend some time tweaking it to run properly on the Eee hardware but today I just want to get the base OS installed. I tried to get it running yesterday, but apparently the Fedora LiveUSB creator for Windows doesn’t like my 32GB Kingston stick that I keep my random crap on, so I stopped at Best Buy today^[1] to pick up a more basic 4GB stick that it wouldn’t choke on. Commence mostly-liveblogging:

• 6:05PM – Got the LiveUSB created on the new stick. Rebooted the PC and. . . Windows XP. Wonderful. Verified in the BIOS that “removable media” is the preferred boot device and turned off “fast boot.” Still booting into Windows. Apparently with the Eee1000HE you sometimes have to hit ESC during POST to choose your boot media manually. After doing so, am now in KDE (running live off the stick). Chose “Install to hard drive” and away we go. KDE looks slicker than the GNOME desktop environment I’m used to.
• 6:10PM – The Anaconda installer looks much the same as ever. Since this Eee came pre-partitioned with two logical drives, I jumped back into Windows and deleted the as-yet-unused D: drive. It will be replaced with ext4 partitions.
• 6:17PM – And. . . done. Base Fedora Install complete. Rebooting Eee to verify Grub bootloader function. . . And it works.
• 6:18PM – Completing setup of NTP and other such.
• 6:20PM – First login to Fedora on the EeePC. Further blogging from the Eee.
• 6:53PM – Up and running on Fedora 11. Ran into some issues with KDE. . . it’s really very different from GNOME. I’m not sure if I like it yet, it’s a little too Windows-like for me. The browser that comes with KDE, Konquerer, is complete crap. I spent a few minutes trying to install Firefox via the GUI and then said “Fuck it” and just ran “yum install firefox” from the terminal. Have I mentioned that I love Red Hat? It’s odd to me that in F11 w/ KDE the graphical interface is on tty1 instead of tty7.

Initial impressions:

• Fedora 11 runs quite well on the EeePC. Issues from Fedora 10 such as a lack of native wireless and power-management support seem to have been resolved in the base Fedora 11 install.
• KDE’s pre-packaged apps are a little too good-looking and gimmicky for me. I use mIRC in Windows, I don’t need my Linux IRC client to be any fancier than that. Gonna have to find another one. I might switch back to GNOME, it was a simpler interface.
• Seriously, Konquerer sucks. It wouldn’t display the WYSIWYG editor for WordPress. I’d almost rather browse with wget and lynx.
• Overall I’m very happy with Fedora so far. I’ll have to screw with it between labs tomorrow.

1. There’s a Fry’s Electronics and a Best Buy on my way home from work. Fry’s has large “No Guns Allowed” signs posted, so I don’t shop there anymore. [↩]

Here’s some stuff I meant to post about but haven’t:

• I guess I’m losing weight, because my old belt doesn’t fit me anymore on the last hole. I headed over to Galco‘s website and ordered a new belt 4″ shorter than my old one, and all is well again. I can wear my gun and my leatherman without my pants falling off. I guess walking everywhere and not eating so damn much is working out well this year.
• HB2474 – the guns in parking lots bill – passed the Arizona House and both of my representatives voted for it. Hopefully it gets a fair shake in the Senate and passes this year. Gov. Brewer will certainly sign it. The way I (and the law)  see it, my car is an extension of my home. Making it illegal to ban guns in private cars doesn’t seem like an invasion of privacy to me.
• Hopefully this weekend I’ll get around to liveblogging a Fedora install on my EeePC. I’m going to try KDE because I’ve only ever used GNOME before.
• Rock Band is getting some Iron Maiden DLC next week! Finally, no more shitty covers of Maiden songs; we’ll be able to rock out to the real deal. Also, next week is the release of Iron Maiden’s new live DVD/documentary Flight 666.
• I think I’ve finally figured out why AT&T is the worst backbone service provider. Level 3 and Qwest have a lot of automated processes for managing BGP policies and such, but AT&T has an understaffed team of fools who do all that work manually. Example: I want to update some policies on 100 Level 3 circuits at once, it takes them less than 24 hours, and only that long because the automated process runs during a maintenance window. If I want to do the same thing for 100 AT&T circuits, it’ll take their team a month to manually make the changes. Then I have to go through and check their work because they inevitably miss something or otherwise cock it up. Even one circuit takes a week if you don’t call and harrass them. The difference: AT&T is a union shop. Unfortunately, I’m not 100% sure that Level 3 is non-union. It’s possible that AT&T just generally sucks.
• Here, more Maiden:

I was listening to episode one of the Vicious Circle podcast at work today (with headphones, of course), and Alan hit on one of my serious pet peeves. He also used an example that I am too familiar with. The peeve in question being where IT departments spend beaucoup bucks on a solution that can be done better for less. The specific example Alan used was Nagios vs. Netcool, two monitoring platforms I’ve used extensively. There are other examples I’ve encountered at work, but this one is too perfect.

Two years ago, before I was a network engineer, I worked as a NOC engineer. The job of a NOC engineer is basically to monitor the network and call someone when something breaks. To facilitate that task, a network operations center will be outfitted with a network monitoring system of some sort. When I started the job we used Nagios as that monitoring system. Nagios is freeware and it runs on Linux. Its system requirements are not burdensome. In our case we were running it on Fedora Core 4 on redundant POS Dell servers. It can run a variety of checks against network equipment and servers from ping checks to SNMP polls and service monitoring (HTTP, SMTP, etc). It’s open source and there are hundreds of add-ons for it from new web front-ends to extra service checks and auto-discovery scripts. With a little scripting know-how, it was easy to configure Nagios to do basically whatever we wanted.

Nagios has one downside: it’s management intensive. Autodiscovery is not a reliable way to find new network equipment, and frankly you should know whenever something new is added to your network. Thus, every new piece of equipment has to be added to the monitoring system manually. Fortunately, doing so is easy. A trained monkey could manage Nagios. With nine NOC engineers, three per shift, each trained in Nagios administration, the management overhead was very light. Each of us was also capable of troubleshooting the monitoring system individually, which is handy when it breaks at 3AM.

Unfortunately, someone higher up decided that real NOCs don’t use Nagios. Nagios is free and everyone knows that free is unreliable and bad. Not to mention unrespectable. Never mind that it worked, required little training to use and only slightly more training to manage, and the program was capable of doing everything we needed it to do. No, we needed something pretty. Something new. Something. . . expensive!

Enter Netcool, a monitoring solution from IBM of respectable breeding. I’m not going to sugar-coat it, I fucking hate Netcool. Nagios ran on any hardware, and it monitored a few thousand nodes of our network without any problem. In order for Netcool to do the same thing, the consultants we hired to install and configure the system and then train us on its use insisted that we needed hyper-expensive Sun servers (but I repeat myself), and of course we needed to run Netcool itself on Solaris X^[1]. It took them a year to figure out how to make it run and to import our network maps, and they were the experts. Nevermind that during the entire time we had a working system that did the same thing. The cost of the endeavour? Let me just say that we could have hired 50 people at my salary for a year.

Nagios has an HTTP front-end that runs in any browser, anywhere. Netcool’s front end uses the Java Runtime Environment, another piece of garbage software that I loathe. A side-rant on Java: I have six different platforms that I need to use Java to manage, and I have to run two different versions of JRE to do that, because half the programs only run on an older version of Java. Every time a new version of the JRE comes out it breaks something new. Fuck Java in the ear. Give me a CLI any day. If I can’t manage it via SSH, you have failed as a vendor.

Anyway, we were promised that Netcool would perform automatic network discovery, cutting down on management time and effort. We were also told that Netcool would use our network diagrams to determine the systems that any given fault affected, and then isolate the root cause without setting off hundreds of alarms to sift through. Both of those claims turned out to be complete bullshit. Cutting down on management overhead? Is that why we hired someone to manage the monitoring software full-time? Is that why instead of having a dozen people who knew our monitoring platform inside and out we now have one, who spent weeks at IBM training for the job? Is that why, millions of dollars and a year and a half after full deployment, we still have a solution that’s only marginally better than Nagios was, and not at all better than it would have been if we had put in the same effort to upgrade that platform?

I’m so glad I don’t have to deal with that horse-shit Netcool software any more. You couldn’t pay me enough to do it again.

1. They were likely full of shit, but management bought what they were selling [↩]

It looks like this is going to be the year of the convention for me. The Blog Bash is this weekend, of course, and today I confirmed that I’ll be attending Cisco Live in San Francisco next month. I’m seriously considering PAX and the Reno GBR as well. Now that I think about it, I was planning on taking two weeks vacation in September anyway. I could hit both events.

With all that traveling, I need a new laptop. A netbook seems ideal due to its portability and no-frills utility, but none of them have traditional serial ports on them. As a network engineer and computer hobbyist, I still use console connections and null modem cables often enough that a laptop without a serial port is a non-starter for me. I’ve used USB-to-RS-232 adapters before, and the ones I’ve tried suck hairy balls. So my question to my readers is this: is there a USB-based solution that I can use with a netbook which actually works half a damn? Or do I just have to get a full-size laptop?

The Big Brown Truck of Happiness just dropped this at my door:

I guess break time is over.

Brady Volpe just finished a long and very informative series of tutorials on how cable modems work. It starts with the basics and gets pretty heavy by the end. If you’ve ever wondered how that little box helps you get the internet, the answer is there.

Well this (PDF) is surprisingly relevant considering that I spent half my weekend writing about the coming IPv6 migration. Here’s something that I find in my inbox this morning via ARIN’s mailing list. Maybe this will help me light a fire under our executives on this issue.

Subject: [arin-announce] ARIN Outreach on IPv4 Depletion

The Board of Trustees has directed ARIN staff to contact, by certified
letter [1], the CEOs of organizations that currently hold IPv4 resources
in its region. The purpose of this effort is to raise executive
awareness of the depletion of IPv4 resources and to encourage the active
adoption of IPv6.

This letter will also serve as notification that, in response to the
approaching depletion of the IPv4 free address pool, the Board has
directed ARIN staff to take additional steps to ensure the legitimacy of
all IPv4 address space requests.

Beginning 18 May 2009, ARIN will require that all applications for IPv4
address space include an attestation of accuracy from an officer of the
organization. For information on this requirement, please see:

https://www.arin.net/resources/agreements/officer_attest.html

Regards,

Leslie Nobile
Director of Registration Services
American Registry for Internet Numbers (ARIN)

[1] https://www.arin.net/knowledge/about_resources/ceo_letter.pdf

As of today I am a Cisco Certified Network Professional. This morning I passed the fourth and final certification exam. Now I can go back to playing video games for a couple weeks before I start studying for my CCIE.

So I promised to answer Borepatch’s questions:

The Fed.Gov is making IPv6 a big deal. How long do you think until (a) large non-DoD migrations occur, or (b) we actually start to run out of IPv4 addresses?

As a member of the cable industry, I can say that for large, non-government, North American IPv6 deployments all eyes are on Comcast. Comcast will likely be the first US ISP to roll out IPv6 on a large (millions of users) scale. Comcast’s push is directly related to the fact that they are completely out of RFC1918 private addresses to use in their network.

For those unfamiliar with Comcast’s problem, the company has about 25 million cable modem subscribers, all of whom need at least two IP addresses. One address is for cable modem management and is not publicly visible, and one address is for the end-user which must be publicly visible. The address for CM management is almost always a private address which is not internet routable. In other words, the address only has significance within an organization, not on the public internet, and so these addresses can be duplicated between organizations. RFC1918 defines the addresses which should be used for this task: 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. There are, in each of these networks, 16777216, 1048576, and 65536 addresses respectively. Since Comcast needs 25 million of these addresses, and there are just under 18 million available, they’ve run into a bit of a problem. Comcast has had to issue about 7 million of its public IP addresses, which are unique and must be assigned by ARIN, to cable modems for management purposes. This is a massive waste of the already limited IPv4 address space. There’s a fascinating (if you’re into this sort of thing) powerpoint presentation by one of Comcast’s IP architects available here which details his proposal for their IPv6 rollout and interim management solution.

So if we need it now, what’s the hold up? In the cable industry, one of the primary problems with deploying IPv6 is that the new protocol isn’t supported over existing cable plants. Cable internet is served with a method called DOCSIS, which is a specification that defines physical and data link layer requirements for transmitting internet packets over cable media. Most cable plants today are DOCSIS 1.1 or 2.0. DOCSIS 3.0, which was ratified in 2006, is required to operate IPv6. DOCSIS 3.0 gear is still relatively difficult to come by in today’s market. It’s also extremely expensive, about twice as expensive as installing DOCSIS2.0 compliant head end gear. But if that were the only thing holding us back, DSL and FTTP providers would be running IPv6 right now and leaving cable in the dust.

Other issues preventing large-scale IPv6 deployments include:

• A lack of support from upstream service providers. Backbone service providers such as AT&T and Level 3 are not currently running IPv6-capable routing protocols throughout their networks. Even if I could turn up IPv6 services tomorrow, I couldn’t send them across the internet in most of the US without translating IPv6 packets into IPv4 packets.
• A lack of support from backend vendors. Many DHCP provisioning solutions currently deployed do not support IPv6. Upgrading is costly.
• Security: IPv6 currently enjoys security by way of obscurity; very few people have any experience at breaking into IPv6 networks. We know where the security holes in v4 are, and we know best practices for plugging them or minimizing their impact. v6 is a virtual unknown. However, one upshot of IPv6 is that it supports IPsec natively.^[1].
• Databases: Every field that used to contain 32 bits for an IPv4 address must now contain 128 bits. Every application that reads from or writes to those databases has to be taught how to parse IPv6 addresses in a meaningful way.
• Apathy. I and other engineers across the country are having a hell of a time convincing management that IPv6 is worth the time and effort now when we know that IPv4 will last us at least another two to four years. It’s not unreasonable to guess that v4 will be around for another decade for the late adopters.

I haven’t been able to find any concrete info^[2], but given that Comcast is pushing its DOCSIS 3.0 rollout fairly hard I expect that Comcast has begun or will begin by the end of this year migrating all of their modem and set-top box management to IPv6. If that goes well, perhaps they will trial issuing IPv6 addresses to end users by the end of 2010 with large migrations by the end of 2011. IPv4 addresses are estimated to be exhausted by the end of 2012. but as large ISPs like Comcast and Verizon start migrating to IPv6, they’ll turn in their IPv4 allocations to ARIN or sell them to other corporations. Ultimately it’s possible that this could delay adoption of IPv6 even further.

While the Federal Government mandated that all of its agencies be IPv6 ready (at least at their core infrastructure) by June of last year, I’ll be very surprised if they require IPv6 transition of private corporations before the time that every major ISP is operating at least a dual-stack architecture, if ever.

1. IPv4 rewrites packets with a second header to use IPsec [↩]
2. It looks like Comcast is only advertising their aggregate IPv6 network (2001:558::/32, ASN 7922) via BGPv6. This could mean that they aren’t using their allocation yet, or it could mean that they’re tunneling all their IPv6 traffic to one location and advertising the prefix there, which is what I would do for management traffic. [↩]