A few weeks back, we discovered that we'll need a new location to host our CPLUG box. Ours is on a T1 at a local company now, but they're redoing a lot of their Internet infrastructure. It'll be somewhat difficult for them to continue our free service.

So, we put out a call for assistance, to see if someone else could host our box. , who had come up to the CPLUG Security Conference (all the way from VA), said that his company would be able to help out. He works for someone I had never heard of, Carpathia.

Grateful for the offer, but not knowing anything about the facility and unsure about having to drive two hours to get there, I waited a bit longer to see if we'd get other offers. I'll get to that later -- but it soon hit me that there was no reason we couldn't place a box down there, even if it was just a backup. So, while preferring to have our primary box within driving distance, it seemed to make sense to take advantage of his offer.

He had gone a step further and offered us a tour of the data center they're in, provided by another company I had never heard of -- Equinix. It sounded like fun, so five of us drove down there to tour the facility and drop off the server.

It was shortly after we arrived that I realized I had grossly underestimated what it was we were being offered. The Equinix facility was huge. You can see a picture of it here. The main datacenter is the large building on the north side of the picture. This datacenter was being used by other companies like Yahoo, Amazon, Capital One, AOL, Verizon, Microsoft (the Windows Update servers), AT&T, and numerous others. There were so many servers inside that you had to shout to be heard. And we saw a bit of everything, Dell's, Compaq's, Sun's, Xserve's, and more. We saw a few servers so large that they were labeled "UNIX Silo's".

Access to the facility is quite secure. It requires a pin and a hand scan by an authorized person to enter the building, and the individual cages that are inside (within a cage is an organizations servers).

They have six gas powered generators in case of a power failure. They also have room after room of batteries for a series of UPS's that they have to keep everything alive until the generators come online. They have two 20,000 gallon tanks of fuel to power them. Further, during an outage, fuel trucks will come once an hour, keeping the fuel no lower than 75%.

A lot of this is also explained on Carpathia's Website, where they give info about the datacenter, including a virtual tour. Apparently, we missed the video game section while we were down there. There's a room off to the side containing video games an pin ball machines :-)

The kicker is that Carpathia has a huge pipe, and is right on the Internet backbone. Our box has a 7GB connection available to it, has a faster ping time than Yahoo or Google -- and is also less hops than Yahoo or Google. Yikes :-)

I'm eager to get this box into production, it doesn't have all our stuff on it quite yet. Thanks to Bob for his generous hosting offer and the tour he gave us.

I've been doing a lot of thinking lately on what distro to use in the server room. Shortly after RedHat dropped support for it's free distro, we moved to Debian. Debian is nice because things tend to work, and upgrading is minimal.

It has it's issues though, Sure, packages are old, but that isn't my big beef with Debian. How many features from new software does one normally run on their server? I know for me it's very little. But there's other issues that have arisen from my usage of Debian. Or perhaps better said, I recognized issues in Debian and other distro's through my usage of Gentoo. Here is an email I recently put together explaining my thoughts on the matter:

Now -- all that being said -- and with everything I had mentioned about
servers a few years back when RedHat dropped it's free products, we're
considering switching the distro on our internal servers.

Believe it or not, we're actually seriously considering Gentoo.

Gentoo has some really interesting features that make it really nice to work
with.

It has absolutally nothing to do with speed. This article here gets into some
of the positive aspects of Gentoo:

  "Gentoo for all the Unusual Reasons"
  http://www.linuxjournal.com/article/7438

The two biggest downsides someone would see to Gentoo on the server are:

 * Newer (possibly less tested) packages
 * Required to compile any package you want to install

Of course, both of those are issues.  It was enough for me to write off
Gentoo as a server platform before having actually used it.

Now that I have it -- nearly every issue I run into with another distro, I
think "Wow, this would be easily solved in Gentoo".  I actually feel that
there's less issues to be run into with the above two "downsides" in Gentoo,
as compared to the issues run into with other distros.

Before I get too much further -- I will state that this isn't necessarily
true for everyone, or always the best option.  I have simply discovered that
it seems to work well for me :-)

I plan on getting around those two issues with Gentoo using a dedicated build
box.  The build box would handle compiling all the packages that the other
servers might want.  It'll store them as binaries (GRP packages).  The other
servers would then install the binaries, not compile the source.

Further, to help the dedicated build box, some of the servers would probably
get distcc (a distributed compiler) put on them to help out with the building
process.  Building packages would only occur at night, so running distcc on
things like our nameservers wouldn't be a problem, and would really speed
things up.

I also would not just install these new packages onto the servers without
first testing the packages on the dedicated box.  The dedicated box would
need to have all the software installed on it that the servers have.

A setup like this would also allow us to make use of Hardened Gentoo.
Hardened Gentoo has a lot of security tools available to it, including memory
and buffer overrun protections, as well as things like SELinux.  SELinux
isn't for everyone.  Even without that, the protections provided by the other
tools is excellent.  A good many of the remote vulnerabilities we're seeing
fall under the categories of being a buffer overflow, or being a poorly coded
web app.  Hardened Gentoo, even without SELinux, can largely take care of the
buffer overflows.  Just take care as to what web apps you're running, and
what they have rights to.  Or run SELinux, and protect everything :-)  But
there's administrative overhead that comes with SELinux deployments.  If
you're willing to learn it, and you have the time, it's a useful tool.

Two things are coming up -- Debian will have a new release out, and we'll need to upgrade our hardware. As opposed to installing the new Debian on this hardware, it might work out to begin a migration to what I feel is a more flexible Gentoo.

Not everything made it into the first story about the Security Conference. This warrants an immediate mention before I forget to write about it. So, we had a dinner after the conference for speakers and those in CPLUG who helped. Of course, Kevin was in attendance. Many of you may remember Kevin as the one who ran the PGP Keysigning Party. You can find Kevin's PGP Key here (also note the presence of an email address... note that, you'll need it in a minute :-)

So, we learned a few important lessons from both this dinner and from Kevin. First, if someone brings wine or champagne, keeping the cork is of the utmost importance. Lest you be forced to learn the rest of these lessons on your own.

If, for some reason you fail to keep a cork for your wine bottle, all is not lost. One could probably get away with Glad wrap and a rubber band. The next point is key! Do not, under any circumstances, try this same trick with champagne!

So why, pray tell, would I suggest this? A single word explains it quite well: Carbonation.

So, we have Kevin, trying to come up with a way to bring the left-over champagne home. We have no corks. And it's illegal to put this in the backseat, so we needed a way to put it in the trunk.

Kevin gets some Glad wrap from the folks at the restaurant, and masterfully puts a layer or two of glad wrap over the top, and has a rubber band for the seal. Uncertain whose trunk this would end up in, I asked if he felt it was sealed well enough. He did, but he sought to prove it.

Kevin diligently turned the bottle upside down with a mild amount of shaking to prove his point.

Click below to see what happens next.

See more ...

Well, yeah, so I'm a little behind. Slowly but surely I'm getting caught up on everything :-)

On Friday evening after the SELinux Symposium, Russell and Manoj hitched a ride back to Harrisburg to check out the CPLUG Conference. That was fun -- they both spent the night, and we got up nice and early the next day to head into the conference.

I feel like I got out of most of the work. By the time I got there, Shana and the rest of the folks who got there early had most of the things handled.

The conference itself went great. I enjoyed each of the speakers. From the input I've received, the attendees found them all enjoyable as well. It sounds as if each of the six speakers presented someones favorite talk. I heard really good things about all of them.

That being said, I believe Trooper Jon Nelson of the Philly LUG had the most popular talk. His topic of computer crime and law caught everyones attention to such a degree that the questions were non-stop. Towards the end of his time slot (when he was only a handful of slides into his talk), someone ultimatly suggested he try and get through the slides instead of answering questions. He managed to zip through most of them, thanks to the pizza being a bit late.

I somehow found myself as organizer, MC, and speaker. I felt vastly underqualified for most of that (no, I'm not fishing for compliments! :-):

• I have never MC'd an event before in my life (the CPLUG meetings don't count, they're small and short :-) Heck, I'm not even sure I quality to introduce the speakers we had, but it all worked out :-)
• I felt odd speaking in between such high caliber presenters. I'm thankful for the opportunity to have done so, it was fun. I'm glad Russell coaxed me into it or I would otherwise never have considered it.

I discovered some interesting things in all this. After a few minutes, I was the least nervous I've ever been in front of an audience. An unusual time to feel that way, for sure. But it gave me a wonderful opportunity to be able to focus on what I was doing rather than being nervous, and I discovered a whole host of things I could do to improve in public speaking.

Rather than getting into all of them right now, I'll say that I'm really considering getting back into the ToastMasters program. I had gone to one meeting before -- but through a series of unfortunate events which left me in crutches for several weeks/months, I never ended up going back :-) (okay, okay, on the walk home, I tripped on the side of the road and twisted my ankle pretty good)

Two quick things regarding this -- I think it's finally time to start putting less info on the slides, and to learn to think while I'm up front. It's easy to load the slides with so much info you don't have to think. Seeing myself next to these other speakers finally allowed me to see that. The part that concerns me is how forgetful I am even when not standing in front of an audience. Without having time to do research and think things through, I often can't think of what it is I want to say on the spot. So the trick is to come up with a way to be able to discuss everything I want to, without actually reading so much off the slides.

Second, I think I want a wireless presentation clicker. I really like the feel of the one I have now, but the cord just keeps getting in the way. Having so much info on the slides was bad enough, not being able to walk around and engage the audience really helped to me recognize where I needed to improve.

This pointer from ThinkGeek looks like it might be a nice one. I've been put off to a lot of these pointers due to their size. This one from ThinkGeek is the size of a PCMCIA card, which is convenient. And it has a laser pointer -- even better :-)

We've been asked by many to do this conference again in the future. We're going to get together again soon to discuss it. My exact thoughts on that warrant another post (as I recognized when I began to type an email on this subject to Dr. Chase. Notably, I still haven't finished that email, but I am highly intersted in doing this again, I had a great time :-) There's a few things I'd like to overcome if we're to do it again next year.

That's all for now. I'll finish up with a link to my CPLUG Security Conference pictures.

This is just a note to potential conference organizers of any sort. At your conference, you probably want to give everyone badges. That's good. When you buy your badges, they'll come in a little Avery box, and the box will be labeled as having 100 inside. So buying two would be plenty, for a conference that has 200 attendees.

Whoopie, I though, as I purchased two boxes (at $50 apiece?!) and set the conference max to 200 people.

Who would have thought to do the math ahead of time? As Andy and I were preparing to print these out for our 200 conference attendees, I happened upon some smaller print, under the part that said "Contains 100 Badges", that stated "6 per page".

I'm not on top of my math skills, so I had to punch that into a calculator. Nope, 100 is not divisible by 6. Hrm. Yeah, it looks like they actually gave us 16 pages of 6. Don't worry, I needed a calculator for that too -- it's 96. So that actually means we're short 8 badges. Well, phooey.

At least 8 people aren't going to show up anyway, right? :-)

The list of talks for the CPLUG Security Conference is looking really good. We now have all but one of the Lightning Talks we wanted. The Lightning Talks should really be a lot of fun, but we need just one more :-)

Some people who read this have considered doing one. If you're interested, I'd be happy to brainstorm some Linux/security related topics with you. There is actually one person (from western MD) who occasionally reads this, who agreed to a talk, then seemed to drop off the face of the earth. Doh! If you're reading this, is your spam filter being over-zealous? :-) Drop me an email :-) (eric@openthought.net)

Barring any takers, I'll mention this in the next batch of email updates being sent out to the attendees.

I actually have a talk I'd like to do sometime, but I don't have time to both prepare it as well as plan the conference. There is a Linux project called Class Based Linux Kernel Resources (CKRM). It's roughly equivalent to file system quotas, but for Linux resources like memory, CPU, and bandwidth.

Since it can limit memory (and process) resources, I'd like to deliver a talk demonstrating that by launching a fork bomb... then opening up OpenOffice to continue the talk :-) When the talk is nearly complete, I'll close down OpenOffice and show the forkbomb still running, but only using the resources it's been allocated by CKRM. Fun fun.

Okay, so CPLUG got a chance to flex it's geek muscle tonight. Outfitted with 12-14 laptops, we braved the outdoors (a significant step for many of us). But we've gotten to far to stop there -- we proceeded to create our own Wireless Mesh Network. We used LocustWorld's bootable Mesh Network CD.

The jist is that each node in the network is an Access Point. So, we start with a base station, connected to the LAN with a cable, and broadcasting with a wireless card. Someone walked 100 feet away, fired up another laptop, which connected to the base station. Someone else walked further out from that, doing the same.

Soon enough, we had moved out of the Intellimark parking lot, and rounded the corner onto the street. So we're working our way down the street with a goal of reaching the other Intellimark building. We managed to spend half the night playing in the street without anyone being hit by a car. And of course, everyone on the node has full Internet access.

However, like true geeks who will never be understood, we did manage to attract the attention of the local police. Sean showed them that even though he was 800 feet from the building, he was still getting signal. They seemed amused. After talking to a few of us to see if we were actually serious, and carried the same story (he actually started asking everyone how their signal was), the two squad cars got together to discuss what a bunch of geeks we are. Boy, they got that right :-) I can't believe it though -- they have computers inside the squad cars, but we didn't think to ask if they could boot up one of our mesh network CD's. We could have gotten a few hundred more feet.

Anyhow, armed with a Zaurus, which would make for a good end node since it couldn't boot the Mesh Network CD (Nate had those honors with a Palm last year), I made my way to the end of the chain. This took me into the other Intellimark parking lot. And then voila, I reach the sidewalk in front of Intellimark. Woohoo! If you look closely, you can see the green signal light on my wireless card. Success!

Earlier on, it was beginning to look like we may only get part of the way to our goal. It turns out Chris had several USB wireless adapters, which seemed to get some incredible range. A few of us booted with them. We also discovered a 75 foot ethernet cable, which allowed us to drag the base station outside. Actually, you can see the laptop behind the door in that picture. It didn't quite make it outside, but the antenna from the wireless card did. And that would be plenty :-) That, combined with all the great folks who were able to bring laptops, hardware, and CD's to make this work, we reached our goal and went from one Intellimark building to the other. In all, it was about 2100 feet. Here you can see a rough map of our network. That map is slightly off -- we cut off a bit of that last corner, plus it shows you going in the far entrance near 11/15. We hadn't gone quite that far.

Thanks to everyone who participated! A special thanks to Nate and Dean for creating all these CD's. As always, thanks to Andy for opening the doors of Intellimark to allow us to meet there. He somehow managed to convince them that we wouldn't break anything if given Internet access.

We're already planning another event for next year. A little more daylight might make the pictures turn out better. And the software didn't work properly with all our wireless cards, we could get even more people on the network with some modifications to the CD. Hooray for Open Source :-)

You can find more pictures at my photo gallery, Nate's gallery, and John's gallery.

For quick and dirty bandwidth monitoring, combining iptraf with rrdtool seems to work really well. this tutorial describes exactly how to get it set up with web based graphs which can monitor per port traffic. It took less than 15 minutes to get that working.

I never realized how easy it is to set up an IRC server until I tried it for the CPLUG and HPM box. It wasn't that much harder to add an SSL option using SSL Wrap. Of course, some people can't get to it due to port restrictions, so it's also got the web interface using CGIIRC. Many CPLUG users have been getting less work done lately :-)

Nate's been pondering an IRC to Email bot. A great Perl project.

It recently hit me that upon setting up the email/web server for the local Linux and Perl groups, I never added a backup MX record to the DNS entry. Which would be a problem this weekend due to scheduled downtime of the datacenter it's being hosted in.

Andy was kind enough to offer their Querx.com server (what in the world is a Querx?) for backup email.

I just went to setup the secondary MX, and realized I never bothered to add *any* MX record. Apparently, email can still get delivered without any MX records listed. Every person on the CPLUG and HPM mailing list has been doing it for three months :-)

CPLUG is much more interesting when more people participate. At least six members are offering to do some talks.

After someone offered to do a five minute talk on their mail server setup experience, we dubbed next month "Mail Server Month", and have since gotten more presentations than we can fit in a single meeting.

Maybe we can make next month "Meat", and the following month "Potatos". We can go over Postfix, MailScanner, Amavis, McAfee, ClamAV, and such during the "Meat" month, and things like SPF, Ask, and other spam filtering techniques during the "Potatos" month. This should be a lot of fun.

The CPLUG had it's first install fest on Tuesday. We had a blast, and a load of people showed up. I'd guess in total there were nearly 25 folks who came by, which is about 15 more than usual. Around 8-10 of those were people coming to a CPLUG meeting for the first time. Some of those were new to Linux, others had used it for awhile.

It worked out really well for me, as I've been needing to do some installs/upgrades for awhile. We have a new server for our mailing list / websites, Todd and I did a Debian install on that. Shana wanted to try something new, I put Mandrake on her computer. And I've been debating for awhile on what to put on my laptop -- since it already had RedHat 9, I decided to just do an upgrade to Fedora to prevent me from having to start from scratch.

All three installs went pretty well. I've gotten pretty familar with Debian in our rollout at work, and that one went pretty smoothly. Fedora wasn't too much different than the standard RedHat install, though the installation itself seemed a bit more polished. I had never used Mandrake before. I was a bit surprised by it's simplicity. Everything from the install to the desktop experience are done very well -- I'm pretty sure my mother could use it, perhaps even install it. It's desktop is much cleaner than other distro's that I've used, though some of that can be chalked up to KDE. For example, RedHat has three menu options which contain settings -- Preferences, System Options, and System Tools. For a lot of users, that could be confusing. If I want to configure my Palm Pilot or Add a Printer, which one would it be in? Palm Pilot is apparently a preference, while adding a printer is a system option. With Mandrake, things like that are all under a single menu item called "Configure". You can just poke around under there until you find what you want.

No, I'm not quite ready to switch my desktop to Mandrake, though I think I might consider recommending that to new users. We'll see how it goes after Shana plays with it for awhile.

Fedora seems just like every other RedHat release, except that it has newer programs. But then, perhaps that's what it's supposed to be. I just need for it to provide some configurable menu's on the desktop, then get out of the way so I can use it. I did finally try a new background for my laptop. One of these days I should get a Digital Blasphemy subscription.

Oh, and I started using Apt with Fedora, I should have started using that awhile ago.

• Freevo - This is a free package that provides Tivo-ish functionality, along with being able to view/play most of the common media file formats.
  
• apt-rpm - Apt, but for RPM files. There's a lot of people who like this, though I'm personally a fan of AutoUpdate.
• Resource Monitoring Tools - We had a good discussion on viewing memory usage of programs, and the difference between RSS and Share as seen in "top"