Sysadmin · I still don't have a title

natura upgraded to lenny

Sat, 28 Feb 2009 00:13:00 +0000

natura.oskuro.net, the home server which still serves this blog, has been suffering hardware problems for some weeks. Apparently the hard drive is failing intermittently, so every now the kernel starts spewing out noisy errors about its main disk dying. If I notice this quickly, it can be rebooted and that normally fixes it for a few more days. But if I don’t, it’ll end up giving nasty bus errors which will make remote logins a challenge. Most processes still work, but the filesystem appears to be gone. It’s easy to know what’s going on if you visit the blog’s url and get some 404, and in that case I can only phone my father and tell him to press the reset button (I’ve tried sysrqd, but I need to open the port in the router and haven’t had chance to do that yet).

So it was time to do something about it, and the other day I installed a dirty 40GB drive on the second IDE controller, in case I could find the time to do somethng about it. Being with an endless pharyngitis that doesn’t seem to get cured entirely, I’ve had some time today to look at it. This evening, I was about to transfer all the system to the new disk (it’s half the size as the broken one, and probably slower, but it hopefully has no bad sectors), but I decided to upgrade the system first.

natura was first installed in late 1997 or at the beginning of 1998, using the Debian bo install media on a Pentium 150MHz, and has gone through seven dist-upgrades which, as far as I can remember, have always worked out without major problems.

The upgrade to lenny hasn’t been an exception. The server has gradually lost many of the services it once hosted, so there aren’t too many services to take care of anymore. All the mail services I setup for my father ended up being deprecated as they started to get used to Hotmail, GMail and so on, and the frequent hardware crashes made me switch them to the Linksys based DHCP server. In the end, the problems I saw after the upgrade were very similar to what I faced when I upgraded to etch:

apache2 restored the 000-default symlink in the sites-enabled dir, which resulted in my website showing the classic “It works” message for a while.
Apache’s suexec support had moved to its own package, and was very well documented in the NEWS file, but I somehow failed to notice for some time and kept wondering what was going on.
The Python upgrade again affected my Pyblosxom install. Fortunately it was a minor problem; I just had to add a coding=utf-8 line to the beginning of my config.py to get it working.
dhcp3-server apparently restored its init scripts and got started again. This time, it got removed.

Such an ancient install will clearly have old, obsolete packages. I installed apt-show-versions to find out what didn’t match my package sources. I found I had every single version of cpp, gcc and g++ from 2.95 to 4.3, and a myriad of obsolete libs. But there were also real gems:

defrag 0.73pjm1-7 installed: No available version in archive
figlet 2.2.1-3 installed: No available version in archive
ipmasqadm 0.4.2-2 installed: No available version in archive
isapnptools 1.26-5 installed: No available version in archive
ms-sys 2.1.0-1 installed: No available version in archive
queso 0.980922b-3 installed: No available version in archive
update 2.11-4 installed: No available version in archive

Spaniards will remember “queso” because it was written by Jordi Murgó and became a classic tool to find out what OS was running on a remote host. “update” was apparently needed to flush your filesystems prior to Linux 2.2.8, and “defrag” is obvious, although leaves me wondering why it was needed at the time.

With the upgrade done successfully, next step is trying to get the system transfered to the spare hard drive. For this, I first partitioned it creating a primary partition using up more or less half of the available space, and setup a LVM volume, leaving some free PE’s in the volume group just in case I want to do snapshots in the future, and formatted it using ext3. I then transfered the system to the new disk and now face the boot challenge.

I haven’t created a boot partition and that should be a double problem: the BIOS is buggy and will only boot from the first 1024 cylinders, and my root is on LVM and GRUB legacy might not like it (but I’m not sure). However, I’ve become a big fan of GRUB2, and know I will be able to boot no matter what my BIOS thinks of my disks and regardless the complex root partition setup I throw at it. The plan is to install GRUB onto the new drive’s MBR, and set it up using the ata module, which should allow to ignore what the BIOS says, and read beyond cylinder 1024 or even boot from CD-ROM. However, this isn’t a setup I haven’t tried before, and a single failure will result in me taking a train to fix it on-site.

So, GRUB experts out there, any suggestions? Of course, for now I guess I can install GRUB in the current drive’s MBR and make it boot the old kernel using the new system as root, but that’s dirty and would just postpone the problem.

natura upgraded to etch

Wed, 25 Apr 2007 10:58:00 +0000

Last week I started the final round of Debian upgrades for the servers I maintain here and there, which is mostly complete today. I haven’t been so lucky with upgrades this time, for a long list of different reasons. In the end, the smoothest upgrades were those boxes I upgraded when etch froze or so.

natura.oskuro.net is the box serving these pages. It’s an old, extremely noisy Pentium 150 which I’ve been intending to replace for a while now. I started the upgrade early on Thursday, knowing it’d take a while (natura takes its time only to read the Dpkg database), and it had apparently finished when I was ready to leave the office.

Three issues:

apache2’s enabled modules were forgotten and I had to reenable them again, plus IIRC the default “It works!” site got reenabled, hiding the blog for a little while.
bind didn’t like some change in one of the default configs and I failed to noticed it had not started. After the reboot, the box was unreachable remotely, but that was OK as I had to go visit my dad that night for his birthday, so it was fixed after one hour or so.
Last, my blog stopped working, and while I thought it’d have something to do with apache2.2, I had no time to have a close look as I had to leave to Lleida this weekend. Yesterday I discovered that for some reason, the old tar.gz PyBlosxom install wasn’t liking the new python2.4, so I decided it was about time to move to PyBlosxom 1.3, from the Debian package. Not without the mandatory “It works!” post published in the gazillion Planets, natura was ready to go.

The very same night of last Thursday, I decided to dist-upgrade the box which serves the Spanish Debian website mirror. That’s the only purpose on the box, so you can imagine the upgrade should have been pretty straight-forward. And so it seemed, until, in the middle of unpacking, dpkg died with a horrible I/O error, and I dropped into an unusable remote terminal with no working commands. Fortunately, apache2 was still up and running, and the web service has been working without interruption since the hard drive crash, albeit with no syncs from www-master.

Today, Sergio visited the campus and had a look. It was a XFS crash, which got cleanly repaired using an install CD. We have an empty partition in the box, and will probably move the system to it temporarily, and back to the RAID, but on ext3. When the box was back online, I just had to resume the upgrade process, make mdadm happy and update lilo.conf before rebooting into the new kernel.

This box uses LILO for some obscure reason I can’t remember too clearly anymore. The box has just one partition on a md array, on two SCSI disks on a aic7xxx-based controller. Can anyone hint me why GRUB would have failed on us back in sarge, and if any fixes in the etch version would work any better? Using LILO here is error prone, and basically feels like a step back. Anyway, www.es.debian.org is now back up and running with updated content.

Sindominio.net had its bi-annual upgrading party last monday, but unfortunately I wasn’t able to help much as when I tried to log into the server, I must have caught the system in the middle of some key lib upgrade or something, and again I was locked in a unusable shell which would only segfault. Given my previous experience, I assumed that something had gone wrong and the box would need to be fixed at the console, and after 20 minutes I gave up helping on that front. Until I noticed, quite a long while later that I was still getting mail from the server. I managed to log in to discover the upgrade was done, with just a few bits remaining to be done. The major issues were encountered with our pam and ldap setup, plus nscd kept dying causing quite a lot of mayhem all over the place. Great work from Seajob, Syvic, nogates, apardo and the rest of the people who handled it! With etch, we can finally move back to an official Debian kernel, something we’ve been longing to do for a long time. The only pending upgrade issue is that we need to move from our old jabber server to either the traditional jabberd 1.x or ejabberd; our current implementation is no longer supported in Debian.

The last of the etch upgrades stories involves Sofcatalà’s servers. The box was running on a CentOS 4.4, which was moved away into a subdir just after booting Debian-Installer, and then lobotomised so it would run as a Linux-VServer under a new Debian etch install. I’ll probably write more details about it soon though, as it could be a maybe less scary alternative to Guillem’s debtakeover.

Yay for etch!

Silent home servers

Thu, 19 Oct 2006 17:36:00 +0000

The computer which hosts this blog is a venerable Pentium 150Mhz, with 64Mb of physical memory and two decently sized disks. It has been running non-stop mostly without hiccups for several years, and I’m quite happy with it, even if the processing power is so scarce I’ve been having to tune down some services as Debian has gotten more resource hungry, dist-upgrade after dist-upgrade.

Natura is my 2nd oldest Debian install, coming back from Ham, and after a while it became a home server when it was replaced by an Athlon 700Mhz at my father’s house. The only hardware incidents are all related to blackouts or storms: two dead disks and one power supply. The CPU died years ago, but I discovered that many months later. I guess it wasn’t so necessary. :)

It is time to replace natura, though. The components are aging and they have become quite noisy, despite my attempts to cleanup the dust. Lately it is so loud that I can’t understand how my dad can actually get work done with that persistent noise in the room. Besides, it’d be good to get just a little bit more of CPU power to do a few things that have been postponed for a while now. I have been looking for offerings in the embedded devices market.

I am looking for a device with the following characteristics:

Silent: this is a must. If fans aren’t involved, that’s great, but I know there are some devices with just one fan for the hard drives, etc. which are really silent too. Noise is the #1 reason I want to get a replacement.
CPU power and RAM: it doesn’t need to be too powerful, but of course an improvement over a Pentium 150 Mhz is expected. :) The minimal RAM would be 128Mb, I guess, and if it’s expandable/replaceable, that’d be a big plus.
Power consumption: I have other boxes around which I haven’t used to replace natura to get more CPU power because I’ve always assumed that it’d be hard to match the Pentium’s power consumption. As it’s up 24/7, I want it to be good in this area. AFAIK, the devices I’m looking for do quite well there, though.
Hard drives: many offerings accept two or four HDs inside the case. I won’t need four, but the possibility of setting up RAID is quite attractive.
Hackable: should be supported by GNU/Linux, and if d-i does a good job on it, bonus!
Price: last, but not least, I’m willing to spend some money on this, but I probably don’t aim for the most expensive devices…

I’ve found that the Thecus YES Box N2100 is one of the most interesting offerings: 2 Gigabit ethernet ports, two internal SATA HD bays, 3 USB ports… but is a bit too expensive: 350€ (without disks). tbm also told me to look at some cheaper PowerPC devices, but I forgot the name right now.

So, dear Lazyweb, what would you recommend as a natura replacement for a home server?

Hard drive failure

Sun, 20 Nov 2005 03:28:00 +0000

On Thursday, I went up to my father’s house to pick up my desktop, now that I finally have Internet access at home so I can have permanent access to it from a more convenient place. While I was there, I decided it’d be a good idea to do an upgrade on my dad’s box.

While I was doing this, the nfs share of the Debian mirror in the home server that serves this blog stopped working, and I couldn’t ssh in, although the console and ICMP appeared to work. Having no monitor attached to that box, I decided to reboot it and resolve the problem in a lame way. Had I known my primary hard drive had decided enough is enough, I would have looked for a monitor instead.

After reboot, the server started doing a very loud and scary noise, which was not new to me. When the BIOS tried to probe the disk, it would not start up and would instead whine like that. When this happened in the past, switching the box for a few minutes and trying again was enough. Not this time, though, it looked like it was the end.

Horrified by the fact that I had no really useful backup of /etc and /home, I tried over and over to get it working. I plugged the HD into another old AT box (which once served as my GNU Hurd playground), but I still got those scary noises. At least I knew it had nothing to do with the mother board.

As it was getting late, I decided to take the HD with me, and another Maxtor of the same size (but not exact model) with me, to have a look in office the next day. I thought I’d have to restort to try to transplant the logic board from the good one to the faulty disk, but before that, I realised I could try mounting it in a USB cage. For some reason or another, this worked, and I quickly saved thetwo partitions it contained with no read errors.

After this, I realised I had not done the copy as root, so I had lost all my ownerships. I plugged the disk again, and re-rsynced. It still worked. Alleviated, I went back home, copied the data to another old disk, and I don’t remember why, I tried mounting the faulty disk again. I got scary noises even using the USB case, and no other tries have been successful, so I think it’s completely dead now, just before my final mount which saved the data.

The box is now back online, using a Seagate disk I had stored in a drawer and with no notes on it about it having any kind of problem. I suspect I will need to do real backups now, because the drive isn’t as safe as it looked…

hda: dma_timer_expiry: dma status == 0x20
hda: DMA timeout retry
hda: timeout waiting for DMA
hda: status error: status=0x58 { DriveReady SeekComplete DataRequest }

hda: drive not ready for command
hda: status timeout: status=0xd0 { Busy }

hda: drive not ready for command
ide0: reset: success

My little P150 needs a 6GB drive. I’ll have to find one somewhere.

ADSL upgrade

Tue, 05 Oct 2004 10:30:00 +0000

Yesterday I wasn’t able to log into my home server from the office, and I assumed the load had skyrocketed again, as it happens every now and then. Timing was quite bad because I’m very busy in the evenings these days, but I went to my father’s house to see what was going on, and when I got there I saw the server was mostly idling. WTF? Shortly after I noticed my local named wouldn’t resolve barely anything, and I couldn’t ssh out, as the connection would hang in the middle of the handshakes. I started looking at my 3com router configuration, seemed ok; rebooted the box, nothing changed; started cursing, which changed nothing either… until I realized it was probably a telco thing. I told my father “it’ll probably be fixed automagically” and left the house.

When I came back from training, I managed to ssh in and quickly tested the downstream speed. As I suspected, the downtime was caused by Telefónica tweaking our stuff to upgrade the ADSL’s of the area from 256/128 to 512/128. Uplink still sucks, but oh well, we got this for “free” (ie, we still pay way to much for crappy connections in Spain, but we’re slowly getting what the rest of Europe seems to have).

Moreover, today the cable company finally opened up the street and installed their stuff to offer their service. 5 or 6 years late…

Hardware disaster

Sun, 05 Sep 2004 18:01:00 +0000

The big storms of the past days not only affected some trees and the cars below them. On Saturday I went to where my main server lives and discovered the primary hard drive had developed a new musical skill: every time the BIOS tried to access it it would reply with a funky loud sound I have never heard before (and no, it’s not the usual high pitch sound of a drive when it’s dying). After getting quite worried about those backups I never made, I managed to make it go quiet and boot again (that was just before I wrote the previous blog entry). I know I shouldn’t be trusting that drive much more, but I have no time to look for a replacement right now. I hope there are no more big storms in some time…

Besides that, my main Debian computer just would not react to the power switch at all. I first thought the power supply was completely fucked, but a few minutes ago I managed to get it back to life after fiddling inside the box, randomly unplugging and plugging power cords back and forth.

In short, it seems my computers are not in their best shape ever. This reminds me (again) that I really need to look for a UPS for the two boxes. Any cheap recommendation that works well with GNU/Linux? And of course, this is the signal that I fucking need to make backups of my stuff!

Aww, Erinn, when I wrote about the storm the other day I didn’t know that you were being evacuated due to Frances. I hope all is well!