From: Prince, Robert
Sent: Monday, November 15, 2004 3:42 PM
To: All/The Kingdom
Subject: RE: server1 & server2 going down for move
Server1 and server2 are back up. sshd on server1 took me a while to get working... the story goes like this:
Once upon a time, CVS over SSH was very very slow. Nobody knew why it was slow. Nobody even knew that it was SSH that was causing the problem. They just swore and groaned whenever they had to do anything with CVS. The whole kingdom suffered.
Then, one day server2's hard disk got fragged. A new hard disk was put in (although the old/bad one was mounted also, and used as swap which caused problems, but that's a horror story for another day), and the latest TopHat distribution was installed. CVS over SSH was magically faster on server2! In fact, it was almost normal!
The handsome Prince (who was very clever) realized that the slowness of CVS was due to the specific version of SSH that was installed on server2 before. He suggested to the IT ogre that perhaps CVS on server1 could also be close to normal in terms of speed if a newer version of SSH was installed, and suggested to the IT ogre that the latest SSH source be compiled and tried on server1.
The IT ogre complained and wailed that things didn't work that way in the World of Magical Windoze. He evaded the Prince for two full days before downloading the SSH source and building it to try the Prince's suggestion. Lo, it worked! Using CVS over SSH on server1 was now almost normal!
However, the next time server1 rebooted, sshd would not start! The villagers surrounded the server room, armed with pitchforks and torches. It sucked. The Prince found that sshd was looking for a directory in a specific user account - the IT ogre's account! The Prince cast the following spell: 'strings /usr/sbin/sshd | grep ogre'
and the output was thus:
/home/ogre/sshd/etc/sshd_config
/home/ogre/sshd/etc/ssh_host_key
/home/ogre/sshd/etc/shosts.equiv
/home/ogre/sshd/etc/ssh_known_hosts
/home/ogre/sshd/etc/ssh_host_rsa_key
/home/ogre/sshd/etc/ssh_host_dsa_key
/usr/bin:/bin:/usr/sbin:/sbin:/home/ogre/sshd/bin
/home/ogre/sshd/etc/sshrc
/bin/sh /home/ogre/sshd/etc/sshrc
/home/ogre/sshd/etc/ssh_known_hosts2
/home/ogre/sshd/etc/moduli
/home/ogre/sshd/etc/primes
It seemed that the IT ogre had compiled SSH in his home directory, and that info had been compiled into the binary. The Prince's friend, the wise wizard, pointed out to him that the IT ogre must have explicitly cast his spell of making such that it used '/home/ogre/sshd' as its install path, since by default it would want to create itself in /usr/local/sbin.
The Prince, aghast at such a twisted use of black magic, cast a healing spell of making which purified sshd, and the kingdom rested easy. Everyone that used CVS over SSH was content.
So, you'll notice the next time you do anything w/SSH on server1, the host keys have changed. Sorry.
The end.