[oclug] a rooted system

[Will Muncy]

    Will be interested to hear the solution to this one!






-Will
I will not forget the face of my Father

  -----Original Message-----
  From: oclug-admin at lists.oclug.on.ca
[mailto:oclug-admin at lists.oclug.on.ca]On Behalf Of Billy Omer
  Sent: April 23, 2001 11:53 PM
  To: oclug at lists.oclug.on.ca
  Subject: [oclug] a rooted system


  Hey all,
  First, a little back ground.

  I just started working for this .com out in (you guest it) Silicon Valley
as their systems administrator.  So basically it's my job to take care of
the cluster fsck they created.

  This morning, my first day back from the Valley (spent a week there with
them doing a number different things), I got a call saying that ssl is not
working on one of their servers.  So I woke up (jet lag will kick your butt
I found out) and noticed that the machine was rebooted around 7am and ssl
was not started because it was sitting there waiting for the passphrase to
be entered (a problem that could only be fixed with an 'expect' script).
Very strange that the machine was rebooted.

  Now, here's the deal

  The machine as rebooted at 4 minutes before 7am PST.  According to syslog,
all users logged in to ssh logged off by 19:38 PST the night before.  fsck
was not invoked, and the other server that is on the same ups was not
rebooted, so that ruled out power failure.  The only other possibilities
were someone rebooted the machine at the console, which is highly doubtful
since the machine is in a locked cabinet in a secure colocation facility, or
someone remotely rebooted the machine without it being logged (either by an
exploit with ftp {changing the runlevel, running the reboot or shutdown
script etc..} or an unlogged ssh connection).

  I then noticed that there was no .bash_history file for root.  I 'touch
.bash_history' and then logged out and back in.  The .bash_history file was
gone.  So, that set off the big red light that something was very wrong here
and finally gave me enough reason to confirm my earlier suspicions of the
machine being rooted (there has been a number of odd problems happening
prior to this, and the owner of the company said that they were compromised
a few months back, I just found that out tonight).

  The only thing I could think of that they did to disable the .bash_history
file from keeping a history of issued commands by the root user was a
modified /bin/bash file.  Since this is a Red Hat machine, I installed the
rpm for bash from the Red Hat ftp site and removed the other version that
was installed.  That did not fix the problem.  However, here is what I
noticed:
  I 'ftp localhost' and login as anonymous.  I then drop to a shell using !,
issue a command or two, the .bash_history suddenly appears and starts to log
the commands I issued.  I exit the subshell and quit out of ftp, the
.bash_history file is still there, but is no longer logging.

  Does anyone have any idea what they could have done to do this?  Any idea
how to fix this?
  Yes I did change the root password, and secured ftp as much as possible,
as well as making double-damn sure nothing was running that shouldn't be.

  The only true fix I can thing of is to wipe and start clean.  However, the
owner feels that is not an option.  There would be a lot of data that would
have to be backed up and restored, and that data it's self could contain
trojains, however I doubt that.

  I would like to wipe the system and do a fresh install (and of Slackware
or Debian, not RH), but I doubt I'll be able to do that.  So, I'm stuck with
having to take care of this and making sure this wont happen again.   Anyone
have any advice or suggestions that I can try?

  Basically all that the system runs is run mysql, sendmail (hardly used),
apache (with mod_ssl), ssh1 and 2, perl and offer cvs services for the dev
team.


  --Bill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://tux.oclug.on.ca/pipermail/oclug/attachments/20010423/281cd4a3/attachment.htm