FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 08-09-2010, 04:25 PM
Paul Hartman
 
Default Rooted/compromised Gentoo, seeking advice

Hi, today when working remotely I ran nethogs and noticed suspicious
network traffic coming from my home gentoo box. It was very low
traffic (less than 1KB/sec bandwidth usage) but according to nethogs
it was between a root user process and various suspicious-looking
ports on outside hosts in other countries that I have no business
with. netstat didn't show anything, however, but when I ran chkrootkit
told me that netstat was INFECTED. I immediately issued "shutdown -h
now" and now I won't be able to take a further look at it until I get
home and have physical access to the box. System uptime was a few
months. It was last updated for installation of a 2.6.33 kernel
(2.6.35 is out now).

I have 3 goals now:

1) Figure out what is running on my box and how long it has been there.
2) Find out how it got there.
3) Sanitizing, or most likely rebuilding the system from scratch.

I won't feel comfortable about doing item 3 until I learn the cause of
1 and 2. Since this is a home PC, it's not mission-critical and I have
other computers so I can afford to leave it offline while I
investigate this security breach, but at the same time it's worrisome
because I do banking etc from this machine. I'll obviously have to
check the status of any other computer on the same network.

My user account has sudo-without-password rights to any command. In
hindsight this risk may not be worth the extra convenience... A rogue
"sudo install-bad-stuff" anywhere over time could have done me in.

Alternatively I was running vulnerable/compromised software. My box
has sshd running, root login in ssh is not allowed, and pubkey only
logins (no passwords). It is behind a wireless router but port 22 is
open and pointing to this box, and a few others needed by other
applications. So I will check out which keys exist on the compromised
machine and make sure I recognize them all. I'll also need to check
the status of any other computer my key is stored on (a mix of linux &
windows, and my mobile phone). Sigh...

I am using ~amd64 and I update deep world about 3 times a week normally.

The computer is only a few months old, but it was created by cloning a
~2-years-old computer. I did emerge -e world as part of the upgrade
process.

If anyone has advice on what I should look at forensically to
determine the cause of this, it is appreciated. I'll first dig into
the logs, bash history etc. and really hope that this very happened
recently.

Thanks for any tips and wish me good luck.
 
Old 08-09-2010, 04:48 PM
Alan McKinnon
 
Default Rooted/compromised Gentoo, seeking advice

On Monday 09 August 2010 18:25:56 Paul Hartman wrote:
> Hi, today when working remotely I ran nethogs and noticed suspicious
> network traffic coming from my home gentoo box. It was very low
> traffic (less than 1KB/sec bandwidth usage) but according to nethogs
> it was between a root user process and various suspicious-looking
> ports on outside hosts in other countries that I have no business
> with. netstat didn't show anything, however, but when I ran chkrootkit
> told me that netstat was INFECTED. I immediately issued "shutdown -h
> now" and now I won't be able to take a further look at it until I get
> home and have physical access to the box. System uptime was a few
> months. It was last updated for installation of a 2.6.33 kernel
> (2.6.35 is out now).
>
> I have 3 goals now:
>
> 1) Figure out what is running on my box and how long it has been there.
> 2) Find out how it got there.
> 3) Sanitizing, or most likely rebuilding the system from scratch.

Here's the bad news:

An intruder probably gained access through a script kiddie script, which has
likely already removed all the logs. Or they have possibly been rotated away
by now.

I would proceed as follows:

1. Keep that machine off the internet till it is reinstalled
2. Fresh reinstall using boot media that you have downloaded and written
elsewhere, plus a portage tree. Don't worry about distfiles - a fresh portage
tree won't use existing copies on that machine if the hashes don't match. So
you can re-use them. If you boot off new install media it is safe to download
new distfiles using it.
3. Keep your old partitions around if you want to do forensics, you can mount
them somewhere when a reinstall is done and peruse them at your leisure.
However, doing that is often a waste of time unless you still have logs. You
can use a scanner like nessus to look things over.
4. And it goes without saying that you should change all passwords and keys
used on that trojaned machine.




> I won't feel comfortable about doing item 3 until I learn the cause of
> 1 and 2. Since this is a home PC, it's not mission-critical and I have
> other computers so I can afford to leave it offline while I
> investigate this security breach, but at the same time it's worrisome
> because I do banking etc from this machine. I'll obviously have to
> check the status of any other computer on the same network.
>
> My user account has sudo-without-password rights to any command. In
> hindsight this risk may not be worth the extra convenience... A rogue
> "sudo install-bad-stuff" anywhere over time could have done me in.
>
> Alternatively I was running vulnerable/compromised software. My box
> has sshd running, root login in ssh is not allowed, and pubkey only
> logins (no passwords). It is behind a wireless router but port 22 is
> open and pointing to this box, and a few others needed by other
> applications. So I will check out which keys exist on the compromised
> machine and make sure I recognize them all. I'll also need to check
> the status of any other computer my key is stored on (a mix of linux &
> windows, and my mobile phone). Sigh...
>
> I am using ~amd64 and I update deep world about 3 times a week normally.
>
> The computer is only a few months old, but it was created by cloning a
> ~2-years-old computer. I did emerge -e world as part of the upgrade
> process.
>
> If anyone has advice on what I should look at forensically to
> determine the cause of this, it is appreciated. I'll first dig into
> the logs, bash history etc. and really hope that this very happened
> recently.
>
> Thanks for any tips and wish me good luck.

--
alan dot mckinnon at gmail dot com
 
Old 08-09-2010, 06:48 PM
Paul Hartman
 
Default Rooted/compromised Gentoo, seeking advice

On Mon, Aug 9, 2010 at 11:48 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On Monday 09 August 2010 18:25:56 Paul Hartman wrote:
>> Hi, today when working remotely I ran nethogs and noticed suspicious
>> network traffic coming from my home gentoo box. It was very low
>> traffic (less than 1KB/sec bandwidth usage) but according to nethogs
>> it was between a root user process and various suspicious-looking
>> ports on outside hosts in other countries that I have no business
>> with. netstat didn't show anything, however, but when I ran chkrootkit
>> told me that netstat was INFECTED. I immediately issued "shutdown -h
>> now" and now I won't be able to take a further look at it until I get
>> home and have physical access to the box. System uptime was a few
>> months. It was last updated for installation of a 2.6.33 kernel
>> (2.6.35 is out now).
>>
>> I have 3 goals now:
>>
>> 1) Figure out what is running on my box and how long it has been there.
>> 2) Find out how it got there.
>> 3) Sanitizing, or most likely rebuilding the system from scratch.
>
> Here's the bad news:
>
> An intruder probably gained access through a script kiddie script, which has
> likely already removed all the logs. Or they have possibly been rotated away
> by now.
>
> I would proceed as follows:
>
> 1. Keep that machine off the internet till it is reinstalled
> 2. Fresh reinstall using boot media that you have downloaded and written
> elsewhere, plus a portage tree. Don't worry about distfiles - a fresh portage
> tree won't use existing copies on that machine if the hashes don't match. So
> you can re-use them. If you boot off new install media it is safe to download
> new distfiles using it.
> 3. Keep your old partitions around if you want to do forensics, you can mount
> them somewhere when a reinstall is done and peruse them at your leisure.
> However, doing that is often a waste of time unless you still have logs. You
> can use a scanner like nessus to look things over.
> 4. And it goes without saying that you should change all passwords and keys
> used on that trojaned machine.

Hi Alan, thanks for the advice.

I just remembered that my DD-WRT router stats page had an anomaly, on
31st of July it showed I had over 700 terabytes of traffic, which is
impossible. Coincidentally, my cable modem stopped working on the same
day, so I wrote it off as a bug or a result of the broken modem. I
replaced the modem and everything seemed to work normally after that.

At this point my mind is running wild thinking of all of the
possibilities. Could the router have been infected? The modem? It'll
still be another 5 or 6 hours before I'm able to lay my hands on the
machine. I'm imagining every doomsday scenario.

My hope is that it was "only" a botnet or ssh-scanner or something,
and not sniffer or keylogger or anything nefarious. I fear I may never
truly be able to know, though.
 
Old 08-09-2010, 06:59 PM
7v5w7go9ub0o
 
Default Rooted/compromised Gentoo, seeking advice

On 08/09/10 12:25, Paul Hartman wrote:
[]
> If anyone has advice on what I should look at forensically to
> determine the cause of this, it is appreciated. I'll first dig into
> the logs, bash history etc. and really hope that this very happened
> recently.
>
> Thanks for any tips and wish me good luck.

AntiVir (Avira) anti-malware scanner has hundreds of Linux rootkit/virus
signatures; you might scan your box with that. It has an on-access,
realtime monitor option as well, which I use it to monitor anything
downloaded and or compiled on my box (in case the distribution screen
gets hacked).

<http://www.free-av.com/en/download/download_servers.php>

Presuming you're rooted, you might first try their stand-alone, linux
live-disk scanner so as to avoid borked kernel and/or core utilities:

<http://www.free-av.com/en/tools/12/avira_antivir_rescue_system.html>
 
Old 08-09-2010, 07:08 PM
Paul Hartman
 
Default Rooted/compromised Gentoo, seeking advice

On Mon, Aug 9, 2010 at 1:59 PM, 7v5w7go9ub0o <7v5w7go9ub0o@gmail.com> wrote:
> On 08/09/10 12:25, Paul Hartman wrote:
> []
>> If anyone has advice on what I should look at forensically to
>> determine the cause of this, it is appreciated. I'll first dig into
>> the logs, bash history etc. and really hope that this very happened
>> recently.
>>
>> Thanks for any tips and wish me good luck.
>
> AntiVir (Avira) anti-malware scanner has hundreds of Linux rootkit/virus
> signatures; you might scan your box with that. It has an on-access,
> realtime monitor option as well, which I use it to monitor anything
> downloaded and or compiled on my box (in case the distribution screen
> gets hacked).
>
> <http://www.free-av.com/en/download/download_servers.php>
>
> Presuming you're rooted, you might first try their stand-alone, linux
> live-disk scanner so as to avoid borked kernel and/or core utilities:
>
> <http://www.free-av.com/en/tools/12/avira_antivir_rescue_system.html>

Was not aware of that one, I'll give it a try. Thanks.
 
Old 08-09-2010, 07:09 PM
Mick
 
Default Rooted/compromised Gentoo, seeking advice

On Monday 09 August 2010 17:25:56 Paul Hartman wrote:
> My user account has sudo-without-password rights to any command.

Ouch!

There have been discussions on this list why sudo is a bad idea and sudo on
*any* command is an even worse idea. You might as well be running everything
as root, right?

You have decided wisely to reinstall because you can't be sure of this OS
anymore.

Please keep us updated on what you find from the forensic analysis.
--
Regards,
Mick
 
Old 08-09-2010, 07:46 PM
Mick
 
Default Rooted/compromised Gentoo, seeking advice

On Monday 09 August 2010 19:59:11 7v5w7go9ub0o wrote:
> On 08/09/10 12:25, Paul Hartman wrote:
> []
>
> > If anyone has advice on what I should look at forensically to
> > determine the cause of this, it is appreciated. I'll first dig into
> > the logs, bash history etc. and really hope that this very happened
> > recently.
> >
> > Thanks for any tips and wish me good luck.
>
> AntiVir (Avira) anti-malware scanner has hundreds of Linux rootkit/virus
> signatures; you might scan your box with that. It has an on-access,
> realtime monitor option as well, which I use it to monitor anything
> downloaded and or compiled on my box (in case the distribution screen
> gets hacked).
>
> <http://www.free-av.com/en/download/download_servers.php>
>
> Presuming you're rooted, you might first try their stand-alone, linux
> live-disk scanner so as to avoid borked kernel and/or core utilities:
>
> <http://www.free-av.com/en/tools/12/avira_antivir_rescue_system.html>

Another idea to help with your forensics would be to bring a netstat and lsof
binary over to your machine and run them to see which actors are running and
trying to get out. That could help you detect what is running on that machine
and google your way from there.

You could also run rkhunter.
--
Regards,
Mick
 
Old 08-09-2010, 08:08 PM
Robert Bridge
 
Default Rooted/compromised Gentoo, seeking advice

On Mon, Aug 9, 2010 at 8:09 PM, Mick <michaelkintzios@gmail.com> wrote:
> There have been discussions on this list why sudo is a bad idea and sudo on
> *any* command is an even worse idea. You might as well be running everything
> as root, right?

sudo normally logs the command executed, and the account which
executes it, so while not relevant for single user systems, it STILL
has benefits over running as root.

RobbieAB
 
Old 08-09-2010, 08:20 PM
Bill Longman
 
Default Rooted/compromised Gentoo, seeking advice

On 08/09/2010 01:08 PM, Robert Bridge wrote:
> On Mon, Aug 9, 2010 at 8:09 PM, Mick <michaelkintzios@gmail.com> wrote:
>> There have been discussions on this list why sudo is a bad idea and sudo on
>> *any* command is an even worse idea. You might as well be running everything
>> as root, right?
>
> sudo normally logs the command executed, and the account which
> executes it, so while not relevant for single user systems, it STILL
> has benefits over running as root.

...excepting, of course, "sudo bash -l" which means you've given away
the keys to the kingdom.
 
Old 08-09-2010, 08:25 PM
Dale
 
Default Rooted/compromised Gentoo, seeking advice

Robert Bridge wrote:

On Mon, Aug 9, 2010 at 8:09 PM, Mick<michaelkintzios@gmail.com> wrote:


There have been discussions on this list why sudo is a bad idea and sudo on
*any* command is an even worse idea. You might as well be running everything
as root, right?


sudo normally logs the command executed, and the account which
executes it, so while not relevant for single user systems, it STILL
has benefits over running as root.

RobbieAB




I don't use sudo here but I assume a admin would only know that a nasty
command has been ran well after it was ran? Basically, after the damage
has been done, you can go look at the logs and see the mess some hacker
left behind. For me, that isn't a whole lot of help. You still got
hacked, you still got to reinstall and check to make sure anything you
copy over is not infected.


Assuming that they can erase dmesg, /var/log/messages and other log
files, whose to say the sudo logs aren't deleted too? Then you still
have no records to look at.


I agree with the other posters tho, re-install from scratch and re-think
your security setup.


Dale

:-) :-)
 

Thread Tools




All times are GMT. The time now is 05:53 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org