FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu User

 
 
LinkBack Thread Tools
 
Old 01-01-2008, 09:00 PM
"Joris Dobbelsteen"
 
Default Server hacked?

Dear,

I'm having a Linux server running Ubuntu 6.06 LTS (under vmware) and it
seems to have some quite weird behaviour. For some reason (or another)
the box seems to create OUTGOING connections to an IRC server from a
supposed kernel address. Below is a snapshot of the netstat output...

Can anyone provide some detailed information about this?
(i.e. confirm my suspicion it is indeed a trojan or something)

If so, I'm desiring to do some more diagnosis/forensics on the box to
get to know what may caused this strange behaviour. Can anyone provide
some help on this?

The box has PostFix, PowerDNS, Apache2 and SSH exposed to the Internet.
Unfortunally its connected to the single LAN segment I have at home.
Fortunally I have a strict firewall that doesn't allow IRC out (I don't
use it, so I do not need to allow it).

Thanks in advance...

- Joris


root@shushan:~# netstat -tnp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 1 192.168.10.xx:52020 216.152.66.54:6667
SYN_SENT 18687/[kjournald]
tcp 0 1 192.168.10.xx:49383 216.152.66.48:6667
SYN_SENT 18599/[kjournald]
tcp 0 1 192.168.10.xx:32936 216.152.66.46:6667
SYN_SENT 11304/[kjournald]
tcp 0 1 192.168.10.xx:56901 216.152.67.49:6667
SYN_SENT 29965/[kjournald]
tcp 0 1 192.168.10.xx:54354 216.152.67.30:6667
SYN_SENT 24235/[kjournald]
tcp 0 1 192.168.10.xx:60278 216.152.66.47:6667
SYN_SENT 15412/[kjournald]
[trusted entries removed]


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-01-2008, 09:26 PM
johanb
 
Default Server hacked?

Joris,

There seems to be some kind of rootkit running on your server. It
concerns syn_sent , towars numbering devices, which inclines scans for
open dcc-servers, to sent packets to. What I would advice you, is to
install a package searching for installed rootkits and scan your system
with ex. chkrootkit or rkhunter.
To answer your question : yes according to my humble opinion it concerns
some kind of rootkit or trojan. (rather rootkit then trojan)
greetz,
Johan

Joris Dobbelsteen schreef:
> Dear,
>
> I'm having a Linux server running Ubuntu 6.06 LTS (under vmware) and it
> seems to have some quite weird behaviour. For some reason (or another)
> the box seems to create OUTGOING connections to an IRC server from a
> supposed kernel address. Below is a snapshot of the netstat output...
>
> Can anyone provide some detailed information about this?
> (i.e. confirm my suspicion it is indeed a trojan or something)
>
> If so, I'm desiring to do some more diagnosis/forensics on the box to
> get to know what may caused this strange behaviour. Can anyone provide
> some help on this?
>
> The box has PostFix, PowerDNS, Apache2 and SSH exposed to the Internet.
> Unfortunally its connected to the single LAN segment I have at home.
> Fortunally I have a strict firewall that doesn't allow IRC out (I don't
> use it, so I do not need to allow it).
>
> Thanks in advance...
>
> - Joris
>
>
> root@shushan:~# netstat -tnp
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> tcp 0 1 192.168.10.xx:52020 216.152.66.54:6667
> SYN_SENT 18687/[kjournald]
> tcp 0 1 192.168.10.xx:49383 216.152.66.48:6667
> SYN_SENT 18599/[kjournald]
> tcp 0 1 192.168.10.xx:32936 216.152.66.46:6667
> SYN_SENT 11304/[kjournald]
> tcp 0 1 192.168.10.xx:56901 216.152.67.49:6667
> SYN_SENT 29965/[kjournald]
> tcp 0 1 192.168.10.xx:54354 216.152.67.30:6667
> SYN_SENT 24235/[kjournald]
> tcp 0 1 192.168.10.xx:60278 216.152.66.47:6667
> SYN_SENT 15412/[kjournald]
> [trusted entries removed]
>
>
>


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-01-2008, 09:48 PM
NoOp
 
Default Server hacked?

On 01/01/2008 02:00 PM, Joris Dobbelsteen wrote:

>
> The box has PostFix, PowerDNS, Apache2 and SSH exposed to the Internet.
> Unfortunally its connected to the single LAN segment I have at home.
> Fortunally I have a strict firewall that doesn't allow IRC out (I don't
> use it, so I do not need to allow it).
>
[snips]
> tcp 0 1 192.168.10.xx:60278 216.152.66.47:6667
> SYN_SENT 15412/[kjournald]
> [trusted entries removed]
>
>

You have been hacked. There are a variety of trojans (linx related) that
use port 6667:

http://www.cert.org/advisories/CA-2002-24.html
<http://www.google.com/search?hl=en&q=Linux+trojan+%2B6667&btnG=Search>
<http://www.symantec.com/security_response/writeup.jsp?docid=2006-021417-0144-99&tabid=2>
<http://www.doshelp.com/Ports/6667.htm>

Is the system fully updated with all the recent Ubuntu patches/updates?
If so, you may want to contact the Ubuntu security team to let them know
and have them take a look.
https://launchpad.net/~ubuntu-security
https://bugs.launchpad.net/~ubuntu-security/
https://bugs.launchpad.net/debian/+source/ircii-pana/+bug/129771
[remote IRC servers can execute arbitrary commands]








--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-01-2008, 10:40 PM
Hal Burgiss
 
Default Server hacked?

On Tue, Jan 01, 2008 at 10:00:19PM -0000, Joris Dobbelsteen wrote:
> I'm having a Linux server running Ubuntu 6.06 LTS (under vmware) and it
> seems to have some quite weird behaviour. For some reason (or another)
> the box seems to create OUTGOING connections to an IRC server from a
> supposed kernel address. Below is a snapshot of the netstat output...
>
> Can anyone provide some detailed information about this?
> (i.e. confirm my suspicion it is indeed a trojan or something)
> SYN_SENT 15412/[kjournald]

Google on kjournald. The box is cracked and likely rooted.

--
Hal


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-01-2008, 11:35 PM
"Joris Dobbelsteen"
 
Default Server hacked?

Johan, NoOp,

Thanks for your comments.
It seems I can be quite lucky, as the damage seems to be rather
contained to a very limited set of my system. The processes are of the
user www-data. So it seems a web site has been hacked instead. (Count
myself lucky this time)

Evidence:
root@shushan:/proc/29965# ls -l
total 0
[snip]
lrwxrwxrwx 1 www-data www-data 0 2008-01-01 23:07 exe -> /usr/bin/perl
[snip]
root@shushan:/proc/29965# ls fd -l
total 10
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 0 -> socket:[543920]
l-wx------ 1 www-data www-data 64 2008-01-01 23:01 1 -> pipe:[543929]
l-wx------ 1 www-data www-data 64 2008-01-01 23:01 2 ->
/var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 3 -> /tmp/.apc.TeL4il
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 4 -> /tmp/.apc.MpL1Yi
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 5 -> /tmp/.apc.2d14Oh
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 6 -> /tmp/.apc.5zT9Eg
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 7 -> /tmp/.apc.Njjgvf
(deleted)
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 8 -> socket:[525267]
lrwx------ 1 www-data www-data 64 2008-01-01 23:01 9 -> socket:[784974]

It seems it limited to the www-data user. Of course the fd/3 file gives
a very good hint that it is indeed apache. The apc comes from PHP. Of
course its odd that the webpage runs perl, since all processes are
supposed to be PHP.
Up to a certain point, this restores my faite (hope) that the system can
be trusted (up to some extend).

Some oddities are these on a site running Joomla are:
===
htdocs/.wonk/motd/USER1.MOTD.old::arcor.de.eu.dal.net 372 _mugo1`dealz
:- You can always reach this server by typing /server arcor.dal.net
6667
htdocs/.wonk/motd/USER2.MOTD.old::arcor.de.eu.dal.net 372 [X]enzo :-
You can always reach this server by typing /server arcor.dal.net 6667
htdocs/.wonk/src/p_client.c:
ap_snprintf(irccontent,sizeof(irccontent),"6667");
htdocs/.wonk/src/p_inifunc.c: if (ern != 0) { user(usernum)->port =
6667; } else { user(usernum)->port = atoi(value); }
Binary file htdocs/.wonk/src/p_client.o matches
Binary file htdocs/.wonk/vi matches
htdocs/dimenso:my $porta="6667";
htdocs/dimenso.1:my $porta="6667";
htdocs/scanner.pl:my $porta="6667";
===

It also has wonk.tar.gz from 2007-03-18.

Anyone familiar with this?

I hope to diagnose when the incident occurred an how to protect against
it better in the future.


At least there are some lessons in this:

* Use one-user-per-website only (easier auditing).
* Deploy you firewall with strict rules.
* Do auditing & automated monitoring.
* Keep longer logs if you don't automatically monitor your systems.
* Keep ALL software up to date (something automatic for websites?)

* Its good policy to deny traffic, except if required for system/website
operation.


I'll be moving all stuff to a new box with Xen and more isolation, these
seem good lessons to get started. Still it seems to be quite a lot of
work for doing it right (or at least better) the second time. Any
suggestions from experienced people that might help me?


Thanks,


- Joris

>-----Original Message-----
>From: ubuntu-users-bounces@lists.ubuntu.com
>[mailto:ubuntu-users-bounces@lists.ubuntu.com] On Behalf Of johanb
>Sent: Tuesday, 1 January 2008 23:26
>To: Ubuntu user technical support,not for general discussions
>Subject: Re: Server hacked?
>
>
>Joris,
>
>There seems to be some kind of rootkit running on your server.
>It concerns syn_sent , towars numbering devices, which
>inclines scans for open dcc-servers, to sent packets to. What
>I would advice you, is to install a package searching for
>installed rootkits and scan your system with ex. chkrootkit or
>rkhunter.
>To answer your question : yes according to my humble opinion
>it concerns some kind of rootkit or trojan. (rather rootkit
>then trojan) greetz, Johan
>
>Joris Dobbelsteen schreef:
>> Dear,
>>
>> I'm having a Linux server running Ubuntu 6.06 LTS (under vmware) and
>> it seems to have some quite weird behaviour. For some reason (or
>> another) the box seems to create OUTGOING connections to an
>IRC server
>> from a supposed kernel address. Below is a snapshot of the
>netstat output...
>>
>> Can anyone provide some detailed information about this?
>> (i.e. confirm my suspicion it is indeed a trojan or something)
>>
>> If so, I'm desiring to do some more diagnosis/forensics on
>the box to
>> get to know what may caused this strange behaviour. Can
>anyone provide
>> some help on this?
>>
>> The box has PostFix, PowerDNS, Apache2 and SSH exposed to
>the Internet.
>> Unfortunally its connected to the single LAN segment I have at home.
>> Fortunally I have a strict firewall that doesn't allow IRC out (I
>> don't use it, so I do not need to allow it).
>>
>> Thanks in advance...
>>
>> - Joris
>>
>>
>> root@shushan:~# netstat -tnp
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State PID/Program name
>> tcp 0 1 192.168.10.xx:52020 216.152.66.54:6667
>> SYN_SENT 18687/[kjournald]
>> tcp 0 1 192.168.10.xx:49383 216.152.66.48:6667
>> SYN_SENT 18599/[kjournald]
>> tcp 0 1 192.168.10.xx:32936 216.152.66.46:6667
>> SYN_SENT 11304/[kjournald]
>> tcp 0 1 192.168.10.xx:56901 216.152.67.49:6667
>> SYN_SENT 29965/[kjournald]
>> tcp 0 1 192.168.10.xx:54354 216.152.67.30:6667
>> SYN_SENT 24235/[kjournald]
>> tcp 0 1 192.168.10.xx:60278 216.152.66.47:6667
>> SYN_SENT 15412/[kjournald]
>> [trusted entries removed]
>>
>>
>>
>
>
>--
>ubuntu-users mailing list
>ubuntu-users@lists.ubuntu.com
>Modify settings or unsubscribe at:
>https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-02-2008, 12:42 AM
Res
 
Default Server hacked?

On Wed, 2 Jan 2008, Joris Dobbelsteen wrote:

> contained to a very limited set of my system. The processes are of the
> user www-data. So it seems a web site has been hacked instead. (Count

Your more important priority is to locate how they got in, else fixing the
system is pointless.

Do you run php, if so what type of programs? Gallery? phpnuke?

> At least there are some lessons in this:
> * Use one-user-per-website only (easier auditing).

Good idea...

Dirs should be 710 for htdocs root
eg: chmod 710 /var/www/vhosts
chmod 710 /var/www/vhosts/example.com
chmod 710 /var/www/vhosts/example.net

Ensure the users who own those domains are the only ones with access,
except group must be web server.
eg: chown -R jack.apache /var/www/vhosts/example.com
chown -R jill.apache /var/www/vhosts/example.net


Use suexec in every virtualhost block in Apache
eg: SuexecUserGroup jack apache

and lock down php... eg:
open_basedir =/var/www:/tmp:/usr/local/lib/php

disable_functions = exec, shell_exec, system, virtual, show_source,
readfile, passthru, escapeshellcmd, popen, pclose, phpinfo


--
Cheers
Res

mysql> update auth set Framed-IP-Address='127.0.0.127' where user= 'troll';

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-02-2008, 01:41 AM
NoOp
 
Default Server hacked?

On 01/01/2008 04:35 PM, Joris Dobbelsteen wrote:
> Johan, NoOp,
>
> Thanks for your comments.
> It seems I can be quite lucky, as the damage seems to be rather
> contained to a very limited set of my system. The processes are of the
> user www-data. So it seems a web site has been hacked instead. (Count
> myself lucky this time)
>

I wouldn't be so sure that you are all that lucky... you never know what
else may have been installed in the process.

In the Windows world I was actually _very_ good at tracking and
eradicating every type of worm, trojan, virus etc., on a system. I took
pride in being able to clean all but one customer system (and on that
one he'd screwed up the drive so bad that it was just cheaper to throw
it out). However, being relatively new to linux (about 1.5 year) I
wouldn't even begin to try and guess what else may have been compromised
on your system. Others (such as Res?) are much more qualified.

However, here are some links that may be of help:

http://ubuntuforums.org/showthread.php?t=510812
https://help.ubuntu.com/community/Security
http://secunia.com/product/12470/?task=statistics

My first thought would be to immediately disconnect the server from
_any_ network until you are confident that you have actually cleaned the
system.


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-02-2008, 02:37 AM
Hal Burgiss
 
Default Server hacked?

On Wed, Jan 02, 2008 at 12:35:23AM -0000, Joris Dobbelsteen wrote:
> It seems it limited to the www-data user. Of course the fd/3 file gives
> a very good hint that it is indeed apache. The apc comes from PHP. Of
> course its odd that the webpage runs perl, since all processes are
> supposed to be PHP.

Possibly its scanner.pl listed below. Probably a port scanner written
in perl for systems that don't come ready equipped with nmap, etc.

> Some oddities are these on a site running Joomla are:
> ===
> htdocs/.wonk/motd/USER1.MOTD.old::arcor.de.eu.dal.net 372 _mugo1`dealz
> :- You can always reach this server by typing /server arcor.dal.net
> 6667

USER1.MOTD.old is the irc bouncer logs. Worth looking at to see how
active this attack was. It might have been purely scripted and the
attacker maybe never did much with it. But then again ...

> Binary file htdocs/.wonk/vi matches
> htdocs/dimenso:my $porta="6667";
> htdocs/dimenso.1:my $porta="6667";

The man page for dimenso?

> htdocs/scanner.pl:my $porta="6667";

A port scanner for scanning other systems.

> It also has wonk.tar.gz from 2007-03-18.

I agree that the most important thing is to find how they got in. Even
if you do a clean install, tighten up everything, but put the same php
applications back, you might still be just as vulnerable.

Another thing to do is to look at the timestamps for things you have
found that you know don't belong. Then look around the rest of the
filesystem for things with the same, or close, timestamps. There may
be other back doors installed scattered around. There may be cron jobs
hidden to restart things that die or are MIA. [kjournald] is something
bogus that was built from source. Where are the pieces to that? If you
have logs that far back, go through those for the date/time and see
what was uploaded and *where*. This was probably done all through
Apache, so the logs should have the evidence (if they exist still).
The logs will probably give you a good idea of exactly how they got
in. And you might get lucky and see what else was accessed (if there
was any sensitive data on the system like in mysql data files, etc).
They almost surely took /etc/passwd, which doesn't give them passwords
but does give them user names, making brute force attacks much simpler
and much more effective.

--
Hal


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-02-2008, 03:07 AM
NoOp
 
Default Server hacked?

On 01/01/2008 07:37 PM, Hal Burgiss wrote:
> On Wed, Jan 02, 2008 at 12:35:23AM -0000, Joris Dobbelsteen wrote:

>
>> It also has wonk.tar.gz from 2007-03-18.
>
> I agree that the most important thing is to find how they got in. Even
> if you do a clean install, tighten up everything, but put the same php
> applications back, you might still be just as vulnerable.
>

*That is the key* to cleaning any system. I can't begin to tell you how
many customer systems I've run across in the past (admittedly Windows)
whereby the customer would reformat, clean, etc., etc., and then reload
the same data back onto the drive from a recent backup that had the
infected data on it. I'd try to explain that even if they installed a
new hard drive the problem would come back every time that they
reinstalled the old infected data.



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 01-02-2008, 12:42 PM
"Joris Dobbelsteen"
 
Default Server hacked?

>-----Original Message-----
>From: ubuntu-users-bounces@lists.ubuntu.com
>[mailto:ubuntu-users-bounces@lists.ubuntu.com] On Behalf Of Res
>Sent: Wednesday, 2 January 2008 2:43
>To: Ubuntu user technical support,not for general discussions
>Subject: RE: Server hacked?
>
>
>On Wed, 2 Jan 2008, Joris Dobbelsteen wrote:
>
>> contained to a very limited set of my system. The processes
>are of the
>> user www-data. So it seems a web site has been hacked instead. (Count
>
>Your more important priority is to locate how they got in,
>else fixing the system is pointless.
>
>Do you run php, if so what type of programs? Gallery? phpnuke?

The exploit was found. System runs PHP with Joomla.
It seems there is an exploit here.

>> At least there are some lessons in this:
>> * Use one-user-per-website only (easier auditing).
>
>Good idea...
>
>Dirs should be 710 for htdocs root
> eg: chmod 710 /var/www/vhosts
> chmod 710 /var/www/vhosts/example.com
> chmod 710 /var/www/vhosts/example.net
>
>Ensure the users who own those domains are the only ones with
>access, except group must be web server.
> eg: chown -R jack.apache /var/www/vhosts/example.com
> chown -R jill.apache /var/www/vhosts/example.net
>
>
>Use suexec in every virtualhost block in Apache
> eg: SuexecUserGroup jack apache

I'm still failing to see how this provides security and what the
implications are. I'm also a bit puzzled how suexec affects file
accesses (those without scripts). I did use CGI and not the webserver
loadable PHP library but didn't get suexec to work to my liking.

>and lock down php... eg:
> open_basedir =/var/www:/tmp:/usr/local/lib/php
>
>disable_functions = exec, shell_exec, system, virtual,
>show_source, readfile, passthru, escapeshellcmd, popen, pclose, phpinfo

Doesn't this break a lot of application? From what I know, at least
Gallery2 does execute shell commands...

Sincerely,

- Joris


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 

Thread Tools




All times are GMT. The time now is 02:45 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org