[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Full-disclosure] one of my servers has been compromized
- To: mitchell <mitchell@xxxxxx>
- Subject: Re: [Full-disclosure] one of my servers has been compromized
- From: "Larry W. Cashdollar" <larry0@xxxxxx>
- Date: Mon, 05 Dec 2011 14:13:13 +0000 (GMT)
<html><body><div>Hi,<br><br>I'd say tell your boss your application has been
compromised right away. Tell them you'll need to rebuild the
entire system from scratch and they'll need to either devise an upgrade path
for virtuemart or find a new ecommerce solution.<br><br>You can't trust a
system once it has been compromised. <br><br>-- larry
C$<br><br><br></div><div><br>On Dec 05, 2011, at 05:35 AM, mitchell
<mitchell@xxxxxx> wrote:<br><br><div><blockquote type="cite"><div
class="msg-quote"><div><span style="color: #222222; font-family:
arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial,
sans-serif">Hi,</span></div><div><span style="color: #222222; font-family:
arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial,
sans-serif"><br></span></div><div><span style="color: #222222; font-family:
arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif">Here is what you
generally need to do in such cases.</span></div><div><span style="color:
#222222; font-family: arial,sans-serif;" data-mce-style="color: #222222;
font-family: arial,sans-serif;" color="#222222" face="arial, sans-serif">1.
Suspend the webapp until you investigate.</span></div><div><span style="color:
#222222; font-family: arial,sans-serif;" data-mce-style="color: #222222;
font-family: arial,sans-serif;" color="#222222" face="arial, sans-serif">2.
Check the web server logs for unusual entries and identify the entry point. You
should have noticed the timestamp of the Perl script in the /tmp directory. Try
looking for entries around this timestamp. Usually, the timestamp would be your
starting point. The files created in the /tmp/.m directory were most probably
put there via an HTTP request.</span></div><div><span style="color: #222222;
font-family: arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif">3. Find all
locations writable by www-data.</span></div><div><span style="color: #222222;
font-family: arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif">4. Touch a file
with a timestamp = the date of the incident.</span></div><div><span
style="color: #222222; font-family: arial,sans-serif;" data-mce-style="color:
#222222; font-family: arial,sans-serif;" color="#222222" face="arial,
sans-serif">5. Find all files -newer than the file you `touch`-ed in the
locations writable by www-data.</span></div><div><span style="color: #222222;
font-family: arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif">6. Identify any
malicious files in the returned listing.</span></div><div><span style="color:
#222222; font-family: arial,sans-serif;" data-mce-style="color: #222222;
font-family: arial,sans-serif;" color="#222222" face="arial, sans-serif">7.
Stat the malicious files and log the data.</span></div><div><span style="color:
#222222; font-family: arial,sans-serif;" data-mce-style="color: #222222;
font-family: arial,sans-serif;" color="#222222" face="arial, sans-serif">8.
Quarantine / remove the malicious file(s).</span></div><div><span style="color:
#222222; font-family: arial,sans-serif;" data-mce-style="color: #222222;
font-family: arial,sans-serif;" color="#222222" face="arial, sans-serif">9.
*Patch* the Web application.</span></div><div><span style="color: #222222;
font-family: arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif">10. Check the
application code for other vulnerabilities.</span></div><div><span
style="color: #222222; font-family: arial,sans-serif;" data-mce-style="color:
#222222; font-family: arial,sans-serif;" color="#222222" face="arial,
sans-serif">11. Allow access to the Webapp.</span></div><div><span
style="color: #222222; font-family: arial,sans-serif;" data-mce-style="color:
#222222; font-family: arial,sans-serif;" color="#222222" face="arial,
sans-serif">12. Check for updates for the application regularly and apply fixes
for any security issues if full upgrade is not possible.</span></div><div><span
style="color: #222222; font-family: arial,sans-serif;" data-mce-style="color:
#222222; font-family: arial,sans-serif;" color="#222222" face="arial,
sans-serif"><br></span></div><div><span style="color: #222222; font-family:
arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif">Unless you patch
the application, the issue will most certainly re-occur.</span></div><div><span
style="color: #222222; font-family: arial,sans-serif;" data-mce-style="color:
#222222; font-family: arial,sans-serif;" color="#222222" face="arial,
sans-serif"><br> </span></div><div><span style="color: #222222; font-family:
arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial,
sans-serif">Regards,</span></div><div><span style="color: #222222; font-family:
arial,sans-serif;" data-mce-style="color: #222222; font-family:
arial,sans-serif;" color="#222222" face="arial, sans-serif"><br></span>--<br>#
m</div><br><div class="gmail_quote">On Mon, Dec 5, 2011 at 12:44, Lucio Crusca
<span dir="ltr"><<a href="mailto:lucio@xxxxxxxxxx";
data-mce-href="mailto:lucio@xxxxxxxxxx";>lucio@xxxxxxxxxx</a>></span>
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0
.8ex; border-left: 1px #ccc solid; padding-left: 1ex;">Hello *,<br> <br> I'm
not new here, but I've mostly lurked all the time through gmane. I never<br>
believed it could happen to me until it actually happened: they compromized<br>
one of my servers. It's a Ubuntu 10.04 server with all security patches<br>
regularly applied. I'm inclined to believe they used some hole in the web<br>
application, which is a old customized Virtuemart version (1.1.3), which is<br>
not upgradable because of the invasive code customizations (I'm not the<br>
author of that code, so I have no clue about what had been changed back<br>
then).<br> <br> Now the problem for me is to track down the security hole. Here
is the email<br> my provider received and forwarded to me:<br> <br> >
Subject: ISP Report; botnet activity on <a href="http://irc.undernet.org";
data-mce-href="http://irc.undernet.org";>irc.undernet.org</a><br> > [...]<br>
><br> > Hello, I am an operator on the irc chat network,<br> > <a
href="http://irc.undernet.org";
data-mce-href="http://irc.undernet.org";>irc.undernet.org</a> and i would like
you to investigate the<br> > owner of the Ip addresses that are listed at
the foot of this<br> > email.<br> ><br> > This/These host(s) have
likely been compromised, and had an<br> > altered/rogue process installed on
it, and was part of a botnet<br> > that was found on our network.<br>
><br> > The exploit or compromise running on this system is likely<br>
> to be an irc bot. Can you please alert the person who is<br> >
responsible, for its security to patch/upgrade, remove the<br> > irc process
and secure their system.<br> ><br> > = Unix System owners =<br> > A
favourite place for hiding the bot(s) is in tmp<br> > and in /var/tmp/ or
/dev/shm/ or in a users home directory<br> > sometimes it may be hidden like
/tmp/". ."/ or similar.<br> ><br> > The bot files can usually be
found by running these one line<br> > commands as the root user.<br>
><br> > find / -exec grep -l "undernet" {} +<br> > find / -exec grep
-l "sybnc" {} +<br> > find / -name "*.set" | perl -pe
's/.\/\w+-(\w+)-.*/$1/' | sort | uniq<br> > find / -name "inst" | perl -pe
's/.\/\w+-(\w+)-.*/$1/' | sort | uniq<br> ><br> > netstat -tanp<br> >
lsof -i tcp:<Port number><br> ><br> > *netstat looking for
connections to remote port 6667 or the<br> > range of ports between
6660-7000 once you find the port you<br> > can use the command, lsof -i
tcp:portnumber to determine<br> > which process/user it is running under,
and terminate it.<br> ><br> > = Windows System Owners =<br> > most
windows bots are mIRC scripted bots and generally<br> > need a file called
mirc.ini to run, you should search for<br> > this file. Run a good antivirus
scanner and firewall.<br> ><br> > This Ip/host may be removed from our
Irc network due to the<br> > risks it presents to our users.<br> ><br>
> Should you need any help with removing the files or bot<br> > process,
feel free to contact me by mail or on our network,<br> > which you connect
to using any irc client and issuing<br> > /server <a
href="http://irc.undernet.org";
data-mce-href="http://irc.undernet.org";>irc.undernet.org</a><br> ><br> >
I look forward to your reply<br> > Scot<br> ><br> > * Affected
host/IPs, capture time is GMT+1: United kingdom<br> > and servers they were
connected to.<br> ><br> > Please note: when resolving server names to IP
Addresses<br> > that all our servers end with .<a href="http://undernet.org";
data-mce-href="http://undernet.org";>undernet.org</a> (for example)<br> > <a
href="http://Tampa.FL.US"; data-mce-href="http://Tampa.FL.US";>Tampa.FL.US</a>.
is actually <a href="http://Tampa.FL.US.undernet.org";
data-mce-href="http://Tampa.FL.US.undernet.org";>Tampa.FL.US.undernet.org</a><br>
><br> > Important: If you reply to this mail needing further<br> >
information, please leave this mail intact, or supply us<br> > with the IP
Address(es) in question, as we reference these<br> > mails by the unique IP
Address<br> ><br> > Time of Capture: DECEMBER 3, 2011 10:03:48 PM<br>
><br> > List of IP address(es) and server it connected to:<br> >
my.server.ip.address (<a href="http://CHICAGO.IL.US";
data-mce-href="http://CHICAGO.IL.US";>CHICAGO.IL.US</a><br> ><br> > <a
href="http://BUDAPEST.HU.EU";
data-mce-href="http://BUDAPEST.HU.EU";>BUDAPEST.HU.EU</a><br> ><br> > <a
href="http://MONTREAL.QC.CA.undernet.org";
data-mce-href="http://MONTREAL.QC.CA.undernet.org";>MONTREAL.QC.CA.undernet.org</a>)<br>
><br> <br> I've run the "find" commands and found a number of file with the
first<br> "find", under /tmp/.m<br> <br> Deleted them, looked up remote
connections with netstat, killed perl<br> processes that where trying to
connect to port 6959 (only trying because<br> I've now set up iptables so that
they actually can't), but those processes<br> kept spawning. Checked crontab of
www-data, found the launcher, removed it.<br> <br> Now the problem is: how do I
pervent further abuse? What should I search in<br> the logs (if anything) to
spot the security hole?<br> <br> TIA<br> Lucio.<br> <br> <br> <br> <br> <br>
_______________________________________________<br> Full-Disclosure - We
believe in it.<br> Charter: <a
href="http://lists.grok.org.uk/full-disclosure-charter.html";
data-mce-href="http://lists.grok.org.uk/full-disclosure-charter.html";>http://lists.grok.org.uk/full-disclosure-charter.html</a><br>
Hosted and sponsored by Secunia - <a href="http://secunia.com/";
data-mce-href="http://secunia.com/";>http://secunia.com/</a><br></blockquote></div><br><div
class="_stretch">_______________________________________________<br>
Full-Disclosure - We believe in it.<br> Charter: <a
href="http://lists.grok.org.uk/full-disclosure-charter.html";
data-mce-href="http://lists.grok.org.uk/full-disclosure-charter.html";>http://lists.grok.org.uk/full-disclosure-charter.html</a><br>
Hosted and sponsored by Secunia - <a href="http://secunia.com/";
data-mce-href="http://secunia.com/";>http://secunia.com/</a><div
style="width:0px;
height:0px;"> </div></div></div></blockquote></div></div></body></html>_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/