Flickus flackus flum

Jacobs tankar om och med kvalitet

Malware under Linux

written by jacob, on Mar 2, 2014 3:16:00 AM.

My computer system was recently penetrated, and this is a report on my findings.

The symptoms of the penetration was that my internet connection came to a standstill. At first, I thought that it was a DDOS attack from the outside, but after disconnecting all my machines and reconnecting them one by one, I realized that it was my primary workstation that was clogging the network.

Netstat revealed some suspicious looking network connections (kill your web browser and any other internet facing programs before checking). Top did not reveal any suspicios processes at first, but after looking for some time, I noticed a couple of processes called atddd and sksapdd. When I disconnected the network, the atdddprocess became very busy for a while.

A search for the process names revealed a number of files in /etc with the process names and some similar names. All in all, I found 6 or 7 files with identical size, owned by root and with s-flags set. I proceeded to remove them.

Next, I tried to find the path the intruders took into the system. In /var/log/auth, I found entries showing multiple login attempts as root. It seems likely that the root password ahd been guessed. I had set the root password to a weak one the same day that the login attempts finally succeeded. This annoys me greatly, because I assumed that root login over ssh would be disabled. I had it disabled in my pervious system, but it turns out that the debian default is to have it enabled. I think this is a major security risk.

At any rate, I edited /etc/ssh/sshd_config to have the line

PermitRootLogin no

I strongly suggest anyone with a computer with ssh access from the internet do the same. If you need to be root, you log in to your own account and the su or sudo to root. This deepens your defenses. An intruder now needs to find your login name, guess your password and then find out how to become root. Not having root login from the internet makes you a smaller target.

Removing the offending processes stopped the network activity, but since there could be more malicious code on my machine, I started investigating things. The /var/log/auth file had too many messages in the form Mar 1 15:20:01 sangiovese CRON[30649]: pam_unix(cron:session): session opened for user smmsp by (uid=0) for me to feel comfortable.The user smmsp is the identity that sendmail uses, so the traffic may be legitimate. On the other hand, I have inly installed sendmail on this machine for development purposes, so I decided to remove the package.

This started a major circus, because aptitude was unable to remove the file, claiming a recursive call in a script called S99DbSecuritySpt. Some investigation revealed that this script was a sodt link in one of the /etc/rc.? directories. The links all pointed to /etc/init.d/DbSecuritySpt which in turn just had one command - /etc/cupsdd. This turned out to be yet another part of the malware, and there were a number of similar copies to the main file. The cupsdd file itself was not removable by root, due to the i flag being set on the file. To remove it, you do chattr -i cupsdd as root, and then you can do rm on the file.

After removing the scripts, I was able to purge sendmail-bin. If you have been targeted, I suggest you do this too, just in case. You can make a clean reinstall after you have dumped the software and the config files.

After this there were some minor bits of cleanup. There were a bunch of lines in /etc/rc.local on the form cd /etc;./ksapdd, that had to be removed. I think they are harmless, since they are after an exit 0 statement, but they should be removed nevertheless. There were also a few config files in /etc with recent timestamps, unfamilair names and unfamiliar contents. I removed them too.

Having gotten this far, I did my best to make sure the system was actually clean from the malware. I did a scan for all fliles that have the s-flag set, and they all looked untouched. It is possible to hide modifications, but the producers of this package do not seem to be that advanced. Finally, I made clean install of debsums and ran debsums -s and debsums -es, carefully checking each locally modified file. I had 6 of them, so it was not a big task.

It is of course possible that the perpetrators have corrupted my debian installation system, but I find it unlikely, considering that they have botched a number of details in the malware setup. I am now reasonably confident that I have eliminated allmalicious code on my disks. There may be hidden processes, but a reboot will make them go away. I will carefully monitor my system for symptoms of remaining security breaches for a while. If it turns out that there are still malicious elements left in my system, I’ll have to go for a clean install.

I have also remembered to reset the root password to a stronger one.

To further reduce the risk of someone accessing the machine from outside, I have limited ssh login to only work for one single accoput on the machine. There is a directive AllowUsers that you can put in /etc/ssh/sshd_config. My firewall only allows access from the outside to port 22 (ssh). This makes the exposure to network attacks fairly small, while still letting me access my machine from outside.

At any rate, I made this writeup in the hopes that it would help someone else in my predicament. If it did, I’d be very happy if you made a comment below. It would tell me that the effort I put into the writeup was not in vain.

Edit: What Terje says in the comment below is important. I did clean my crontab, but forgot to write about it. Doing crontab -r as root does the job.

Comments

  • Thanks Jacob!
    Same shi* happened to me yesterday.
    My server had weak root password and
    was penetrated from some Chinese IP.
    Your post helped me to identify and fix problem!
    Thans again!

    Comment by Ivan Ilves — Mar 12, 2014 3:40:59 PM | # - re

  • Thanks, great info,

    BUT: I found that you must also clean out the root crontab!! The code to re-infect is there and it does re-try to infect every hour…

    I did a ‘crontab -r’ as root.

    Comment by Terje — Mar 14, 2014 6:04:27 AM | # - re

  • hi, i came here after search for DbSecuritySpt in google, after some troubles updating my debian system i found that file that looks so suspicius. You can imagine the rest, thanks to your info i make a quick clean and i find the bastard still connected to my sshd from 116.10.189.246 No idea about infection vector, i hope is not through official debian repositories.

    Comment by tul — Mar 21, 2014 1:54:59 AM | # - re

  • I just found a beagle bone black I loaded up with debian got rooted; I am not sure how I was compromised; no root password was set; only thing I can think of is I had SSH forwarded before I removed default beagle user? Im 90% sure I did that after removing the default account but I cant be completely sure but I had fail2ban installed right away.

    Unsettling we all seem to be running up to date debian.. I am going to remove default user, ensure no root password set and open it to the internet for a while and see what happens before restoring all my work.

    Comment by R — Apr 6, 2014 10:09:09 AM | # - re

  • It’s a botnet called “BillGates”
    https://github.com/ValdikSS/billgates-botnet-tracker

    Comment by ValdikSS — Apr 13, 2014 4:30:41 PM | # - re

  • Comment by gabriel — Apr 30, 2014 12:57:53 AM | # - re

  • Compromised as well on one of my F19 installs. IP localized in US and China, not that it matters that much.

    Thanx for all the info it was really helpful!

    I had all the same symptoms, messed up crontab (tens of pages long), rc.local messed up, files created in /etc and executed by cron that claimed my network connection and a lot of cpu.

    Some of the files created under /etc had the immutable attribute set.

    Another strange thing was that the sticky bits was set for / and it was opened for the world , ‘drwsrwsrwt’. I discovered this as it makes makes sendmail complain at startup: ‘ NOQUEUE: SYSERR(root): /etc/mail/sendmail.cf: line 87: fileclass: cannot open ‘/etc/mail/local-host-names’: World writable directory’

    2 hours wasted on this headache, wishing for a face-2-face with the responsible in company of a baseball bat.

    Comment by blackened — May 14, 2014 1:59:08 AM | # - re

  • Jacob,
    I recommend you some things:
    1 - If you are comfortable, change to CentOS in Production Server instead of Debian-like ones. 2 - Install Fail2Ban[0]. It’ll looks up ur logs and block malicious activity.
    3 - Some changes to your sshd_config:
    a) Port 22; change it to a 20k+ port, i.e. 22222
    b) LoginGraceTime 120; change it to 30(sec). it’ll stop manual attempts and the Fail2Ban will stop automated ones.

    If you have more people accessing this server, You can use Public/Private Keys[1]; better than passwords.

    If only You access this server, you can block all incoming connections from another country IP that isn’t from Sweden[2]… If you use VPN, you have to specify the IP or IP range.

    Best regards from Brazil
    ----
    [0] www.fail2ban.org/wiki/index.php/Main_Page
    [1] https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys–2
    [2] apple.stackexchange.com/questions/34091/how-to-restrict-remote-login-ssh-access-to-only-certain-ip-ranges

    Comment by Gutem — Aug 11, 2014 10:04:13 PM | # - re

    • Thanks for the hints, but I’m quite comfortable running Debian. The reason I keep a port open at all is that I might want to log in from somewhere out in the world. Not being able to do that is much more annoying than having to remove vermin from the system occasionally.

      Comment by jacob — Aug 18, 2014 11:42:11 PM | # - re

  • Someone else had root access to your server. You should NOT trust in this server anymore. Attackers could have installed a rootkit or anything else in this machine. I’d recommend you to reinstall it completely.

    Best,
    Fernando

    Comment by Fernando — Aug 13, 2014 10:13:36 PM | # - re

    • This is always a balance. How critical is the system? How likely are the attackers to be competent enough to hide a proper root kit? What other measures have I taken to ensure that they haven’t tampered with essential parts of the system? I use some standard precaution and some private tricks to be able to determine if my system is compromised or not.

      Comment by jacob — Aug 18, 2014 11:47:11 PM | # - re

  • Came here thanks to running nethogs and immediately spotting the process /boot/pro and googling it.

    Update on how it works from my box:

    • same entry point: /etc/init.d/DbSecuritySpt (and every rc.d) but pointing to /boot/pro instead of /etc/cupsdd
    • crontabs empty, save for the couple of routines I set.
    • no sendmail was ever installed, so nothing from that end.
    • no cupsdd file

    The culprits: weak root password, PermitRootLogin set to yes and outbound connections allowed in apache settings.

    Running Elementary OS Luna.

    Thank you so much, and all the best to everyone infected! It was tanking my upload to the point of no connectivity for anyone else in the network.

    Comment by Andres — Sep 4, 2014 5:53:45 AM | # - re

  • Seems I spoke too soon. Nethogs is showing “unknown TCP” along with “/boot/pro”. I’ll give it another go tomorrow.

    Comment by Andres — Sep 4, 2014 5:58:16 AM | # - re

  • outbound connections allowed in apache settings.

    I meant iptables settings.

    In other news, found a /usr/bin/zl copy of /boot/pro

    A grep of the entire system found a /boot/pro entry in /bin/reset.sh, which contained a simple set of instructions:

    #!/bin/sh
    cp -rf /tmp/get/pro /boot/
    /boot/pro
    /bin/kill.sh

    And which, in turn, was being called by /bin/check.sh, which contained:

    #!/usr/bin/env sh

    count=`ps -ef | grep -v grep | grep pro | wc -l`

    if [ $count -gt 5 ]; then
    /bin/reset.sh
    fi
    /bin/reset.sh

    kill.sh, on the other hand, had:

    #!/bin/sh killall rewgtf3er4t atddd xfsdxd b26h b26 dsfrefr gfhjrtfyhuf sdmfdsfhjfe ferwfrre pojie sksapd gfhddsfew sfewfesfsh sfewfesfs cupsdd atdd cupsddh ksapd ksapdd Bill xfsdx cupsddd kysapdd sksapdd kysapdd skysapdd skysapd kysapd meiun 111

    Last, but most importantly, I found /usr/bin/chattr to have been deleted. Without this guy I wouldn’t have been able to delete any of the aforementioned files.

    I’ll make sure to post any updates, but hopefully won’t.

    Comment by Andres — Sep 5, 2014 1:32:31 AM | # - re

  • I found connections still outgoing, so I reinstalled the system.

    Comment by Andres — Sep 7, 2014 4:35:32 PM | # - re

  • Hi, I have same issue, but I don’t know how to get rid of it, could you please help me? I’m dying with this thing.

    Comment by Gen — Sep 10, 2014 9:36:27 AM | # - re

  • A similar issue with DbSecuritySpt and suspicious scripts on Debian was recently also reported in https://www.barbosa.xyz/notes/4/virus-infection-on-linux-server.

    A fresh re-install seems to be the safer way to permanently get rid of it.

    Comment by Richard Lawrence — Jan 23, 2017 8:14:05 PM | # - re

Leave a Reply