My computer system was recently penetrated, and this is a report on my findings.
The symptoms of the penetration was that my internet connection came to a standstill. At first, I thought that it was a DDOS attack from the outside, but after disconnecting all my machines and reconnecting them one by one, I realized that it was my primary workstation that was clogging the network.
Netstat revealed some suspicious looking network connections (kill your web browser and any other internet facing programs before checking). Top did not reveal any suspicios processes at first, but after looking for some time, I noticed a couple of processes called atddd and sksapdd. When I disconnected the network, the atdddprocess became very busy for a while.
A search for the process names revealed a number of files in /etc with the process names and some similar names. All in all, I found 6 or 7 files with identical size, owned by root and with s-flags set. I proceeded to remove them.
Next, I tried to find the path the intruders took into the system. In /var/log/auth, I found entries showing multiple login attempts as root. It seems likely that the root password ahd been guessed. I had set the root password to a weak one the same day that the login attempts finally succeeded. This annoys me greatly, because I assumed that root login over ssh would be disabled. I had it disabled in my pervious system, but it turns out that the debian default is to have it enabled. I think this is a major security risk.
At any rate, I edited /etc/ssh/sshd_config to have the line
I strongly suggest anyone with a computer with ssh access from the internet do the same. If you need to be root, you log in to your own account and the su or sudo to root. This deepens your defenses. An intruder now needs to find your login name, guess your password and then find out how to become root. Not having root login from the internet makes you a smaller target.
Removing the offending processes stopped the network activity, but since there could be more malicious code on my machine, I started investigating things. The /var/log/auth file had too many messages in the form
Mar 1 15:20:01 sangiovese CRON: pam_unix(cron:session): session opened for user smmsp by (uid=0) for me to feel comfortable.The user smmsp is the identity that sendmail uses, so the traffic may be legitimate. On the other hand, I have inly installed sendmail on this machine for development purposes, so I decided to remove the package.
This started a major circus, because aptitude was unable to remove the file, claiming a recursive call in a script called S99DbSecuritySpt. Some investigation revealed that this script was a sodt link in one of the /etc/rc.? directories. The links all pointed to /etc/init.d/DbSecuritySpt which in turn just had one command - /etc/cupsdd. This turned out to be yet another part of the malware, and there were a number of similar copies to the main file. The cupsdd file itself was not removable by root, due to the i flag being set on the file. To remove it, you do chattr -i cupsdd as root, and then you can do rm on the file.
After removing the scripts, I was able to purge sendmail-bin. If you have been targeted, I suggest you do this too, just in case. You can make a clean reinstall after you have dumped the software and the config files.
After this there were some minor bits of cleanup. There were a bunch of lines in /etc/rc.local on the form
cd /etc;./ksapdd, that had to be removed. I think they are harmless, since they are after an exit 0 statement, but they should be removed nevertheless. There were also a few config files in /etc with recent timestamps, unfamilair names and unfamiliar contents. I removed them too.
Having gotten this far, I did my best to make sure the system was actually clean from the malware. I did a scan for all fliles that have the s-flag set, and they all looked untouched. It is possible to hide modifications, but the producers of this package do not seem to be that advanced. Finally, I made clean install of debsums and ran debsums -s and debsums -es, carefully checking each locally modified file. I had 6 of them, so it was not a big task.
It is of course possible that the perpetrators have corrupted my debian installation system, but I find it unlikely, considering that they have botched a number of details in the malware setup. I am now reasonably confident that I have eliminated allmalicious code on my disks. There may be hidden processes, but a reboot will make them go away. I will carefully monitor my system for symptoms of remaining security breaches for a while. If it turns out that there are still malicious elements left in my system, I’ll have to go for a clean install.
I have also remembered to reset the root password to a stronger one.
To further reduce the risk of someone accessing the machine from outside, I have limited ssh login to only work for one single accoput on the machine. There is a directive
AllowUsers that you can put in /etc/ssh/sshd_config. My firewall only allows access from the outside to port 22 (ssh). This makes the exposure to network attacks fairly small, while still letting me access my machine from outside.
At any rate, I made this writeup in the hopes that it would help someone else in my predicament. If it did, I’d be very happy if you made a comment below. It would tell me that the effort I put into the writeup was not in vain.
Edit: What Terje says in the comment below is important. I did clean my crontab, but forgot to write about it. Doing crontab -r as root does the job.