UPDATE: The
Linux Foundation and its sub-domains, including
Linux.com have announced they too have been hacked and it appears to be related to the breach of kernel.org, discovered on August 28.
As previously reported, one of the developers with root privileges, was found to have a trojan on his personal computer.
Linux Foundation infrastructure including LinuxFoundation.org, Linux.com, and their subdomains are down for maintenance due to a security breach that was discovered on September 8, 2011. The Linux Foundation made this decision in the interest of extreme caution and security best practices. We believe this breach was connected to the intrusion on kernel.org.
We are in the process of restoring services in a secure manner as quickly as possible. As with any intrusion and as a matter of caution, you should consider the passwords and SSH keys that you have used on these sites compromised. If you have reused these passwords on other sites, please change them immediately. We are currently auditing all systems and will update this statement when we have more information.
We apologize for the inconvenience. We are taking this matter seriously and appreciate your patience. The Linux Foundation infrastructure houses a variety of services and programs including Linux.com, Open Printing, Linux Mark, Linux Foundation events and others, but does not include the Linux kernel or its code repositories.
Please contact us at info@linuxfoundation.org with questions about this matter.
The Linux Foundation
Kernel.org Hacked
Sept. 2, 2011
Kernel.org - the Official Linux Kernel Archive announced on its website that it's servers have been compromised. The statement reads, in part :
“Security breach on kernel.org
Earlier this month, a number of servers in the kernel.org infrastructure were compromised. We discovered this August 28th. While we currently believe that the source code repositories were unaffected, we are in the process of verifying this and taking steps to enhance security across the kernel.org infrastructure. "
This revelation is disturbing on many levels. Once again, it proves that even Linux is vulnerable to attack, despite its reputation for superior security. It proves that even the most experienced sys admins can fail in their efforts to ensure the integrity of their networks. It shows the vulnerability of simple human error, like a trojaned laptop, can have wide ramifications and over-dependence on a flawed web security system, like SSL cert signing, leaves all systems at risk. (See this excellent article regarding the recent compromise of Dutch company DigiNotar which an intruded generated 531 fake certificates, including for popular sites such as Google, Facebook and Skype.)
What Happened?
Sometime before August 12 an intruder gained access to the Hera server at kernel.org using a compromised user credential. Starting around August 19th, user interactions were logged, and a trojan startup file was added to rc3.d. (Interestingly, an attempt was made to attack 3.1-rc2, but this was blocked for some reason.) SSh files, including openssh, openssh-server and openssh-clients were altered and running live.
It appears that one of the developers with root access may have had a trojan on his personal computer which allowed an attacker to gain access to his ssh key and root password. The same trojan was found on the servers Hera and odin1, with possible infections of three others.
The intrusion was detected on August 28 when xnest /dev/mem starting throwing errors.
Xnest is an X Window system server and was not installed on the infected machine.
After the intrusion was detected, the boxes were taken offline and reinstalled from clean backups. The folks at kernel.org were open about the breech and posted details on their website right away. They explain how using git, the exposure to the public should be minimal.
Git was developed by Linux Trovalds in 2007 as a fast, efficient distributed revision control system. It is used to keep track of software revisions and allow collaboration on projects among developers across diverse networks.
One of the features of git is its cryptographic authentication of revision history. As
Wikipedia explains:
"The Git history is stored in such a way that the name of a particular revision (a "commit" in Git terms) depends upon the complete development history leading up to that commit. Once it is published, it is not possible to change the old versions without it being noticed. The structure is similar to a hash tree, but with additional data at the nodes as well as the leaves."
So each revision is named according to its revision history and each of its parent revision histories all they way up to the top level. This creates a 'tree' of revision hashes which contains the entire history of the project. If you change anything, it creates a cascade of changes to the hashes which will throw errors when attempting to merge the changes into the main repository. Attempting to force the changes would result in a separate branch.
So what does all this mean to Linux users? Are they at risk? Generally, the risks appear small. Most Linux users get their updates through their distos package manager and these are generally signed and maintained on their own servers. Most of the maintainers of these distors use git to update their packages. Therefore, most users can consider themselves safe.
I have seen some discussion as to whether a user who directly downloaded a kernel tarball using ftp or http between August 12 and August 28, possibly bypassing git, could have received an infected copy of the file. I have found no clear answer to this question as yet. If you have recently built your own kernel, using files from kernel.org, downloaded via ftp or http, you might consider a re-install to be safe. Also, look for errors involving xnest, as this was the clue which led to detection of the intrusion. As noted on the website:
"Trojan initially discovered due to the Xnest /dev/mem error message w/o Xnest installed; have been seen on other systems. It is unclear if systems that exhibit this message are susceptible, compromised or not. If you see this, and you don't have Xnest installed, please investigate."
Linux users like to believe we are nearly immune from malware attack. Occasionally, reality proves this not to be true.
A bit of history:
Intrusion into four servers that host the project's bug-tracking system, mailing lists and various Web pages.
In case you believe those were ancient history, that it can't happen today, consider these:
August 2007: Ubuntu had to take 5 of its 7 production servers offline after they were being used to attack other servers. The effected servers were found to be out of date on security patches and using insecure protocols (FTP without SSL).
The point here is that no OS is 100% secure. No sysadmin is 100% perfect. Human error, when combined with millions of lines of complex code, will occasionally fail. For now, the Linux community appears to remain safe and git, developed by Linus himself, has done its job.
