Ext4 Data Corruption Bug and Solution

Just when you thought it was safe...

Ext4 (the fourth extended file system) has been the gold standard for the Linux kernel ever since it was declared "stable" in October 2008. It was the direct descendant of ext3 (released November 2001) which introduced journaling to the previously unjournaled ext2 file system that has been with us since 1993. Ever since its release into the wild, ext4 has proven to be fast and reliable. I've installed it onto more computers than I can count, and never had any reason to complain about it...until recently. But a serious bug has crept into the ext4 stable release, causing data corruption on some computers running Linux kernels 3.4, 3.5, and 3.6.

I first stumbled across this problem about six weeks ago when I was writing this review of Bodhi Linux. After installing Bodhi, it initially ran well, but by the fourth or fifth reboot problems developed. The first glitch: I couldn't log in while running Xorg, but could log in if I used text mode. A few reboots later and I couldn't log in at all -- in fact, all I saw was a blank, black screen. Needless to say, I was nonplussed and had a few email exchanges with the developer, who insisted that Bodhi was innocent. I, of course, occupied the moral high ground and nearly gave Bodhi a scathing review -- until I tried installing Lubuntu which soon developed similar problems. Properly chastised, I meekly retreated and gave Bodhi the fine review it deserved, and looked elsewhere for the cause of this debacle.

My assumption at this point was that my hard drive was probably failing. However, I had an older version of Ubuntu installed on the same hard drive and it ran smooth as silk. As it turned out, that installation was on a partition formatted with XFS, a file system that used to be my favorite because it was considerably faster than ext3. I never had any complaints about XFS and only abandoned it because most new distros no longer include the formatting command mkfs.xfs plus the necessary XFS utilities for error-checking. However, kernel support for XFS is very much alive, so it is still a viable option.

I broke out an old Slackware CD and reformatted my ext4 partition with XFS, then reinstalled Bodhi. It ran without a hiccup. Although I still wasn't confident that my hard drive was OK, I was glad to see it running well and wasn't going to look a gift horse in the mouth. Then just last week I was installing the recently released Ubuntu 12.10 on a friend's new computer when the same thing happened. Ubuntu ran great until the third reboot, when it too began to act flaky. At first it gave me nasty error messages, but on subsequent reboot attempts it just crashed and burned. Remembering my prior experience with Bodhi, I broke out that Slackware CD and reformatted with XFS, reinstalled and all was well.

If I was seeing this, I assumed there were others in the same boat, so I went searching. With just a little bit of googling, I found this October 24 post by Michael Larabel on the Phoronix web site: EXT4 Data Corruption Bug Hits Stable Linux Kernels. You should probably read Michael's post, but in a nutshell what it says is that starting from Linux kernel 3.4 and continuing, some users are seeing file data corruption on partitions formatted with ext4.

I should stress that not everyone is experiencing this issue. Indeed, I make a habit of installing Linux on friends' computers and only two of them formatted with the latest ext4 have so far developed this problem. On the other hand, that is two too many. I am always claiming that Linux is rock-solid reliable, so it's more than a little embarrassing when I install it for someone only to see it blow up shortly thereafter.

The chief maintainer of ext4, Theodore Y. "Ted" Ts'o , is well aware of the problem and hard at work on a solution. Ted is one of the most knowledgeable people in the world when it comes to file systems and I don't doubt for a minute that he will sort out this issue. However, tracking down the bug, issuing a fix and putting it through rigorous testing takes time.

On the one hand I am somewhat reluctant to write this news story, since having a large community of beta-testers (even unwilling ones) is helpful to solving this problem. On the other hand, some of us value our data more than life itself (did I just say that?), so we need a solution right now. Thus, I would like to make a few suggestions to those of you who have been bit by this bug, or are at least very concerned about it.

At the minimum, your various removable backup devices (external USB hard drive and USB memory sticks) should be formatted with either FAT32 (the factory default in almost every case) or ext2. I personally prefer ext2 as it supports such niceties as file ownership and uses space more efficiently. In fact, I've been using ext2 for this purpose long before this ext4 bug ever hit, because there really is no need for journaling on a backup device.

On your hard drive itself, a minimum precaution would be to have a separate /home partition formatted with a file system other than ext4. This would at least protect your precious data even if the root partition crashes and burns. However, most people (present company included) tend to put everything under the "/" root partition, /home included. Furthermore, it's kind of inconvenient to have your root file system crash, though again I stress that only a minority of users are actually experiencing this. If you decide do as I've done and go with XFS, there are a few tricks to installing it. As I already said, practically every distro comes with a kernel with XFS support baked in, but you need the formatting and maintenance utilities. You can hunt around for a live CD that already has mkfs.xfs installed. I used Slackware 13, but there are many others.

Alternatively, you could boot up even the latest Ubuntu live CD and install package XFSprogs before you run the installer. The Ubuntu installation program allows you choose XFS under the "something else" option for disk partitioning, but if you don't have XFSprogs installed, it won't be able to format the partition with that file system.

You may find that after you've installed your favorite distro on an XFS partition, on first boot-up you'll be greeted with this disheartening error message (as I saw on Ubuntu):

    Serious errors were found while checking the disk drive for /. Press I to ignore, S to skip mounting or M for manual.

As scary as that message looks, there was nothing wrong with the disk. The problem was simply that since package XFSprogs was not installed by default, there were no XFS file system checking utilities. Thus, file system integrity could not be checked on boot-up and the attempt to do so failed. The simple solution is to press I to "ignore" and then after boot-up, install XFSprogs. On subsequent restarts, the error message should be absent.

Now that I've hyped XFS, I should probably mention that there are a few other ancient alternatives passed down to us by our ancestors. Why not simply go back to good old ext3? Probably not a good idea, for the reason it was abandoned in the first place - it was slow. Indeed, the old ext2 file system was significantly faster than ext3, but it lacked a journal. Aside from lackluster performance, ext3 had a few other drawbacks which you can read about here.

Another competitor in the file system wars is ReiserFS. Once very popular, it was the default on a number of distros I used (SUSE, for example). However, the murder conviction of chief developer Hans Reiser put a dark cloud over the whole project and no one really likes to talk about it. Nevertheless, the Linux kernel still has support, and if you want to try it look for the package reiserfsprogs.

One more alternative is JFS (the Journaled File System). This is an open-source implementation of a commercial file system that was used on IBM's trusty old AIX operating system. From what I understand, it works well enough though I've never used it personally. I don't know any distros that default to JFS, but it's still supported in the Linux kernel. The package you'll need is jfsutils. Using JFS might at least give you the warm fuzzy feeling of knowing that it was developed by a major brand-name tech company.

On the other hand, my personal favorite, XFS, was also a corporate spin-off. Developed by the now-defunct SGI (Silicon Graphics, Inc), it was the star file system of the IRIX OS and was considered very advanced for its time. Now that SGI is no longer with us, all development work on XFS has ceased. Given the fact that such development has introduced bugs into ext4, that might be a good thing.