Tags: debian, sysadmin, encryption
Update: The situation seems to have improved, see this comment.
I use both in my desktop and laptop Debian testing (currently codenamed 'wheezy'). For backups, I have been using an AES256 encrypted ZFS external USB hard drive, with date-based snapshots and deduplication. I have two hard drives, one of which I keep off-site, doing weekly (or so) backups. I have all the same set-up in my laptop, and I have done some drill back-up recoveries from it (from the latest backup, not from the snapshots, as zfs-fuse doesn't support .snapshot folder, yet).
This solution has been working quite well until I went to
Argentina for teaching. There,
while I upgraded my Debian testing on the laptop, FUSE stop working
properly in strange ways. When I got back to Canada and tried to
backup my desktop, zfs-fuse was also non-working. The problem took me
quite a bit of work to fix, I'm documenting here the problem and its
solution (in a nutshell: loop-AES, in the package loop-aes-utils,
diverts /bin/mount to a version that is incompatible with
FUSE, solution: don't use loop-AES and move to dm-crypt).
In Debian there are several "flavors" for the distribution: stable, testing and unstable. Stable is well-tested, with timely security updates and all around rock solid (or that is the expectation at least). Nothing new happens in Debian stable for about year and a half to two years. In the mid-time, all the new packages, testing of new versions, etc., happens in Debian unstable (Debian 'sid'). When new versions of packages are uploaded to unstable, they automatically migrate into testing if there are no serious bugs filled against the upload in those first two weeks. I have been using testing for years and it is overall a much "happier" experience that testing as it usually Just Works (TM) and has very recent software.
Interestingly, when I start getting more involved in Debian, I realized testing is a little bit the worst of both worlds: you get no security updates (as in stable) and you get no interest from the developers (as in unstable). If you try to submit a bug against a package in testing, chances are you'll be asked to upgrade to unstable and see what happens. In general, submitting a bug against a package in testing when there is a newer version in unstable is a waste of everybody's time (and counter-intuitive, as the distro is called 'testing'!). Still, once in a blue moon a horrible update comes in unstable and the system breaks in a horrible way. I will stick to testing until I am comfortable with checking #debian-devel in OFTC before doing an upgrade. But over time using testing is becoming more and more of a nuisance. And this issue of testing vs. unstable is very related to my loop-AES blues, keep reading.
As usual in the FLOSS world, there is a variety of competing
technologies to do anything, in this case disk encryption. I have had
encrypted containers in my machine for a while and when I started
using them, loop-AES was a recommended solution. The way loop-AES works is
by "butchering" the loop functionality in the kernel (the
functionality that allows you to mount a file as if it were a device)
with extra encryption-fu. Therefore, you make sure your kernel has the
loop device as a module, install the Debian package loop-aes-source,
then use the module assistant to create a custom Debian package with
the new loop.ko, install it and then you are ready to rumba. Of
course, that does it at the kernel level, so you need modified
user-space tools (mount, losetup, swapon, etc) to access this new
functionality (selecting encryption algorithm, setting up keys,
encryption parameters, etc). These modified versions of existing tools
comes in the package loop-aes-utils, which diverts
/bin/mount (and others) to their modified
versions.
Loop-AES is half the story, the other half is FUSE, File System in User-space, a kernel module that allows you to have the file system driver in, well, user-space. Even though it doesn't have particularly good press, two file systems I rely on are FUSE-based: NTFS-3G and ZFS-FUSE. The NT file system moved out of the kernel due to a number of issues (the driver in Linux 2.4 caused data corruption and NTFS is structured as a database, ensuring data consistency is easier using recursion, which is usually unacceptable in kernel code). On the other hand, ZFS is Free Software, developed by then SUN Microsystems but its license under the CDDL, which is GPL-incompatible. There, using FUSE allows for a technical solution to a legal problem. The original ZFS code is kernel-quality, so the Debian version based on a FreeBSD kernel, Debian/kFreeBSD incorporates ZFS directly in the kernel (and this relates to a last-ditch solution I considered but didn't have to try).
My loop-AES blues started when I upgraded my laptop back in Argentina and couldn't access the NTFS partition on the dual boot system. That got made even worse due to the fact that GrUB won't be able to recognize the NTFS partitions without being able to mount them (that might be a bug, I should look into reporting it) and then deleted the bootloader entry to boot the Windows partition (so I had to recreate the chainloading by hand, not very difficult, but not pretty either). If that happens to you, just add to /etc/grub.d/40_custom the following lines:
menuentry "windows" {
set root=(hd0,2)
chainloader +1
}
Replace hd0,2 for your partition: (hd0,2) is
/dev/sda2. More worryingly, I couldn't access from Debian anymore an
encrypted TrueCrypt
partition which I share between the two OSes. Annoying, but none of
them was a show-stopper. Unbeknown to me, these were all loop-AES
blues. I could see it was a FUSE problem, but I couldn't find anything
in the Debian
BTS.
When I got back home, it took me some days to merge back my
files from the laptop (yes, I should be using git more comprehensively
next time) and then I was ready to backup the desktop. Then the
loop-AES blues hit hard: I would mount the encrypted partition fine
with losetup -p0 -e aes256 /dev/loop7 /dev/sdc1 but then
zpool import zfs_backup will croak:
cannot mount 'zfs_backup': Input/output error. Make sure the FUSE module is loaded.
A very unhappy puppy, indeed. Good news were, zfs list
will see my data, everything seemed to be there. I
tried with my laptop, but the running kernel had no loop-AES module,
and trying to re-compile it failed miserably (there I missed a very
good clue), so I tried with an old kernel which had it and it wouldn't
work. Exactly the same error message as above. Searching for it on-line
wouldn't help either. In syslog I found zfs-fuse: kstat:
fuse_mount error - trying to umount but those went away when I
started running zfs-fuse with
--no-kstat-mount. Adding debug to
fuse-mount-options in /etc/zfs/zfsrc didn't
make a difference (and there was a good clue I missed
completely as the diverted /bin/mount wasn't using /etc/zfs/zfsrc).
By then I was fairly down on my loop-AES blues. No bugs filled, no
idea where the bug was, no clear culprit. Besides, both FUSE and
loop-AES didn't seem to be very popular en the BTS. One had a recent
request to be orphaned,
the other had a FTBS bug
saying it was unfit for Wheezy. It seemed I had based my
backups in technology that was going away. Fast. But AES256 and ZFS
are available outside GNU/Linux. So just when I was resigned to set-up
a kFreeBSD box to re-enable my backups, I read in some random bug post
a request to upgrade /bin/mount.
"Here goes nothing", I said to myself. If I were to submit a bug
report, I needed to be running unstable. So I upgraded my laptop to
unstable. Now, I couldn't use encryption with the old kernel anymore:
loop-aes-utils conflicted with
fuse. Then it all made sense, the diverted version of
/bin/mount was breaking FUSE and taking it out made
NTFS-3G and TrueCrypt work again! Sadly, I needed the encrypted loop to mount the
ZFS.
Luckily, AES256 is not anything particularly strange cryptographically speaking (and according to Manning it doesn't take the government more than a few weeks to break) so I followed a very good migration guide and found a great script that provides loop-AES using the dm-crypt capabilities of modern kernels (local copy).
That was the end of my loop-AES blues. Moral of the story: if things aren't working... switch to unstable.
I have all the disk on my laptop running dm-crypt partitions :-D
permalinkIs the overhead really worth it?
permalink
I don't like the idea of getting my laptop stolen... but given that possibility, I don't like either the idea of someone who could access or steal my laptop being able to read the data I store in.
It is not really important nor sensitive, it is private and I love my privacy :-)
I imagine the troubles I could have entering a country like .us when a marine asks me about why my laptop has password protected access. And even worse, once the disk were decrypted and nothing interesting found inside, the time I could spend giving explanations for not being arrested :-/
Couldn't you just keep the standard Debian utils-linux-ng (mount etc) in place for fuse and Debian package dependency joy and just compile and install the loop-aes patched utils in say /opt/bin?
Then use the patched utils from the command line when you need loop-AES. I'm about to try this out myself (not on Debian) so we'll see.
From what I read, support for mounting loop-AES 3.x encrypted partitions in the dm-crypt stuff is rather experimental.
Also: loop-AES is reported to be quite a bit faster than dm-crypt in some circumstances, which is not surprising since it's (a) in the kernel, and (b) monolithic-coded. There is also an argument that loop-AES's multikey design makes it more difficult to reverse-engineer a plaintext key from watching changes in ciphertext (or something like that, I'd have to find the reference).
Besides which loop-AES has had years to mature under one developer and a reputation for rock solid reliability on large datastores. Not sure that dm-crypt has that track record yet.
No, I haven't, but the idea sounds intriguing. I don't know enough
about the interaction between regular losetup and the loop-AES patched
one, but it might be something that can be looked upon.
Very interesting info about Loop-AES vs. dm-crypt. I haven't had any
noticeably slow down but my use case is dominated by the speed of the
USB connection anyway. The possibilities of losing data are indeed
worrisome so I might look into your suggestion at some moment.
Thanks for your post!
Hi,
Good news. The conflict between loop-AES patched utils and fuse is now been *fixed* in Ubuntu natty and oneiric and also fixed upstream in loop-AES sources. Not sure about Debian.
See my post here:
http://sourceforge.net/tracker/?func=detail&aid=3421774&group_id=28891&atid=394668
Ref:
https://launchpad.net/ubuntu/+source/loop-aes-utils/2.16.2-1ubuntu1
https://launchpad.net/ubuntu/+source/loop-aes-utils/2.16.2-2ubuntu1
Please spread the word. This is no longer a reason for mass rollback of loop-AES in the main distros or for downgrading fuse to fuse-2.8.1.
Phil
Thanks so much for the information. I have update the blog post
pointing to your comment.
My main concern is that, as indicated by the changelog in one of your
links, the package has been orphaned (the current maintainer at loop-aes-utils
is marked as the Debian QA group, compatible with an orphaned
package), so its future in Debian is uncertain.
You seem to feel very strongly about Loop-AES, would you consider
maintaining the package? It is not a crazy amount of work, you can
ask the previous maintainer, he might be willing to sponsor your
uploads.
Thanks again for your post :-)
Hi
I know that Max Vozeler unfortunately has retired as maintainer of the loop-AES Debian packages - I saw his post about this on a Debian bug tracker.
I do think loop-AES has been sold short and some distros have acted far too quickly to remove it. A conflict like you experienced with Fuse is not surprising because loop-AES, while famously leaving the rest of the kernel intact, forks both mainline loop module and mount utilities. This means Jari (loop-AES author) has to consider which changed features in the mainline loop module and utilities he will continue to track in loop-AES and which he will not.
It really would have been a lot easier (and created fewer issues) if loop-AES had just been adopted by the mainline kernel maintainers years ago once it was obvious it was the state of the art at the time. Instead they "maintained" a dangerously broken cryptoloop implementation in kernel, who knows why - I suspect because they were just waiting for the device mapper stuff to mature enough so they could forget all about encryption. There must have been an element of policy enforcement - device mapper was the ordained religion (it had been planned as such) but the fact remains that loop-AES was clearly superior to dm-crypt for years.
Actually I think there has been far too little debate about the relative strengths and weaknesses in these two approaches as dm-crypt has improved and gained ground; certainly there has not been enough discussion to just dump it on "oh dm-crypt does that better anyway" grounds. A device mapper advantage is that filesystems can be non-destructively resized without drama. That's the only real advantage I see. As for overhead, with loop-AES on a machine less than ten years' old you won't ever notice any. As an experiment, it's possible to alter my script tripl (http://sourceforge.net/projects/tripl/), which aids in the set up of multiple encryption, to do (say) nine layers of encryption. When I did so I was able to play video through many encryption layers of loop-AES using different ciphers without any noticeable latency on an average aging laptop. Loop-AES is efficient.
I often think something must have gone wrong with kernel politics (of which I have only a distant understanding) and early loop-AES. I know that Jari was not shy in criticizing the borken cryptoloop implementation but why should he have been. It was bad and someone needed to tell people it was bad and it should have been fixed if dm-crypt wasn't yet ready for showtime (which is wasn't). There were also email list squabbles with the cryptsetup developers who I think responded poorly and reactively.
I have thought about offering to do the Debian packages, dunno. I'll think about it.