But we’re not looking at the virtues of dd here. We’re looking at its annoyances and dd has one particularly glaring annoyance – a lack of display of progress. You could tell dd to start imaging your multi-terabyte hard-drive and not have any indication of how far it has gone – you just have to wait until it finishes. The dd command only outputs some information right at the very end of its job, which could well be several hours later. The only indicator that you have that something is happening is your hard-drive light madly flashing away.

Luckily while dd doesn’t show progress during its tasks, it can be prodded externally to give up information about itself as it runs, and we can achieve that by using the kill command without actually killing the dd command’s execution.

Now the kill command doesn’t just terminate tasks. It can also send other standard POSIX signals to applications. Many applications are written to acknowledge certain signals and perform an action on receiving them. In the case of dd, there is one signal that will make it output its summary information to the console as though it had finished, but rather than terminate, it keeps on going. The signal in question is USR1.

Here’s how to make use of it:

1. First we need a dd process running. Let’s say we are imaging our first fixed hard-drive /dev/sda to an image file called MyHDDBackup.img using the following command:

   $ sudo dd if=/dev/sda of=/home/jbloggs/MyHDDBackup.img

2. Now while that is running, open up a second terminal and type in the following followed by Enter:

   $ sudo kill -USR1 `pgrep ^dd`
   

   Breakdown of the above command:
   

   • sudo runs this command as root. In this case since we’re using dd as root, we also need to run the kill command as root or we won’t be able to send anything to dd.
   • kill is the command we are using to send dd a signal with.
   • -USR1 is the signal we are sending to dd.
   • pgrep is a program that will show you the process ID of a given application, in this case we use the argument “^dd” to say “locate any program that starts with the text ‘dd’ only” otherwise we would get other processes that containg the letters “dd” together in their name if we left out the chevron character. Note that the pgrep command is enclosed in “`” quotes so that it is executed and returns just a process ID number to the kill command.

3. Now have a look at the terminal that dd is running in. It has suddenly output some data about itself, specifically, how much of the task had been completed, but dd is still running! This is great – but how can we regularly have dd tell us where it’s up to without having to manually enter the above command each time? Simple, by combining it with the watch command:

   $ watch 'sudo kill -USR1 `pgrep ^dd`'
   

   The watch command will will re-execute the kill command once every two seconds, giving you near-realtime status to what dd is doing. If you want to change how often the watch command polls for, you can change it with the -n parameter, for example to change the polling to every 5 seconds, you would use:

   $ watch -n5 'sudo kill -USR1 `pgrep ^dd`'

While not providing any means of showing you how long it will take for dd to ultimately finish, you can now make use of the transfer speed shown along with the data processed thus far and calculate a rough idea of when dd will complete its task now.

Pat yourself on the back. You can now rest easy knowing that you can find out at anytime where dd is up to.

APC make some great UPS products and they all have the ability to communicate with a host PC to advise of its state, eg: on mains, on battery, fault, etc. The support software is available aplenty for Windows and Mac, but what about Linux?

This guide will show you how to hook up an APC UPS to an Ubuntu-based Desktop or Server PC and allow your UPS to email you when mains power has failed, when mains power has been restored, and also give your PC ample opportunity to shutdown when battery on the UPS gets to a critical low.

Pre-requisites:

• Ubuntu-based PC. I am using Ubuntu 10.04 Lucid Lynx in this guide.
• An APC UPS. I’m using a Back-UPS RS 800 in this guide.
• Details about your ISP’s mail server, or your own local mail server.

Configuration:

1. Setup your APC UPS as outlined in the instruction manual. Ensure that the USB or serial cable is connected from the UPS to the PC. This is what the UPS uses to communicate with the PC.
   .
2. On your Ubuntu PC, go into a Terminal. If you are using Ubuntu Desktop, you can do this by going to the Applications menu, then Accessories and then Terminal. If you are using Ubuntu Server, simply login to your server by the console or SSH in remotely.
   .
3. The Ubuntu repositories contain a UPS daemon specifically designed with APC UPS units in mind called APCUPSD, so let’s install it:

   $ sudo apt-get install apcupsd

4. Once installed, we need to configure it to suit your UPS. Bring up the configuration file by typing in:

   $ sudo nano /etc/apcupsd/apcupsd.conf

5. In the editor, scroll down to UPSNAME and give the UPS a name, eg:

   UPSNAME MyAwesomeUPS

6. Scroll down to UPSCABLE and change the parameter to the type of cable you are using to connect to your UPS. In my case, I use a USB cable that came with the UPS, so I changed this line to read:

   UPSCABLE usb

7. The next line to modify is the UPSTYPE line. Again, because I’m using USB I specify:

   UPSTYPE usb

   If you have a DEVICE /dev/ttyS0 line after the UPSTYPE line, comment it out by adding a hash symbol at the start of the line like so:

   #DEVICE /dev/ttyS0

8. We do not need to modify anything else, so press CTRL+X, then “Y” and then Enter to save your changes and exit the Nano text editor.
   .
9. Now we need to tell the system that the UPS daemon has been setup and is essentially ready to go. Type in:

   $ sudo nano /etc/default/apcupsd

10. In the file that appears, modify the ISCONFIGURED line to say Yes”, ie:

    ISCONFIGURED=yes

11. Press CTRL+X, then “Y” and then Enter to save your changes and exit.
    .
12. We’re pretty much done here. Start the daemon now with the following command:

    $ sudo service apcupsd restart

    …or simply reboot your PC. Your PC is now monitoring the state of the UPS.
    .

13. Let’s check the UPS now. Type in the following command:

    $ apcaccess

    …and you will get output similar to the following:

    $ apcaccess
    APC      : 001,043,1045
    DATE     : Tue Feb 01 00:02:36 EST 2011
    HOSTNAME : lamaar
    VERSION  : 3.14.6 (16 May 2009) debian
    UPSNAME  : LAMAAR
    CABLE    : USB Cable
    MODEL    : Back-UPS BR  800
    UPSMODE  : Stand Alone
    STARTTIME: Mon Jan 31 23:43:37 EST 2011
    STATUS   : ONLINE
    LINEV    : 250.0 Volts
    LOADPCT  :  40.0 Percent Load Capacity
    BCHARGE  : 100.0 Percent
    TIMELEFT :  19.9 Minutes
    MBATTCHG : 5 Percent
    MINTIMEL : 3 Minutes
    MAXTIME  : 0 Seconds
    OUTPUTV  : 230.0 Volts
    SENSE    : Medium
    DWAKE    : 000 Seconds
    DSHUTD   : 000 Seconds
    LOTRANS  : 194.0 Volts
    HITRANS  : 264.0 Volts
    RETPCT   : 000.0 Percent
    ITEMP    : 29.2 C Internal
    ALARMDEL : Always
    BATTV    : 27.4 Volts
    LINEFREQ : 50.0 Hz
    LASTXFER : Low line voltage
    NUMXFERS : 0
    TONBATT  : 0 seconds
    CUMONBATT: 0 seconds
    XOFFBATT : N/A
    SELFTEST : NO
    STATFLAG : 0x07000008 Status Flag
    SERIALNO : xxxxxxxxxxxx  
    BATTDATE : 2001-09-25
    NOMOUTV  : 230 Volts
    NOMINV   : 230 Volts
    NOMBATTV :  24.0 Volts
    NOMPOWER : 540 Watts
    FIRMWARE : 9.o5 .I USB FW:o5
    APCMODEL : Back-UPS BR  800
    END APC  : Tue Feb 01 00:02:41 EST 2011
    $

14. Pat yourself on the back, you’re basically done.

But hang on, what about setting up email notifications? Read on.

Setting up Email Notifications

By default, APCUPSD is configured to use sendmail to send emails, but sendmail is finicky. There is a better way using a similar application called sendemail instead.

Why use sendemail instead of sendmail? Well, sendemail is a simple command-line SMTP client. You construct a simple message with a recipient and off it goes, but the key difference here is that sendemail is a CLIENT, not a SERVER. This means for sendemail to work, you need a third-party mail server to send it through. You may already have a mail server on your network running Postfix or similar. If not, your ISP will most certainly have one that you can use.

1. Let’s start by installing sendemail:

   $ sudo apt-get install sendemail

2. Now we need to configure apcupsd to use it. If you have a look inside the /etc/apcupsd directory, you will see several important files as follows:
   • changeme – notifies you if the UPS battery needs changing.
   • commfailure – notifies you if the PC loses communication with the UPS.
   • commok – notifies you if lost communication is restored with the UPS.
   • onbattery – notifies you if the UPS engages battery mode due to mains power failure.
   • offbattery – notifies you if the UPS returns to mains mode after a mains power failure.
     .
3. Let’s modify the onbattery message:

   $ sudo nano /etc/apcupsd/onbattery

4. You will notice there are some lines in this file that define some variables followed by a message and then finally a command that sends an email using the sendmail app. Since we are not using sendmail, we need to modify a number of areas. Replace the content of the entire text file with the following:

   #!/bin/sh
   #
   # This shell script if placed in /etc/apcupsd
   # will be called by /etc/apcupsd/apccontrol when the UPS
   # goes on batteries.
   #
   SYSADMIN=jbloggs@mydomain.com.au
   APCUPSD_MAIL="/usr/bin/sendemail"
   HOSTNAME=`hostname`
   MSG="Ubuntu PC Power Failure !!!"
   #
   (
   echo " "
   echo " ====================================="
   echo " POWER FAILURE ON $HOSTNAME !!"
   echo " ====================================="
   echo " "
   echo " The UPS on your Ubuntu PC has experienced a power problem that has required the UPS to engage battery mode."
   echo " "
   echo " Do not panic! Remain calm..."
   echo " "
   echo "Current UPS status:"
   echo " "
   /sbin/apcaccess status
   ) | $APCUPSD_MAIL -u "$MSG" -f MyUbuntuPC@mydomain.com.au -t $SYSADMIN -s mail.myisp.com:25
   exit 0

   So what have we done here? The first part of the file sets some variables, namely the email address(es) of the people required to be notified. If you need to send to more than one recipient, separate email addresses with a comma, eg:

   SYSADMIN=jbloggs@mydomain.com.au,fredsmith@thatotherdomain.com

   We then define the path to the sendemail app, the hostname of the PC running the UPS daemon, a subject line for the email in question, and then a big email body explaining what’s going on, in this case that the UPS battery mode has been engaged.

   All this is then piped into the sendemail application with a fake sender’s address to give you an idea where it came from and we also specify the mail server that the email will be sent via. In this case, we are using our ISP’s mail server that has the address mail.myisp.com
   .

5. Save your changes by pressing CTRL+X, then “Y” and then Enter.
   .
6. You can now test the script by simply executing it (we don’t have to disconnect the UPS’ mains cable to trigger this until we’re ready to test that). Execute the file using the command:

   $ sh /etc/apcupsd/onbattery

7. Check for any errors and then check to see if you got an email in your mailbox advising that the UPS is on battery (the UPS of course is not, we’re just testing the email).
   .
8. If all is well, continue to change the other four message files to be similar to the above, changing the message as you go of course.
   .
9. Pat yourself on the back – you now have custom, explicit messages to let you know what’s going on with the UPS.

Setting up your own mail server to use instead

If, in your testing, you discover that your ISP’s mail server does not allow you to randomly spam messages to it using fake from addresses, you have two choices – either specify a valid email address as the sender’s address, or you can setup a simple email server of your own using Postfix:

$ sudo apt-get install postfix

In the case of setting up your own email server, simply go with Postfix and set it up as an “Internet Site”. The basic configuration is good enough to deal with our needs, but make sure your basic network security is adequate to prevent outsiders from trying to use your mail server to send unsolicited mails, or spam, through it.

This article will not describe how to setup or secure Postfix as it is beyond the scope of this article, however you do need to modify the outgoing mail server name in all your communication messages to reflect the change of name or internal IP address of your mail server instead of mail.myisp.com.

Testing

You can test your daemon setup easily enough by doing any of the following:

• Remove mains power cable from UPS.
• Remove the USB cable to the UPS.
• Verify that an email is sent for either of the error conditions above.

Enjoy!

This article discusses how I used my recently purchased LG GGC-H20L Blu-ray ROM drive to successfully read and watch movies using Ubuntu Intrepid.

DISCLAIMER: This article describes decrypting BD titles using an Intel or AMD based PC with Ubuntu Linux. While you can use a PlayStation3′s BD drive to read and decrypt a title using known decryption keys using the PS3 version of Ubuntu, at this time of writing you cannot use Ubuntu installed on a PlayStation3 console to identify unknown decryption keys of a given BD title because the application used to derive those keys from the disc is not available for the PPC processor used by the PS3. You must use a consumer BD-ROM drive on an Intel or AMD based PC instead.

Hang on, you say – if there is no support for playback of Blu-ray movies on Linux, then why buy a Blu-ray drive if you can’t watch the movies? Well, I might not be able to watch them directly, but I can certainly rip the little buggers and watch a file version of it instead. But wait again, you say, if there’s no official Blu-ray support, and you can’t watch the discs directly, then how on earth do you rip them?? I’m glad you asked, and even if you didn’t ask, I’m about to tell you anyway.

First up, a little Blu-ray 101.

Blu-ray movies feature Digital Rights Management (DRM). Like DVD’s before it, most Blu-ray movies are encrypted. This is to stop those naughty pirate types from making illegal backups of the movie and giving or selling it to their mates. In the case of DVD’s, however, there was only one decryption key which was eventually discovered and from then on allowed all DVD’s to be easily decrypted using a simple library (called libdvdcss2). The movie industry as a whole were not impressed by this and insisted that the next format be more difficult (preferably impossible) to decrypt, so Blu-rays (and HD-DVD’s, but I’ll concentrate on Blu-ray for this article) each have a different decryption key now. But to complicate this further, this key is kept hidden from you by an authentication mechanism to ensure that there isn’t a repeat of the De-CSS scandal that brought the DVD encryption scheme undone.

Each and every player out there, hardware or software, has a unique player authentication key which is passed to the Blu-ray optical drive, essentially like giving your passport to Customs at the country border, to validate whether or not you are legally authorised to  playback a movie. If the drive has not got a blacklisted record of your authentication key, and the key is accepted as a generally valid key that has been paid for, THEN the drive will give up the movie’s decryption key and movie playback can commence.

Blacklist?? What blacklist? Well, like DVD players, it’s not difficult to pick up a Blu-ray player’s authentication key that is used to prompt the drive for the disc’s decryption key, after all, it has to be held in memory somewhere. Once an industry authority discovers that an authentication key has been compromised, it is added to a blacklist so that it will not work anymore. This is why hardware Blu-ray players need to be firmware-upgradable, and why software players need to be upgraded to the next version periodically with patches, etc, so that new, non-blacklisted authentication keys can be provided.

OK, that sounds all well and good, but if the Blu-ray drive itself is doing the blacklisting, how exactly does it know when a given authentication key is no longer valid? Simple – its blacklist will get updated with the next latest-release movie you buy.

Say what?

Every Blu-ray movie you buy has a little file on the disc under the “AACS” folder called “ContentRevocation.lst”. This file contains a complete list of blacklisted authentication keys for the drive to update itself with, and – get this – you can’t stop the drive loading it. Well, actually to be more accurate this file is simply a copy of the same list that is actually hidden in a non-tamperable, non-user-readable area elsewhere on the disc for the drive to read, but basically the instant you stick that movie disc in, the hidden version of this file is read and the drive automatically updates its blacklist right away with any new blacklist data, even before the disc icon appears on your desktop. Sneaky, huh?

So the next time your legal (or more specifically, pirated) copy of PowerDVD or whatever tries to playback a movie, all of a sudden you’ll see an error message instead saying that your player’s authentication key has been revoked – thus the movie is now unplayable. What’s worse is that you won’t be able to watch any of your older discs that worked previously either! It’s this exact reason that many people have called for the Blu-ray (and HD-DVD) formats to be boycotted.

But not all is lost. Remember, this is an encryption technology created by Man, and therefore can be broken by Man with a bit of help from the Open Source Community at large. With a little help from a few external resources including the Ubuntu Community Documentation and the Doom9 forums, I discovered a plethora of projects by various people, from dumping discs to breaking the encryption and authentication.

On the surface are general applications to dump the movie, decrypted, to a file. One of the best projects is DumpHD which is a Java app that provides a nice easy to use GUI that can rip a movie with a minimum of fuss (see detail about how to use it later in this article), however it requires that you already know what the decryption key for a given movie is which you can only obtain if you have authenticated with the drive. If you have the decryption key already, however, then authentication is not necessary and you can rip the movie right away without a problem, so this project is heavily supported by people posting up various decryption keys for all sorts of movie releases. The problem with this approach, however, is that different countries usually get different releases of the same movie, so for example a release of “Batman Begins” in Australia might have a completely different decryption key to the release of “Batman Begins” in America or Europe. This is not always the case of course, a good example being the Ewan McGregor movie “The Island” – the Australian release is actually the UK release, right down to the age-rating and film-office classification markings on the disc itself – only the box bears any Australian ratings markings!

Click for full size (228K)

So, how do you find out the decryption key of your locally purchased movie then? You don’t want to keep buying a commercial player, especially one that doesn’t run under Ubuntu, just to get a valid authentication key. There’s got to be a better way! Well, there is! How about we just bypass the authentication procedure altogether? How? Again, through another great contribution on the Doom9 forums.

The LG GGC-H20L drive is but one of many Blu-ray/HD-DVD drives which have had their firmware reverse-engineered. Firmware is largely just a computer program that operates the drive. Since the firmware is upgradable to fix bugs and add new features to the drive, it means the program can be altered by a third party. A Doom9 contributor has provided modified firmware for various popular drives, including the GGC-H20L that effectively allow the drive to ignore the authentication procedure no matter what player authentication key is provided, blacklisted or not, thus making the drive give up the decryption key for the Blu-ray movie currently inserted every time!

EDIT 10th Jan 2012: It appears the firmware is no longer available from the Doom9 forums as it has not been updated in quite some time. Here is a mirror of the v1.03 modified firmware to suit the LG GGC-H20L only.

In the case of the LG GGC-H20L that I use, the firmware is provided as a Windows Executable file. There are three ways to run this, either via a native Windows installation on your PC, a virtualised Windows installation on your PC, or via the Wine compatibility layer. I successfully upgraded my drive using the Wine option as follows:

WARNING: The following information can damage or even brick your LG Blu-ray drive if not followed correctly, or if you have a power failure during firmware update. You proceed at your own risk and I will not be held responsible for any damage incurred by your drive, or for loss of hair being torn out, by following these instructions.

1. You need a normal Wine installation. If you’ve never installed it before, then type in:

   $ sudo apt-get install wine

   …and this will install the Ubuntu-repository version of Wine.

2. Since we’ll be modifying a physical device, only root can do that, so we will need to use sudo to execute the firmware upgrade, however Wine will not work as root until we change permissions of your Wine configuration in your Home directory, so type in the following to make root the owner of your Wine configuration:

   $ sudo chown -R root:root ~/.wine

3. Now execute the downloaded firmware file:

   $ sudo wine GGC-H20L_1.03_VolumeID_Patch.exe

4. The Upgrade GUI will appear. Click on the button to commence update and let it do its thing. After a minute or two, it will tell you that the drive firmware has been successfully updated. At this point, you will need to reboot your PC, but before you do, remember to change the owner of your Wine installation back to yourself. If your login was “jbloggs”, then you’d type in:

   $ sudo chown -R jbloggs:jbloggs ~/.wine

Upon returning from restart, every time you query the drive for the inserted disc’s decryption key, the drive will now happily give it to you without question. Nice.

So how do we query the drive for that decryption key anyway? A simple tool to do this is aacskeys (version 0.4.0c at this time of writing, but check further along the thread for newer releases) written by another Doom9 member, which queries the drive and tells you its Volume Unique Key and its Disc ID, which you can then copy and paste into DumpHD’s keys config file and happily begin dumping your movie.

$ ./aacskeys /media/cdrom aacskeys 0.3.6 by arnezami, KenD00 Volume Unique Key:              5D9BCD44522B6940F8705400DA612ED9 Unit Key File Hash (Disc ID):   837487B4D6F614D5B4D5F566387B41C2D284F393 $ 

If your drive’s firmware was not patched, instead of seeing the Volume Unique Key and Disc ID, you would get this error message instead: “The given Host Certficate / Private Key has been revoked by your drive.”.

We now need to take the output data and copy it to DumpHD’s “keydb.cfg” file. Each key is placed on its own line in the following format:

DISC ID = Movie Title | D | YYYY-MM-DD | V | VOLUME UNIQUE KEY

Thus in the example above, we would enter:

837487B4D6F614D5B4D5F566387B41C2D284F393 = The Island | D | 2007-03-22 | V | 5D9BCD44522B6940F8705400DA612ED9 

Now technically it appears that the date is largly irelevant, and most people just use 0000-00-00 instead of a real date (which is supposed to be the file date of the “Unit_Key_RO.inf” file in the “AACS” folder of the disc). I have tested this and I can’t see any difference in how DumpHD handles the disc.

Once you have finished editing the keydb.cfg file, save it.

NOTE: DumpHD (below) can now use AACSKeys directly, saving you having to edit to the keydb.cfg file manually, because DumpHD now does all the work for you. See this article for more information.

Since DumpHD is a Java application, you will need Java installed to run it. If you haven’t already got it installed, you can install it with the following command:

sudo apt-get install java-common

…or more realistically you should install it as part of the ubuntu-restricted-extras package which also installs a number of other useful packages:

sudo apt-get install ubuntu-restricted-extras

Once that is done, launch DumpHD by simply running the “dumphd.sh” script by either double-clicking on it and select “Run” when prompted, or in a terminal, change to where you extracted the DumpHD program and type in:

$ sh ./dumphd.sh

When loaded, you will be presented with the following interface:

Click for full size (63K)

At the top-right of the window, there is the Source Browse button. Click on it and a new window will appear. In that window, type in or select “/media/cdrom” and then click OK. After a few seconds, the disc should be identified and you will see the window change as follows:

Click for full size (85K)

Now all you need to do is click on the Destination Browse button, specify a place to store the decrypted movie and then click the Dump button to start the whole process! Once finished, you will have every video title found on the disc dumped in its originally encoded (video-wise) format, but without DRM. You can then use MPlayer to play these files directly. Generally the movie itself is the largest file, so in the case of my example, it’s the “00000.m2ts” file. I can play it simply with:

$ mplayer -fs 00000.m2ts

The -fs parameter plays the movie full-screen. I can toggle audio tracks using the hash (#) key (as for some reason, English is generally not the first audio track on the disc).

Remember that Blu-ray movies are very large. In my example, “The Island” is a good 21GB! It’s now up to you to decide whether or not you want to provide storage for your rips of this size, or whether or not you want to compress them down to save space. In my case, I was able to compress the movie down to about 4GB with negligible quality loss at a bitrate of 1200 using the x264 codec. I’ll probably increase this bitrate and allow the filesize to go to 8GB so I can maintain as near-perfect image quality to the original Blu-ray as possible. I personally choose to preserve the audio tracks as-is without down-converting them – they really don’t eat up that much space – a few hundred megabytes only.

Happy ripping!

Woah! After I’ve ripped the movie, it plays but it’s all corrupted! What’s going on?

The movie you have ripped is likely protected using BD+ protection. This is where some or much of the movie is deliberately corrupted to annoy you. I have written a guide on how to deal with this and correct the corruption here.

• Backup to an image file
• Clone to another hard-drive (eg: building multiple identical workstations) either directly or via an image file
• Data recovery (it’s safer and easier to examine an image file than risk further damage to the hard-drive itself)

Linux has a neat little command that can do this for us called simply “dd”. It is completely filesystem independent, so you can backup any hard-drive regardless of whether it was Linux formatted, Mac formatted or Windows formatted. It copies the drive bit by bit, sector by sector, not file by file.

WARNING: The following article describes commands that can destroy the information on your hard-drive with next-to-no-chance of being able to rescue what’s left. Make sure that you have full backups of your data prior to using these commands. You use these commands at your own risk!

First up, your drive/partition being backed up or written to should be unmounted before you begin. If you were trying to backup the partition containing your system’s root filesystem, you should boot up your system on a LiveCD so you can access your root partition unmounted.

You then need to determine which drive or partition you want to backup. Easiest way to do this is to fire up a terminal session and type in:

$ sudo fdisk -l

…which will list all your drives, their partitions and sizes. Note down the device name of the drive or partition you wish to backup, eg: /dev/sda for the whole of your first drive or /dev/sda1 for just the first partition of that drive, and you’re ready to begin backup.

At its simplest, backing up a drive takes two basic parameters – input device and output device or file as follows:

$ sudo dd if=/dev/sda1 of=/dev/sdb1

…will copy the content of the first partition on drive sda to the first partition of drive sdb (destroying whatever was in /dev/sdb1 previously).

$ sudo dd if=/dev/sda1 of=~/mybackup.img

…will copy the content of the first partition on drive sda to a file called mybackup.img in your Home directory.

If you wish to backup an entire drive rather than just a partition, drop the number on the drive name, ie: /dev/sda1 becomes /dev/sda instead.

To restore the image to a drive, simply reverse the parameters:

$ sudo dd if=~/mybackup.img of=/dev/sda1

How simple is that? The only issue with this method of backup, though, is that if you were to backup a 10GB partition containing only 5GB of actual data, the resulting image file will still be 10GB as dd does not simply copy active data from the drive, it copies everything.

So with that in mind, let’s compress the image, so any empty space on the drive doesn’t take up unnecessary space in the image file. We can do this by piping the data from dd into gzip before writing it to a file:

$ sudo dd if=/dev/sda1 | gzip -9 > ~/mybackup.img.gz

…and to restore, we simply pipe the decompressed data into dd again for writing to the drive:

$ gunzip -c ~/mybackup.img.gz | sudo dd of=/dev/sda1

But wait! There’s more!

This just scratches the surface of what you can do with dd. For example, you can also use dd to destroy a drive’s content before you dispose of, or sell the drive by using the following:

$ sudo dd if=/dev/random of=/dev/sda1

…which will write random data to the partition sda1 using the virtual random device, completely obliterating it.

Or, if you want a more uniform destruction, try:

$ sudo dd if=/dev/zero of=/dev/sda1

…which will – no surprises here – write zeros across the entire partition.

Or perhaps you have a drive that isn’t playing nice, for example Microsoft Windows can get stroppy when working with, or removing partitions created by a foreign OS. The easiest way to fix this is to simply destroy the Master Boor Record (MBR). You can achieve this by specifying a blocksize parameter. The MBR is inside the first 512 bytes of the drive, so all we have to do is blank those 512 bytes out using:

$ sudo dd if=/dev/zero of=/dev/sda bs=512 count=1

The bs parameter specifies the block size in bytes (in this case 512) and the count parameter specifies how many blocks of 512 bytes we’re doing (only one in this case, the first 512 bytes of the drive where the MBR resides).

Any PC will now view this hard-drive as a brand new, never partitioned drive.

Rescuing a dying hard-drive

The dd command is particularly useful for recovering data on a drive that’s about to become a paperweight. As long as it can power up and identify itself to the system, you can attempt data recovery by taking an image of the drive and then try to recover the files from that image.

Since dying hard-drives are typically sown with errors, we need to tell dd to ignore them and keep pulling as much data off as possible. Taking our first imaging example, we add a couple of parameters to the command as follows:

$ sudo dd if=/dev/sda of=~/mybackup.img conv=noerror,sync

This will make a full backup of all partitions on drive /dev/sda to a file called mybackup.img in your Home directory, and the conv parameter tells dd to ignore any errors found during the read by skipping over them. The sync parameter tells dd to replace the errors with null values – blanks – so that the resulting image file is the same data length as the drive.

Once the backup is complete, you can then mount the image to view its content. As long as the MBR (the first 512 bytes of the hard-drive) were not damaged, you will be able to now recover most, if not all your files, without worrying about further failure to the drive or odd behaviour due to the physical errors on the drive itself.

To mount the image file, you do the following. First we need a mountpoint – a place to reference the mounted image volume:

$ mkdir /dev/shm/myrecoveryimage

This creates a directory called “myrecoverytimage” in your system’s RAM drive. Before you point out that you have less physical RAM than the capacity of your hard-drive, don’t worry – this is just a mount-point. The image will still be physically read from wherever you put it on another drive.

Now we mount the image. Let’s say we backed up a standard Linux ext3-formatted drive:

$ sudo mount -t ext3 -o loop ~/.mybackup.img /dev/shm/myrecoveryimage

…or for a Windows NTFS formatted backup image:

$ sudo mount -t ntfs -o loop ~/.mybackup.img /dev/shm/myrecoveryimage

We have told the system to mount an image file of filesystem type Ext3 (or NTFS). We have to specify the loopback device because an image file is not a block device like a hard-drive is. We then specify the filename of the image we are wanting to mount, and finally where we are mounting it to.

You can now simply switch over to the mountpoint directory and see the content of the image file as though it were any normally mounted hard-drive.

$ cd /dev/shm/myrecoveryimage
$ ls -l

There are a plethora of filesystems that the mount command can manage including FAT, NTFS network filesystems etc. Refer to the man page for the mount command for more details.

When you have finished with the image, you can unmount it with:

$ sudo umount /dev/shm/myrecoveryimage

…and that will make the /dev/shm/myrecoverimage directory revert back to an ordinary empty directory again (which, being in the RAM disk, will be lost when you reboot).

Happy dd’ing!