Georgia Hilton MPE CAS MPSE Prod/Dir/Editor

Film Services Professional ( Producing / Editing / Delivery )


From Matt at Digidesign. Concerning Lost Files.

Posted on February 28, 2010 at 12:51 AM

From Matt at Digidesign. Concerning Lost Files. Very cool! Ifeel your pain. I had a very similar problem recently. I backed up asession then put it in the Trash and Emptied and then realized I hadaudio files in that session folder that were from a different session.Since I had used Save Session Copy In to do the backup, those filesweren't copied. So they were gone. Or so I thought.

First and most important thing: Do NOT use the hard disk for anything!!Don't even launch Pro Tools with the thing connected. Pro Tools mayvery well update the .ddb file on that drive which could overwriteimportant data. Just leave it unconnected until you are ready toattempt recovery, which I will describe next.

You will need three programs (well, two actually, but a text editor makes life simpler):

1. Terminal (found in your Applications:Utilities folder)

2. HexEdit (freeware)

3. TextEdit (in your Apps folder) or, my favorite, TextEdit (by Haxial)

You will also need an extra hard disk or two, and a Pro Tools systemcapable of playing the same audio files you are trying to recover(i.e.: if you're trying to get back 96kHz files, you need a 002 or HDrig.) For simplicity's sake we're going to refer to your precious harddisk as the Source Disk and the extra hard disks as your DestinationDisk(s).

Here's what you're going to do

1. Read raw data off of the hard disk, 1GB at a time, to a second hard disk.

2. Attach audio file wrapper data to each raw data file. Do this THREEtimes if you are trying to recover 24-bit audio data. Do it TWICE ifyou're recovering 16-bit data. I'll explain.

3. Import the new audio files into Pro Tools.

4. Manually comb through the audio files in Pro Tools to find the datayou want. This part of the process may take quite a while and test yourpatience, but if the data is really important, you'll do it.

Some caveats (in no particular order)

1. This is going to take time. Quite a bit of time. If you need thisdone quickly, and you're getting paid, it might be better to send it toa data recovery company and pay big $$. That's up to you.

2. This is going to take up a lot of disk space. The bigger the harddisk you are trying to recover, the more space you'll eventually need.As a rule of thumb, figure on 4x the amount of disk space from theoriginal drive. If you don't have this much extra space, you can do itin chunks, but some of the time-saving techniques won't be as useful.

3. You need to know what type of audio files you're looking for. Fileformat, sample rate and bit depth. If you are trying to recover a wholebunch of different file types or you don't know the types, you're infor a REALLY LONG HAUL. You'll basically have to repeat this entireprocedure for each file type. If you're willing to do that you musthave some really important audio files to recover. Good luck.

4. Again (worth repeating), do NOT write to your affected disk driveuntil you have recovered your audio or given up. This is reallyimportant. Since you emptied the trash, the computer doesn't know wherethose audio files are and could write over them without warning if yousave something to that disk.

How to do it

1. Connect your Source and Destination Disks to your computer.

2. Boot up.

3. Launch Terminal.

4. In Terminal type su. Enter your password. You're now in superuser mode. Be careful.

Now you need to find out some info via unix:

5. In Terminal type df. This will show the mount point of your drives.On far right is the name of the disk; e.g. /Volumes/MyDiskName. On farleft is "unix device mount point", e.g. /dev/disk1s9. 1 is the disknumber (e.g. 1st disk found since booting). 9 is the partition number.

6. Find your disk in this list by the name and note down the mountpoint. This is how you will tell unix where to read raw data from.

Here goes the main recovery effort. I suggest you read ahead before you actually type this stuff.

7. In Terminal type dd if=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/01 count=2m

Okay, what's all this about?

dd This is a unix command that allows raw data reading/writing.

if= Tells dd command that this is the Input File.

/dev/rdisk1s9 This is your mount point that you found in step 5. Thenumbers will probably be different on your system. The r is added in totell dd that you want to do a raw disk read. The r is very important!

of= Tells dd command that this is the Output File.

/Volumes/MyDestinationDisk/01 Here's where you need to put in yourDestination Disk name. Keep the /Volumes/ part since this is the sameon all OS X systems and then type the name of the hard disk volume youwant to write to. Add a slash after the name and then type a file namefor the new raw data file you're going to create. I use numbers, like01 since this command will be executed many times. Each time I increasethis file number by 1 to keep things organized. (Unix scripters willsee opportunity for automation here but doing it manually gives thesame results.)

count=2m This tells dd how much data you want to read. 2m=1GB. (Unix deals in 512 Kilobyte chunks.)

8. If this all looks fine and dandy to you, press Enter. You won't seemuch going on but if you look at your drive access lights you'll seethat reads and writes are occuring.

Now you may be wondering how unix knows which 1GB of data to read offof your drive? Simple. It just reads the first 1GB of data. So how doyou get it to read the 2nd GB? or the 3rd? Or the 49th? Easy. Just addan additional command at the end of the dd line that says skip=2m. Thistells dd to start reading raw data 1GB from the beginning of the disk.You'd use this to create your second raw data file. Your third filewould need skip=4m added to it. The fourth will need skip=6m. Etc. Ahandy equation for this is of=N skip=(N*2-2)m. I.e.: Your Output Filenumber is N and the number in the skip part is N*2 - 2.

So your second raw data recovery will look like:

dd if=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/02 count=2m skip=2m

Your third raw data recovery will look like:

dd if=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/03 count=2m skip=4m

Your fourth raw data recovery will look like:

dd if=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/04 count=2m skip=6m

And so on. Until you've recovered all your data or run out of diskspace on Destination Disk. If you have to stop in the middle, make noteof where you left off and DISCONNECT your Source Disk.

How to speed up the raw data recovery process.

You may have noticed that each 1GB of data takes a long time torecover. I don't suggest you use your computer for any other tasksduring this process so you probably want to do this stuff late at nightor whenever the computer is not in use. But you don't want to be therebabysitting the thing all night long. There's a solution. You can typecommands into unix one after another if you seperate them with asemicolon. That's what I use TextEdit for. It's much easier to copy andpaste a whole bunch of those commands into TextEdit, then scrollthrough and change the Output File names (the numbers) and the skipcommands. Then you can copy and paste out of TextEdit back intoTerminal. You'll end up with something like this:

dd if=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/01 count=2m; ddif=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/02 count=2m skip=2m; ddif=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/03 count=2m skip=4m; ddif=/dev/rdisk1s9 of=/Volumes/MyDestinationDisk/04 count=2m skip=6m; etc.

I find that resizing the TextEdit window so that one dd command fitsperfectly on a line also helps in making the edits correctly.

Be careful! This is a powerful technique for getting things done in abig batch, but if you make a small typing error you can do disastrousthings to your hard disks. Or you can accidentally write each new rawdata file over the previous one, which means you won't really know whatyou recovered and you'll have to start over. If you know how to writeshell scripts in unix, I'm sure you'll be automating this wholeprocess. I found it more satisfying to be able to see each and everycommand written out before I let them loose on my system.

Also, don't forget that you need to do the superuser mode change eachtime you launch Terminal. And you should recheck the df command to makesure the mount points didn't change (they can every time the systemboots.)

You're one-third of the way there.

The next step is to add the audio file wrappers so you can import thesenew raw data files into Pro Tools, but first you need to create thewrapper files. There are two files to make a wrapper, one that comesfirst, then your raw data, then a second wrapper. I'll call thesewrappers header and footer. You'll use the same footer for each rawdata file, but you'll need to create three headers for each raw datafile. Why? Pro Tools records 24-bit data. 24-bit data is broken up intothree 8-bit chunks. Since you recovered raw data, you don't know whatthe right ordering of the three chunks. The only way to make sure youcan get all audio data back is to create three versions of each rawdata file, each version being a different ordering of the 8-bit chunks.So you'll need to create three header files, the second one being 1byte longer than the first, and the third being 2 bytes longer than thefirst. When you attach the raw data to each of these, the byte orderingwill start in each of the three possible places allowing you to findall the audio data that may be in the raw files. Sounds a littlecomplicated but it's really not.

Creating header and footer files

1. Disconnect your precious Source Disk!!!

2. Launch Pro Tools.

3. Create a session in the same file format, sample rate and bit depthas the data you're trying to recover. If you don't know, you'll have tocreate headers and footers for each file type and do everything fromhere on over and over until you find your data. If this is yoursituation, think long and hard about how important those files are.Unless you have original recordings of spacealiens I suggest you forgettrying to recover and re-record stuff. If you DID have originalrecordings of spacealiens and you didn't make a backup copy immediatelythen I suggest you sell your computer and get a job in a lessintellectually demanding field. That said, mistakes happen. I made sucha mistake. Luckily, I remembered the file type, sample rate, bit depthand even the number of channels of what I was looking for. So I pressedon.

4. Make a selection on your timeline that will create an audio file 1GB in size.

5. Press Option-Shift-3 to create a blank audio file with your selection.

6. You'll need to do this several times to zero in on the exact timeneeded to create a 1GB file. I'm not sure how exact you have to be butI got it to the exact file size through trial and error. Come to thinkof it, it's probably better to have a little bigger file than exactly1GB because the headers and footers will take a little space. Oh well.My recovery worked well. I got back over 99% of my lost data (about 4hours worth) and that was good enough for me.

7. Quit Pro Tools once you have your 1GB audio file. Name that file something like 1GBAudioFile.

8. Now launch HexEdit.

9. Open 1GBAudioFile in HexEdit. A sea of numbers will fill the window. Don't flinch.

10. HexEdit shows the raw hex data of the open file on the left sideand the "human readable" version on the right. Sometimes you can makeout words on the right side, but for audio data it just looks likegarbage. Scroll down through the data until you see a large amount ofzeroes that goes on forever. This is the audio data. Since you createda blank audio file, it's just zeroes in there, so it's easy to seewhere it begins and ends.

11. Find the beginning of the audio data.

12. Copy all the data from just before the audio data all the way to the beginning of the file.

13. Create a new file in HexEdit and copy this data into it.

14. Name this file "HeaderA".

15. In HeaderA file, copy and paste a single 00 to the end of the file.

16. Name this file "HeaderB".

17. In HeaderB file, copy and paste a single 00 to the end of the file.

18. Name this file "HeaderC". You now have three header files, each one byte longer than the previous one. Now for the footer.

19. Find the end of the audio data in the original 1GBAudioFile. It'sthe where the zeros all end and you get a few non-zero numbers showingup. There will be more zeros after this (where the waveform drawingdata is stored) so make sure you scroll in far enough to get the end ofactual audio data. This could take a while since you're looking at ALOT of data.

20. Copy all the data from the end of the audio data to the end of the file.

21. Create a new file and copy this data into it.

22. Name this file "Footer".

23. Quit HexEdit.

24. Eat something, take a break.

25. Copy the three Header and one Footer files into the same folder with your raw audio data files.

26. Launch Terminal.

27. Type cd and then drag the folder containing your raw audio filesinto the Terminal window. It will autofill the pathname for you. HitEnter. Now you are in the same directory (folder) as your raw audiofiles so commands will look for files in this directory.

28. Type cat HeaderA 01 Footer > 01A; cat HeaderB 01 Footer >01B; cat HeaderC 01 Footer > 01C This is three commands in a row(note the semicolons.) Each one says concatenate file HeaderA then file01 then file Footer in that order into a new file called 01A. Then dothe same for B, then C. So you now have three copies of your raw datafile with the audio file wrappers on them. They are now ready to beimported into Pro Tools.

29. Repeat step 28 for every raw data file you recovered. You can queueup a lot of these commands with semicolons just like you did with thedd command earlier, as long as you have the disk space. As you can nowsee, you'll have three more 1GB files on your drive for each 1GB rawdata file, making 4GB of data. This is where the 4x space requirementcomes from. If you're doing an 80GB drive, you'll eventually need 320GBof space to check every bit of space on the disk for audio (no punintended.) Breaking it up into smaller chunks, like doing 10GB of atime might be necessary if you don't have lots of free disk space. Andremember, don't use your original precious hard disk!! Don't even thinkabout writing to it until you are completely done recovering data fromit.

30. If you are trying to recover 16-bit data, then you only need tocreate two copies of each raw data file. There are only two bytes of 8bits each in this case.

The moment of truth

You have three 1GB files for each 1GB of space on your original disk.Now you need to put them in Pro Tools and start listening to them andlooking at them to identify audio data and save it.

1. Launch Pro Tools.

2. Create a new session at the same sample rate, bit depth, format as your recovered data.

3. Import files 01A, 01B and 01C to Audio Tracks.

4. Let the waveforms redraw. If they don't automatically redraw, select them in the Region Bin and force the redraw.

5. You are now looking at the the first 1GB of data from your drive,presented with three different byte orderings. You will probably seelarge regions of what looks like solid full code noise (looks like abrick when zoomed out). Interspersed with these bricks you'll hopefullysee what looks like regular audio.

6. Turn down your speakers!!! You're going to be hearing some unpleasant noises coming from your system. Do not use headphones.

7. Solo one of the tracks and start playing the parts that look likeactual audio data. If it sounds right, then go ahead and select anddelete the regions on the other two tracks that are in the same placeas the good audio data. You're deleting what is the same data as thegood stuff, just shifted one or two bits so what should be the bottom 8bits of your audio is now the top 8 bits or something like that. Goahead and listen to some of it. Depending on the original material,there may be some interesting stuff in there especially if you likenoise and distortion.

8. Repeat step 7 for each of the three tracks until you've gotten ridof all the stuff that you know is junk. You'll be left with mostly realaudio and probably some unknown sections that don't appear to haveaudio on them in any of the tracks. Listen to the unknown stuff anddetermine if it's of any value. Most likely it's junk.

9. Repeat it all with the each and every file you recovered and created.

10. You may have noticed while going through the audio that some stuffwill get cut off at the end of a file and then pick up at the beginningof the next one. This is helpful so you can edit stuff back together.You'll also notice that you've lost all meaningful timing relationshipsbetween tracks that were recorded at the same time, or edited togetherlater. You'll have to reassemble your session manually. This may soundnightmarish, but if this data is really that important to recover andyou've come this far, you can probably recreate the session even betterthan you did originally. Good luck.

The End


geo __________________

ms georgia hilton mpse cas

Categories: None