While this medium is not widely used for audio or video, there is, I suspect, still a large amount of ancillary data kept in this format. I will look through my collection and see if there is anything else I need to capture to my servers before my drives die.  I would not entrust anything important to this format.

We stopped using these about three or four years ago, with the last holdout being school work brought home. Now, even that is done on USB keys.

The Lunar Orbiter tape digitization folks have just posted a commentary that bears reading by all archivists who are holding tapes. You may link to it here. The main site is www.moonviews.com

NASA, in their press conference yesterday held at The Newseum, admitted that the original 14-track 1-inch instrumentation (IRIG) tapes that contained the slow-scan video direct from the moon were most likely recycled and reused for later missions. Apparently, over 350,000 reels of instrumentation tape were recycled by NASA over time. No one apparently thought to preserve the 45-odd reels of the original moon walk.

The loss of the original IRIG tapes of the moonwalk is truly sad because this data could be re-converted to standard television formats using far superior methods than were available in 1969. There may be 2-inch helical Ampex VR-660 video tapes still extant of the slow-scan data,  but those have not surfaced. It appears that all surviving copies of the moonwalk videos are ones that had gone through optical standards converters. An optical standards converter is one that has a monitor displaying the image in real time in the transmitted standard and a television camera taking a picture of that monitor using the desired standard. Even the Australian Broadcasting Corp. tapes would have gone through this type of device, although they would be in PAL rather than the U.S.’s NTSC versions.

Lowry Digital is doing a great job of restoring what they have, but the Polaroid screen shot that survives of the slow-scan monitor is alluring of what could have been preserved. More information is available on the Parkes website and from NASA.

Vigilant migration of data as new storage techniques become available is the only way to assure long-term preservation. Even if the IRIG tapes are found, we are almost at the point where the tapes would be un-decipherable. I think one of my machines could play them (I say think as I’ve never tested it to full 500 kHz bandwidth), but I don’t have the specialized video decoder. NASA apparently preserved some equipment should the tapes ever show up.

This also raises another spectre. We MUST be selective as to what we keep in our archives because if we keep everything we won’t be able to afford it–or find it. This is one of the key jobs that archivists do. However, blindly following retention practices, as was done by NASA for the IRIG Apollo 11 tapes, needs to be tempered by historians as well. Certain small subsets of data (moonwalk slow scan video) are much more important than others (astronauts’ blood pressure and other biometrics throughout the entire flight).

All organizations who keep archives need to address this. In a generation (or less) if we save everything, it will become an overwhelming burden and the high points will be lost if they are not properly indexed.

Unit #1 for client audio projects is a Netgear ReadyNAS NV+ with four 500 GB drives, providing about 1.3 TiB of storage.

Unit #2 for personal projects and general data is a Thecus N5200 Pro with four 1000 GB drives, providing about 2.7 TiB of storage.

These two units are then duplicated off-site and connected by a fibre optic link (currently running at 100 Mb/s while the rest of the network is running at 1000 Mb/s).

One of the reasons for the upgrade to the Thecus units for personal storage is that they can grow to 3.6 TiB of storage with the addition of one 1000 GB drive to each. We are in the midst of a film scanning marathon that is seeing me scanning my Velvia slides to 36 MB files and my son scanning our family negatives to 18 MB files. We’re running the Nikon Coolscan 5000 ED in multiple shifts between the two of us. It was my son’s summer job.

With the addition of this storage and a subtle change in my storage model, I decided that it was time to re-think how the NAS units got loaded. I am still ingesting to a local hard drive; 750 GB for the main audio workstation and 250 GB for the photo workstation.

I use ViceVersa Pro from TGRMN and have loved it for many years. I use it to compare the work or release folders on the local machine with the target shares on the NASes. It then pushes the files that are older or non-existent on the NASes to both NASes. In the case of the audio workstation, it can push both client projects and personal projects to the two NASes and their off-site mirrors. We do NOT propagate deletes.

In order to avoid heat-soak of the RME Multiface AD/DA converters attached to the audio computer, I wanted to shut it down after a push session. Rather than just use ViceVersa’s scheduled run system and run the backup sometime when I am sleeping, I wanted to start the backup manually as soon as I was done and then have the machine shut down.

There is an undocumented command within Windows XP called “Shutdown”. Click on run and then enter “cmd” to open a DOS box. Type “Shutdown” (w/o the quotes) and hit enter. This will provide a list of parameters.

To shut down the computer enter
shutdown -s
To abort the shutdown enter
shutdown -a
And to change the default 30 second timeout to 60 seconds enter
shutdown -s -t 60

You can add a script for ViceVersa to run after it’s done, so I added the shutdown command to a batch file. I made this ViceVersa configuration an auto-run shortcut on my desktop. Right above it, I made another shortcut with shutdown’s abort command in it.

Now, when I’m done for the day, I can run the file pushout routines and still have the computer shut down as soon as it’s done–while I’m sleeping.

If you want to do off-site with NAS units but without the interconnection and the constant checking and alerting–in other words, you want a backup unit that just sits there off-site and is brought together with the main one every few weeks–then I would strongly suggest RAID-6 for both. This is the configuration I would suggest considering if you live in an area subject to major earthquakes and/or wildfires. It’s obvious that my California mentality hasn’t fully left me!

For “office” type work, all of the files reside on the local Thecus NAS. We run an update routine from our 24/7 Mail client PC to backup those documents nightly. The mail itself is pushed out as per the above model. Again, we do not propagate deletes.

A complete set of backups from local to off-site also run nightly on the photo computer, although that is not kept up every night. All this is done with ViceVersa Pro.

One of the neat things we do with ViceVersa is for JPG files in the images shares, we do not propagate changes from the local to the remote NAS. In that way, the image on the remote NAS is more like a “negative” for the family digital cameras that shoot JPGs. In that way, if one of my sons edits an image and wants the original back–and violates our rule that derivatives are renamed–we still have the original. I’d rather risk losing the derivative image than the original should the local NAS fail. We also do that with WAV files as we have had some minor modifications made to WAV file metadata with earlier versions of software. It does not seem to be an issue with the current software.

It appears that I am not alone in thinking this is a good product as it has resurfaced as the U-Print CDP78, now in black, and available from many online distributors. The cartridges appear to be interchangeable with the Primera. I can now suggest that this is a good alternative for safe, long-lasting, and reasonably attractive text labeling of CDs and DVDs. It appears that the Teac P11 is also similar. The last time I checked, the Casio required manual rotation of the disk, rather than the Primera’s automatic rotation.

Please provide me with any feedback pro or con—preferably as comments to this post.

The general theory behind a checksum (or “Message Digest” = “MD”) is that it provides a unique 128-bit number for each and every file, based on its content. If one bit changes, the MD5 checksum (sometimes called “hash”) changes. The checksum is repeatable, does not permit discovery of two different files that produce the same checksum, and is non-reversible (i.e. you can’t create the content from the checksum).

Common practice in audio archiving is to use 128-bit checksums, although 160-bit checksums are available. The other common practice is to take the entire filename, including extension, and add a second “MD5″ extension to that. The *.*.md5 file contains the MD5 checksum plus generally the filename.

An alternate configuration is to create one file named checksums.md5 for the entire folder to save downloading effort.

While there are many options for creating MD5 checksums, for Windows, FastSum is a convenient and low-cost package.

At any point in the future, to verify your files, you create a new MD5 checksum and compare it to the ones that were created when the file was created. If they match, you know your file has not changed. The likelihood of both the original file and the checksum file changing so that they still appear valid is essentially nil.

Some applications, like D-Space, check the integrity of all the files in the system on a regular basis using MD5 checksums.

If you receive WAV files from me, more than likely, you will also receive MD5 checksums in the same package, be that on a DVD, hard drive, or via the Internet.

It only takes a minute to create these and it truly does offer peace of mind.

Oh, and the discussion started about images, but it pertains equally to audio and video. Ike was certainly clear on his thoughts about data tape. As I see it (without following it too closely), the marketplace is consolidating around LTO and appears to be shrinking, so maybe he’s right. I’ve stayed away from data tape in favour of an all-disk solution (for approximately 3 TB of storage at the moment).

When looking at storage for audio, consider a track-hour (i.e. a mono program) at 96,000 samples per second, 24 bits, is about 1 GB, uncompressed. 1 TB is about 1,000 track-hours of material at the normal high-resolution sample rate and bit depth…and that currently fits on one physical drive.

The recent experience: A client project generated about 70 GB of files so hard-drive delivery made sense–especially since I had two gently used Western Digital Passport 80 GB drives sitting on the shelf. There would be no infant mortality here. The client (on one continent) and the producer (on another continent) were both to receive copies. I shipped them off and the producer (on the same continent as I) received his copy first and mentioned to the client he couldn’t read it on his Mac. The client had told me he was sure that the Mac would read NTFS, so based on my conversation with Eric, I decided, with client approval, to send NTFS. A few days later, the client received his copy and is pleased with the files and was able to open them without a hitch on his Mac.

As of this writing, the issue of the producer’s disk is still up in the air. I have offered to take it back, reformat it as FAT32, reload the files with MD5 checksums, verify the checksums, and return the drive to him. I think the producer is looking for a Mac utility that will permit reading the disk as-is. I hope so, less work here, but still I’m willing to undertake this to maintain client satisfaction.

In the meantime, I was searching for an easy way to format a drive larger than 32 GB in FAT32 which is locked out by design in Windows XP. It turns out that there is a DOS-box utility, FAT32FORMAT, that works like a charm. If the drives are combo USB/FireWire, using the Sound Devices 722 to format them would be a choice, but for USB-only drives, that is not an available option. PLEASE be careful as this utility will destroy all data on the disk. Use it with care.

After posting the above, the producer gave up trying to read the disk. I reformatted it FAT32, reloaded it, sent it back to him and he was happy.

For those of us who are working at a much smaller scale, I have provided references on what I do for fairly robust storage on a budget. Please see these two attachments: description and map. It shows a unified (I hope) approach useful to small archives and businesses.

On March 12, 2006, Chris Goosman asked:

What does it take to point the family logins to the centralized “My Documents”? Does each user get their own “My Documents” folder or is it truly a central space that all users share?

Your page has inspired me to write up a similar document for the home/studio/office network here including my backup strategy.

I currently use a backup server with 750GB of Raid-5 running Retrospect, but I need a longer term archive storage and I found one of those LaCie NAS boxes like you use (the local one, with the 10/100 and firewire interfaces, not the newer one with gigabit and USB 2.0) for a great price and it’s on the way. I will be trying out ViceVersa which looks neat.

For the network, is your Netgear switch a layer 2 switch? I think I’ll be moving toward a layer 2 switch to handle a few different VLANs. I’m using Vonage for my own voice service which I’d like on it’s own VLAN, plus my wifes office computers on their own VLAN, my studio on a VLAN, and then soon I plan on putting together an Astrisk PBX and would want it on it’s own VLAN. But this is too much computer geek talk and not enough audio geek talk, so I’ll stop now.

I replied:

Hi, Chris,

Each “My Documents” is a separate “My Documents” folder in each user’s own logon – as is a “Favorites” which includes “Links” so no matter where you log in your IE looks the same. I move: “Favorites” and “My Pictures” and “Desktop” into the root of “My Documents” so they’re visible. Also move “Music” and “Videos”. I use Microsoft’s TWEAK UI PowerToy http://www.microsoft.com/windowsxp/downloads/powertoys/xppowertoys.mspx to do this.

My hierarchy is root/family/person

That way, I can back up root/family and get everyone. I also have set permissions so my wife and I can see everyone’s space. My wife is worried about deleting stuff, so I made mine read only to her.

I am not using any managed switches, and don’t intend to go VOIP in the near future, but Skype may be in my future for long calls. The only thing I have (that I’m not using) that is at all fancy is the Netgear FVS-318 firewall which also has VPN capabilities, but I don’t have a VPN client installed (and yes, I know VLAN and VPN are different things).

Since I currently have dual firewalls “in series”, my kids’ laptops when they get them (years) will be connected between the two firewalls and I’ll have a small NAS box at that point to house their documents, and then pull it across the firewall with VVPro.