A Small SHN and MD5 FAQ

SHN Resources | Lossless Sources Database | Live Music Archive
http://research.umbc.edu/~hamilton/shnfaq.html
Please update bookmarks older than 7/02 (change old "alumni" to "research" in url)
Last update 1/28/03

The FAQ below gives a quick overview pitched toward new folks, whether ftp downloaders or regular mail traders. For another simple explanation, see Allen's new Basic .shn and Trading Information. There are additional FAQs at etree.org (beginner and detailed level also at their older site, especially oriented toward downloaders), U2SHN.com, benharpershn, and Phishhook.com.

The Japan Times has recently published a well-written article with sidebar.

What are SHNs?

Shorten (extension .shn) is an audio compression scheme written by Tony Robinson of SoftSound that can compact wav files without subtracting out any frequencies. It's "lossless"- everything in the original wav is there, so that full quality is maintained. For many traders of live music, Shorten files (shns) therefore have an undisputed advantage over "lossy" mp3-format files. A shn file made from a wav is quite a bit larger than the corresponding mp3, with a size perhaps ~50-70% of the orginal wav.

Although many people just download shns, keep in mind that these files can also be traded through the mail on disc very easily. The overview below focuses on working with shn files that are already created by others. Once you are familiar with that side of things, it will become obvious how to create shns sets on your own.

Ethical note: For historical reasons, much of the music available in Shorten format is of trade-friendly bands. To make it easier for all of us to continue to enjoy this hobby, please respect the artists. Please, no bootlegging (selling of trade-allowed music, stealing of commercial releases) through shns. Thanks!

Where can you get SHNs?

The accompanying SHN Resources page lists places to download, trade, distribute or discuss shn-format music of trade-friendly artists. Notably, the Internet Archive has a new and growing Live Music Archive of shns.

Information about some specific shns circulating for a number of trade-friendly artists is listed in the "Lossless Sources in Circulation" Database (aka the "shn db"). There are no actual shns in that database, just information about them, although the Database now features crossreferences to the Archive's music.

What does a "SHN set" look like?

If you are collecting shns of live music, you will receive various data files together in a group. Files you may see:

.shn: Binary files with compressed music data. Since shn files are meant to be decompressed to .wav for burning onto audio CDs for listening, they will be organized by CD layout. Typical shns are  named by band abbreviation, show date, disc and track, such as  ph2000-04-20d1t01.shn.
.txt: A plain text file with information about the shns- CD tracklist, recording details, credits, flaw notes, etc. They're typically named by band abbrev plus show date, such as ph2000-04-20.txt.
.md5: Plain text files with checksum information for the shns. The checksum strings inside are like 32-character ID numbers for each shn, used to check the integrity of the shns and to "fingerprint" them. People create md5s at the same time they create the shns. The files are typically named by band abbrev, show date, and often disc, such as ph2000-04-20d1.md5. Within the md5 file there should be a separate line for each shn. See below for their important use.
.skt: Optional files used in support of playing shns "on the fly" on a computer. See below.

When you download a shn set, you will at least want all .shn, .md5 and .txt files that you see in the directory you are after.

What do you do with SHNs once you get them?

1) Make sure you have all the files you need. Check the txt file against the shns you have. Are all the tracks there?

2) Required: Use the handy md5 files to verify the integrity of your shns. Software to "check md5s" may be bundled with the Shorten program or may be available separately (as Md5sum). Why check md5s? Since shns are really big, they can get corrupted more easily than smaller files during data transfer or even during data CD burning. A shn file copied without glitches should always be identical to its parent shn, and have the same checksum "fingerprint". If the shn was changed through corruption, it will no longer match its original fingerprint and the check will fail. If the md5 check fails for a shn, redownload that shn (or even try "resume" in your ftp program- you may have a simple truncation), or recopy the shn from your data disc and try again. If it fails repeatedly, contact your host operator or trading partner- his own copy may be defective. If there are no md5s to check, you take your chances with errors.

If the md5s you received just don't seem to work right, read these tips on md5 file format.

3) Optional: Play the shns directly on your computer. They'll only sound as good as your computer speakers though, and may sound no better than mp3s that way.

4) Recommended: Decompress the shns to wavs, then burn audio CDs for best listening quality. Don't delete the shns, md5s or txt if you plan to go to the next step:

5) Recommended: Archive the shn set. It's a great idea to save the shn set itself by burning all of the files (.shn, .md5, .txt) to data-format discs for storage and trading as data. A typical 3-disc audio show will fit on 2 shn data discs. Many people would rather trade music as original shn sets than as audio CDs. But if you do trade audio, archiving shns also makes it easy to produce multiple, identical audio copies painlessly without any troublesome, error-prone audio extraction. Note "reshn" from audio discs is strongly discouraged and will draw criticism for various reasons; save the original shn set only.

For extra carefulness, run an md5 check on the archived shns after the burn, to make sure they all burned OK.

What SHN and MD5 software is available?

To check md5s and to decompress shns into wavs, you need appropriate software. Some choices are listed below along with a few other support tools. Etree.org also has a new software page with advice pointers (plus an older version). If you use any of the noncommercial, modified versions of Shorten listed below, please read Tony Robinson's original license agreement. What about errors?

When you decompress a shn you may occasionally see an error. If so, did you check the shns against the md5s first (see above)? If the md5 check fails, you know there will be some kind of problem. If you have a corrupt shn, it's best to delete it and obtain an uncorrupted copy. Here are some typical error reports and what they mean:

If the md5 check passes but you still get an error message indicating corruption, it's an unfortunate situation. The shn may have been corrupted at the time it was compressed from wav. This happens rarely but does occur. The typical remedy is to have the original "seeder" create a fresh shn for distribution.

Can you play SHNs directly on your computer?

You can listen to shns "on the fly" on your computer without decompressing them to wavs first. Bear in mind that after downloading the shns, you should still use the md5s to check the file integrity (see above) before playing. Corrupt files may not play correctly. If you can't use the methods below, you can always decompress and use other tools to play the uncompressed audio data as usual. However, note that shns or wavs played on typical computer speakers don't sound much better than mp3s. Some choices: Coders have produced different versions of Shorten that create "seek tables", either attached to the shns or as separate ".skt" files. This lets ShnAmp and xmms-shn users skip ahead during play. Note that although the audio portions are not altered, these additions will change the md5 fingerprint of any shn file, relative to the non-seeking shn version. For detailed explanation- and cautions about creating seek tables- see the etree.org seeking FAQ. What if the resource links above don't work?

Additional mirror or archive sites for some of the tools listed above may be found here:

What about other lossless compression schemes?

Shorten is currently the standard lossless compression method used in trading communities. Many versions are free, it's usable on nearly all platforms, and it has a large pool of experienced users. Below are a couple of other file types in use. If you are further interested in the topic, try here or here for starters. Both sites have recent comparisons including recent schemes.
  • FLAC (extension .flac) is free, open source and under active development. It is usable on many platforms including Windows, Unix and OS X. FLAC also supports 24-bit as well as the 16-bit audio taken by Shorten. This has strong future potential. Etree.org is just starting to try it out- more info. (Also, see Mike Wren's site.)
  • Monkey's Audio (extension .ape) by Matthew T. Ashland is also free, open source and under active development. It is currently available in Windows and is being developed for other platforms. Monkey's Audio also supports 24-bit as well as 16-bit audio, and supports other schemes including Shorten. This also has strong future potential.
  • MKW (extension .mkw) was in development by Michael K. Weise. This is not comparable in tightness of compression to Shorten and more importantly, it cannot be used cross-platform (Windows only). The scheme is folded into Michael's popular Shorten tool  (see above, often itself nicknamed "MKW") and is sometimes used by accident. If you get a show as .mkw, consider using the mkwACT tool to change it from mkw> wav> shn, for all to enjoy.
  • SHN Resources | Lossless Sources Database | Live Music Archive

    If you have comments or suggestions, email Diana and Greg at hamilton@umbc.edu
    This site and its collections of data copyright © 1999-2003 by Greg and Diana Hamilton. Permission is granted for noncommercial use only.