Parsing WAVE files correctly in Haskell.

Wavy over WAVE

Playing with audio data is a ton of fun and something that I believe that Haskell could do very well. Processing audio data safely and efficiently seems to fit very well into Haskell’s model so, overy a year ago, I started working on and off on a WAVE file format parser. I have been working on it very infrequently (bus trips and other spare time) and I rewrote it once but today I am pleased to announce the release of the very first version of my ‘wavy’ package that lets you extract data from WAVE files in Haskell. The features of this release include:

  • Methods to Parse and Assemble Wave Files
  • Support for different orderings of RIFF chunks (via the riff library I wrote previously)
  • A split between the parsers for the container format and the data allowing efficient metadata parsing.
  • The ability to pase the data into Int64 or Float formats so that you can handle the data in whichever way that you please.
  • Example programs that make use of the library for your perusal and use.

Things which the library is currently missing include:

  • RIFX support
  • Direct support for maintaining RIFF chunks that are not mentioned in the WAVE specification.

Getting the Code

The code is on hackage as the wavy library so you can install it by:

cabal update
cabal install wavy

Please feel free to give it a try. Probably the best way to quickly see it working is by finding a WAVE file on your machine:

locate '*.wav'

And then passing that wave file as the first argument into wave-info. For example:

$ wave-info /Applications/Steam.app/Contents/MacOS/Friends/friend_join.wav
File: /Applications/Steam.app/Contents/MacOS/Friends/friend_join.wav - 2s
     Audio Format:  Microsoft PCM
     Channels:      2
     Sample Rate:   44100
     Bits Per Sample:  16
     Byte Rate:  176400
     Block Alignment:  4

   INFO Metadata
     Creation Date: 2010-03-11
     Engineers:       - Kelly Thornton
     Creation Software: Sony Sound Forge 9.0

As you can see it can parse an audio file very quickly. The wave-info program is very efficient because the library is lazy and does not parse the data chunk unless you specifically require it.

Using the Library

I would recommend that you start by looking at the executables in the libraries source code for examples of how this library can be used in your applications. The wave-info source code can be found on BitBucket.

Once you have finished doing that then you can have a read throug the documentation on Hackage to get a full understanding of what methods the library provides.

If you have any questions then please do not hesitate to contact me or comment on the blog.

(This blog post was produced using pandoc)


Tutorial: The basics of cipher-aes

Cipher AES Tutorial

In this guide we are going to have a quick run through the cipher-aes library to show how it can be used in your own projects. We will just stick to simple ECB encryption and leave other encodings for a future guide.

Note: In order to follow along with this guide you will need a working knowledge of Haskell (Understanding the first few chapters of Real World Haskell should do the trick)

Import all the Things

But first lets get the imports out of the way:

module Main where

import qualified Crypto.Cipher.AES as CCA
import qualified Data.ByteString as B
import qualified Data.ByteString.Char8 as BC
import qualified Data.ByteString.Base16 as B16
import Data.Char (chr)

You will notice that I like to qualify everything to give you a better idea of which libraries these functions are coming from.

How much seed data do we need?

Whenever you wish to begin the encryption you need to provide a seed. You can see that in the type of the initAES method:

initAES :: Byteable b => b -> AES

For the sake of this guide we are going to use a 32 byte hash in order to initialise the AES cipher. This library supports starting with 16, 24 and 32 byte hashes. In most programs that you write this initial seed is going to be called a "secret key" or "shared key"; it will often be stuck in a configuration file in a secure location and it should certainly be kept out of the source code. So developers often like specifying this secret key as a big hexadecimal String. However, even though you could read this string straight into a ByteString that would not be correct because each hexidecimal character only has 16 unique points of data (4-bits).

So the quesiton is: how many hexidecimal characters do we need in order to get 32 bytes of random seed data?

Lets just use Haskell to do it for us. We know that there are four bits of data in each hexidecimal character:

bitsPerHexChar :: Integer
bitsPerHexChar = 4

There are 8 bits per byte:

bitsPerByte :: Integer
bitsPerByte = 8

We want 32 bytes of seed data:

requiredSeedBytes :: Integer
requiredSeedBytes = 32

With that information we can work out how many hexidecimal characters are required:

requiredHexChars :: Integer
requiredHexChars = (requiredSeedBytes * bitsPerByte) `div` bitsPerHexChar

Now this only works because either requiredSeedBytes or bitsPerByte is divisible by 4. In our case they both are.

Loading the seed from Hexidecimal input

Now that we know that we need to load requiredHexChars many characters in order to initialise the AES encryption. However this is going to be given to us in this form:

type HexStream = B.ByteString

Where every byte in the stream has 4 bits of unique data. However, the initAES function expects the data to be a little more compressed than that. It expects 8 bits of unique data per per byte to make a true seed for the AES algorithm:

type InputSeed = B.ByteString

And before we even try to convert from one type to the other we should validate that our inputs are correctly formated Hexidecimal strings (validate your inputs):

validHex :: HexStream -> Either String HexStream
validHex input = if inputLength == requiredHexChars
   then Right input
   else Left $ "Expected " ++ show requiredHexChars ++ " hex characters but instead recieved " ++ show inputLength
      inputLength :: Integer
      inputLength = fromIntegral . B.length $ input

Then we will want a function that converts from one to the other and squishes two 4 bit chunks together to form an 8 bit number. We can do that like so by using the base16-bytestring library:

toSeed :: HexStream -> Either String InputSeed
toSeed input = if B.null errors 
   then Right seedData 
   else Left "The input data was not made up of Hexidecimal characters."
      (seedData, errors) = B16.decode input

We can quickly join the two to have a useful function for loading seed data from hex streams:

toValidSeed :: HexStream -> Either String InputSeed
toValidSeed input = toSeed =<< validHex input

In order to actually make use of these methods we need to have a hexidecimal string to start off with as our input. Here is one that I generated for this guide (you will need to generate another truly random one for your production applications):

chosenHexStream :: HexStream
chosenHexStream = BC.pack "e5d6834e0e52a78a47fc1c8887ca0e0ecd0863df89e6a3eebf7085bd131bb854"

We can then use this seed to create an InputSeed but it might not parse so we want to encode that in the types:

potentialInputSeed :: Either String InputSeed
potentialInputSeed = toValidSeed chosenHexStream

Using the AES library

Now that we have our input seed we can initialise an AES encryption context:

aesEnc :: Either String CCA.AES
aesEnc = fmap CCA.initAES potentialInputSeed

And with it we can start to do some encryption! Woo! However, we need some data to try and encrypt…hmmm. Lets make some random test strings:

testData0 :: B.ByteString
testData0 = BC.pack $ "It might seem crazy what I’m about to say" 
   ++ "Sunshine she’s here, you can take a break"
   ++ "I’m a hot air balloon that could go to space"
   ++ "With the air, like I don’t care baby by the way"

testData1 :: B.ByteString
testData1 = BC.pack "B-b-b-baby, you just ain't seen n-n-nothin' yet Here's something that you never gonna forget"

testData2 :: B.ByteString
testData2 = BC.pack $ "There's a calm surrender to the rush of day" 
   ++ "When the heat of a rolling wind can be turned away"

So, with this test data we can now try and run some encryption. Lets look at the most basic encryption method:

encryptECB :: AES -> ByteString -> ByteString

Okay, that type seems pretty self explanatory, give me an AES context and the thing that you want to encrypt and I’ll run some ECB encryption over it and give the result to you in a Strict ByteString. So you would think that we could just do something like this:

broken :: Either String B.ByteString
broken = fmap (flip CCA.encryptECB testData0) aesEnc

After all, it even compiles! But, surprisingly, that does not work. Instead you get the following error message:

Encryption error: input length must be a multiple of block size (16). Its length is: 173

As you can see the cipher-aes library apparently requires that all data be aligned to a block size of 16 bytes. I know that the correct units are bytes based on the fact that testData0 is 173 units long. To solve this problem we need to make sure that we are always giving the encryptECB function a bytestring that always meets that boundary. The most sensible way I can think to do that is to use a zero padded bytestring. So lets try and build a funciton that will do that for us:

type PaddedByteString = B.ByteString

zeroPadData :: B.ByteString -> PaddedByteString
zeroPadData input = input `B.append` padding
      padding = BC.replicate requiredPadding (chr 0)
      requiredPadding = case inputLength `mod` 16 of
         0 -> 0
         x -> 16 - x
      inputLength = B.length input

Now with this new zeroPadData function we can build an encryption function that will always work:

safeEncryptECB :: CCA.AES -> B.ByteString -> B.ByteString
safeEncryptECB enc input = CCA.encryptECB enc (zeroPadData input)

With this new safe encryption function we can encrypt all of our test data. However, before we do that lets first try and show what happens when you go in the other direction.

Decrypting your AES data

Now that we have function that can encrypt our data we really want a function that can go in the other direction and decrypt it. Lets take a look at the decryptECB function from the cipher-aes library:

decryptECB :: AES -> ByteString -> ByteString

Once again this one is pretty simple, given an AES content and an encoded string of data it will decode it back into the original format. So lets try and write a function that will do the reverse of the operation that we did in the safeEncryptECB function. It is important to note that the data also needs to come back in 16 byte blocks otherwise it cannot be decoded. We can ensure that and use types to handle the errors instead of the error command:

safeDecryptECB :: CCA.AES -> B.ByteString -> Either String B.ByteString
safeDecryptECB enc encodedData = if alignment /= 0
   then Left $ "Error: encoded data should have been 16 byte aligned but was off by: " ++ show alignment
   else Right . fst . BC.spanEnd (== (chr 0)) $ CCA.decryptECB enc encodedData
      alignment = B.length encodedData `mod` 16

This function handles encrypting and decrypting the data safely. Which is especially important because often the encrypted data has the potential to be modified by other systems before coming back to us.

However, it is really important to note that the safeEncryptECB and safeDecryptECB functions are not inversions of eachother. Specifically, if you have an input string that legitimately has trailing nul characters and you encrypt it and then decrypt it then those characters will be stripped in the final output. That is something to be wary of. However, the set of strings that do not end in nul characters will be invertable by these functions.

Now that we have all of this we can really bring it all together.

Bringing it all together

Now that we have put in all of that effort we can really bring it all together with a function that will encrypt the data, show it to us encrypted, and then decrypt it again. Lets give that a try:

-- A hobbits tale...
thereAndBackAgain :: CCA.AES -> B.ByteString -> IO ()
thereAndBackAgain enc input = do
   putStrLn $ "Data is: " ++ show input
   let encData = safeEncryptECB enc input
   putStrLn $ "Encrypted data: " ++ show encData
   case safeDecryptECB enc encData of
      Left error -> putStrLn error
      Right originalData -> putStrLn $ "Original data was: " ++ show originalData

newline = putStrLn ""

We can run that little snippet of code on our test data and watch as it encrypts and decrypts our data. But we also want to make sure that invalid data is handled correctly too, so why don’t we also run something that actually modifies the data before trying to decrypt it again and see what happens:

showErrorsHappening :: CCA.AES -> IO ()
showErrorsHappening enc = do
   let testData0Enc = safeEncryptECB enc testData0
   print $ safeDecryptECB enc (testData0Enc `B.append` (BC.pack "modified"))

And now that we have done that all that is left is to actually run the code:

main = do
   putStrLn "Welcome to the cipher-aes guide by Robert Massaioli."
   putStrLn $ "We calculated that we need " ++ show requiredHexChars ++ " hexidecimal characters in the seed in order to initialise the AES cipher."
   case aesEnc of
      Left error -> putStrLn $ "Error parsing chosenHexStream: " ++ error
      Right aes -> do
         putStrLn "Parsed successfully."
         thereAndBackAgain aes testData0
         thereAndBackAgain aes testData1
         thereAndBackAgain aes testData2
         putStrLn "Showing an error happening by data that is too long:"
         showErrorsHappening aes
         putStrLn "All tests ran successfully"

This guide is actually a literate Haskell source file so you can run it and watch the results printed on the screen. Please check out the repository on BitBucket for this guide to give it a try. If you have any questions then please post them below and thankyou for reading!

(Note: This guide was designed to be converted into HTML for use in WordPress via pandoc.)