I am trying to parse binary data using pipe-attoparsec in Haskell. The reason why pipes (proxies) are used is to interleave reads using parsing to avoid using large memory for large files. Many binary formats are based on blocks (or chunks), and their sizes are often described by a field in a file. I'm not sure what the parser is called for such a block, but this is what I mean by "sub-parser" in the name. The problem is to implement them in a compressed form without a potentially large amount of memory. I came up with two alternatives that in each case will not work.
Alternative 1 is to read the block into a separate bytestring and run a separate parser for it. Although short, a large block will cause high memory.
Alternative 2 is to continue parsing in the same context and keep track of the number of bytes consumed. This tracking is error prone and seems to infect all the parsers that are generated in the last block processor. For a garbled input file, it can also spend time parsing further than indicated by the size field before the tracked size can be compared.
import Control.Proxy.Attoparsec
import Control.Proxy.Trans.Either
import Data.Attoparsec as P
import Data.Attoparsec.Binary
import qualified Data.ByteString as BS
parser = do
size <- fromIntegral <$> anyWord32le
Right result <- parseOnly blockParser <$> P.take size
return result
(result, trackedSize) <- blockparser
when (size /= trackedSize) $ fail "size mismatch"
return result
blockParser = undefined
main = withBinaryFile "bin" ReadMode go where
go h = fmap print . runProxy . runEitherK $ session h
session h = printD <-< parserD parser <-< throwParsingErrors <-< parserInputD <-< readChunk h 128
readChunk h n () = runIdentityP go where
go = do
c <- lift $ BS.hGet h n
unless (BS.null c) $ respond c *> go
source
share