Geek asks and answers

Lazy ByteString created using the Socket handle cannot be consumed and GCed is lazy

I am writing an application to transfer a network file. Using Lazy ByteString as an Intermediate

import qualified Data.ByteString.Lazy as BSL

When building BSL from a local file, put the BSL in the Handle of Socket:

BSL.readFile filename >>= BSL.hPut remoteH  -- OK

It works great. Memory usage is permanent. But to get data from Socket, then write to a local file:

BSL.hGet remoteH size >>= BSL.hPut fileH bs  -- starts swapping in 1 second

I see that memory usage continues to grow, BSL takes the size of bytes of memory. Worse, for a large size that exceeded my physical memory size, the OS immediately begins to replace.

I have to recursively get ByteStrings segments. This is normal.

Why does BSL behave like this?

+3
2

hGet - . , .

hGetContentsN , readFile hGetContentsN.

:

hGetContentsN :: Int -> Handle -> IO ByteString
hGetContentsN k h = lazyRead -- TODO close on exceptions
  where
    lazyRead = unsafeInterleaveIO loop

    loop = do
        c <- S.hGetSome h k -- only blocks if there is no data available
        if S.null c
          then do hClose h >> return Empty
          else do cs <- lazyRead
                  return (Chunk c cs)


hGet :: Handle -> Int -> IO ByteString
hGet = hGetN defaultChunkSize

hGetN :: Int -> Handle -> Int -> IO ByteString
hGetN k h n | n > 0 = readChunks n
  where
    STRICT1(readChunks)
    readChunks i = do
        c <- S.hGet h (min k i)
        case S.length c of
            0 -> return Empty
            m -> do cs <- readChunks (i - m)
                    return (Chunk c cs)

hGetContentsN.

+4

, - , conduit enumerator. - :

import Data.Conduit
import Data.Conduit.Binary

main = do
    let filename = "something"
    remoteH <- getRemoteHandle
    runResourceT $ sourceHandle remoteH $$ sinkFile filename

Handle, , network-conduit - :

runResourceT $ sourceSocket socket $$ sinkFile filename
+2
source

More articles:


All Articles