Geek asks and answers

Ensuring data integrity with Python Boto and Amazon Glacier concurrent.ConcurrentUploader?

I use Python and Boto in a script to copy multiple files from my local drives, convert them to .tar files, and upload to AWS glacier.

I based my script on: http://www.withoutthesarcasm.com/using-amazon-glacier-for-personal-backups/#highlighter_243847

What concurrent.ConcurrentUploader uses

I am just wondering how confident I am that the data is all in Glacier after successfully receiving the ID? Does concurrentUploader contain any hash checks to ensure that all bits are received?

I want to delete files from my local disk, but I'm afraid that I have to implement some kind of hash check ... I hope this happens under the hood. I tried and successfully extracted a couple of archives and was able to knead. Just trying to be very careful.

Does anyone know if there is a check under the hood that all parts of the transmission have been successfully downloaded? If not, does anyone have sample python sample code, how to implement hash-check loading?

Many thanks!

Boto Concurrent Uploader Docs: http://docs.pythonboto.org/en/latest/ref/glacier.html#boto.glacier.concurrent.ConcurrentUploader

UPDATE: Looking at the actual Boto code ( https://github.com/boto/boto/blob/develop/boto/glacier/concurrent.py ), line 132 shows that the hashes are calculated automatically, but I don’t understand what

[None] * total_parts

. , ?

+3
2

, .

http://docs.aws.amazon.com/amazonglacier/latest/dev/api-multipart-complete-upload.html

API, , " " - sha256 sha256 MiB , - . , ( , , sha256 - ), " multipart" .

API Glacier , "" , , .

+3

, . :

[None] * total_parts

, total_parts None. None . , , - .

, , API Glacier.

+1

More articles:


All Articles