Geek asks and answers

How to Find Zero Byte Files in Amazon S3

Is there a way to programmatically find a null file with bytes in Amazon S3?

The total bucket size is more than 100G,
it is unlikely that I will synchronize with the server, and then run

find . -size 0 -type f
+7
source share
5 answers

There is no direct process for finding files with zero bytes in amazon s3. You can do this by specifying all the objects, and then sort them by size, then you can get the whole zero file size.

if you want to get a list of all files with a zero size, you can use the Bucket Explorer and list the objects of the selected bucket, and then click on the size header (sort by size) so that it combines the size of the files with zero byte.

. Bucket Explorer.

+3

s3cmd awk .

: s3cmd 4 , , , . ( 3) 0 ( 4). ...

$ s3cmd ls -r s3://bucketname | awk '{if ($3 == 0) print $4}'
s3://bucketname/root/
s3://bucketname/root/e

, 4 , .

$ s3cmd ls -r s3://bucketname | awk '{if ($3 == 0) print}' 
2013-03-04 06:28         0   s3://bucketname/root/
2013-03-04 06:28         0   s3://bucketname/root/e

, , .

+9

Boto:

from boto import S3Connection
aws_access_key = ''
aws_secret_key = ''
bucket_name = ''
s3_conn = S3Connection(aws_access_key, aws_secret_key)
s3_conn.get_bucket(bucket_name)
for key in bucket.list():
    if key.size == 0:
        print(key.key)

Boto ( ) 1000 ( aws), , .

+1

, :

hdfs dfs -ls -R s3a://bucket_path/ | grep '^-' | awk -F " " '{if ($4 == 0) print $4, $7}'
0

JMSE Query:

aws s3api list-objects --bucket $BUCKET --prefix $PREFIX --output json --query 'Contents[?Size=='0']'
0

More articles:


All Articles