AWS S3 Ruby

Get Amazon S3 bucket item number and filesize for more than 1000 items with the AWS Ruby SDK

16 Apr 2016
2 minutes read

Getting the overall number and filesize quota of all your items in an S3 bucket is more complicated than you’d think, especially if your bucket contains more than a thousand items.

How to

First you need to install the AWS-SDK gem.

gem install aws-sdk

With the AWS-SDK included, you can access your bucket and retrieve a list of objects inside it:

@aws_count = 0
@aws_size  = 0

s3_object_count(s3_bucket_objects)

The counting itself, needs to be done recursively, as each call to list_objects_v2 will only return a maximum number of 1000 objects:

# Get back S3 object listing, starting at start_after
def s3_bucket_objects(start_after = nil)
  s3_connection = Aws::S3::Client.new(
    region: ENV["aws_region"],
    credentials: Aws::Credentials.new(<your_aws_access_key_id>, <your_aws_secret_access_key>)
  )

  s3_connection.list_objects_v2(
  bucket: <your_aws_bucket>,
    start_after: start_after
  )
end

# Recursively count objects from an S3 object listing
def s3_object_count(bucket_objects)
  bucket_objects.contents.each do |object|
    unless object.nil?
      @aws_count += 1
      @aws_size += object.size
    end
  end

  return if bucket_objects.is_truncated == false

  s3_object_count(s3_bucket_objects(bucket_objects.contents.last.key))
end

@aws_count and @aws_size now should reflect the number of objects and your filesize quota in bytes.

If you are using this with Rails, you can use the view helper number_to_human_size inside your views to output @aws_size in a more human friendly format, like ‘1.61 GB’.