Moving S3 bucket to another AWS account in another AWS region

Standard

AWS does not support moving S3 buckets, so actually we will be syncing the content from original account/bucket to a new account/bucket.

Note: the following instructions are for a simple scenario – there in an app that only reads/writes data from/to S3. So we copy data from old bucket to a new bucket without any special ACLs.

Assuming you have new account and bucket created, you will need following to start the process:

  • Old and new bucket names
  • AWS cli installed
  • (Optionally) ARN of IAM user that will be used to sync the data

Prepare SOURCE account

Login to SOURCE account management console, go to IAM / Users / particular user you would like to use for doing the transfer and set following Inline policy to ensure an user from SOURCE account can read from SOURCE bucket and write to DESTINATION bucket – change old and new bucket name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::_old_bucket_",
                "arn:aws:s3:::_old_bucket_/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::_new_bucket_",
                "arn:aws:s3:::_new_bucket_/*"
            ]
        }
    ]
}

If you intend to transfer multiple buckets, just list them all in Resource sections above.

Now while you are in user details you can take ARN of the user if you do not have it already. It is on user details page and looks like:

"arn:aws:iam::_aws_account_id_:user/_username_"

Prepare DESTINATION account

Login to DESTINATION account management console and set a policy on DESTINATION bucket to allow user from SOURCE account write to DESTINATION bucket:

{
	"Version": "2012-10-17",
	"Statement": [
    {
      "Sid": "AccountsTransfer",
      "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::remastermedia-files-paveltuma",
        "arn:aws:s3:::remastermedia-files-paveltuma/*"
      ],
      "Principal": {
        "AWS": [
          "arn:aws:iam::_aws_account_id_:user/_username_"
        ]
      }
    }
  ]
}

Set DESTINATION bucket default object ownership to be the DESTINATION account. Go to DESTINATION bucket / Permissions / Object ownership and set it to Bucket owner preferred.

Perform sync

Setup aws cli to use the user that will be used to sync data and then perform:

aws s3 sync s3://_old_bucket_  s3://_new_bucket_ --region _new_region_ --acl bucket-owner-full-control

Depending on a volume of your data it may take long, so I recommend to execute it in a terminal multiplexer such as tmux.

Unicode filenames note

You may experience errors when syncing the files with some unicode characters – not all, but emoji mostly in my case. Basically it will say copy failed:

copy failed: s3://_old_bucket_/prefix/🔥.wav to s3://_new_bucket_/prefix/🔥.wav An error occurred (InvalidRequest) when calling the CopyObject operation: Couldn't parse the specified URI.

It seems this is an issue of S3 that it does not handle these filenames when syncing between different AWS regions.

The easiest workaround in my case was to download the files from SOURCE bucket to a local filesystem and re-upload them into DESTINATION bucket. But of course this may not apply to you.