|
Discussion Forums
Posts:
98
Registered:
3/15/07
|
|
|
|
Amazon S3 Copy
Posted:
Mar 17, 2008 12:53 PM PDT
|
|
|
We've received a lot of customer feedback asking that we add support to Amazon S3 for copying an object. We have an initial design for this feature that we wanted to share with the community in order to get feedback. We're soliciting feedback publicly on this forum and we'll also be following up privately with a few customers who have been asking us for this feature through other channels.
We've designed this feature so that it directly supports or helps enable the following functionality without the need to download and re-uploads objects to Amazon S3:
Copy
- create a new object from an existing one.
Update Metadata
- change the metadata for an existing object by providing the changed metadata and specifying the same source and destination for the copy.
Rename
- change the name of an object by specifying the new name when copying the object and then use the delete operation to delete the originally named object.
Move
- move the object to another bucket by specifying the new bucket when copying the object and then use the delete operation to delete the object in the original bucket.
A document describing the current design can be found here:
http://doc.s3.amazonaws.com/proposals/copy.html. Please note that you should not start coding against the contents of this document. The design may change before the feature is publicly available.
Looking forward to your comments,
The Amazon S3 Team
|
|
Posts:
69
Registered:
3/16/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 17, 2008 2:06 PM PDT
in response to: Alyssa@AWS
|
|
|
Any comment on bucket/item access control changes that might go with this? For instance, it might be nice to grant "CopyOnly" on an object to allow access to resources - the account requesting the copy might not have direct read access to the object, but, by being able to request a Copy, can then download the content of the original source on their own bandwidth bill from their own bucket, and delete it when done.
Perhaps, actually, a CopySource (on an object) and CopyDest (presumably, on a bucket, since bucket permissions are the only thing that matter to writes right now) permission, thus allowing a partial implementation of the "write only buckets" many are asking for to help prevent unauthorized deletes or fat-fingering.
|
|
Posts:
752
Registered:
2/8/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 17, 2008 4:04 PM PDT
in response to: Alyssa@AWS
|
|
|
Thanks !!
This is going to help a lot to folks who are using S3 for media hosting and need to change URLs or metadata.
Saurabh
|
|
Posts:
752
Registered:
2/8/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 17, 2008 4:48 PM PDT
in response to: Alyssa@AWS
|
|
|
Two questions:
1. Is it required for source and target buckets to be owned by same user? Will we be able to add read permission to an object owned by user A, and then use credentials of user B to copy the object in user B's bucket using this copy API?
2. Just trying my luck with this second question here
 Any possibility to support copying multiple objects into one target object?
copy a + b + c => to => d
(This command works on windows command prompt and a feature like this can enable us to upload a large file in smaller chunks and then merge it on S3).
Thanks
Saurabh
(
http://www.bucketexplorer.com/)
|
|
Posts:
377
Registered:
7/17/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 17, 2008 7:35 PM PDT
in response to: Saurabh Dani
|
|
|
What is the effect of passing a Content-MD5 parameter? Can this be used to ensure that an out-of-date copy of an object is not copied, ie, atomically only perform the copy if the MD5 is the specified value?
@Saurabh,
>1. Is it required for
> source and target buckets to be owned by same user?
"To copy an object, you must provide us with a source bucket, source object, destination bucket, and destination object. You must be authenticated, and must have read access to the source object and write access to the destination bucket."
This doesn't say anything about the ownership, so I'd assume you'd just need access...
|
|
Posts:
644
Registered:
12/28/07
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 5:18 AM PDT
in response to: Alyssa@AWS
|
|
|
The proposal looks thorough to me. It's going to be useful to have the choice of getting a response when the copy starts or getting a response when the copy is done.
The extra white space is a nice touch, although the proposal is quite vague about the details.
|
|
Posts:
377
Registered:
7/17/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 7:53 AM PDT
in response to: Mocky Habeeb
|
|
|
> The proposal looks thorough to me. It's going to
> be useful to have the choice of getting a response
> when the copy starts or getting a response when the
> copy is done. The extra white space is a
> nice touch, although the proposal is quite vague
> about the details.
I imagine the white space is just so the connection stays open while the copy is performed. (It would probably be a bad idea to rely on it for any reason, as it's possible they might optimize it to be fast enough to not need the whitespace later...)
|
|
Posts:
644
Registered:
12/28/07
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 10:53 AM PDT
in response to: bd_
|
|
|
I imagine the white space is just so the connection stays open while the copy is performed. (It would probably be a bad idea to rely on it for any reason, as it's possible they might optimize it to be fast enough to not need the white space later...)
This may be the case since they do talk about the possibility of a time out in the case of a large file copy.
If it is, I think the official docs (that will be released when this feature goes live) should make this more clear because the only actual mention of this white space in the proposal is in the section on REST and the REST api doesn't wait for the copy to complete and isn't in jeopardy of time out.
The SOAP section (where the timeout concerns are) never mentions this white space, and if these are keep-alive data trickled through the pipe, I think this should mentioned directly in the section about SOAP.
I don't think relying on this is an issue. There isn't much you can do with it, other than know it's there.
|
|
Posts:
5
Registered:
4/12/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 11:03 AM PDT
in response to: Alyssa@AWS
|
|
|
Just read through it, looks great.
|
|
Posts:
5,320
Registered:
3/19/07
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 11:20 AM PDT
in response to: Alyssa@AWS
|
|
|
What do you end up with if you copy an object in an "eventually consistent" environment? Do you eventually get a copy of the object that was most recently PUT prior to the COPY, or do you eventually get a copy of the object version that just happened to exist on the responding node at the time the copy was issued?
|
|
Posts:
281
Registered:
4/10/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 2:24 PM PDT
in response to: Alyssa@AWS
|
|
|
> Since we want to mitigate the possibility for time outs during long copy
> operations, the response to a copy is slightly different than other operations
> in Amazon S3—once Amazon S3 initiates the actual copy, it will respond
> with a 200 response.
Why not send back one or more "199 In Progress" responses? This is exactly the sort of situation for which 1xx Informational responses were created -- providing information prior to the arrival of the final response -- and HTTP/1.1 clients are required to accept unexpected 1xx responses prior to the expected response.
|
|
Posts:
214
Registered:
1/22/08
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 2:57 PM PDT
in response to: Alyssa@AWS
|
|
|
Is there any reason to name 2 different operations for
move/
rename
, but only 1 name for
copy
?
Is that because
rename
is a in place operation?
|
|
Posts:
158
Registered:
2/10/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 4:13 PM PDT
in response to: Alyssa@AWS
|
|
|
Hi--
Thank you for all the great feedback so far.
Saurabh: You can copy any objects that you have permissions to read, the requestor does not need to own either of the buckets, and our standard access control policies will enforce the access to the source and destination.
bd and Mocky: The reason for adding the whitespace is, as bd speculates, to ensure that your socket does not time out while we are making the copy. Note that it is not intended to be a progress monitoring mechanism. The reason we didn't take the same approach for SOAP is because there is not a standard mechanism that allows us to do this in SOAP.
bd: The Content-MD5 header will apply to the entity referenced in the x-amz-copy-source header; while Amazon S3 will ensure that the MD5 of the source object and destination object is the same, you can further ensure that the MD5 you think the source object should have. That said, the goal of the Content-MD5 is to verify the correctness of the data and not to validate the consistency of Amazon S3.
Allen: The object consistency for Copy is the same as our consistency model described in our documentation (
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/ConsistencyModel.html).
We hope this helps clarify some of our thinking behind the feature. We look forward to more feedback. Thanks!
The Amazon S3 Team
|
|
Posts:
5,320
Registered:
3/19/07
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 4:18 PM PDT
in response to: Eric@AWS
|
|
|
Hello Eric,
I'm very familiar with that page of the Developers' Guide because I wrote it. That page however does not answer the question.
Allen
|
|
Posts:
73
Registered:
4/10/06
|
|
|
|
Re: Amazon S3 Copy
Posted:
Mar 18, 2008 5:34 PM PDT
in response to: Eric@AWS
|
|
|
1. Could you clarify the protocol for cancelling a copy request? Does aborting the reading of the response stream (mid-"whitespace") signify an abort? Would the client be charged for the entire copy?
2. +1 for Saurabh's #2 above. This would be a nice workaround for the lack of the ability to specify range headers on PUT object requests.
3. Similarly, it would also be nice to have the ability to utilize the standard http range header for a partial copy. e.g. copy bytes 1024-2048 from A into B
Make sure this new feature is bulletproof before releasing it to the public - I suspect it will be
hammered
... which is probably why you've waited so long on this in the first place. : )
Thanks,
- John
|
|
|
|
