|
Discussion Forums
|
Thread: Content-Range in PUT
|
|
|
Replies:
21
-
Pages:
2
[
1
2
| Next
]
-
Last Post:
Jan 29, 2010 3:10 PM
by: Barry B. Anderb...
|
|
|
Posts:
4
Registered:
6/1/06
|
|
|
|
Content-Range in PUT
Posted:
Jun 1, 2006 11:04 PM PDT
|
|
|
Does Amazon intend to impliment the Content-Range header on PUT requests? It has some use for appending and altering small sections of large files.
I appreciate that Range has been implimented for GET requests. Thanks.
-Charles
|
|
Posts:
192
Registered:
2/8/06
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 2, 2006 9:09 AM PDT
in response to: Charles Pritchard
|
|
|
It is certainly one of the features we're considering. Can you describe in more detail how you would use it?
|
|
Posts:
102
Registered:
5/8/06
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 5, 2006 1:49 AM PDT
in response to: John Cormie
|
|
|
I don't know what the original questioner was thinking of, but here is one example of my own:
I want to maintain, on S3, a backup copy of my database file, which is 2.5GB in size.
The database contains large stretches of data that will stay unchanged for long periods. Updates on any particular day are unlikely to exceed 1MB.
I can easily maintain two copies of the file on my computer - (1) the current file and (2) the "S3-image" - the copy of the file that I know that S3 is storing for me. Then, on a regular basis, I can do the following:
1. Compare the database (1) with my S3-image (2).
2. Where the data differ, send content-range PUT requests to Amazon S3.
3. When S3 has acknowledged each PUT request, update my local S3-image so that it corresponds to what S3 actually has.
Thus we could maintain a backup of a very large dataset with very little actual data transmission.
If you implemented a COPY request in S3 then that would also help here, since then I could maintain earlier versions of the S3 backup by asking S3 to make a copy before I sent the first PUT.
Most of these things can be done by splitting the file into smaller chunks and storing those chunks as objects on S3, but it would be far better to be able to avoid these software heroics - not least because the fact that "one object = one file" would mean that no special software is needed when restoring from the backup!
I'm already using this architecture to implement a "download the database from the server to a workstation" feature - because it only downloads the changed bytes, it makes backup possible even over slow communication channels.
I'd be very happy to work with you if you wanted anyone to try out a beta implementation of any of this.
|
|
Posts:
4
Registered:
6/1/06
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 12, 2006 12:03 AM PDT
in response to: John Cormie
|
|
|
Incremental updates, would be well-spiffy.
I'd like to keep the "simple" that S3 has.
Simple, simple, simple.. That's the acronym, right?
I'd use the PUT for a few nasty things; I'd like to be able to push content into the same Object from several servers at once. Nasty, I know.
As poster #2 detailed... Content-Range allows us to patch files, as well.
Usage number three: I'd like to append data, because I'm fickle.
|
|
Posts:
13
Registered:
5/15/07
|
|
|
|
Re: Content-Range in PUT
Posted:
May 15, 2007 11:03 PM PDT
in response to: Charles Pritchard
|
|
|
Any update on whether this would be supported and in what timeframe?
My use cases is for doing a partial update of files on Amazon. This is especially important for bigger files.
Message was edited by: bassamt
|
|
Posts:
105
Registered:
10/23/06
|
|
|
|
Re: Content-Range in PUT
Posted:
May 16, 2007 2:04 AM PDT
in response to: bassamt
|
|
|
I would like to register my interest.
In my application (Sissle) I have an option to use the get ContentRange to retrieve large files. This has the obvious advantage that on failure the download can proceed from the point of failure, repeating only the last failed segment. If the network is "playing up" the download can be restarted a day later.
From the users perspective the put version is more useful than the get version.
Most users have slower upload speeds than download. in my case 0.72 MegaBits upload to 6.0 MegaBits download.
I have far more fears that my uploads will fail than my downloads will fail (10 to 1 as that is the time differential between the two).
As I have implemented assembling large files from smaller get ContetRange chunks I can understand Amazon's reluctance to do this from a technical view point, but it is possible and many people out there have the download version working.
Dave
|
|
Posts:
13
Registered:
5/15/07
|
|
|
|
Re: Content-Range in PUT
Posted:
May 18, 2007 10:43 AM PDT
in response to: Charles Pritchard
|
|
|
I would appreciate someone from Amazon responding on wether content-range will be support in PUTs and in what timeframe. Your response factors into an architectural decision we are trying to make.
Thanks!
|
|
Posts:
13
Registered:
5/15/07
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 13, 2007 6:28 PM PDT
in response to: Charles Pritchard
|
|
|
anyone?
|
|
Posts:
1
Registered:
2/2/08
|
|
|
|
Re: Content-Range in PUT
Posted:
Feb 2, 2008 1:13 PM PST
in response to: bassamt
|
|
|
I also am in need of this feature, my current server to server integration from Salesforce.com httpCallout to S3 only functions with a body size of 32000, but i can send multiple PUT requests.
Without access to Content-Range, this integration will be hopelessly limited.
Thanks
|
|
Posts:
8
Registered:
12/13/07
|
|
|
Posts:
177
Registered:
3/28/06
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 18, 2008 8:33 PM PDT
in response to: Charles Pritchard
|
|
|
It's nearly impossible to upload a large object because something will disturb the connection and there's no way to save a partially uploaded thing, upload in ranges, resume, or append. So the practical size limit of an object is in fact only a couple few hundred megs, at least coming in from outside the amazon neighborhood.
I kind of can't believe it's 2008 and this is still not supported. (On the other hand there's Copy support now, which used to be my biggest gripe. So now Resume is :)
|
|
Posts:
5,320
Registered:
3/19/07
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 18, 2008 9:23 PM PDT
in response to: greg13070
|
|
|
This is a lot harder than you think, due to the distributed nature of S3. A PUT can be received by any S3 node. If two different nodes receive a PUT on the same object, they resolve the difference and synchronize with each other by using the timestamp of the PUT. In the present implementation, there is one timestamp associated with each object. Now imagine you can PUT ranges, and each PUT can still go to any S3 node. In order to synchronize, you need one timestamp per range of bytes that was PUT. In addition, the node that receives the PUT might not have the whole object, so it needs to keep track of which ranges it has and which it doesn't have. The process of sync'ing two nodes now involves swapping, comparing and combining these lists as well as the data ranges. It quickly gets unwieldy.
|
|
Posts:
177
Registered:
3/28/06
|
|
|
|
Re: Content-Range in PUT
Posted:
Jun 18, 2008 9:31 PM PDT
in response to: Allen
|
|
|
Oh, no... I already knew it was hard. But I'm a dumb, cranky user and I want features! :)
Anyway, full range put isn't even needed. Minimally we just need the ability to merge data between nodes in some fashion.. like Copy but with (at least) 2 sources. Then the rest could be done in the clients.
I'm not saying that gets around what makes it hard. But it could be done in such a fashion to minimize the difficulty, in keeping with the general S3 design.
|
|
Posts:
64
Registered:
7/16/08
|
|
|
|
Re: Content-Range in PUT
Posted:
Jul 16, 2008 3:45 AM PDT
in response to: Charles Pritchard
|
|
|
This is the single most important issue for me with respect to S3.
My goal is to be able to mount an S3 bucket as a local filesystem with reasonable performance. This can be accomplished using existing S3 functionality, but any change to a large file (for example changing 1 byte of a gigabyte file) will require the re-upload of the entire updated version of the file.
It would be exceedingly more efficient to use a ranged PUT operation to update only the modified bytes of the file.
I realize that it could be very complicated for the S3 servers to support ranged PUTs and achieve data consistency, however, I think that simplified data consistency semantics are all that is needed to provide the most useful functionality. If S3 returned an error for any simultaneous ranged PUT requests for the same key, that would be sufficient for enabling much better performance for S3 mounted filesystems.
|
|
Posts:
88
Registered:
3/15/07
|
|
|
|
Re: Content-Range in PUT
Posted:
Jul 16, 2008 5:54 AM PDT
in response to: Bryan J. Ischo
|
|
|
This is a feature we continue to think about. It's clear that supporting this would make using Amazon S3 easier and/or more efficient in several different scenarios.
Allen explains some of the challenges in supporting it, there are others as well. As a result, it's not something we plan on adding for 2008, but we will be reconsidering and reevaluating again for 2009.
We appreciate all of the community feedback and commentary on features. Please keep it coming.
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
