|
Resources
Sample Code and Libraries
Another S3 Bash Interface
 |
Put, get, delete, and list files using nothing more than curl, openssl, and coreutils.
| Submitted By: |
nescafe5
|
| AWS Products Used: |
Amazon S3 |
| Language(s): |
Other |
| License: |
GNU General Public License (GPL version 2) |
| Source Control Access: |
git clone git://fnordovax.org/another-s3-bash/ |
|
|
|
Originally designed as a code library for a larger project, this contains all the tools needed to do basic single-user file management on S3. Also includes a standalone hmac script to make up for the lack of hmac functionality exposed in the openssl command interface. Only tested on Ubuntu 7.10, should work in any recent UNIX environment that has curl and openssl installed.
Installation: Download, unpack, and copy the s3 and hmac commands to a location in your path.
Usage:
- s3 ls bucket -- lists the contents of a bucket
- s3 put bucket file -- uploads file into bucket using the same name.
- s3 get bucket file -- downloads file from bucket using the same name.
- s3 rm bucket file -- deletes a file on S3
- s3 rmrf bucket -- deletes all files from bucket.
UPDATED: Replaced GNUism in using the date(1) command, and replaced use of the non-POSIX stat(1) command with wc(1). This should greatly improve cross-platform portability.
|
Posts:
1
Registered:
4/1/08
|
|
|
|
Another S3 Bash Interface - Portability
Posted:
Apr 14, 2008 7:10 AM PDT
|
|
|
So, on OSX wc -c for my secret access key file returns 41. stat on the file returns 4096, which is 40*1024.
So, perhaps something like this would work better:
sak=$((`stat -r ${S3_SECRET_ACCESS_KEY} | awk '{split ($0,a," "); print a[13]}'`/1024*10))
|
|
Posts:
1
Registered:
12/4/07
|
|
|
|
Re: Another S3 Bash Interface - Portability
Posted:
Apr 15, 2008 5:49 PM PDT
in response to: Laran Evans
|
|
|
Unfortunately, the stat command is not portable -- that is why I removed it when adding some portability fixes last year.
If wc -c is returning 41 characters, then there are 41 characters in your secret key file. If you can move the cursor in your favorite text editor to a different line when editing the secret key file, it has an unneeded newline in it.
|
|
Posts:
24
Registered:
2/12/08
|
|
|
|
Re: Another S3 Bash Interface - Portability
Posted:
Jun 19, 2008 4:54 PM PDT
in response to: Laran Evans
|
|
|
I had the same issue initially, and it also was fixed by removing the new line.
|
|
Posts:
6
Registered:
4/24/08
|
|
|
|
Re: Another S3 Bash Interface - Portability
Posted:
Jul 19, 2008 2:21 AM PDT
in response to: Laran Evans
|
|
|
The script should check for the file size limit of 5GB before upload.
|
|
Posts:
1
Registered:
4/27/08
|
|
|
|
Re: Another S3 Bash Interface - Portability
Posted:
Jan 13, 2009 10:17 PM PST
in response to: Laran Evans
|
|
|
quick note: README says 'list' is a command, in fact it is 'ls'
|
|
|
|
Off site backups in 5 minutes, Dec 29, 2007 1:44 PM
Reviewer: johnpfeifer4
This is an excellent way to perform off site backups of critical data such as subversion repositories, database dumps, etc. It took about 5 minutes to configure and use.
|
|
Great on Linux, some problems on OSX, Dec 30, 2007 8:27 AM
Reviewer: David Salgado
This is a great utility, but I don't like the way it forces you to keep your secret key in a file as opposed to an environment variable. Also, the script checks the secret key's existence and length before it runs, and the arguments to "stat" which it uses to check the file size do not work on Mac OS X (tiger).
Other than those minor nitpicks, this is a great and very useful utility.
|
|
Helpful but not portable, Jan 15, 2008 9:09 PM
Reviewer: alb@quandary.org
I am confident that these scripts will prove very useful, but they are not portable to all Unices without some modification first. Under NetBSD 3.1_RC3, the arguments to "date" will require modification, as well as those to "stat" - and that is as far as I have gotten; I am still working on the latter and waiting to find out what other obstacles, if any, will pop up.
|
|
Nice work, Mar 1, 2008 10:08 AM
Reviewer: T. Reeder
Love the simplicity of it!
|
|
I also had problems on OSX, Apr 14, 2008 6:33 AM
Reviewer: Laran Evans
I'm not sure why, but my S3 secret access key comes to 41 bytes when saved to a file. There are no newlines or other odd characters in place. Just the key.
So I tweaked the script to check for the secret access key to be `wc -c $sak_file` == 41. I then did `s3 ls $mybucket` and nothing printed out. No error, nothing at all. There are definitely files in the bucket.
Also, `s3 --help` doesn't return anything. This would be a nice addition.
Good start though. All the heavy lifting is done. Nice work.
|
|
Works great on Mac OS X, Jun 27, 2008 2:38 AM
Reviewer: mckenfra
Had the same 41-character access file problem in Mac OS X (Tiger), but managed to create a 40-character file with the following command:
perl -e "print 'TYPE_YOUR_ACCESS_KEY_HERE'" > access_key_file
After that, everything worked fine - thank you very much! Does anyone know why there aren't official Amazon versions of these S3 scripts?
|
|
I just got warning: 'header', Jan 28, 2009 9:50 PM
Reviewer: sportics
Sorry, this are my first steps. During calling the script (mac os x server) i just got:
/dev/fd/63:1: warning: 'header' requires parameter
Ok, i thougt it's just a warning, but nothing more happens.
|
|
Great but ..., Feb 22, 2009 10:46 AM
Reviewer: wolfgang127us
If this script is executed on a machine whose LANG is not C such as ja_JP, it cannot make a proper date header and signature. Thus, it will end with ACCESS DENIED error in the respose of S3.
I took a couple of hours to figure this out...
So plz add "LANG=C" at the top of the script or state the info in this page.
Thanks !!
|
|
|
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
