This article provides a tutorial on integrating the Amazon S3 REST API for Ruby with the Ruby on Rails web application framework to create a web management user interface for Amazon Simple Storage Service (Amazon S3).

By Dominic DaSilva
2007-01-16

The S3 on Rails web application demonstrates the basic Amazon S3 operations:

• Logging into Amazon S3
• Listing all buckets
• Creating a bucket
• Listing all objects in a bucket
• Bucket actions (delete and setACL)
• Object actions (delete and setACL)
• Uploading an object to a bucket
• Downloading an object from a bucket

The full S3 on Rails application, including source code, is provided and runs on Ruby 1.8.2 and Rails 1.1.6.

Prerequisites for Running S3 on Rails

In order to build and run S3 on Rails, you will need to make sure you have the following tools installed on your machine:

• Ruby version 1.8.2
• Rails version 1.1.6

Amazon Simple Storage Service (Amazon S3)

Amazon Simple Storage Service (S3) is Amazon's unlimited data storage service which is exposed via web service APIs. The Amazon S3 service provides an extremely affordable data storage solution. For more information on Amazon S3, please go to the Amazon S3 web page, where you can sign up for the service and begin using Amazon S3.

REST: Web Services Light

Amazon S3 is exposed via two forms of Web Service APIs: REST and SOAP. REST, which stands for REpresentational State Transfer, makes use of the HTTP protocol POST, GET, PUT, and DELETE operations to perform Create, Read, Update, and Delete (CRUD) operations on a web resource. S3 on Rails uses the Amazon S3 Library for REST in Ruby to integrate with Amazon S3 and perform operations on Amazon S3 data.

Ruby on Rails: The red hot web application framework built on the Ruby language

Ruby on Rails is a free open-source framework for developing web applications. It was created by David Heinemeier Hansson, a partner of 37 signals, as a web application framework for building their web applications. He open-sourced Ruby on Rails under the MIT license and it is now the fastest growing web application framework. It is a full-stack framework for developing database-backed web applications according to the Model-View-Control pattern. Ruby on Rails provides Ajax support in the view layer, a request and response controller, and a domain model wrapping the database. All that is needed to deploy a Rails web application is a web server and a database.

Anatomy of a Rails Application

Rails is based on the MVC (Model-View-Controller) design pattern. A Rails application uses Controllers, Models, and Views to get data from a data source and present it to the user on an HTML web page.

Data Source (S3) -> Model -> Controller -> View (RHTML) -> HTML page

Users update the data on the HTML page, which is then passed back to the data source.

HTML page -> View (RHTML) -> Controller -> Model -> Data Source (S3)

Controllers and Models are Ruby classes. Views are RHTML files, which are text files with HTML and embedded Ruby code. The core of any Rails web application is the Rails configuration file. This file contains definitions of controller actions to be invoked for specific URL mappings of the web application. Rails connects these pieces together at runtime. The S3 on Rails configuration file, routes.rb , sits in the config/environments directory of the web application.

Rails Models

Rails uses Models to hold the data presented in a web page. An RHTML page references model data and renders it to an HTML page. Rails models reside in the app/models directory of the web application.

Rails Controllers

Rails Controllers perform the business logic of the application. They retrieve data from a data source and use it to populate a model so that the RHTML file can use that data to render an HTML page. Rails controllers reside in the app/controllers directory of the web application.

S3 on Rails

S3 on Rails provides a web interface to the basic Amazon S3 operations:

• Logging into Amazon S3
• Listing all buckets
• Creating a bucket
• Listing all objects in a bucket
• Deleting a bucket or object in a bucket
• Setting an object or bucket ACL
• Uploading an object to a bucket
• Downloading an object from a bucket

Let us look at how each of these operations are implemented within S3 on Rails.

Logging into Amazon S3

S3 on Rails uses a YAML file for Amazon S3 parameter configuration. The file s3_config.yaml located in the config directory contains the following Amazon S3 configuration parameters:

• Access Key ID
• Secret Access Key
• Use SSL flag
• Bucket Prefix

Note: YAML is a markup language used by Rails for data serialization, configuration settings, log files, Internet messaging, and filtering.

The contents of the s3_config.yaml file are below:

aws_access_key: !!! ENTER YOUR ACCESS KEY HERE !!!
aws_secret_access_key: !!! ENTER YOUR SECRET ACCESS KEY HERE !!!
use_ssl: false
bucket_prefix:

S3 on Rails uses one controller, S3Controller, and one model, S3Model. When the S3Controller is instantiated, it calls the S3Model.init() method.

class S3Controller < ApplicationController
 model :s3_model

 S3Model.init(@config_file)

The S3Model.init() method reads in the s3_config.yaml configuration file. It then creates an S3::AWSAuthConnection class instance, which holds a connection to Amazon S3:

 # bucket_prefix: use this as the prefix for buckets;
 # will be set from bucket_prefix in aws_keys.yaml if defined; otherwise defaults to aws access key
 def self.init(config_file, bucket_prefix=nil)
  config = File.open(@config_file) { |f| YAML::load(f) }
  @aws_access_key = config['aws_access_key']
  @aws_secret_access_key = config['aws_secret_access_key']
  @ssl = config['use_ssl'] || false
  if @ssl
   @logger.debug("SSL is enabled")
  else
   @logger.debug("SSL is disabled")
  end
  @bucket_prefix = bucket_prefix || config['bucket_prefix'] || @aws_access_key
  @conn = S3::AWSAuthConnection.new(@aws_access_key, @aws_secret_access_key, @ssl)
 end

S3::AWSAuthConnection is the AWSAuthConnection class contained in the S3 module. This module contains the Amazon S3 Library for REST in Ruby and is contained in the file lib/S3.rb

Listing All Buckets

The index method of S3Controller retreives the bucket listing for the connected AWS account. It delegates to the S3Model.list_buckets method. The list of buckets is stored in the @buckets variable. Tthe list of buckets is paginated for display, the current page being stored in the @entries variable:

 # default controller method
 def index
 if @params[:page] == nil
   @count = 0
   @bucket_page = 1
  else
   @bucket_page = @params[:page]
  end
  @buckets = S3Model.list_buckets
 @entry_pages, @entries = paginate_collection @buckets, { :per_page => 20, :page => @params[:page] }
 @count = (@entry_pages.current.to_i - 1) * 20
 end

The list_buckets method of S3Model calls the AWSAuthConnection.list_all_my_buckets method. This method returns a ListAllMyBucketsResponse, whose entries attribute reader is used to return the list of buckets:

 # lists buckets
 def self.list_buckets
  @conn.list_all_my_buckets.entries
 end

The index.rhtml file is used to display the list of buckets:

<h2>Your S3 account has <%= @buckets.size %> buckets, viewing <%= @entries.size %></h2>

<% if @entries.size > 0 %>
<div class="left-aligned">
  <p align="center">
  <% for bucket in @entries %>
    <% @count = @count + 1 %>
    <% if (@count % 100) == 0 %>
      <br><br>
    <% end %>
    <span id="bucket_count">[<%= @count %>]</span>
    <span id="bucket"><%= link_to bucket.name, :action => 'list_bucket', :bucket_name => bucket.name, :bucket_page => @bucket_page %></span>
   <span id="ops">
    [ <%= link_to 'Public', :action => 'make_bucket_public', :bucket_name => bucket.name %> |
      <%= link_to 'Private', :action => 'make_bucket_private', :bucket_name => bucket.name %> |
      <%= link_to 'DELETE', { :action => 'delete_bucket', :bucket_name => bucket.name }, :confirm =>
      "Are you sure you want to delete '" + bucket.name + "'?\nAll items contained in it will be deleted!" %> ]
    </span>
    <br >
  <% end %>
  </p>
  </div>
  <%= link_to '<< First', { :page => 1 } %>
  <%= link_to '< Prev', { :page => @entry_pages.current.previous } if @entry_pages.current.previous %>
  <%= link_to 'Next >', { :page => @entry_pages.current.next } if @entry_pages.current.next %>
  <%= link_to 'Last >>', { :page => @entry_pages.length } %>
<% else %>
  <p>
  <b>You have no Buckets</b>
 </p>
<% end %>

The @buckets and @entries variables of the S3Controller are accessed and used to display the list of buckets.

Creating a Bucket

The index.rhtml page also contains a form that is used to input the name of a bucket to be created:

<fieldset>
<legend>Create</legend>
<div>
<%= start_form_tag :action => 'create_bucket' %>
<p><span id="prompt">Bucket name:</span> <%= text_field 'bucket', 'name' %> <%= submit_tag 'Create bucket' %></p>
<%= end_form_tag %>

</div>
</fieldset>

The action of this form is declared to be the S3Controller.create_bucket method:

 # create a bucket
 def create_bucket
  @bucket_name = params[:bucket][:name]
  result = S3Model.create_bucket(@bucket_name)
  logger.debug(result)
  if result == 'OK'
   flash[:notice] = "Bucket '" + @bucket_name + "' successfully created"
  else
   flash[:notice] = "Could not create bucket '" + @bucket_name + "' - Error: " + result
  end
  redirect_to :action => 'index'
 end

The bucket name is passed to the S3Model.create_bucket method:

 # creates a bucket
 def self.create_bucket(bucket)
  @conn.create_bucket(bucket).http_response.message
 end

This method calls the AWSAuthConnection.create_bucket method. The response message is stored in the result variable, which is used to set the flash variable displayed in the view:

<% if not flash.empty? -%>
  <div id="flash"><p><%= @flash[:notice] %></p></div>
<% end -%>

Listing All Objects in a Bucket

Note: The S3 on Rails application uses the term 'key' in place of object.

The list_bucket method of S3Controller retreives the object listing of a bucket: It delegates to the S3Model.list_bucket method. The list of objects is stored in the @bucket_list variable. The list of objects is paginated for display, with the current page being stored in the @entries variable:

 # list a bucket
 def list_bucket
  if @params[:page] == nil
   @count = 0
  end
  @bucket_page = @params[:bucket_page]
  @bucket_name = params[:bucket_name]
  @bucket_list = S3Model.list_bucket(@bucket_name)
  @entry_pages, @entries = paginate_collection @bucket_list, { :per_page => 5, :page => @params[:page] }
  @count = (@entry_pages.current.to_i - 1) * 5
 end

The list_bucket method of S3Model calls the AWSAuthConnection.list_bucket method. This method returns a ListBucketResponse, whose entries attribute reader is used to return the list of objects:

 # lists contents of a bucket
 def self.list_bucket(bucket)
  @conn.list_bucket(bucket).entries
 end

The list_bucket.rhtml file is used to display the list of objects for a bucket:

<h2>Bucket <em><%= @bucket_name %></em> has <%= @bucket_list.size %> items, viewing <%= @entries.size %></h2>

<% if @entries.size > 0 %>
<div class="left-aligned">
  <p align="center">
  <% for entry in @entries %>
   <% @count = @count + 1 %>
   <% if (@count % 100) == 0 %>
    <br><br>
   <% end %>
   <span id="bucket_count"><%= @count %>.</span>
   <span id="item"><%= public_link @bucket_name, entry.key %></span>
   <span id="ops">
    [ <%= link_to 'Public', :action => 'make_key_public', :bucket_name => @bucket_name, :key => entry.key %> |
     <%= link_to 'Private', :action => 'make_key_private', :bucket_name => @bucket_name, :key => entry.key %> |
      <%= link_to 'DELETE', { :action => 'delete_key', :bucket_name => @bucket_name, :key => entry.key }, :confirm => "Are you sure you want to delete '" + entry.key + "'?" %> ]
   </span>
   <br >
  <% end %>
  </p>
</div>
  <%= link_to '<< First', { :bucket_name => @bucket_name, :page => 1 } %>
  <%= link_to '< Prev', { :bucket_name => @bucket_name, :page => @entry_pages.current.previous, :bucket_page => @bucket_page } if @entry_pages.current.previous %>
  <%= link_to 'Next >', { :bucket_name => @bucket_name, :page => @entry_pages.current.next,:bucket_page => @bucket_page } if @entry_pages.current.next %>
  <%= link_to 'Last >>', { :bucket_name => @bucket_name, :page => @entry_pages.length } %>
<% else %>
  <p>
  <b>Bucket is empty</b>
  </p>
<% end %>

The @bucket_list and @entries variables of the S3Controller are accessed and used to display the list of buckets.

Bucket Actions (delete and setACL)

The deletion of a bucket is handled by the S3Controller.delete_bucket method. It delegates to the S3Model.delete_bucket method. The response message is stored in the result variable, which is used to set the flash variable displayed in the view:

 # delete a bucket
 def delete_bucket
  @bucket_name = params[:bucket_name]
  result = S3Model.delete_bucket(@bucket_name)
  logger.debug(result)
  if result == 'No Content'
   flash[:notice] = "Bucket '" + @bucket_name + "' successfully deleted"
  else
   flash[:notice] = "Could not delete bucket '" + @bucket_name + "' - Error: " + result
  end
  redirect_to :action => 'index'
 end

 # make a bucket public
 def make_bucket_public
  @bucket_name = params[:bucket_name]
  S3Model.set_acl_public_read(@bucket_name)
  flash[:notice] = "Set bucket '" + @bucket_name + "' ACL to 'public-read'"
  redirect_to :action => 'index'
 end

 # make a bucket private
 def make_bucket_private
  @bucket_name = params[:bucket_name]
  S3Model.set_acl_private(@bucket_name)
  flash[:notice] = "Set bucket '" + @bucket_name + "' ACL to 'private'"
  redirect_to :action => 'index'
 end

The delete_bucket method of S3Model calls the AWSAuthConnection.delete_bucket method. This method returns a Response. The response message is stored in the result variable, which is used to set the flash variable displayed in the view:

 # deletes a bucket
 def self.delete_bucket(bucket)
   @conn.delete_bucket(bucket).http_response.message
 end

 # sets ACL to public-read
 def self.set_acl_public_read(bucket, key={})
  if key == {}
   acl_xml = @conn.get_bucket_acl(bucket).object.data
   @logger.debug('--- ACL before ---')
   @logger.debug(acl_xml)
   updated_acl = S3Helper.set_acl_public_read(acl_xml)
   @logger.debug('--- ACL after ---')
   @logger.debug(updated_acl)
   @logger.debug('------------------')
   @conn.put_bucket_acl(bucket, updated_acl).http_response.message
  else
   acl_xml = @conn.get_acl(bucket, key).object.data
   @logger.debug('--- ACL before ---')
   @logger.debug(acl_xml)
   updated_acl = S3Helper.set_acl_public_read(acl_xml)
   @logger.debug('--- ACL after ---')
   @logger.debug(updated_acl)
   @logger.debug('------------------')
   @conn.put_acl(bucket, key, updated_acl).http_response.message
  end
 end

 # sets ACL to private
 def self.set_acl_private(bucket, key={})
  if key == {}
   acl_xml = @conn.get_bucket_acl(bucket).object.data
   @logger.debug('--- ACL before ---')
   @logger.debug(acl_xml)
   updated_acl = S3Helper.set_acl_private(acl_xml)
   @logger.debug('--- ACL after ---')
   @logger.debug(updated_acl)
   @logger.debug('------------------')
   @conn.put_bucket_acl(bucket, updated_acl).http_response.message
  else
   acl_xml = @conn.get_acl(bucket, key).object.data
   @logger.debug('--- ACL before ---')
   @logger.debug(acl_xml)
   updated_acl = S3Helper.set_acl_private(acl_xml)
   @logger.debug('--- ACL after ---')
   @logger.debug(updated_acl)
   @logger.debug('------------------')
   @conn.put_acl(bucket, key, updated_acl).http_response.message
  end
 end

The S3Controller.delete_bucket action then redirects to the index action, which is handled by the S3Controller.list_buckets method, resulting in the diplay of the updated list of buckets.

The setACL methods for a bucket are handled by the S3Controller.make_bucket_public and S3Controller.make_bucket_private methods. These methods delegate to the S3Model.set_acl_public_read and S3Model.set_acl_private methods respectively.

Object actions (delete and setACL)

The deletion of an object is handled by the S3Controller.delete_key method. It delegates to the S3Model.delete_key method. The response message is stored in the result variable, which is used to set theflash variable displayed in the view:

 # delete a key
 def delete_key
  @bucket_name = params[:bucket_name]
  key = params[:key]
  S3Model.delete_key(@bucket_name, key)
  flash[:notice] = "Deleted item '" + key + "'"
  redirect_to :action => 'list_bucket', :bucket_name => @bucket_name
 end

 # make a key public
 def make_key_public
  @bucket_name = params[:bucket_name]
  key = params[:key]
  S3Model.set_acl_public_read(@bucket_name, key)
  flash[:notice] = "Set item '" + key + "' ACL to 'public-read'"
  redirect_to :action => 'list_bucket', :bucket_name => @bucket_name
 end

 # make a key private
 def make_key_private
  @bucket_name = params[:bucket_name]
  key = params[:key]
  S3Model.set_acl_private(@bucket_name, key)
  flash[:notice] = "Set item '" + key + "' ACL to 'private'"
  redirect_to :action => 'list_bucket', :bucket_name => @bucket_name
 end

The delete_key method of S3Model calls the AWSAuthConnection.delete method. This method returns a Response. The response message is stored in the result variable, which is used to set the flash variable displayed in the view:

 # deletes a key
 def self.delete_key(bucket, key)
  @conn.delete(bucket, key).http_response.message
 end

The S3Controller.delete action then redirects to the list_bucket action, which is handled by the S3Controller.list_bucket method, resulting in the display of the updated list of objects for that bucket.

The setACL methods for an object are handled by the S3Controller.make_key_public and S3Controller.make_key_private methods. These methods delegate to the S3Model.set_acl_public_read and S3Model.set_acl_private methods, respectively.

Uploading an Object to a Bucket

The upload of an object to a bucket is handled by the S3Controller.upload_file method. It delegates to the S3Model.put_file method:

 # upload a file
 def upload_file
  @bucket_name = params[:upload][:bucket_name]
  file_to_upload = params[:upload][:filename]
  if file_to_upload == ''
   flash[:notice] = "No upload file specified"
  else
   base_filename = file_to_upload.original_filename
   file_data = file_to_upload.read
   content_type = file_to_upload.content_type.chomp
   S3Model.put_file(@bucket_name, base_filename, file_data, { 'Content-Type' => content_type })
   flash[:notice] = "Uploaded file '" + base_filename + "'"
  end
  redirect_to :action => 'list_bucket', :bucket_name => @bucket_name
 end

The put_file method of S3Model calls the AWSAuthConnection.put method. This method returns a Response. The response message is stored in the result variable, which is used set the to flash variable displayed in the view:

 def self.put_file(bucket, key, data, headers)
  @conn.put(bucket, key, data, headers)
 end

The S3Controller.upload_file action then redirects to the list_bucket action, which is handled by the S3Controller.list_bucket method, resulting in the display of the updated list of objects for that bucket.

Downloading an Object from a Bucket

The list_bucket.rhtml displays the objects in a bucket as hyperlinks. Clicking on the object name downloads the object from Amazon S3:

 <span id="item"><%= public_link @bucket_name, entry.key %></span>

S3Helper

The helper/s3_helper.rb file contains the S3Helper module. This module contains the helper methods public_link, set_acl_public_read, set_acl_private and set_acl_public_read_write. The public_link method is used to build a public URL for an Amazon S3 object:

 # returns public URL for key
 def public_link(bucket_name, key='')
  url = File.join('http://', S3::DEFAULT_HOST, bucket_name, key)
  str = link_to(key, url)
  str
 end

The set_acl_* methods are used to set a bucket or object ACL. They use the Ruby REXML library to update the ACL XML definition and return it to the caller.

Running S3 on Rails

You can download S3 on Rails using the attached file (below) . First, unzip the archive file and change directory to the S3onRails directory. It is assumed you have Ruby (version 1.8.2 was used to build S3 on Rails), and Rails (version 1.1.6 was used to build S3 on Rails) installed on your system. Change directory to the S3onRails directory and run the command ruby script\server. You can then point your browser to http://localhost:8080/S3onRails/, which brings up the S3 on Rails start page.

S3 on Rails Screenshots

The following screenshots show the S3 on Rails user interface, consisting of two main screens:

S3 on Rails Bucket Listing

Once logged into the application, the user gets a listing of their Amazon S3 buckets.

S3 on Rails item listing

Upon selection of a bucket (clicking on the bucket name), the user gets a listing of the items in that bucket.

Dominic Da Silva is the President of SilvaSoft, Inc. , a software consulting company specializing in Java, Ruby, and .NET-based web and web services development. He has worked with Java since the year 2000 and is a Linux user from the 1.0 days. He is also Sun Certified for the Java 2 platform. Born on the beautiful Caribbean island of Trinidad and Tobago, he now makes his home in sunny Orlando, Florida.