 |
By Richard Monson-Haefel (www.monson-haefel.com) and PJ Cabrera
Abstract
Amazon Web Services™ (AWS) provides an affordable, high-performance, scalable network of applications that anyone with an Amazon.com account can access and put to work. In this article, Richard Monson-Haefel and PJ Cabrera discuss using Java to access AWS and provide code samples for performing basic Java-based functions available with AWS solutions.
Introduction to Amazon Web Services
If you are like most Java developers--or software developers in general--you probably have had some great ideas for new web businesses over the years. The question is, what stopped you? You had a great idea, you know how to program, but you didn't have an infrastructure on which to run your system. To scale software beyond a single desktop application, you need computers--sometimes a lot of them--and bandwidth beyond your home DSL or cable access. Amazon Web Services (AWS) can solve these problems by providing an instant infrastructure that leverages the same systems that Amazon.com uses to do business every day.
AWS provides an affordable, high-performance, scalable network of applications that anyone with an Amazon.com account can access and put to work. In this article, I will introduce you to the following four services:
Refer to aws.amazon.com for a complete list of AWS solutions.
Amazon S3, Amazon SQS, Amazon EC2, and Amazon SimpleDB provide an instant infrastructure that anyone can use to build a new web application. Amazon S3 supports the storage of massive amounts of data with high bandwidth. Amazon SQS provides a messaging backbone that can support the exchange of thousands of messages per minute. Amazon EC2 provides a massively scalable infrastructure on which to run applications. Amazon SimpleDB works in conjunction with Amazon S3 and Amazon EC2 to provide quick, efficient storage and retrieval of your data to support high-performance web applications.
These infrastructure services--Amazon S3, Amazon SQS, Amazon EC2, and AmazonSimpleDB--provide most, if not all, of the infrastructure needed to compete with larger organizations. What's missing is the huge upfront costs. Whichever of these services you choose, you pay only for the processing power, storage space, and bandwidth that you actually use.
Accessing AWS: SOAP vs. REST
All AWS solutions are accessible over the Internet, through either Simple Object Access Protocol (SOAP) or representational state transfer (REST). Both of these techniques involve the use of HTTP as the communication protocol, but the semantics of using SOAP and REST couldn't be more different.
With the SOAP interface, AWS products are organized into custom-built operations or methods that can be called over the network, similar to the way that remote objects can be called by using Java Remote Method Invocation (RMI) or Common Object Resource Broker Architecture (CORBA). The big difference between SOAP and Java RMI or CORBA is that the application protocol is XML, which is embedded in the body of HTTP requests. SOAP has seen widespread acceptance and adoption throughout the industry, making it an attractive option for Java developers because Java-SOAP application programming interfaces (APIs), including the industry standard Java API for XML-based Web Services (JAX-WS), already exist.
REST also uses HTTP but is very different from SOAP. As you probably already know, web browsers use HTTP GET requests to retrieve resource representations (for example, HTML web pages) from a web site and use HTTP POST requests to submit data (for example, HTML form data) to a web site. HTTP also defines several other standard operations, including DELETE, HEAD, PUT, TRACE, and CONNECT, even though web browsers do not use them. Unlike SOAP, in which you define all kinds of custom methods for each type of operation, REST-based web services support only a subset of the standard HTTP operations (that is, GET, POST, HEAD, PUT, and DELETE) to perform all the operations needed to interact with a web service. Four of these methods--GET, POST, PUT, and DELETE--map fairly closely to the Create, Read, Update, and Delete (CRUD) semantics that typically are associated with relational databases. You can think of REST as a kind of data-access methodology, whereas SOAP is a process-control methodology.
REST isn't as popular in the IT industry as SOAP is, but its use is growing rapidly. In many cases, REST is a lot simpler to implement and use than SOAP--so much so that major web service providers, including AWS, frequently offer access to their web services through both SOAP and REST protocols.
Using Java with Amazon S3, Amazon SQS, Amazon EC2, and Amazon SimpleDB
Until now, I've been speaking somewhat abstractly about AWS. Now I'll use some examples to make the concepts and services more concrete.
Amazon S3
By using Amazon S3, anyone on the Internet can store an unlimited number of objects (for example, documents, images, movies, or text)--as much as 5 GB each--on the highly reliable and fast storage infrastructure that Amazon uses for its own business. Amazon S3 is accessible through either a simple SOAP interface or through REST. You can add, retrieve, and delete objects by using SOAP or REST over HTTP. Amazon S3 also supports the use of the BitTorrent protocol, and Amazon plans to add support for other protocols in the future.
Today, the most popular Java API for Amazon S3 is JetS3t, an open source Java library available at Java.net. The following code is an example of a Java program that copies all the files from a directory on your hard disk to your Amazon S3 storage account.
import java.io.File;
import org.jets3t.service.S3Service;
import org.jets3t.service.impl.rest.httpclient.RestS3Service;
import org.jets3t.service.model.S3Bucket;
import org.jets3t.service.model.S3Object;
import org.jets3t.service.security.AWSCredentials;
public class MyBackup {
public static void main(String[] args) throws Exception {
// Step 1: Get command-line arguments
String myAccessKey = args[0];
String mySecretKey = args[1];
String directoryName = args[2];
/* Step 2: Create a credentials object and service to access
your S3 account */
AWSCredentials myCredentials =
new AWSCredentials(myAccessKey, mySecretKey);
S3Service myService = new RestS3Service(myCredentials);
/* Step 3: Create a new bucket named after a normalized directory path,
and include my Access Key ID to ensure the bucket name is unique */
String bucketName =
directoryName.replace('\\', '_').replace('/','_').replace(':', '_');
S3Bucket myBucket =
myService.createBucket(myAccessKey + "." + bucketName);
// Step 4: Add files from specified directory to bucket
File directory = new File(directoryName);
File[] files = directory.listFiles();
for (int i = 0; i < files.length; i++) {
if (files[i].isFile()) {
S3Object object = new S3Object(myBucket, files[i]);
myService.putObject(myBucket, object);
}
}
}
}
You can use the Amazon S3 service as a backup service for your personal or corporate computers; as a storage location for massive amounts of multimedia files such as movies, songs, and digital images; or as a storage space for any other forms of data that can be transferred as discrete packaged data (files).
Amazon SQS
Amazon SQS provides a queue-based, message-oriented middleware (MOM) for processing unlimited messages that have unlimited senders and receivers. As the name suggests, the service is simple to use, requiring familiarity with only nine operations divided into four APIs: CreateQueue, SendMessage, ReceiveMessage, and DeleteMessage. You can access Amazon SQS by using SOAP or REST. Message bodies can be as much as 256 KB in length, thus easily accommodating most alert and event notification use cases.
The following code is an example of a Java program that creates a new queue, sends two messages, retrieves and deletes those messages from the same queue, then deletes the queue before exiting. This program uses the Amazon AWS SQS Java library.
import com.amazonaws.queue.*;
import com.amazonaws.queue.model.*;
public class MyMessenger{
public static void main(String[] args) throws Exception{
new MyMessenger(args[0],args[1]);
}
public MyMessenger (String myAccessKey, String mySecretKey)
throws Exception {
AmazonSQS myService = new AmazonSQSClient(myAccessKey, mySecretKey);
String queueName = "MyQueue"+myAccessKey
// Step 1: Connect to SQS and create a new queue
createQueue(myService, queueName);
// Step 2: Send two messages to the queue
sendMessage(myService, queueName, "Hello World!");
sendMessage(myService, queueName, "Hello again!");
// Step 3: Wait while messages are stored by SQS
Thread.currentThread().sleep(2000);
// Step 4: Read all messages from queue and print and delete each one
java.util.List<Message> messagesList = getMessages(myService, queueName);
if (messagesList != null) {
for (Message message : messagesList) {
System.out.println(message.getMessageBody());
deleteMessage(myService, queueName, message.getReceiptHandle());
}
}
// Step 5: Remove queue
deleteQueue(myService, queueName);
}
private void createQueue(AmazonSQS service, String queueName) throws Exception
{
CreateQueue request = new CreateQueue().withQueueName(queueName);
service.createQueue(request);
}
private void sendMessage(AmazonSQS service, String queueName, String message)
throws Exception
{
SendMessage request = new SendMessage().withQueue(queueName).withMessage(message);
service.sendMessage(request);
}
private java.util.List<Message> getMessages(AmazonSQS service, String queueName)
throws Exception
{
java.util.List<Message> messagesList = null;
ReceiveMessage request = new ReceiveMessage().withQueueName(queueName);
ReceiveMessageResponse response = service.receiveMessage(request);
if (response.isSetReceiveMessageResult()) {
messagesList = response.getReceiveMessageResult().getMessage();
}
return messageList;
}
private void deleteMessage(AmazonSQS service, String queueName, String receiptHandle)
throws Exception
{
DeleteMessage request = new DeleteMessage().withQueue(queueName).withReceiptHandle(receiptHandle);
service.deleteMessage(request);
}
private void deleteQueue(AmazonSQS service, String queueName) throws Exception
{
DeleteQueue request = new DeleteQueue().withQueueName(queueName);
service.deleteQueue(request);
}
}
Queue-style messaging systems such as Amazon SQS are extremely useful when you need to spread the work of processing data across many computers. For example, you could send the bucket name and object identifier of files in Amazon S3 to be processed in parallel by hundreds of computers. Because the queue locks each message until it is deleted or times out, only one computer will process any given message and therefore any data object in Amazon S3. Imagine using this service to spread out software unit tests across an entire community of users. A task that might take hours on one machine could be reduced to minutes.
Amazon EC2
Amazon EC2 is a fancy name for a rather simple but powerful service. In essence, AWS will host the Linux- and OpenSolaris-based server-side applications that you build, as long as you package them properly and deploy and manage them by using predefined Amazon EC2 SOAP or REST web services. The data centers that support Amazon EC2 are massive and can scale to meet extremely heavy loads, because Amazon is providing you with the same infrastructure on which it supports billions of transactions each year. Your application can scale as needed--you could support hundreds of thousands of users--and you pay only for the processing power used and the data transferred by your application.
AWS provides tools for packaging applications into Amazon Machine Images (AMIs), which can be stored on Amazon S3 and executed by using Amazon EC2. You can create AMIs based on just about any Linux distribution, as well as OpenSolaris. Amazon EC2 supports either 32- or 64-bit versions of these operating systems. You can choose from pre-made AMIs and modify them to meet your needs, or you can create your own AMI by using a set of Amazon-provided tools to take a snapshot of your Linux server or workstation. The AMIs can contain any packages (e.g., web and application servers, message-oriented middleware) that you need for your application.
Amazon EC2 offers several different instance "sizes", for different workloads. You are billed differently depending on which instance size you pick when launching your instances. You can also pick among several Linux kernels and start-up RAM disks provided by Amazon and select partners. Amazon EC2 offers Availability Zones and Elastic IP Addressing options for better management of fault-tolerance and availability within your applications.
One important differentiator between Amazon EC2 and other hosting providers is that with the Amazon EC2 API, you can request more instances of your AMIs to be brought up, and they are usually serviced in less than 15 minutes. With other hosting providers, you have to make your request of a new server at least a few hours, if not days, in advance. This scalability feature, coupled with monitoring services, lets your application scale as necessary, bringing up more instances when traffic goes up and bringing them down when they're no longer needed.
Using the API is necessary only when you're deeply integrating Amazon EC2 services into a product or solution. For interactive use of Amazon EC2, Amazon provides command-line utilities that run on any modern platform with Java. The Elasticfox Firefox extension for Amazon EC2 provides a nice GUI frontend for launching Amazon EC2 instances, searching for an AMI among the many publicly available from Amazon and the community, dealing with group permissions, and other tasks.
Typica is an open source Java library that provides an Amazon EC2 API, among other AWS APIs. The following example lists the AMIs available to a user, as well as the list of instances that the user has run most recently and those instances' current status.
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Properties;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.xerox.amazonws.ec2.ConsoleOutput;
import com.xerox.amazonws.ec2.GroupDescription;
import com.xerox.amazonws.ec2.Jec2;
import com.xerox.amazonws.ec2.ImageDescription;
import com.xerox.amazonws.ec2.ImageListAttributeItem;
import com.xerox.amazonws.ec2.ImageListAttribute.ImageListAttributeItemType;
import com.xerox.amazonws.ec2.KeyPairInfo;
import com.xerox.amazonws.ec2.ReservationDescription;
import com.xerox.amazonws.ec2.ReservationDescription.Instance;
public class Ec2Sample {
private static Log logger = LogFactory.getLog(Ec2Sample.class);
public static void main(String [] args) throws Exception {
String myAccessKey = args[0];
String mySecretKey = args[1];
Jec2 ec2 = new Jec2(myAccessKey, mySecretKey);
// describe images
List<String> params = new ArrayList<String>();
List<ImageDescription> images = ec2.describeImages(params);
System.out.println("Available Images");
for (ImageDescription img : images) {
if (img.getImageState().equals("available")) {
System.out.println(img.getImageId()+"\t"+img.getImageLocation()+"\t"+img.getImageOwnerId());
}
}
// describe instances
params = new ArrayList<String>();
List<ReservationDescription> instances = ec2.describeInstances(params);
System.out.println("Instances");
for (ReservationDescription res : instances) {
System.out.println(res.getOwner()+"\t"+res.getReservationId());
if (res.getInstances() != null) {
for (Instance inst : res.getInstances()) {
System.out.println("\t"+inst.getImageId()+"\t"+inst.getDnsName()+"\t"+inst.getState());
}
}
}
}
}
Amazon SimpleDB
Amazon SimpleDB provides a simple API for storing multiple data sets, querying those data sets, and returning results quickly. You organize your data into domains, which are analogous to individual databases in a relational database product. All your data creation commands and queries are run against a particular domain. Domains are made up of individual data items, and each item is made up of attribute-value pairs that describe each item. An item is like an individual row in a database table, attributes are like column names in the table, and the value part of the attribute-value pair is like the particular value of a column and row.
One way in which Amazon SimpleDB is not like a relational database is that Amazon SimpleDB domains do not have schemas. Each item in a an Amazon SimpleDB domain can contain a unique set of as many as 256 attributes, differently named from all other attributes in the items of a particular domain. Therefore, Amazon SimpleDB domains are extremely flexible, able to hold any kind of items with all kinds of sets of attributes. Compared to Amazon SimpleDB, relational databases are strict and inflexible, in the sense that the database can hold values only as prescribed in the database schema.
The lack of a schema means that you have the flexibility of not predefining different data for your items, but rather can change them dynamically as your application requires. You can add attributes to the items that are used in your application to support new features, or you can remove them if your code no longer needs them. You can release your code without worrying about migrating your database because unlike other databases, Amazon SimpleDB domains and items don't need to have a schema updated to match how your code uses the data.
Another way in which Amazon SimpleDB is not like a relational database is that the attributes of an Amazon SimpleDB item can have multiple values. This feature can be used to implement multiple states for the same item. For example, a manufacturing item can have a status attribute with values of "in production", "on hold", and "waiting for raw materials" (or appropriate look-up codes, if you want to normalize your data to ease maintenance of your status strings).
Another important consideration is that an attribute's value is always a string. These values are collated in lexicographical order. You cannot specify that a particular attribute holds numbers or dates; all values are treated as text. Other articles about Amazon SimpleDB can help you understand this concept and manage your data with lexicographical collation in mind.
The following code sample uses the Amazon AWS SimpleDB Java library to create and delete an Amazon SimpleDB domain and to populate an item with a number of attribute-value pairs.
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Properties;
import com.amazonaws.sdb.*;
import com.amazonaws.sdb.model.*;
public class TestSimpleDB {
public static void main(String [] args) throws Exception {
/* Usage: TestSimpleDB AWS_Access_Key AWS_secret_key new_SDB_domain */
String myAccessKey = args[0];
String mySecretKey = args[1];
String mySDBDomainName = args[2];
AmazonSimpleDBConfig config = new AmazonSimpleDBConfig();
config.setSignatureVersion("0");
AmazonSimpleDB myService = new AmazonSimpleDBClient(accessKeyId, secretAccessKey, config);
/* if you haven't used SimpleDB before, no domains should be listed */
listDomains(myService);
System.out.println();
/* Creating a domain, and creating an item and attributes on the new domain */
System.out.println("Creating Amazon SimpleDB domain "+mySDBDomainName);
createDomain(myService, mySDBDomainName);
List<ReplaceableAttribute> list = new ArrayList<ReplaceableAttribute>();
// list.add(new ReplaceableAttribute( attributeName, replacePreviousValue?, value ));
list.add(new ReplaceableAttribute("name", false, "John"));
list.add(new ReplaceableAttribute("street", false, "123 South Street"));
list.add(new ReplaceableAttribute("state", false, "New Mexico"));
list.add(new ReplaceableAttribute("gender", false, "male"));
list.add(new ReplaceableAttribute("age", false, "25"));
createItem(myService, mySDBDomainName, "person1", list);
/* the newly created domain should be listed */
System.out.println();
listDomains(myService);
System.out.println();
System.out.println("Deleting Amazon SimpleDB domain "+mySDBDomainName);
deleteDomain(myService, mySDBDomainName);
/* the domain should not be listed any more */
System.out.println();
listDomains(myService);
}
static void createDomain(AmazonSimpleDB service, String domainName) throws Exception
{
CreateDomain request = new CreateDomain(domainName);
CreateDomainResponse response = service.createDomain(request);
}
static void createItem(AmazonSimpleDB service, String domainName, String item, List<ReplaceableAttribute> list)
throws Exception
{
PutAttributes request = new PutAttributes().withDomainName(domainName).withItemName(item);
request.setAttribute(list);
PutAttributesResponse response = service.putAttributes(request);
}
static void deleteDomain(AmazonSimpleDB service, String domainName) throws Exception
{
DeleteDomain request = new DeleteDomain(domainName);
DeleteDomainResponse response = service.deleteDomain(request);
}
static void listDomains(AmazonSimpleDB service) throws Exception {
/* Listing the domains belonging to our AWS account */
System.out.println("My Amazon SimpleDB domains:");
String nextToken = "";
while (nextToken != null) {
ListDomainsResult result = getDomains(service, nextToken, 10);
nextToken = null;
if (result != null) {
java.util.List<String> domainNamesList = result.getDomainName();
for (String domainName : domainNamesList) {
System.out.println(domainName);
}
nextToken = result.getNextToken();
}
}
}
static ListDomainsResult getDomains(AmazonSimpleDB service, Sting nextToken, int maxDomains)
{
ListDomains request = new ListDomains(maxDomains, nextToken);
ListDomainsResponse response = service.listDomains(request);
ListDomainsResult result = null;
if (response.isSetListDomainsResult()) {
result = response.getListDomainsResult();
}
return result;
}
}
Conclusion
Amazon S3, Amazon SQS, Amazon EC2, and Amazon SimpleDB provide a powerful, instant infrastructure for all kinds of applications that you, or your company, can build and deploy without concern for scalability, availability, or throughput. These infrastructure services will provide an excellent foundation for new web applications designed for hundreds, thousands, or millions of concurrent users.
Learning More About AWS
This article highlights a few aspects of working with AWS. Here are a few more resources available to developers to help you learn more.
AWS Resources
- Learn more about each web service at the AWS web site.
- The Developer Connection web site for AWS developers includes forums on AWS, a solutions catalog with examples of what your peers have built, and more.
- Part of the Developer Connection web site, the Resource Center has links to tutorials, code samples, technical documentation, and other resources for building your application on AWS.
Great Resources for Java Developers
Java and AWS In Action
These web sites use AWS and Java:
About the Authors
Richard Monson-Haefel is a Senior Analyst for Burton Group, a technical analyst company. He has been consulting in enterprise Java since 1996, and has served on the executive committee and the J2EE 1.4 (JSR-151), EJB 2.1 (JSR-153), and EJB 3.0 (JSR 220) expert groups for the Java Community Process. Richard is the award-winning author of the bestselling Enterprise JavaBeans (O'Reilly), J2EE Web Services (Addison-Wesley), and is the coauthor of Java Message Service (O'Reilly).
PJ Cabrera is a freelance software developer specializing in Ruby on Rails e-commerce and content management systems development. PJ's interests include Ruby on Rails and open-source scripting languages and frameworks, agile development practices, mesh networks, compute clouds, XML parsing and processing technologies, microformats for more semantic web content, and research into innovative uses of Bayesian filtering and symbolic processing for improved information retrieval, question answering, text categorization, and extraction. You can reach him at pjcabrera at pobox dot com, and read his weblog at pjtrix.com/blawg/.
|
Sign in to the AWS Management Console
Create an AWS Account
