|
Discussion Forums
|
Thread: Database Functionality: What Do You Need?
 |
This question is not answered.
Helpful answers available: 2.
Correct answers available: 1.
|
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 14, 2006 8:12 AM PDT
in response to: Martin@AWS
|
|
|
> Database Functionality: What Do You Need?
As I see it it depends on who you talk to.
You could try to build a super databse with serveral interfaces (SQL, XPath, etc...) and have all the features to please everyone. It would be a super complex project but if you can handle it, go ahead because the success would be huge.
On the other hand you could try to build a limited functionality that does only xyz. In that case the product will lack many thing that people want to do but it would be easier to build. The success might not be at the end of the road.
So what to do? Go for the complex stuff, you have no choice. You cannot go against people's expectation. If I talk SQL, I expect full functionality and not just a limited subset that I have no use for.
How to do it?
In my opinion, the database service is just a subset of the generic EC2 vision.
It is a manage database solution just for you. "Just for you" means that your database runs on some EC2 instances (at least 2 for reliability and high availability) and is able to scale .... but no to infinity.
For example Oracle RAC can manage serveral TB on up to 100 nodes. No more.
EC2-database, unlike S3, should have realistic and clear limitation.
What is EC2-database?
From the user perspective, you give me a user/password and port to connect to and I open my jdbc/odbc connection and that it is.
I would never have access to the shell of the machine.
This is what the service is about:
- you keep the database up
- you replicate the data
I would know that:
- I can store up to X TB of data
- If I want my database to scale with load, I can add up to XX nodes. More nodes = more simultaneous requests. (may be there could be an option to scale the nodes automatically based upon certain criteria).
Advantages:
- it is the real thing. A real DB. (May be you could even choose a BD from serveral ones?)
- if you do complex queries that never end, you're not bothering anyone else. It happen on instances dedicated to you.
- Always on. Reliability, high availability, failover... it's EC2-database problem. As a user I just talk to an ip or a DNS name. One for the whole cluster.
- Based on EC2 current architecture. 'Just' need to add the database management stuff. Your clients would proabably have to do it anyway again an again.
Please comment.
|
|
Posts:
14
Registered:
4/3/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 15, 2006 6:58 AM PDT
in response to: Martin@AWS
|
|
|
I've been looking at ec2 as a target infrastructure for deploying an existing enterprise app. The app is already written and makes assumptions about drivers and databases which I could go in and change (probably just configuration and some coding) to run on an Amazon Super DB but now I have an instance that is different than what's in production today, what's QA'd, etc. So, if you offered DB services I'd be excited (actually *really* excited, you guys should definitely press-on here) but probably wouldn't be able to use them any time soon. The fewer things I have to do to re-provision my app on your infrastructure the better the value proposition and the lower my overall risk. I can re-deploy what I have today, unchanged, on ec2 if I can install all my own bits, which includes the currently supported db.
The biggest problem I have today, as others have mentioned, is persistent storage. I won't rehash this.
However, I have a request; if I can't have on-demand, good-performance, persistent virtual disk arrays (i.e., arrays in the sense that I could spec level of performance and reliability like you do with raid levels today...), then if Amazon could facilitate and support more OS-transparent access to S3, that would go a long way for making my life easier.
It'd be nice if there were a tight back-end coupling between ec2 and s3 at the filesystem layer (e.g., fine-grained synchronization) but even if you only provided webdav access natively at the s3 side then that would help. It would eliminate the double-proxy setups I have today (fs->dav + dav->s3 == two points of failure, two caches, generally questionable reliability...).
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 15, 2006 7:14 AM PDT
in response to: jim@culbert.net
|
|
|
> The fewer things I have to do to re-provision my app on yourinfrastructure the better the value proposition and the lower myoverall risk.
This is exactly why I think they should go down the real database path. When I say "real" I mean the full SQL thing, not just a subset of the language.
|
|
Posts:
3
Registered:
9/7/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 15, 2006 8:44 AM PDT
in response to: jim@culbert.net
|
|
|
I'd like to chip in and support Jim's statements.
EC2 is of most use to me if I can reprovision existing applications onto it. If those apps need "porting" to run against an Amazon-supplied DB interface, the cost isn't worth it. Developer time is more expensive - and harder to acquire - than alternative hosting.
Best regards,
Stu
|
|
Posts:
14
Registered:
4/3/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 15, 2006 11:19 AM PDT
in response to: againnickname
|
|
|
Even if they implement the "full sql thing" it doesn't help an oracle/sqlserver/db2 based application that makes use of proprietary database capabilities.
While the app I have in mind I can probably port, I'm out of luck in cases where my solution depends on a third party.
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 15, 2006 11:56 PM PDT
in response to: jim@culbert.net
|
|
|
> it doesn't help an oracle/sqlserver/db2 based application that makes use of proprietary database capabilities.
True. And the DB you mention are probably too expensive.
If MySQL can be used at no cost for this type of setup, it might be a good option.
If we take out the issue of price, Amazon might offer a choice for serveral DB.
If the issues of replication is not done at the database level, it could be done in a database independent way (à la
http://sequoia.continuent.org/HomePage).
This would simplify the setup/configuration of the database: just install a single database instance with no knowledge that it is in a cluster and that's it!
The replication/syncronization/load-balancing/failover would be done on a layer on top of it.
As an added bonus you could cluster any database, even the ones that do not support clustering!
Antother bonus is that you do
not
have to have expert knowledge on the replication processes of any database.
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 16, 2006 12:20 AM PDT
in response to: againnickname
|
|
|
I'm just thinking: to a certain extend, amazon might even propose an option to let people create their own DB instance with whatever database they want.
Then you register this instance with the EC2-database and Amazon will take care of the replication/syncronization/load-balancing/failover (as it would be done on a layer on top of it, independently of the DB).
Thus, you would have to take care of the database's licensing costs and not Amazon. [EC2-database would not launch more instances than you have specified].
I think that this idea is becoming more and more final, ready for implementation.
Unless you tell me that this is not possible because of XYZ.
|
|
Posts:
323
Registered:
2/8/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 16, 2006 2:39 AM PDT
in response to: againnickname
|
|
|
I think that this idea is becoming more and more final, ready for implementation.
I don't think so. From my point of view it's not even an idea - it's a fantasy. Database replication/clustering on such a scale isn't something that's done overnight, if at all. Did you know that most enterprise Oracle clusters aren't "replication clusters" at all? Most of them are failover clusters on OS level (database goes down on machine 1, get's started on machine 2) with some kind of SAN in the background.
I for my part would be happy if Amazon could provide some kind of ACD (atomic, consistent and durable) datastore with indexing capabilities. I think this could be done pretty fast, as it could leverage what has already been done by the AWS team (S3).
Sascha
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 16, 2006 6:08 AM PDT
in response to: S. Matzke
|
|
|
Thank you for your comment.
I'ce never said that a layer of top of a SQL database that handles repication/failover/synchronization would be easy. I'm probably not aware of all the technical difficulties and complexity of such a project. Is it doable? I don't really know. Amazon should ask to some database specialist (the guys who build the replication system) to make an imformed decision if they were interested in such idea.
All I know is that if the interface of "a database functionality" service is the SQL language, people will expect to use their sql knowledge. This implies that they will expect that the DB works just like Oracle, or MySQl, or Postgresql, etc...
As you said, this might be hard or even impossible to do.
On the other hand, they could build what you say : ACD (atomic, consistent and durable) datastore with indexing capabilities). That would be great too but there is a catch: you have to develop your application specifically for this new service. I'm probably willing to do that, I'm willing to adapt if the service offers clear benefits, but I guess that many people will not.
Also, there are probably other alternatives that we have not discussed here.
In the end, it all depends on the real objectives of Amazon which we ignore at this time...
|
|
Posts:
102
Registered:
5/8/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 16, 2006 6:42 AM PDT
in response to: againnickname
|
|
|
Why do Amazon have to do this?
What is there to prevent someone else from building a suitable layer on top of EC2, and selling it, if there is enough demand?
Is there something missing from the EC2 architecture or business model that makes this impossible?
|
|
Posts:
2,027
Registered:
3/15/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 16, 2006 7:04 AM PDT
in response to: "mkochanski"
|
|
|
I agree. I would much rather have the fine developers at Amazon working on cool new services and adding some of the really critical features to existing services than groveling around trying to figure out the NP-Hard problem of providing fully-redundant, high-availability SQL databases in whatever flavor you may want.
If people really need that, it sounds like an opportunity for a third-party to build a nice business.
Mitch
|
|
Posts:
29
Registered:
9/17/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 17, 2006 5:57 PM PDT
in response to: Martin@AWS
|
|
|
For me it would simply be the ability to run Oracle Express (which is now free) with persistent storage and a static ip address.
|
|
Posts:
29
Registered:
9/17/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 17, 2006 6:46 PM PDT
in response to: Keith Leng
|
|
|
Actually, having read a bit more about the relationship betwen EC2 and S3, I can see that I could set up a pretty standard Oracle backup strategy. S3 would hold a backup of the database and redo logs would be shipped across as they are created. Exposure to dataloss would be regulated by controlling the frequency of redo log switches.
I'm really just looking around for a good, cheap dedicated system. From what little I've read so far, it looks as though EC2/S3 is well worth considering.
|
|
Posts:
9
Registered:
2/9/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 18, 2006 7:06 PM PDT
in response to: Martin@AWS
|
|
|
Hey all -
Just wanted to thank everyone for contributing. Please continue to expand on this thread! We are watching this discussion closely...
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Database Functionality: What Do You Need?
Posted:
Sep 18, 2006 10:23 PM PDT
in response to: Martin@AWS
|
|
|
Well, if you're watching I wanted to add my conclusion so far of this thread.
What Do We Need? I depends on the people. Some want more functionality than other. Some people know exactly what they want up to the version of the software and other are willing to adapt.
Nevertheless, shall we agree guys that something is better than nothing?
For my part, I'm learning to use Lucene. It is a simple and powerful API. I was using the DB to store
and
search data.
Now I'm off loading the search part to Lucene and will only (mostly) search by primary key on the DB.
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
