|
Discussion Forums
|
Thread: Alpha version of grid resource management
|
|
|
Replies:
8
-
Pages:
1
-
Last Post:
Oct 23, 2008 7:26 PM
by: Edward Johnson
|
|
|
Posts:
22
Registered:
11/7/06
|
|
|
|
Alpha version of grid resource management
Posted:
Mar 9, 2007 6:00 AM PST
|
|
|
Hi,
I just finished the first component for a grid resource management system - a DNS Load Balancing component. It's available on:
http://code.google.com/p/dnsbalancer
The dnsbalancer is a set of java programs for load distribution and node management in a dynamic Grid (i.e. AWS EC2). DNS balancer uses DNS to distribute the load amongst nodes. It can make session replication in a cluster (often encountered with application servers, e.g. JBoss) obsolete because it records the client's IP address and will always forward a particular client to the same server.
It currently works with PowerDNS, a powerful yet easily configured nameserver.
The load balancing director communicates with the backend processes via RMI. Scripts are included that run on the nodes and can propagate address changes of the nodes to the director.
The NS entries can be read from a XML file and the load balanced records can be obtained from a database.
Currently, there are some tools that allow to administer the Director via RMI.
--
Please note, this is a proof-of-concept implementation - seems to handle load ok - however it has not been thorougly tested due to time constraints...
I am also trying to implement a dynamic resource allocation model - i.e. to monitor and predict load of nodes and start or stop nodes as load changes.
There are sample shell scripts and a working (also powerDns) config files included, so you can get started rapidly. It allows both CNAME and A records to be used for LB - I use a shell script to monitor the ip address and notify the director when changes (DHCP renewal) is available...
Many thanks to HeadCase Humanufactoring Inc. for making this project possible.
Best,
~flo
|
|
Posts:
4
Registered:
3/7/07
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 9, 2007 10:49 AM PST
in response to: Flo
|
|
|
What a great project- While there are load managers out there currently, the ability to keep track of load across the farm and automatically create new instances through EC2 is great!
I'd love to see more done with this..
Is there any support for fallover? Ie, noting if a machine dies, and then stopping traffic to it?
What about fallover or LB of the LBer? Can I set up several instances of it, and have them share a DB? (ns1, ns2, etc)
Other blind feature requests- (Having not seen it- I'm not trying to be rude, I just think it's cool)
* A way to set maximum number of instances to start
* A way to query the server, so I can find out what server it's created
* Integrated E-mail alerts or a nagios plugin
*
Keep us up to date on your progress- Having Ec2 start/stop support built in a LB would make things much easier to automatically scale to Slashdot/Digg situations.
-Colin
|
|
Posts:
22
Registered:
11/7/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 9, 2007 11:14 AM PST
in response to: e2ven
|
|
|
Hi Colin,
thanks for your post. I am currently working on a monitoring component that will check if the server responds. However, given that there are full blown monitoring applications (like Nagios), it might be useful to integrate rather than re-implement. For my case, I only need basic monitoring features - so a simple HTTP request might do.
Some node-dumping is already implement. The resource allocation however is not yet done - mainly because there are still a lot of open questions which I hope will be resolved after some discussions on the project home.
So if anyone is interested in working on this project - feel free to contact me!
I am running the current setup - with one SSL cert for X instances. For that, I use simple "A" records and RoundRobin - however that is EASILY changeable to any other type of true Load Balancing.
Cheers,
Florian Leibert
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 12, 2007 2:51 AM PDT
in response to: Flo
|
|
|
> it records the client's IP address and will always forward a particular client to the same server.
This is a really great idea. I have never thought about that.
|
|
Posts:
232
Registered:
8/24/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 12, 2007 3:12 AM PDT
in response to: Flo
|
|
|
I have a question for you.
You spread the load thanks to ips so you would like to have ideally a situation where
1 ip = 1 user
so that the load can be spread optimally.
However you get the ip from the DNS client right? As most/all users delegate the resolution of domain name to a recursive DNS servers, the ip that you receive is the ip of the recursive DNS server and not the ip of the client. If I am right, that means that
1 ip = many users
.
If "many users" is superior to the capacity of a server... what do you do?
I think that this is a real threat to this idea. My ISP for instance has many clients and only a few recursive DNS servers.
|
|
Posts:
69
Registered:
9/1/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 12, 2007 6:27 AM PDT
in response to: Flo
|
|
|
flo, but the loadbalancer itself you are currently running outside of EC2, right ? (At least as long as EC2 has no static IP's, I assume)
Roberto
|
|
Posts:
22
Registered:
11/7/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 12, 2007 10:47 AM PDT
in response to: againnickname
|
|
|
Hi,
you are correct with your statement.
However, I still think this makes sense for many small and mid-sized applications - especially if they're for an international and specifically NOT for a regional audience.
If you have around 5-10 instances and want to distribute the load, that should work ok.
For a advanced setup you should use a cluster inside the cloud but you can still use the major DNS LB methods - especially to avoid having to purchase multiple SSL certs...
I am currently putting together a service for session replication across cluster groups - this will be the ideal solution but is a little bit more complex, especially because one has to use TCP Gossip not multicasting. That should solve the problem - with the cost of a little overhead.
Best,
~flo
|
|
Posts:
22
Registered:
11/7/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Mar 12, 2007 10:52 AM PDT
in response to: rsaccon
|
|
|
I am currently running it outside of the cloud - for the reason you mentioned. Depending on the demand and what Amazon does w.r.t. static ips, someone might decide to run a delegate-NS-service - maybe Amazon could give us some static ips so we can test this
best,
~flo
|
|
Posts:
8
Registered:
8/25/06
|
|
|
|
Re: Alpha version of grid resource management
Posted:
Oct 23, 2008 7:26 PM PDT
in response to: Flo
|
|
|
It seems Amazon Elastic IP's fit the bill now.
Any chance of registering a public AMI with the latest version of your balancer?
Cheers,
Ed
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
