Grid computing and Windows
-
Does anyone know if there is an easy way to connect a computer of PCs for Grid computing? I have done a bunch of googling already and they all seem specific and I didn't really find an easy solution. I don't want to spend too much time on it if there isn't something easy. Maybe I am better off using rackspace but I haven't used that yet. Apparently, for rackspace you need to build the machine then save it when you are done so you don't keep getting hourly charged? [NOTE: this is not for C++ or language compiles; it is for game engine (Gtkradiant) builds]
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
Could you use a Beowulf Cluster[^] and maybe see if you can run a VM across the cluster or use Windows HPC Server[^] though it wouldn't be cheap.
People are more violently opposed to fur than leather because it's safer to harass rich women than motorcycle gangs
-
Could you use a Beowulf Cluster[^] and maybe see if you can run a VM across the cluster or use Windows HPC Server[^] though it wouldn't be cheap.
People are more violently opposed to fur than leather because it's safer to harass rich women than motorcycle gangs
Interesting, I guess I am looking for something like Windows HPC Server but I don't have the minimum requirements. I decided to just try a 32GB rackspace Win2k8R2 and see how much money I throw at that. That will get me 8 core instead of the 4 core i5 I am using right now. Thanks.
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
-
Interesting, I guess I am looking for something like Windows HPC Server but I don't have the minimum requirements. I decided to just try a 32GB rackspace Win2k8R2 and see how much money I throw at that. That will get me 8 core instead of the 4 core i5 I am using right now. Thanks.
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
If you do decide to do it yourself later have a look at the free version of the VMWare vSphere Hypervisor[^], you install it on each node in the cluster which gives you a pool of resources that can be assigned to a virtual machine running normal windows. Though if you don't have the hardware on hand that's where it can get expensive. :-D
People are more violently opposed to fur than leather because it's safer to harass rich women than motorcycle gangs
-
Does anyone know if there is an easy way to connect a computer of PCs for Grid computing? I have done a bunch of googling already and they all seem specific and I didn't really find an easy solution. I don't want to spend too much time on it if there isn't something easy. Maybe I am better off using rackspace but I haven't used that yet. Apparently, for rackspace you need to build the machine then save it when you are done so you don't keep getting hourly charged? [NOTE: this is not for C++ or language compiles; it is for game engine (Gtkradiant) builds]
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
-
Not for a project though. I am running an application that takes lots of CPU.
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
In the past, I supported an application that put tasks in a database, it had hundreds of servers that ran the same service. Each service would pick up x tasks from the DB at a time.(It would lock the tasks up for 5 minutes then reopen them if nothing happened to them in that time. So if a server crashed, the tasks weren't lost.) Then run x threads to solve the x tasks. This had an infinite loop that looked at how many threads were active and if x more wasn't more than the y limit it would do it again until it retrieved 0 tasks and then the task would sleep for a second. If you can separate your tasks into discrete parts, you can put it into a DB and use it as a central controller. No limit to the number of computers involved. If you can't separate it into tasks, what make you think adding cores is going to solve the problem?
-
In the past, I supported an application that put tasks in a database, it had hundreds of servers that ran the same service. Each service would pick up x tasks from the DB at a time.(It would lock the tasks up for 5 minutes then reopen them if nothing happened to them in that time. So if a server crashed, the tasks weren't lost.) Then run x threads to solve the x tasks. This had an infinite loop that looked at how many threads were active and if x more wasn't more than the y limit it would do it again until it retrieved 0 tasks and then the task would sleep for a second. If you can separate your tasks into discrete parts, you can put it into a DB and use it as a central controller. No limit to the number of computers involved. If you can't separate it into tasks, what make you think adding cores is going to solve the problem?
KP Lee wrote:
If you can't separate it into tasks, what make you think adding cores is going to solve the problem?
"Does anyone know if there is an easy way to connect a computer of PCs for Grid computing?" So your answer is "no"? Hypervisors seem to do a good job of scaling back and forth cores, so my question was if there was something similar, however linking PCs together not just onboard cores.
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
-
Does anyone know if there is an easy way to connect a computer of PCs for Grid computing? I have done a bunch of googling already and they all seem specific and I didn't really find an easy solution. I don't want to spend too much time on it if there isn't something easy. Maybe I am better off using rackspace but I haven't used that yet. Apparently, for rackspace you need to build the machine then save it when you are done so you don't keep getting hourly charged? [NOTE: this is not for C++ or language compiles; it is for game engine (Gtkradiant) builds]
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
-
Take a look at Utilify. It's a flexible system, and does what was mentioned earlier: put tasks in a database table and hand off to PCs and/or servers.
"It is a very easy-to-use application framework for developers to create distributed / grid applications rapidly. " Thanks. Checking it out now.
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
-
KP Lee wrote:
If you can't separate it into tasks, what make you think adding cores is going to solve the problem?
"Does anyone know if there is an easy way to connect a computer of PCs for Grid computing?" So your answer is "no"? Hypervisors seem to do a good job of scaling back and forth cores, so my question was if there was something similar, however linking PCs together not just onboard cores.
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
JoeSox wrote:
So your answer is "no"?
My answer is yes or no. In order for multi-threading to work, they have to do distinct things. (IE one thread can run through a complete process and finish its task without needing information from the other task.) If they share items, they need to handshake the sharing. (I've locked it, you can't have it. I've unlocked it, you can have it. It's locked, I need to wait.) You can't have one huge process and say, "OK, break it into threads for me." Some tools like SQL are designed to multi-thread, but as far as I know that is built into the number of CPUs locally available. It still separates it into tasks. (You get this part of the table, U2 get another part, all of you feed your data into a mashup process.) If your code is designed to multi-thread, then yes, you can split it up to run on several CPUs. If it isn't, then no, you can't. I've got an application with three threads designed into it. Two of them are basically idle most of the time. One of them takes up 100% of one of my dual cores for 2 minutes. So my machine shows 50% active if nothing else is going on. I've figured out how I could split that active thread up into 3 to 40,000 different threads, but it would take a complete redesign of how that one thread currently works. For 3 threads, one could divy up the tasks into 2 plates, then it would start one thread with one plate and start another with the 2nd plate. It would take a few seconds to fill each plate with 20,000 tasks, start the threads, monitor the status of the threads. Or one could put one task each on both plates and ask each to run up to 70 times with a variable load for each task. So, I've got a variety of ways to do it, all I have to do is redesign my code first. This is a toy app, so I'm not about to do that. (Anything not commercial is a toy app. Even if no money is involved, anything in consumer demand is commercial) So, do you have discreet tasks ready to divvy up?
-
JoeSox wrote:
So your answer is "no"?
My answer is yes or no. In order for multi-threading to work, they have to do distinct things. (IE one thread can run through a complete process and finish its task without needing information from the other task.) If they share items, they need to handshake the sharing. (I've locked it, you can't have it. I've unlocked it, you can have it. It's locked, I need to wait.) You can't have one huge process and say, "OK, break it into threads for me." Some tools like SQL are designed to multi-thread, but as far as I know that is built into the number of CPUs locally available. It still separates it into tasks. (You get this part of the table, U2 get another part, all of you feed your data into a mashup process.) If your code is designed to multi-thread, then yes, you can split it up to run on several CPUs. If it isn't, then no, you can't. I've got an application with three threads designed into it. Two of them are basically idle most of the time. One of them takes up 100% of one of my dual cores for 2 minutes. So my machine shows 50% active if nothing else is going on. I've figured out how I could split that active thread up into 3 to 40,000 different threads, but it would take a complete redesign of how that one thread currently works. For 3 threads, one could divy up the tasks into 2 plates, then it would start one thread with one plate and start another with the 2nd plate. It would take a few seconds to fill each plate with 20,000 tasks, start the threads, monitor the status of the threads. Or one could put one task each on both plates and ask each to run up to 70 times with a variable load for each task. So, I've got a variety of ways to do it, all I have to do is redesign my code first. This is a toy app, so I'm not about to do that. (Anything not commercial is a toy app. Even if no money is involved, anything in consumer demand is commercial) So, do you have discreet tasks ready to divvy up?
I am not programming anything. I am using GtkRadiant 1.5 and desire more CPU
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
-
Does anyone know if there is an easy way to connect a computer of PCs for Grid computing? I have done a bunch of googling already and they all seem specific and I didn't really find an easy solution. I don't want to spend too much time on it if there isn't something easy. Maybe I am better off using rackspace but I haven't used that yet. Apparently, for rackspace you need to build the machine then save it when you are done so you don't keep getting hourly charged? [NOTE: this is not for C++ or language compiles; it is for game engine (Gtkradiant) builds]
Later, JoeSox CPMCv1.0 - Last.fm - MyFriendfeed - CPForAndroid++
The answer is, no. If you don't control the code you're using, and it isn't one of those megabuck programs that is set up for cluster operations (such as SQL Server), then there is no easy way to get a set of PCs to break up a task being done by one PC.
"It's not what you don't know that will hurt you the most, it's what you think you know that isn't so." - Unknown