Considering starting a software project - anything like this exist?
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
The reason there are lots of distributed build systems for C++ is because of the compilation model of C++: Separate Compilation. This means that each source file (technically each compilation unit) is compiled separately and the resulting object files are only linked together in the final stage (generally by a separate tool known as a linker). Because each compilation unit is compiled separately and requires no communication with the the other compilations this process can be easily parallelized.
Steve
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
On the surface it sounds interesting, but it sounds like you need to do some sort of profiling to figure out where the bottlenecks really are. If the bottlenecks are deep within VS itself, then I don't know how much you could help it along? But definitely sounds interesting.
¡El diablo está en mis pantalones! ¡Mire, mire! SELECT * FROM User WHERE Clue > 0 0 rows returned Save an Orange - Use the VCF! Personal 3D projects Just Say No to Web 2 Point Blow
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
yes, in fact, I think it's already part of msbuild (or maybe part of VS2010 msbuild) you can get a huge speed increase by making all your projects output to the same bin directory. the only caveat being that web projects need to still output to individual bins
-
The reason there are lots of distributed build systems for C++ is because of the compilation model of C++: Separate Compilation. This means that each source file (technically each compilation unit) is compiled separately and the resulting object files are only linked together in the final stage (generally by a separate tool known as a linker). Because each compilation unit is compiled separately and requires no communication with the the other compilations this process can be easily parallelized.
Steve
Yeah, I'm aware of that. I know .NET doesn't do things this way; the best parallelism we can get is having MSBuild build each project using multiple cores. There isn't file-level parallelism. However, in my anecdotal experience, many .NET solutions have several projects that can be built independently. Potential speed up there. And while MSBuild supports building projects in parallel to a degree, the local disk becomes a bottleneck. By spreading the work across machines, I theoretically could save significant build times.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
modified on Tuesday, September 15, 2009 12:27 PM
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
I have never worked on a project that took more than a minute to build. Where have I been all my life?
I have been trying for weeks to get this little site indexed. If you wonder what it is, or would like some informal accommodation for the 2010 World Cup, please click on this link for Rhino Cottages.
-
On the surface it sounds interesting, but it sounds like you need to do some sort of profiling to figure out where the bottlenecks really are. If the bottlenecks are deep within VS itself, then I don't know how much you could help it along? But definitely sounds interesting.
¡El diablo está en mis pantalones! ¡Mire, mire! SELECT * FROM User WHERE Clue > 0 0 rows returned Save an Orange - Use the VCF! Personal 3D projects Just Say No to Web 2 Point Blow
Right. I would certainly do some profiling before even going to a proof-of-concept. Now I would skip Visual Studio altogether, of course, going directly to MSBuild system to do the builds. The gains I'd hope to achieve are building independent projects on multiple machines in parallel.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
yes, in fact, I think it's already part of msbuild (or maybe part of VS2010 msbuild) you can get a huge speed increase by making all your projects output to the same bin directory. the only caveat being that web projects need to still output to individual bins
Yeah, I'm aware of that. The NDepend creator, whose name escapes me at the moment, blogged about this recently. Still, I suspect very few developers take advantage of this, and fewer that can actually save time by doing it this way, as there are some notable exceptions to the rule. Even if we can save time by outputting to the same bin directory, I think we can get a good speed up by doing a distributed build. Theoretically, anyways.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
I have never worked on a project that took more than a minute to build. Where have I been all my life?
I have been trying for weeks to get this little site indexed. If you wonder what it is, or would like some informal accommodation for the 2010 World Cup, please click on this link for Rhino Cottages.
Our current project here at work takes about 2:30 to build. Even 1:00 is insane, considering we do many builds per day. Make a change to some low-level component, then suddenly all these dependent projects need to build, and you go on a coffee break as VS thrashes your disk for a while. The idea is to make even large solutions (30-50 projects) build in seconds, rather than minutes. The idea is to eliminate coffee breaks. :)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
How does that compare to IncrediBuild[^]?
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
-
Yeah, I'm aware of that. The NDepend creator, whose name escapes me at the moment, blogged about this recently. Still, I suspect very few developers take advantage of this, and fewer that can actually save time by doing it this way, as there are some notable exceptions to the rule. Even if we can save time by outputting to the same bin directory, I think we can get a good speed up by doing a distributed build. Theoretically, anyways.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
well, i was able to improve the build times considerably where i worked. we had 200+ projects all outputting to individual directories. it took over 1.5 hours to build. i changed the projects to all output to the same directory. reduced build time to under 5 minutes. yes, 5 minutes. using 10k & 15k rpm raptor drives also helped. but yeah, theoretically you could speed things up with a distributed build. the problem is dependencies.
Judah Himango wrote:
blogging on the intarwebs
lol. i still get a kick out of that US Rep. who called them the itarTUBES :laugh: :-D
-
How does that compare to IncrediBuild[^]?
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
My project has similar goals, albeit a bit more lower-level on the "what to send to/from server" level. However, mine is targeted at .NET code. I checked out IncrediBuild when doing some research last month. AFAICT, it really works on C/C++ only. They claim it does builds for .NET, but it doesn't do distributed builds then, essentially canceling it's usefulness for .NET. (If I'm wrong about this, anyone, please show me documentation/examples otherwise.)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Our current project here at work takes about 2:30 to build. Even 1:00 is insane, considering we do many builds per day. Make a change to some low-level component, then suddenly all these dependent projects need to build, and you go on a coffee break as VS thrashes your disk for a while. The idea is to make even large solutions (30-50 projects) build in seconds, rather than minutes. The idea is to eliminate coffee breaks. :)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
allows for natural coffee breaks!
------------------------------------ "Men may make bad decisions, immoral decisions or just plain wrong decisions, but at least they make decisions. Women on the other hand..." Patrick Kielty 2006
-
How does that compare to IncrediBuild[^]?
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
to quote from the site: "builds by distributing C/C++ compilation" Judah wants to distribute C# compilation.
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
I have found that extremely large projects with slow build times can usually be optimized to have fast build times. But that is just my experience. I can tell you, however, that projects with slow build times tend to extract better work from less experienced developers because they are forced to think about their actions a lot more.
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. A man said to the universe: "Sir I exist!" "However," replied the universe, "The fact has not created in me A sense of obligation." --Stephen Crane
-
Our current project here at work takes about 2:30 to build. Even 1:00 is insane, considering we do many builds per day. Make a change to some low-level component, then suddenly all these dependent projects need to build, and you go on a coffee break as VS thrashes your disk for a while. The idea is to make even large solutions (30-50 projects) build in seconds, rather than minutes. The idea is to eliminate coffee breaks. :)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
Judah Himango wrote:
2:30 to build. Even 1:00 is insane
Is that hh:mm or mm:ss ?
-
Our current project here at work takes about 2:30 to build. Even 1:00 is insane, considering we do many builds per day. Make a change to some low-level component, then suddenly all these dependent projects need to build, and you go on a coffee break as VS thrashes your disk for a while. The idea is to make even large solutions (30-50 projects) build in seconds, rather than minutes. The idea is to eliminate coffee breaks. :)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
Sure must be some kind of application. If you often have to rebuild so many projects that it takes that long to build, perhaps more thought needs to go into the dependencies to make them not quite so tight. Independence can be a good thing :) I do think the idea of a simple distributed build is a good idea though. It is amazing the functionality was not built in many years ago.
Rocky <>< Recent Blog Post: Chocolate Chip Cookies!
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
How much C# code do you have really? In my experience, C# compiles a lot faster than C++ in the order of magnitudes. I have a project with about 120k+ lines of C++ code, and it takes roughly 2-3 minutes to compile on a quad core machine. I don't have nearly as much C# code, only 10k+ maybe. I do know that the C# code compiles blazingly fast though. I think the IDE spends most of its time in starting the C# compiler, rather than waiting for it to complete. I'm really curious to know how much C# code you have. You ought to have the proverbial shitload of code if compilation times are a problem for you...
-
My project has similar goals, albeit a bit more lower-level on the "what to send to/from server" level. However, mine is targeted at .NET code. I checked out IncrediBuild when doing some research last month. AFAICT, it really works on C/C++ only. They claim it does builds for .NET, but it doesn't do distributed builds then, essentially canceling it's usefulness for .NET. (If I'm wrong about this, anyone, please show me documentation/examples otherwise.)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
Ah - sorry, hadn't noticed that. As I do C++ pretty exclusively, a lack of C# didn't really catch my eye :-). I can imagine there might be issues with C# - making sure you've got the same framework versions, referenced assemblies etc, whereas with C/C++ you only involve files (unless you're using #import).
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
-
Ah - sorry, hadn't noticed that. As I do C++ pretty exclusively, a lack of C# didn't really catch my eye :-). I can imagine there might be issues with C# - making sure you've got the same framework versions, referenced assemblies etc, whereas with C/C++ you only involve files (unless you're using #import).
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
Stuart Dootson wrote:
I can imagine there might be issues with C# - making sure you've got the same framework versions, referenced assemblies etc
Certainly. The good news is .NET frameworks can install safely side-by-side from 1.0 to the latest. And the latest MSBuilds can target 2.0, 3, 3.5, and 4. So the versioning story there is good, I'm not expecting too many headaches there.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango