Considering starting a software project - anything like this exist?
-
well, i was able to improve the build times considerably where i worked. we had 200+ projects all outputting to individual directories. it took over 1.5 hours to build. i changed the projects to all output to the same directory. reduced build time to under 5 minutes. yes, 5 minutes. using 10k & 15k rpm raptor drives also helped. but yeah, theoretically you could speed things up with a distributed build. the problem is dependencies.
Judah Himango wrote:
blogging on the intarwebs
lol. i still get a kick out of that US Rep. who called them the itarTUBES :laugh: :-D
ahmed zahmed wrote:
well, i was able to improve the build times considerably where i worked. we had 200+ projects all outputting to individual directories. it took over 1.5 hours to build. i changed the projects to all output to the same directory. reduced build time to under 5 minutes. yes, 5 minutes.
Awesome! I will pass this on to my team here at work. We have maybe 30-40 projects including unit test projects. Takes about 2 and half minutes to build. Not terrible, but when you consider that we do hundreds of builds a day, it adds up.
ahmed zahmed wrote:
the problem is dependencies.
Yes, that's the trick. I'm going to be testing my theory on some big open source .NET projects and see what kind of dependencies can be isolated.
ahmed zahmed wrote:
using 10k & 15k rpm raptor drives also helped.
I believe it. You know, if this idea pans out, I might build some fresh machines with high-speed HDs or SSDs. The builds in the cloud could occur very quickly to get the results back to the client. Just dreaming for now, though. :-)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Judah Himango wrote:
2:30 to build. Even 1:00 is insane
Is that hh:mm or mm:ss ?
2 minutes and 30 seconds. :-) Doesn't sound like much, but when you do many builds per day, and each developer is doing many builds per day, it adds up.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Sure must be some kind of application. If you often have to rebuild so many projects that it takes that long to build, perhaps more thought needs to go into the dependencies to make them not quite so tight. Independence can be a good thing :) I do think the idea of a simple distributed build is a good idea though. It is amazing the functionality was not built in many years ago.
Rocky <>< Recent Blog Post: Chocolate Chip Cookies!
Rocky Moore wrote:
Sure must be some kind of application. If you often have to rebuild so many projects that it takes that long to build, perhaps more thought needs to go into the dependencies to make them not quite so tight. Independence can be a good thing
Agreed. And we are doing some project-combining next week to make fewer but larger assemblies. Nonetheless, we have unit test projects that need to be separate assemblies, we have client projects separate from server projects (these also cannot be combined), add-in framework assemblies which cannot be combined, shared libraries between client and server, and some server-side components that deliberately are separated from the server assembly so they can be run on a separate machine, and so on. What I'm trying to say is, on larger projects, solutions tend to have many projects, some of which cannot be combined, yet many could be built in parallel.
Rocky Moore wrote:
I do think the idea of a simple distributed build is a good idea though. It is amazing the functionality was not built in many years ago.
Cool, thanks, that's encouraging. Alright, I will pursue this idea further.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Our current project here at work takes about 2:30 to build. Even 1:00 is insane, considering we do many builds per day. Make a change to some low-level component, then suddenly all these dependent projects need to build, and you go on a coffee break as VS thrashes your disk for a while. The idea is to make even large solutions (30-50 projects) build in seconds, rather than minutes. The idea is to eliminate coffee breaks. :)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
I went through and converted a bunch of code to my extremist form of include file protection the other day and saw an enourmous improvement in compile speed. In many cases I have seen very large improvements. Basically it amounts to using the include gaurd ifdefs outside the include statement so that they aren't always blindly included if they don't need to be.
-
I went through and converted a bunch of code to my extremist form of include file protection the other day and saw an enourmous improvement in compile speed. In many cases I have seen very large improvements. Basically it amounts to using the include gaurd ifdefs outside the include statement so that they aren't always blindly included if they don't need to be.
-
Judging from other things you have said this will probably not have such a big effect on your code.
This project would do .NET compilations - C#, VB.NET, F#, etc.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Jörgen Sigvardsson wrote:
How much C# code do you have really?
Last I checked, 270,000 LOC. It's probably closer to 300k now.
Jörgen Sigvardsson wrote:
I do know that the C# code compiles blazingly fast though.
It does. The compiling isn't really the issue. You do a compile, it finishes near instantly, then the disk churns as it spits out assemblies and dependencies. Then dependent projects compile, spit out assemblies and depenedencies, etc. until all changed/depedent-on-changed projects are rebuilt. This takes significant time when done locally. One thing I've found is that if you have fewer projects but larger assemblies, build times decrease. However, often times you can't combine projects: separate assemblies for client and server, separate assemblies for shared data between client/server, utility libraries, add-in projects that require their own assemblies, independently-maintained libraries (e.g. a .NET project including the Lucene.NET open source search engine), and so on. What I'm saying is, you often cannot combine many projects. This causes build/run times to slow way down.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
I understand your predicament. Basically, it's the overhead of not compiling (shuffling stuff into place) that increases build time. I think you would have to partition the solution in such a way that the distribution nodes can work independently of each other. This may be hard if the projects are tightly coupled to each other. Perhaps you could find a subset of projects that you could build on a single node, and then parallelize the resulting forrest of projects over several nodes. Post some updates here in the Lounge if you come up with something :)
-
Jörgen Sigvardsson wrote:
How much C# code do you have really?
Last I checked, 270,000 LOC. It's probably closer to 300k now.
Jörgen Sigvardsson wrote:
I do know that the C# code compiles blazingly fast though.
It does. The compiling isn't really the issue. You do a compile, it finishes near instantly, then the disk churns as it spits out assemblies and dependencies. Then dependent projects compile, spit out assemblies and depenedencies, etc. until all changed/depedent-on-changed projects are rebuilt. This takes significant time when done locally. One thing I've found is that if you have fewer projects but larger assemblies, build times decrease. However, often times you can't combine projects: separate assemblies for client and server, separate assemblies for shared data between client/server, utility libraries, add-in projects that require their own assemblies, independently-maintained libraries (e.g. a .NET project including the Lucene.NET open source search engine), and so on. What I'm saying is, you often cannot combine many projects. This causes build/run times to slow way down.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
If you really have a lot of assemblies, particularly ones that have dependencies on each other, and you are not compiling to a common directory and marking the references "Copy local = false", then you are wasting a lot of time duplicating references to each project build directory. File i/o is eating you alive, not to mention wasting a lot of disk real-estate. Changing so you build to a common output folder, and making sure references aren't copied, could save a lot of time. As a test, write a little utility to edit all the project files to fix up references and change output directory and then try a build. Surely your project files are in some source control system, so if it doesn't help enough, no harm done, just revert all the projects.
-
This project would do .NET compilations - C#, VB.NET, F#, etc.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
If you really have a lot of assemblies, particularly ones that have dependencies on each other, and you are not compiling to a common directory and marking the references "Copy local = false", then you are wasting a lot of time duplicating references to each project build directory. File i/o is eating you alive, not to mention wasting a lot of disk real-estate. Changing so you build to a common output folder, and making sure references aren't copied, could save a lot of time. As a test, write a little utility to edit all the project files to fix up references and change output directory and then try a build. Surely your project files are in some source control system, so if it doesn't help enough, no harm done, just revert all the projects.
Yep, we know. We figure that's a big source of the time taken in build times. We going to do a project reorganization next week that will combine many assemblies into fewer, larger assemblies. (Nonetheless, there are some projects that cannot be merged.) I'll additionally push for a common output directory as you suggest. Still worth pursuing a distributed build system for .NET projects?
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Driven by really slow build times in our .NET project here at work, and slow build times on the projects I've done consulting for, I'm considering building a Visual Studio add-in + web service that would allow you to perform a distributed build for .NET projects. Performing a build would go like this: -You click Build->Distributed Build in Visual Studio. -The VS add-in would detect which files have changed since the last build. -It send only the changed bytes of those files compressed to the web service. -The web service, having previously analyzed your project dependence hierarchies, figures out which projects need to be rebuilt. -Spreads the builds across multiple machines, each building some independent project(s). Then: -If you're just trying to build the software, it sends back the build result, success or error messages. -If you're trying to actually run the software, it sends only the changed bytes of the assemblies, compressed, back to the VS add-in. I'm thinking I could get super fast build times doing this, much faster than doing a build locally. Obviously some proof-of-concept is in order. Two Questions for you CPians: -Is there any existing distributed build systems for .NET? I see lots of C/C++, but can't find any for .NET projects. -Is this a worthwhile project? .NET projects generally build fast...until you have a large solution with lots of projects.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
Are you using WPF? If so, you're probably aware of the issue that cyclic references between XAML and C# cause the project to be compiled twice. Unfortunately such cyclic references are usually unavoidable. Recently I've been trying to figure out a way to do XAML compilation without the second C# compile run, instead injecting the BAML into the compiled assembly. I've been able to make place inside the Resources segment in an assembly (which means updating a lot of RVAs that point to stuff after the Resources segment); and indeed assemblies "enlarged" this way run on .NET; but for some reason PEVerify doesn't like them and I don't know why. Still, for local development it might be interesting to use this approach, falling back to the double compile run only for release builds. I'll try to continue work on this...
-
Yep, we know. We figure that's a big source of the time taken in build times. We going to do a project reorganization next week that will combine many assemblies into fewer, larger assemblies. (Nonetheless, there are some projects that cannot be merged.) I'll additionally push for a common output directory as you suggest. Still worth pursuing a distributed build system for .NET projects?
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
Judah Himango wrote:
Still worth pursuing a distributed build system for .NET projects?
I don't see how you could efficiently resolve dependency issues - you might end up losing just as much time shuttling/copying dependent assemblies around. An SSD[^]for the output directory would probably help more, and possibly cost about the same as deploying a distributed build system.
-
Are you using WPF? If so, you're probably aware of the issue that cyclic references between XAML and C# cause the project to be compiled twice. Unfortunately such cyclic references are usually unavoidable. Recently I've been trying to figure out a way to do XAML compilation without the second C# compile run, instead injecting the BAML into the compiled assembly. I've been able to make place inside the Resources segment in an assembly (which means updating a lot of RVAs that point to stuff after the Resources segment); and indeed assemblies "enlarged" this way run on .NET; but for some reason PEVerify doesn't like them and I don't know why. Still, for local development it might be interesting to use this approach, falling back to the double compile run only for release builds. I'll try to continue work on this...
Daniel Grunwald wrote:
Are you using WPF?
Not much. We're mostly WinForms on the client, but we're integrating with some WPF components now.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Judah Himango wrote:
Still worth pursuing a distributed build system for .NET projects?
I don't see how you could efficiently resolve dependency issues - you might end up losing just as much time shuttling/copying dependent assemblies around. An SSD[^]for the output directory would probably help more, and possibly cost about the same as deploying a distributed build system.
Rob Graham wrote:
I don't see how you could efficiently resolve dependency issues
It will be tough. I'll see what can be done. I do plan on having 10k RPM HDs or SSDs in a RAID on the build machines. So even if the projects don't have much or any independent bits of work to be done, we can still at least do faster builds than a typical dev machine.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Rob Graham wrote:
I don't see how you could efficiently resolve dependency issues
It will be tough. I'll see what can be done. I do plan on having 10k RPM HDs or SSDs in a RAID on the build machines. So even if the projects don't have much or any independent bits of work to be done, we can still at least do faster builds than a typical dev machine.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
How will you arbitrate the builds - that is, who gets to build and who doesn't.
-
How will you arbitrate the builds - that is, who gets to build and who doesn't.
Which projects get to build, you're asking? On the client (VS add-in) side, I'd monitor changed source files. When the server (web service) gets these changes, it figures out which projects need to be built based on the project dependence hierarchy tree generated on the server during the one-time initialization of the project.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
-
Rocky Moore wrote:
Sure must be some kind of application. If you often have to rebuild so many projects that it takes that long to build, perhaps more thought needs to go into the dependencies to make them not quite so tight. Independence can be a good thing
Agreed. And we are doing some project-combining next week to make fewer but larger assemblies. Nonetheless, we have unit test projects that need to be separate assemblies, we have client projects separate from server projects (these also cannot be combined), add-in framework assemblies which cannot be combined, shared libraries between client and server, and some server-side components that deliberately are separated from the server assembly so they can be run on a separate machine, and so on. What I'm trying to say is, on larger projects, solutions tend to have many projects, some of which cannot be combined, yet many could be built in parallel.
Rocky Moore wrote:
I do think the idea of a simple distributed build is a good idea though. It is amazing the functionality was not built in many years ago.
Cool, thanks, that's encouraging. Alright, I will pursue this idea further.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
Judah Himango wrote:
What I'm trying to say is, on larger projects, solutions tend to have many projects, some of which cannot be combined, yet many could be built in parallel.
Yeah, I was not talking as much about combining them, but rather how them different assembles interact, that they can be more loosly coupled I understand that it depends on the application.
Rocky <>< Recent Blog Post: Chocolate Chip Cookies!
-
Yeah, I'm aware of that. I know .NET doesn't do things this way; the best parallelism we can get is having MSBuild build each project using multiple cores. There isn't file-level parallelism. However, in my anecdotal experience, many .NET solutions have several projects that can be built independently. Potential speed up there. And while MSBuild supports building projects in parallel to a degree, the local disk becomes a bottleneck. By spreading the work across machines, I theoretically could save significant build times.
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango
modified on Tuesday, September 15, 2009 12:27 PM
I have found that runing RAID 0 on the source code folders (and bin folders) have a very noticable impact on build times... HDD is a huge problem, especially when running top end CPU's and plenty RAM. I haven't played with SDD yet but RAID 0 works very well. Sounds like a very interesting project though, one worth doing as a 'lurning curve' if you have the time, even if it isn't hugely successful...
____________________________________________________________ Be brave little warrior, be VERY brave
-
I have never worked on a project that took more than a minute to build. Where have I been all my life?
I have been trying for weeks to get this little site indexed. If you wonder what it is, or would like some informal accommodation for the 2010 World Cup, please click on this link for Rhino Cottages.
-
Our current project here at work takes about 2:30 to build. Even 1:00 is insane, considering we do many builds per day. Make a change to some low-level component, then suddenly all these dependent projects need to build, and you go on a coffee break as VS thrashes your disk for a while. The idea is to make even large solutions (30-50 projects) build in seconds, rather than minutes. The idea is to eliminate coffee breaks. :)
Religiously blogging on the intarwebs since the early 21st century: Kineti L'Tziyon Judah Himango