Going big
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
It's a hard task to say the least. As you progress through large projects new problems are encountered and new avenues opened. I find myself planning, designing and reviewing more and more frequently the larger the project is. IMO This is absolutely critical as no-one can hold the scope of the project within their own head, no matter how well they think they can(I include myself here :)) It really is a matter of changing your programming practices. Analyse, plan and design before any coding. Given this you may have to re-code aspects of the project as you inevitably know how best to solve the problem when you have finished :-D Ant.
-
It's a hard task to say the least. As you progress through large projects new problems are encountered and new avenues opened. I find myself planning, designing and reviewing more and more frequently the larger the project is. IMO This is absolutely critical as no-one can hold the scope of the project within their own head, no matter how well they think they can(I include myself here :)) It really is a matter of changing your programming practices. Analyse, plan and design before any coding. Given this you may have to re-code aspects of the project as you inevitably know how best to solve the problem when you have finished :-D Ant.
> know how best to solve the problem when you have finished :laugh: Very true, even on smaller projects. Even then you can recode (or refactor if you like buzzwords) and find a better way. The first revision is major, big improvements. The next improves quite a bit as well but not as much as the first. After a few revisions though the improvements decrease considerably and you end up shaving-bits. When did you face the first project you knew you had to get serious for and how did you manage? regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
I think a good project understanding by ALL part of the team is necessary, from the administrative section, to the marketing, to the QA, to the R'n'D team. from a more management point of view, you need to be more precise, generate a bit more of documentation ( not too much, but enough to keep track of the different phase of the projects ), and make the information flow as much as possible, from you to the project leader, to the management ( and vice-versa ). having meeting as soon as possible if a problem occurs, so that everyone knows what happening, and no bad surprises come. also, if you can build a team right at the beginning, is the best thing, having to hire people at different phases, will make thing difficult, training, different coding practice, technical level, ... also having a good technical leader is important, if you're not the one. if you're the one technical leader, on which all is resting on your shoulders, you need to spend a good amount of time right at the beginning to learn, and learn, and prototype things, and this should be clear to management, that the time taken at the beginning, even if it does not produce immediate results, is important and normal, and safe.
Maximilien Lincourt Your Head A Splode - Strong Bad
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
Paul Watson wrote: In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. I think one of the biggest changes is the one of perspective. On a big project, you are still solving problems and implementing solutions, but they are different problems and solutions than those on a small/medium project. The basic tools of analysis, design, implementation and testing are the same. They are just applied to a different set of problems and goals. The key is not going to be adjusting your development style, (although that may need some firming up), but your biggest payoff will be in properly identifying and understanding the requirements and the problems. Big project problems are different than small project problems, and when you can shift your focus to those, you are well on your way. When you look back, you won't notice that you have made any sort of transition. You just solved the problems and implemented their solution. Good luck. Dave "You can say that again." -- Dept. of Redundancy Dept.
-
I think a good project understanding by ALL part of the team is necessary, from the administrative section, to the marketing, to the QA, to the R'n'D team. from a more management point of view, you need to be more precise, generate a bit more of documentation ( not too much, but enough to keep track of the different phase of the projects ), and make the information flow as much as possible, from you to the project leader, to the management ( and vice-versa ). having meeting as soon as possible if a problem occurs, so that everyone knows what happening, and no bad surprises come. also, if you can build a team right at the beginning, is the best thing, having to hire people at different phases, will make thing difficult, training, different coding practice, technical level, ... also having a good technical leader is important, if you're not the one. if you're the one technical leader, on which all is resting on your shoulders, you need to spend a good amount of time right at the beginning to learn, and learn, and prototype things, and this should be clear to management, that the time taken at the beginning, even if it does not produce immediate results, is important and normal, and safe.
Maximilien Lincourt Your Head A Splode - Strong Bad
>if you're the one technical leader, on which all is resting on your shoulders Yes, and I am the rest of the team on this project. So learn, learn and learn I will have to do. Unfortuanately, and scarily, there is nobody here that can mentor me, no uber-developer to show me the way. I am supposed to be that for the other guys here :) I am confident when it comes to gathering requirements, managing expectations, using the right resources etc. etc. It is the technical jedi master bits I need to figure out. Actual coding practices. Like Mac mentions below, thread and connection pooling or master pointer void or some such. Heck, just threading full stop I need to learn. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
What do you mean by 'sticky-tape-and-twine methodology'? You might realize that your approach is not so bad, it just needs some minor adjustments thats all. The hardest part about 'going big' is realizing that you have to change your attitude and style. Clearly you do not suffer from this problem. Drinking In The Sun Forgot Password?
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
I think Chris may have some recent experience with this...:-D I'm sure that he never planned on having 1 million+ members, certainly not in the first version of the code. Some people think of it as a six-pack; I consider it more of a support group.
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
You need bigger sticky-tape-and-twine methodology. ;P //Start of joke Never comment ur code. If it was hard to write, it should be hard to understand !!! //End of joke
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
I'm in the opposite situation, having moved from a site that handles millions of requests a day to a site that handles 1/100 of that traffic. First understand the underlying rules that govern performance and scalability. Operations per second times the average time per operation equals the number of concurrent operations. Just like a schedule you can only control 2 of the 3 parameters. For a scalable system the max number of concurrent operations is dependent on hardware, it isn't artificially constrained by code. So, as long as you don't have anything like session or a slow cache constraining the number of concurrent operations, your only option is to handle each request more quickly. Also, understand how slow operations and fast operations interact on a server. If you have a max of 30 threads on a server, one operation that takes 1 sec and one that takes 10 seconds. If your server gets 10 req/sec for the fast operation, it takes 10 threads to handle the fast operations. If the slow operation gets 2 req/sec then it takes 20 threads, it all runs fine. However as you add traffic to the slow process say to 3 req/sec, then it requires all 30 threads. What ends up happening at that point is that most fast operations end up waiting on a slow operation and their average response time shoots up to 10 seconds and now you have 13 req/sec with an average response time of 10 seconds which requires 130 threads and the server goes down in flames. The solution is to try to group your fast and slow operations in seperate thread pools if possible. In ASP you could do this with seperate applications, or with server 2003 you can do this with asp.net. The basic rules are 1. Minimize network traffic. Only make 0-1 requests to back end services per page. 2. Cache where it makes sense, but watch out for putting constraints on concurrent operations. 3. Learn about indexes and how to write queries that maximize index usage. This is the #1 sql rule for performance. 4. Analyize your site frequently. I have sql scripts written to process sql profiler data to look for total time spent on each stored procedure and also the number of different parameters that it was called with to look at caching procedures that are always called with the same parameters. 5. Instramentation is key. Make sure you capture and record all exceptions. I also have a script that captures any page that takes longer then a set amount of time to run. 6. Set backend command timeouts to no more then 2-3 times your max allow
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
Chances are, what you are trying to do can be refactored into another problem that somebody, somewhere has solved. The trick is in pattern recognition - seeing that your particular problem looks an awful lot like some other problems, and finding resources for those other problems. This is one time where working at a big organization can help... sometimes your problem is similar to the one the team upstairs has been working on for a while, and you can pool resources. Other times, you have to scour the net... CodeProject, etc., and see what's out there and how you might be able to use it (ideas or actual code) for your issues. I guess it mostly boils down to design and requirements gathering. Knowing exactly what is required ("This sever has to support 300,000 hits a day!") will help immensely in defining the problems, and will help in the design phase - you will be able to identify and break down issues that will plague you ("Argh! We can't be allocating 10 meg of memory per connection now, we have to find another way!"). This will cut the problem down into bite-size pieces that can be more easily digested. And sometimes, you truly are venturing into new territory, and you will just have to brainstorm and figure out creative solutions. And then you write articles about how you solved your problem and post them on CodeProject. Just my rambling thoughts. :) "Fish and guests stink in three days." - Benjamin Franlkin
-
I think Chris may have some recent experience with this...:-D I'm sure that he never planned on having 1 million+ members, certainly not in the first version of the code. Some people think of it as a six-pack; I consider it more of a support group.
Roger Wright wrote: I'm sure that he never planned on having 1 million+ members, certainly not in the first version of the cod Remember, that's why CodeProject moved to this[^] hosting system. A more distributed enviroment can handle it better. :-D "Fish and guests stink in three days." - Benjamin Franlkin
-
The time has come in my development career where I am faced with an application that my usual sticky-tape-and-twine methodology is just not going to cut it. In the past most of my app. design has been focused on flexibility, maintainability and deadlineability. For small projects those are more important than scalibility, performance and the other requirements of large applications. Let's face it, you can fool yourself about your development skills quite easily when working on small to medium sized projects. Virtually any combination of code will make the wheel turn. Then you run into a project that needs to be efficient, scaleable and enterprise ready. It needs to handle thousands of users or be able to crunch through hundreds of thousands of SQL records without locking the server up. That is when you find your normal code starts unravelling; other services are locked out, SQL goes into a deadlock, the server falls over and what worked fine with 100 simply does not work with 100k. You can throw hardware at it but eventually the client wonders why he needs three top-line servers to handle a fistful of users. Now I know there are some hectic developers on this site. Guys who have made the leap from the blissful ignorance of sticky-tape-and-twine development to can-run-a-nuclear-power-plant dev. My question is how did you do it? What changed and made you step up a gear? Were there any useful resources or references that helped you? Programming is pretty much programming whatever language or environment you use and IMO "best practices" carry over quite easily. Too clarify; I am asking about you as personal developer not as in working on a specific project and working with peers etc. I can carry on fumbling my way through but fumbling will never make that leap I reckon. regards, Paul Watson Bluegrass South Africa Ian Darling wrote: "and our loonies usually end up doing things like Monty Python." Crikey! ain't life grand?
My 2 cents ... your mileage may vary. After years of looking at million of lines of others code, writing millions your self, you should be able to see what the habits of successful large scale developers are - emulate them. Develop/adopt a robust coding style (e.g. init vars in func's, sanity check all parameters to public methods, ...) and get in the habit of sticking to it regardless of size of project. As mentioned by others : big-picture, plan for scaling, ... it's all a matter of doing a design up front, not just making it up as you go along. This is probably the biggest change you can make and, like the coding style, get in the habit of doing a design up front regardless of the project size ... projects tend to grow and, without a solid design up front, you're beggin' for trouble. Do a rough design for more than the specs require, anticipate future requirements. Do a final design that is scaled back to what is spec'd - keep your rough design in your files. This will help the design grow in a clean manner as new requirements are added. As for server stuff, i would strongly suggest getting up to speed on IOCP (IO completion ports). There are a number of good articles here. It helps handle scaling socket connections, job queuing, async file access, ... it's like a slightly tarnished silver bullet for scalable development. As for SQL stuff, well that's a whole different arena ... but if you're going to deal with it it's worthwhile digging in and reading the MSSQL online books. So, to answer your questions : >> My question is how did you do it? See above. >> What changed and made you step up a gear? Partly ego - looking at millions of lines of code and saying "psshhh!, i can do better than that" :). Partly the perfectionist part of my personality (want to know how it all works). Partly the drive to want to know more, have more ($). >> Were there any useful resources or references that helped you? Books, newsgroups, sites like CP, going through any code i could find, following tangents when learning a new topic - e.g. wanted to learn how to load graphic files, leads to learning about the different formats, leads to colors (and the 4 ways MS represents them), leads to image manipulation and image filters, ... Anyways, i consider it to be a matter of hard work to 'step up a gear'. Good luck, i'm sure you're up to the task ... although there is a LOT to be said for blissful ignorance. :) ...cmk Save the whales - collect the whole set