Long operation [modified]
-
Hi, I have a form that takes a number from the user and loops through this number to do some calculations and database access:
for (int i=0; i <NUM; ++i){ //do calculation // access database }my problem here is that: for example when NUM=1000, it takes 20 seconds to complete, when I tried 1500 it took more than 60 seconds which is unrealistic because each loop takes the same time. I tried to stop the loop when i=1000 but NUM is still = 1500 to see when it does finish, I was shocked to see that 60 seconds was required to do the 1000!! how could this possibly be? P.S: I did NUM=1200 and 1400 and they all took 20 seconds to complete the 1000!! why does 1500 is different? another thing, is multithreading useful to speed up long operations? I have read about it and know that it is useful when I want more than one process to be executed at the same time, but what if I only have one process, does it speed it up or something? thanks, -- modified at 21:46 Wednesday 25th April, 2007 -
Hi, I have a form that takes a number from the user and loops through this number to do some calculations and database access:
for (int i=0; i <NUM; ++i){ //do calculation // access database }my problem here is that: for example when NUM=1000, it takes 20 seconds to complete, when I tried 1500 it took more than 60 seconds which is unrealistic because each loop takes the same time. I tried to stop the loop when i=1000 but NUM is still = 1500 to see when it does finish, I was shocked to see that 60 seconds was required to do the 1000!! how could this possibly be? P.S: I did NUM=1200 and 1400 and they all took 20 seconds to complete the 1000!! why does 1500 is different? another thing, is multithreading useful to speed up long operations? I have read about it and know that it is useful when I want more than one process to be executed at the same time, but what if I only have one process, does it speed it up or something? thanks, -- modified at 21:46 Wednesday 25th April, 2007The only thing that speeds up long operations is more powerful machine. Multithreading can only prevent long operations blocking other operations, and to be picky, it will actually slow the long operation down by a minuscule factor. The whole point there is being able to do other things while the long operation is churning away in the background.
-
The only thing that speeds up long operations is more powerful machine. Multithreading can only prevent long operations blocking other operations, and to be picky, it will actually slow the long operation down by a minuscule factor. The whole point there is being able to do other things while the long operation is churning away in the background.
Brady Kelly wrote:
The only thing that speeds up long operations is more powerful machine
I strongly disagree. You can apply a better algorithm, choose a better implementation, or improve parallellism, which is able to reduce the overall time provided there were spare CPU cycles to begin with (indicating part of the code is either waiting on some resource, or just sleeping for some reason). For some resources, behavior is non-linear; AFAIK sockets use some resources that are available to some extent, and include a delay before getting freed automatically. :)
Luc Pattyn [My Articles]
-
Brady Kelly wrote:
The only thing that speeds up long operations is more powerful machine
I strongly disagree. You can apply a better algorithm, choose a better implementation, or improve parallellism, which is able to reduce the overall time provided there were spare CPU cycles to begin with (indicating part of the code is either waiting on some resource, or just sleeping for some reason). For some resources, behavior is non-linear; AFAIK sockets use some resources that are available to some extent, and include a delay before getting freed automatically. :)
Luc Pattyn [My Articles]
OK, maybe my answer was grossly over simplified, and for that it should also been much shorter.:-O
-
OK, maybe my answer was grossly over simplified, and for that it should also been much shorter.:-O
thanks for the info guys. :)
-
Hi, I have a form that takes a number from the user and loops through this number to do some calculations and database access:
for (int i=0; i <NUM; ++i){ //do calculation // access database }my problem here is that: for example when NUM=1000, it takes 20 seconds to complete, when I tried 1500 it took more than 60 seconds which is unrealistic because each loop takes the same time. I tried to stop the loop when i=1000 but NUM is still = 1500 to see when it does finish, I was shocked to see that 60 seconds was required to do the 1000!! how could this possibly be? P.S: I did NUM=1200 and 1400 and they all took 20 seconds to complete the 1000!! why does 1500 is different? another thing, is multithreading useful to speed up long operations? I have read about it and know that it is useful when I want more than one process to be executed at the same time, but what if I only have one process, does it speed it up or something? thanks, -- modified at 21:46 Wednesday 25th April, 2007I'm betting this loop can be made faster if the access to the DB is optimized. What I mean by that is: Does the DB really need to be accessed each time through the loop? If the answer is no, then create a conditional that will only access the DB when it is necessary. Another way to look at this is to do all of the DB access at once. In other words, split your processing up. Do the calculations and store them in a manner that is efficient for later DB access once the calculations have been finished (after the calculation loop is completed). Without knowing much about what the calculation is, why the DB access needs to be done, and how the data in the DB is affected, I can't give a good answer on how to split the processing up. Phil
-
I'm betting this loop can be made faster if the access to the DB is optimized. What I mean by that is: Does the DB really need to be accessed each time through the loop? If the answer is no, then create a conditional that will only access the DB when it is necessary. Another way to look at this is to do all of the DB access at once. In other words, split your processing up. Do the calculations and store them in a manner that is efficient for later DB access once the calculations have been finished (after the calculation loop is completed). Without knowing much about what the calculation is, why the DB access needs to be done, and how the data in the DB is affected, I can't give a good answer on how to split the processing up. Phil
pbraun wrote:
Does the DB really need to be accessed each time through the loop?
Yes, it does. It accesses the DB seeking for some different info each time. About doing all the DB access at once, I can see what do u mean, but does it really make a difference? I can split the loop into 2 loops(with the same lenght). the first one is to access the DB and the other to do the calculations.
pbraun wrote:
Do the calculations and store them in a manner that is efficient for later DB access once the calculations have been finished
Actually, my calculations depends on the values retrieved from the DB, that is, I have to access the DB first, get some values, do calculations, have result and then next loop. Also, the data in DB is never affected. This process only retrieves records. I can simplfy the process as following:
for (int i=0; i<NUM ; ++i){ "SELECT * FROM TABLE WHERE CONDITION"; // condition differs at each loop[i] while (reader->Read()){ //reader->GetValue; //do calculation depending on (GetValue) } reader->>Close(); } -
pbraun wrote:
Does the DB really need to be accessed each time through the loop?
Yes, it does. It accesses the DB seeking for some different info each time. About doing all the DB access at once, I can see what do u mean, but does it really make a difference? I can split the loop into 2 loops(with the same lenght). the first one is to access the DB and the other to do the calculations.
pbraun wrote:
Do the calculations and store them in a manner that is efficient for later DB access once the calculations have been finished
Actually, my calculations depends on the values retrieved from the DB, that is, I have to access the DB first, get some values, do calculations, have result and then next loop. Also, the data in DB is never affected. This process only retrieves records. I can simplfy the process as following:
for (int i=0; i<NUM ; ++i){ "SELECT * FROM TABLE WHERE CONDITION"; // condition differs at each loop[i] while (reader->Read()){ //reader->GetValue; //do calculation depending on (GetValue) } reader->>Close(); }Hmmm. From that description, it seems that it would be better to send one query to the DB that retrieves all of the necessary data from the DB, then do the processing (in your case, calculations). It is my opinion, that the constant queries to the DB is costing you the time. Of course, you will then have the problem of memory management if your number of data becomes sizable. In that case, you may want to complicate your algorithm by reading in up to a predetermined number of records, doing some of the calculations, then reading in the next batch of records, and so on until finished. This will minimize the DB access. Phil
-
Hmmm. From that description, it seems that it would be better to send one query to the DB that retrieves all of the necessary data from the DB, then do the processing (in your case, calculations). It is my opinion, that the constant queries to the DB is costing you the time. Of course, you will then have the problem of memory management if your number of data becomes sizable. In that case, you may want to complicate your algorithm by reading in up to a predetermined number of records, doing some of the calculations, then reading in the next batch of records, and so on until finished. This will minimize the DB access. Phil
pbraun wrote:
it seems that it would be better to send one query to the DB that retrieves all of the necessary data from the DB, then do the processing (in your case, calculations).
I'm afraid that it is not possible in my case, my condition is already complex enough at each loop :((, also it depends on some data outside the loop ...
pbraun wrote:
the constant queries to the DB is costing you the time
Yes, the queries are costing much of the time! I would like to ask also does the .NET framework do anything behind the scene when a long operation is in run? I have encountered a warning when "another" long operation was running:
The CLR has been unable to transition from COM context 0x1a2008 to COM context 0x1a2178 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment (STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.Thanks for your help Phil, I really appreciate it :-D