Looking for ideas ................
-
I am looking for suggestions on how to do something described below: - a web service that will be called by a web application about 200,000 times in a 15-minutes interval. - Each call will generate a string of data that needed to be recorded to the central database. Someone suggested to me: - the data generated by each call should temporary be sitting in memory. - A timer should wake up periodically to push the data to a queue, before the memory is too full. - The queue will write the data to the central database at a daily scheduled time. I would like to get some idea of what technique / technology that I can use to implement those suggestion., and to make the web service efficient. I am new to .NET and MS technology. I hope that someone can just point me to some directions and will be very much appreicate. Thanks. A Newcomer
-
I am looking for suggestions on how to do something described below: - a web service that will be called by a web application about 200,000 times in a 15-minutes interval. - Each call will generate a string of data that needed to be recorded to the central database. Someone suggested to me: - the data generated by each call should temporary be sitting in memory. - A timer should wake up periodically to push the data to a queue, before the memory is too full. - The queue will write the data to the central database at a daily scheduled time. I would like to get some idea of what technique / technology that I can use to implement those suggestion., and to make the web service efficient. I am new to .NET and MS technology. I hope that someone can just point me to some directions and will be very much appreicate. Thanks. A Newcomer
200,000 times in 15 minute intervals? Are you sure you really need to use a webservice for that? That is 200,000 HTTP requests and responses. Plus all the SOAP packaging, serialisation of the data, deserialisation, authenticating... At the least I know one IIS box is not going to come anywhere near handling that. You could optimise the data factory to hell and back but it is the IIS and webservice part that will tank. Why does it have to be 200,000 transactions? Can you not send more data with fewer transactions? I just think that no amount of in-memory storage, "queuing of memory" etc. ideas will help. The database could handle the writes easily enough, but not IIS. Maybe you can explain what you are trying to do and we can come up with a better architecture and set of technologies for this :)
Paul Watson
Bluegrass
Cape Town, South AfricaRobert Edward Caldecott wrote: My father-in-law calls yer man bits "weasels"
-
200,000 times in 15 minute intervals? Are you sure you really need to use a webservice for that? That is 200,000 HTTP requests and responses. Plus all the SOAP packaging, serialisation of the data, deserialisation, authenticating... At the least I know one IIS box is not going to come anywhere near handling that. You could optimise the data factory to hell and back but it is the IIS and webservice part that will tank. Why does it have to be 200,000 transactions? Can you not send more data with fewer transactions? I just think that no amount of in-memory storage, "queuing of memory" etc. ideas will help. The database could handle the writes easily enough, but not IIS. Maybe you can explain what you are trying to do and we can come up with a better architecture and set of technologies for this :)
Paul Watson
Bluegrass
Cape Town, South AfricaRobert Edward Caldecott wrote: My father-in-law calls yer man bits "weasels"
-
Actually 200,000/15 minutes is the total hits in the peak time of the day for a farm of 6-7 servers. We are a brokeage firm that offers on-line quotes to clients. We would like to log the usage. A Newcomer
farm of 6-7 servers That is better. Also I misunderstood you originally, I thought you meant every 15 minutes you get 200,000 requests. Not 200,000 per 15 minutes :) We are a brokeage firm that offers on-line quotes to clients. Ok. Well if I think of anything I will post.
Paul Watson
Bluegrass
Cape Town, South AfricaRobert Edward Caldecott wrote: My father-in-law calls yer man bits "weasels"
-
I am looking for suggestions on how to do something described below: - a web service that will be called by a web application about 200,000 times in a 15-minutes interval. - Each call will generate a string of data that needed to be recorded to the central database. Someone suggested to me: - the data generated by each call should temporary be sitting in memory. - A timer should wake up periodically to push the data to a queue, before the memory is too full. - The queue will write the data to the central database at a daily scheduled time. I would like to get some idea of what technique / technology that I can use to implement those suggestion., and to make the web service efficient. I am new to .NET and MS technology. I hope that someone can just point me to some directions and will be very much appreicate. Thanks. A Newcomer
-
I am looking for suggestions on how to do something described below: - a web service that will be called by a web application about 200,000 times in a 15-minutes interval. - Each call will generate a string of data that needed to be recorded to the central database. Someone suggested to me: - the data generated by each call should temporary be sitting in memory. - A timer should wake up periodically to push the data to a queue, before the memory is too full. - The queue will write the data to the central database at a daily scheduled time. I would like to get some idea of what technique / technology that I can use to implement those suggestion., and to make the web service efficient. I am new to .NET and MS technology. I hope that someone can just point me to some directions and will be very much appreicate. Thanks. A Newcomer
I feel you can use the MSMQ of Microsoft.. to log into it.. (From the webservice)..Now write a bunch of message handlers..(exes /windows service) that keep pulling out messages from the queue and write into DB.. This might work.
Visit me:
http://www2.domaindlx.com/earavi/
When you know something.. its meant to share with others :-) for otherwise that knowledge has no worth:-)
mail me:
aravinthan@rediffmail.com