Accessing file from a remote location
-
Hello All, I am writing an application to get some zipped files from a remote location [over WAN but within the domain]. The file data is huge [around 1.5 GB]. I need to unzip the contents of the file to a temporary location and then read an unzipped database to get required information. Currently what I do is this: 1. I have written the code using FindFirstFile/FindNextFile. 2. Download the files to a temporary location. 3. Unzip the contents of the file. 4. Read the extracted database file. 5. Display the result. But this process is slow and communicating over the network is slow as well. I understand that the download speed will be a major factor in getting the files faster, but is there any other tweak that I can follow to make this work in a better way? P.S. I cannot do this on a on-demand basis as the datastructure in the zipped file doesn't support this and is not likely to be changed.
You talk about Being HUMAN. I have it in my name AnsHUMAN
-
Hello All, I am writing an application to get some zipped files from a remote location [over WAN but within the domain]. The file data is huge [around 1.5 GB]. I need to unzip the contents of the file to a temporary location and then read an unzipped database to get required information. Currently what I do is this: 1. I have written the code using FindFirstFile/FindNextFile. 2. Download the files to a temporary location. 3. Unzip the contents of the file. 4. Read the extracted database file. 5. Display the result. But this process is slow and communicating over the network is slow as well. I understand that the download speed will be a major factor in getting the files faster, but is there any other tweak that I can follow to make this work in a better way? P.S. I cannot do this on a on-demand basis as the datastructure in the zipped file doesn't support this and is not likely to be changed.
You talk about Being HUMAN. I have it in my name AnsHUMAN
How about unzipping "on the fly", so as you receive the file from the network, start unzipping it right away getting along as it downloads?
> The problem with computers is that they do what you tell them to do and not what you want them to do. < > If it doesn't matter, it's antimatter.<
-
Hello All, I am writing an application to get some zipped files from a remote location [over WAN but within the domain]. The file data is huge [around 1.5 GB]. I need to unzip the contents of the file to a temporary location and then read an unzipped database to get required information. Currently what I do is this: 1. I have written the code using FindFirstFile/FindNextFile. 2. Download the files to a temporary location. 3. Unzip the contents of the file. 4. Read the extracted database file. 5. Display the result. But this process is slow and communicating over the network is slow as well. I understand that the download speed will be a major factor in getting the files faster, but is there any other tweak that I can follow to make this work in a better way? P.S. I cannot do this on a on-demand basis as the datastructure in the zipped file doesn't support this and is not likely to be changed.
You talk about Being HUMAN. I have it in my name AnsHUMAN
Is there more than one zip file? Do you've to get all the zip files and then extract them to a single large file (as in a large file was split into several small ones by a compression utility)?
"Real men drive manual transmission" - Rajesh.
-
Is there more than one zip file? Do you've to get all the zip files and then extract them to a single large file (as in a large file was split into several small ones by a compression utility)?
"Real men drive manual transmission" - Rajesh.
Affirmative! There is more than one zip file I need to extract but not to a single large file. All zip's have to be downloaded and unzipped individually.
You talk about Being HUMAN. I have it in my name AnsHUMAN
-
Affirmative! There is more than one zip file I need to extract but not to a single large file. All zip's have to be downloaded and unzipped individually.
You talk about Being HUMAN. I have it in my name AnsHUMAN
So you can use something like the classic producer-consumer design. A thread keeps downloading zip files and another thread will simultaneously extract the downloaded zip files (and perhaps another thread will process the extracted files, depending on how fast all these things happen).
"Real men drive manual transmission" - Rajesh.
-
So you can use something like the classic producer-consumer design. A thread keeps downloading zip files and another thread will simultaneously extract the downloaded zip files (and perhaps another thread will process the extracted files, depending on how fast all these things happen).
"Real men drive manual transmission" - Rajesh.
Rajesh R Subramanian wrote:
producer-consumer design
Can you provide some good links for the above Design pattern with Code Samples?
-
Rajesh R Subramanian wrote:
producer-consumer design
Can you provide some good links for the above Design pattern with Code Samples?
pix_programmer wrote:
Can you provide some good links for the above Design pattern with Code Samples?
Producer/Consumer[^] http://en.wikipedia.org/wiki/Producer-consumer_problem[^] http://developer.amd.com/documentation/articles/pages/125200687.aspx[^] PS: Your fundamentals on multi-threading must be strong to work on this. Because it's very easy to make a mistake otherwise.
"Real men drive manual transmission" - Rajesh.
-
Rajesh R Subramanian wrote:
producer-consumer design
Can you provide some good links for the above Design pattern with Code Samples?
I think the producer consumer example is the way to go, but maybe a version of that using asynchronous
workerswould help the most. What I have in mind would limit the threaded complexity to a minimum while still achieving very good performance. If you are using Qt then my answer would be slightly different but in short I would do like this Assuming I understood you correct and you do not have access to c++11 or concurrency library features likefutureorasynch run., I am also assuming that it is OK that the zip files are unzipped in a non-deterministic order. 1. A thread, maybe the main thread, tells a worker to start downloading one or several zip files. 2. As soon as a zip file is downloaded it is handed over to another worker that is unzipping the file at the wanted location 3. When a zip file is downloaded and unzipped then this is communicated back to the original calling thread 4. Communication between "participants" is done solely through message queues. No shared memory except the queue, that way data sharing and thread synchronization issues are minimal. 5. The "design pattern" I have in mind for this is the active object. There is plenty of material if you google for it. My favourite is Herb Sutters take on it. I wrote my version of active object, which should be easy to use, maybe after making minimal change to use whatever thread library you use. Another example., although more of the fire and forget nature than your scenario., is the asynchronous logger g2log that I recently wrote about. Check in the code how the messages are passed from the g2logworker to it's thread internal and you will see how easy it is -
Hello All, I am writing an application to get some zipped files from a remote location [over WAN but within the domain]. The file data is huge [around 1.5 GB]. I need to unzip the contents of the file to a temporary location and then read an unzipped database to get required information. Currently what I do is this: 1. I have written the code using FindFirstFile/FindNextFile. 2. Download the files to a temporary location. 3. Unzip the contents of the file. 4. Read the extracted database file. 5. Display the result. But this process is slow and communicating over the network is slow as well. I understand that the download speed will be a major factor in getting the files faster, but is there any other tweak that I can follow to make this work in a better way? P.S. I cannot do this on a on-demand basis as the datastructure in the zipped file doesn't support this and is not likely to be changed.
You talk about Being HUMAN. I have it in my name AnsHUMAN
Sounds like you need a client/server approach. Since a majority of your time is spent downloading, and only for the sake of opening the file, leave the file on the server and simply send the server a request to unzip the file and extract the necessary information from it.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous