I had a problem like that but with xml reading and writing. As more bots crawled the site, at a very fast pace, my program would crash, because the file was still open. I had to really fine tune the program about 10 times, with more verification, and to open, read and close as fast as possible, then work in memory only, when done open write and close, then dump the memory and cleanup. I know it's not the same problem, but in principal, the code had to be as efficient as possible.
