what is the best algorithm for an information retrieval system?
-
hi. I am developing an IS and I was wondering on how I can load the heavy amount of data in a more efficient way. I was asked to also integrate an algorithm but I don't know where to put it and what to use. The program does much more in SQL queries than C# algorithmic codes. If you know what I mean. To explain it further, if I have about 1 million data entries in my database. How can I possibly load them in a jiffy?
-
hi. I am developing an IS and I was wondering on how I can load the heavy amount of data in a more efficient way. I was asked to also integrate an algorithm but I don't know where to put it and what to use. The program does much more in SQL queries than C# algorithmic codes. If you know what I mean. To explain it further, if I have about 1 million data entries in my database. How can I possibly load them in a jiffy?
harcaype wrote:
if I have about 1 million data entries in my database. How can I possibly load them in a jiffy?
Why would you? I doubt you need them all at once.
-
harcaype wrote:
if I have about 1 million data entries in my database. How can I possibly load them in a jiffy?
Why would you? I doubt you need them all at once.
i need them because new entries add up about every second. So when new entries are added up, I have to update/refresh it. And refreshing them again takes awhile to load. Also, I have a quick search option in the program, so if for example I will incorporate an algorithm for that search. Do you know the best to use? Because instead of searching all possibilities, it would just go straight to the search. I shouldn't be placing an algorithm actually, but I was required to do so. :|
-
i need them because new entries add up about every second. So when new entries are added up, I have to update/refresh it. And refreshing them again takes awhile to load. Also, I have a quick search option in the program, so if for example I will incorporate an algorithm for that search. Do you know the best to use? Because instead of searching all possibilities, it would just go straight to the search. I shouldn't be placing an algorithm actually, but I was required to do so. :|
harcaype wrote:
So when new entries are added up, I have to update/refresh it.
There is no way you are displaying over a million rows on the screen simultaneously. At best you are displaying the aggregate result of those rows. In that case you should consider doing whatever calculation you are doing in the database itself. If you do have a million rows available to your user then you might want to consider implementing some sort of paging where the system only gets the rows that are needed for the current display.
harcaype wrote:
Also, I have a quick search option in the program, so if for example I will incorporate an algorithm for that search.
If you are doing this for a million rows, again a paging option is probably the best. You might want to have some sort of predictive fetch if you don't want the application to pause when scrolling. i.e. When you have scrolled two thirds the way through the current list, perform an asynchronous database query to fetch the next page worth of data. By the time the user has scrolled there it should be available.
Recent blog posts: * Event Organisation (Feedback) * LINQ to XML (part 4) * Scottish Developers June Newsletter My Blog
-
harcaype wrote:
So when new entries are added up, I have to update/refresh it.
There is no way you are displaying over a million rows on the screen simultaneously. At best you are displaying the aggregate result of those rows. In that case you should consider doing whatever calculation you are doing in the database itself. If you do have a million rows available to your user then you might want to consider implementing some sort of paging where the system only gets the rows that are needed for the current display.
harcaype wrote:
Also, I have a quick search option in the program, so if for example I will incorporate an algorithm for that search.
If you are doing this for a million rows, again a paging option is probably the best. You might want to have some sort of predictive fetch if you don't want the application to pause when scrolling. i.e. When you have scrolled two thirds the way through the current list, perform an asynchronous database query to fetch the next page worth of data. By the time the user has scrolled there it should be available.
Recent blog posts: * Event Organisation (Feedback) * LINQ to XML (part 4) * Scottish Developers June Newsletter My Blog
i agree with Colin, let database server handle it, but if you insist to use custom retrieval system you must develop data retrieval management component that support memory table (to store your data) in your application and your database server needs to support events on insert, update and delete data and catch that events to your application, this means you must manage some delta handler to synchronize data between your application and database server hope it helps
dhaim programming is a hobby that make some money as side effect :)
-
i agree with Colin, let database server handle it, but if you insist to use custom retrieval system you must develop data retrieval management component that support memory table (to store your data) in your application and your database server needs to support events on insert, update and delete data and catch that events to your application, this means you must manage some delta handler to synchronize data between your application and database server hope it helps
dhaim programming is a hobby that make some money as side effect :)
Mbah Dhaim wrote:
but if you insist to use custom retrieval system
Why are they insisting on it? What is their justification? What benefits does it bring? What is the impact on other applications in the system. (i.e. If your application is taking massive quantities of memory, what's left for other things)
Recent blog posts: * SQL Server Memory Usage * Monitoring change in XML data * Debugging into the .NET Source code My Blog
-
i agree with Colin, let database server handle it, but if you insist to use custom retrieval system you must develop data retrieval management component that support memory table (to store your data) in your application and your database server needs to support events on insert, update and delete data and catch that events to your application, this means you must manage some delta handler to synchronize data between your application and database server hope it helps
dhaim programming is a hobby that make some money as side effect :)
-
i need them because new entries add up about every second. So when new entries are added up, I have to update/refresh it. And refreshing them again takes awhile to load. Also, I have a quick search option in the program, so if for example I will incorporate an algorithm for that search. Do you know the best to use? Because instead of searching all possibilities, it would just go straight to the search. I shouldn't be placing an algorithm actually, but I was required to do so. :|
harcaype wrote:
i need them because new entries add up about every second. So when new entries are added up, I have to update/refresh it. And refreshing them again takes awhile to load.
So why keep the data in two places? Just keep it in the database, so that you only have one place where you need to update it.
harcaype wrote:
Also, I have a quick search option in the program, so if for example I will incorporate an algorithm for that search. Do you know the best to use?
Use the database. Searching the data is exactly what it's built for.
Despite everything, the person most likely to be fooling you next is yourself.
-
Mbah Dhaim wrote:
but if you insist to use custom retrieval system
Why are they insisting on it? What is their justification? What benefits does it bring? What is the impact on other applications in the system. (i.e. If your application is taking massive quantities of memory, what's left for other things)
Recent blog posts: * SQL Server Memory Usage * Monitoring change in XML data * Debugging into the .NET Source code My Blog
i agree with you, to much things to manage :(
dhaim programming is a hobby that make some money as side effect :)