Continued Values Collection/List/Dictionary [modified]
-
Som Shekhar wrote:
I really appreciate your efforts.
Nice to see someone who's biting into a subject, instead of just asking for code :)
Som Shekhar wrote:
But the question is to know if I am missing something fundamentally? If there is an implementation already present? Like we have Hashtables, List, Dictionaries for various purposes. Is there any other tool that I missed which can handle such a case?
Not that I know. Yes, we got generic lists that can take all kinds of data, and we got observable lists that give you a notification if anything changes. But no list that's specialized in doing an incremental save. I think that most of us would cache the result, storing redundant values. It's a waste of memory, I know, but we often make these kind of trades. If you got some spare CPU-time, then it might make sense to add this optimization. You'll lose a bit of speed reconstructing the data at a particular index, but return you'd have some extra memory. The guys who work with Windows Mobile might have more experience with this, as they have less resources and actually need to think about using them effectively. On my desktop, I don't mind wasting a megabyte or so, if it means that I can spend my time on more critical issues.
Som Shekhar wrote:
Or, may be there could be a need to develop such a list which could only record changes, do all the calculations internally, fast enough to match those of dictionary/indexed methods.
Again, recalculating the data will (logically seen) cost more processor-time than just reading it. Then again, the time that it takes might be neglectable, and it may also be true that you win back a fair amount of memory. That would depend on the amount of data, and the amount of 'holes' that one has to move through to get the 'last known values' for the columns on that particular index. At the start of this thread, I would have advised against that on the assumption that there's not much to gain. I'm not that sure anymore. The only way to get a definite answer is by building a prototype and measuring the results. Therein lies another consideration; would it be worth to spend the time on building such a prototype?
I are Troll :suss:
Eddy Vluggen wrote:
Nice to see someone who's biting into a subject, instead of just asking for code
Coding is easy. Concepts are difficult to grasp. If you know the direction, you can reach anywhere. If you only know the target,god save you.
Eddy Vluggen wrote:
It's a waste of memory
Memory is not really an issue. I am building an application for a bigger use and hence using all kind of hardware resources onto it. I can tell my clients to use better hardware. This means a good speed CPU and a good amount of RAM. Hence I really don't mind 1-2 MB extra here. I am already looping to create lookup-ready directory. Hence that is already covered. As I mentioned, The trouble comes when multiple of such calculations happen together. I am currently working on multi-threading of different instances. Atleast to save some more time. Let me give you a link of another problem that i posted. You would see the use of such a datatype there. http://www.codeproject.com/Messages/3304858/What-will-be-the-height-of-fluid-columns-in-a-vari.aspx[] In this problem, calculation of fluid height is needed. There are multiple fluid columns and many such tubes. With drag and drop functionality :( Usually working with already implemented concepts is always better. Consider using a dictionary vs. implemented List with key.
Eddy Vluggen wrote:
would it be worth to spend the time on building such a prototype?
You would be surprised that i have come across such situation more than 4-5 times already while designing my applications. I usually work on disconnected database system and speed is a primary concern in loading and saving data. I initially worked with datatables which worked fine when my application was young. As it grew older, datatables are damn slow. I moved to dictionary. So far, they are fine. Even today, i experience a max lag of 0.5-0.6 sec on a drag drop operation which isn't too much to worry about. By multi-threading, i hope to reduce it to around 0.1-0.2 which should be manageable. But it is good to keep up with concepts. Usually a parallel solutions does wonders and tha
-
Eddy Vluggen wrote:
Nice to see someone who's biting into a subject, instead of just asking for code
Coding is easy. Concepts are difficult to grasp. If you know the direction, you can reach anywhere. If you only know the target,god save you.
Eddy Vluggen wrote:
It's a waste of memory
Memory is not really an issue. I am building an application for a bigger use and hence using all kind of hardware resources onto it. I can tell my clients to use better hardware. This means a good speed CPU and a good amount of RAM. Hence I really don't mind 1-2 MB extra here. I am already looping to create lookup-ready directory. Hence that is already covered. As I mentioned, The trouble comes when multiple of such calculations happen together. I am currently working on multi-threading of different instances. Atleast to save some more time. Let me give you a link of another problem that i posted. You would see the use of such a datatype there. http://www.codeproject.com/Messages/3304858/What-will-be-the-height-of-fluid-columns-in-a-vari.aspx[] In this problem, calculation of fluid height is needed. There are multiple fluid columns and many such tubes. With drag and drop functionality :( Usually working with already implemented concepts is always better. Consider using a dictionary vs. implemented List with key.
Eddy Vluggen wrote:
would it be worth to spend the time on building such a prototype?
You would be surprised that i have come across such situation more than 4-5 times already while designing my applications. I usually work on disconnected database system and speed is a primary concern in loading and saving data. I initially worked with datatables which worked fine when my application was young. As it grew older, datatables are damn slow. I moved to dictionary. So far, they are fine. Even today, i experience a max lag of 0.5-0.6 sec on a drag drop operation which isn't too much to worry about. By multi-threading, i hope to reduce it to around 0.1-0.2 which should be manageable. But it is good to keep up with concepts. Usually a parallel solutions does wonders and tha
Som Shekhar wrote:
Coding is easy.
I'm looking at a buglist right now which tells me that it's not as easy as English.
Som Shekhar wrote:
As I mentioned, The trouble comes when multiple of such calculations happen together. I am currently working on multi-threading of different instances. Atleast to save some more time.
Have you seen the article[^] on the
AForge.Parallel.For-class? It might help in building a prototype to measure against :)Som Shekhar wrote:
In this problem, calculation of fluid height is needed. There are multiple fluid columns and many such tubes. With drag and drop functionality
True, but it would also make an impressive interface :cool:
Som Shekhar wrote:
Usually working with already implemented concepts is always better. Consider using a dictionary vs. implemented List with key.
I'd try to mirror the concept of a database in-memory; creating a list of the records, and the equivalent of an index. IQueryable[^] springs into mind.
Som Shekhar wrote:
You would be surprised that i have come across such situation more than 4-5 times already while designing my applications. I usually work on disconnected database system and speed is a primary concern in loading and saving data. I initially worked with datatables which worked fine when my application was young. As it grew older, datatables are damn slow. I moved to dictionary. So far, they are fine. Even today, i experience a max lag of 0.5-0.6 sec on a drag drop operation which isn't too much to worry about.
This post[^] confirms that although databases manipulate data very fast, your results are faster.
-
Som Shekhar wrote:
Coding is easy.
I'm looking at a buglist right now which tells me that it's not as easy as English.
Som Shekhar wrote:
As I mentioned, The trouble comes when multiple of such calculations happen together. I am currently working on multi-threading of different instances. Atleast to save some more time.
Have you seen the article[^] on the
AForge.Parallel.For-class? It might help in building a prototype to measure against :)Som Shekhar wrote:
In this problem, calculation of fluid height is needed. There are multiple fluid columns and many such tubes. With drag and drop functionality
True, but it would also make an impressive interface :cool:
Som Shekhar wrote:
Usually working with already implemented concepts is always better. Consider using a dictionary vs. implemented List with key.
I'd try to mirror the concept of a database in-memory; creating a list of the records, and the equivalent of an index. IQueryable[^] springs into mind.
Som Shekhar wrote:
You would be surprised that i have come across such situation more than 4-5 times already while designing my applications. I usually work on disconnected database system and speed is a primary concern in loading and saving data. I initially worked with datatables which worked fine when my application was young. As it grew older, datatables are damn slow. I moved to dictionary. So far, they are fine. Even today, i experience a max lag of 0.5-0.6 sec on a drag drop operation which isn't too much to worry about.
This post[^] confirms that although databases manipulate data very fast, your results are faster.
Parallel.For looks promising. Will dig into it. Currently, since i have already implemented multithreading, I guess no need to implement that for now.
Eddy Vluggen wrote:
True, but it would also make an impressive interface
Oh you bet. These days, looks may not be everything but that is what sells the first. I guess, that is it for now... I gotta be happy with multithreading for now. Since no other implementation already exist in this area. It was great having some meaningful conversations. :)
-
Som Shekhar wrote: "I have added my comments to the question itself. There are some more areas to look at." Hi Som, Have edited the first code example to make the internal list of "current values to replace with if incoming item# is null" private and not static. Will be able to review your comments later tonight (I live at GMT +7 by the way) to try to understand what you mean by "retrieving them back" : isn't the test example I show in the code ... where the keys and list values are being read out in a foreach loop ... and printed to the console ... an example of retrieving back the values ? If I want the 3rd. item in the List in the dictionary t1 which is accessed by the key #4 : and I access it via : t1[4][2] Isn't that retrieving ? Namaste, Bill
"Many : not conversant with mathematical studies, imagine that because it [the Analytical Engine] is to give results in numerical notation, its processes must consequently be arithmetical, numerical, rather than algebraical and analytical. This is an error. The engine can arrange and combine numerical quantities as if they were letters or any other general symbols; and it fact it might bring out its results in algebraical notation, were provisions made accordingly." Ada, Countess Lovelace, 1844
I guess you sent me a message. I didn't see it here though.. What I was looking for was a framework implementation to sort my issue. I have done something on my own already and it works currently. Just a speed improvement was needed. I have already worked on multithreading and some decent work is already done... so only thing better would be a root level work.
-
Parallel.For looks promising. Will dig into it. Currently, since i have already implemented multithreading, I guess no need to implement that for now.
Eddy Vluggen wrote:
True, but it would also make an impressive interface
Oh you bet. These days, looks may not be everything but that is what sells the first. I guess, that is it for now... I gotta be happy with multithreading for now. Since no other implementation already exist in this area. It was great having some meaningful conversations. :)
Som Shekhar wrote:
Oh you bet. These days, looks may not be everything but that is what sells the first.
Sad, but true.
Som Shekhar wrote:
I guess, that is it for now... I gotta be happy with multithreading for now.
You're dividing your workload over multiple CPU's, there's not much room for improvement there. If you get unhappy in the future, try Brahma[^], that would give you the option to offload some work from the CPU to the GPU, abusing the graphics-card.
Som Shekhar wrote:
It was great having some meaningful conversations.
Yup, engaging a conversation is simply more fun than posting an answer. Good luck with your venture :)
I are Troll :suss:
-
Som Shekhar wrote:
Oh you bet. These days, looks may not be everything but that is what sells the first.
Sad, but true.
Som Shekhar wrote:
I guess, that is it for now... I gotta be happy with multithreading for now.
You're dividing your workload over multiple CPU's, there's not much room for improvement there. If you get unhappy in the future, try Brahma[^], that would give you the option to offload some work from the CPU to the GPU, abusing the graphics-card.
Som Shekhar wrote:
It was great having some meaningful conversations.
Yup, engaging a conversation is simply more fun than posting an answer. Good luck with your venture :)
I are Troll :suss:
Brahma looks interesting!!! Now, that we are talking of multi threading, why is it that we need to code for multi-threading? If you look at any operating system that can support multiple processors, it automatically distributes work onto different cores. Can't there be a framework which can do the same without the need to code differently? If a user defines a program priority as "High" or "RealTime" it only increases the share of thread time to current process. But no change in threading... Am I missing something?
-
Brahma looks interesting!!! Now, that we are talking of multi threading, why is it that we need to code for multi-threading? If you look at any operating system that can support multiple processors, it automatically distributes work onto different cores. Can't there be a framework which can do the same without the need to code differently? If a user defines a program priority as "High" or "RealTime" it only increases the share of thread time to current process. But no change in threading... Am I missing something?
Som Shekhar wrote:
Am I missing something?
The perversion to launch a new thread from a Visual Basic 6 application, I hope :)
Som Shekhar wrote:
If a user defines a program priority as "High" or "RealTime" it only increases the share of thread time to current process. But no change in threading...
..and "realtime" isn't really realtime, but just the name of the highest level of priority.
Som Shekhar wrote:
If you look at any operating system that can support multiple processors, it automatically distributes work onto different cores.
Though it feels that way, it's an illusion. A program is made up of a logical set of commands/instructions that get executed one after another. That's reflected in our applications; we expect that the second instruction won't be execute before the first instruction. A short example;
10 A$ = "Hello"
20 B$ = "World"
30 PRINT A$ + " " + B$These three lines of code should be considered atomic, meaning that you don't want to distribute them over 2 different people to interpret. This is a task that can't be divided. One processor had access to it's own cache, and it's memory. Windows was created and started to fake multitasking. Applications would run (to line 20 in our example), get thrown into the deep-freeze, another application would be defrozen and run, ad infinitum. Do that very fast, and it seems to become a fluid movement. Threads we're already there; it was preferred to launch your own thread instead of spawning a new process if you needed to do some additional tasks. Using a thread would cost less resources and they behaved like an additional proces, owned by some other (main)-thread. Fibers were introduced also, but those never gained popularity. You wanted this processing to happen in "some other place" than the thread that ran your interface. Every Windows-application has a method that's called "WndProc", which Windows calls now and then to inform your application of mouse-movements that have occurred, or that certain parts of the form need to be repainted. Let's extend our example application;
10 REM Example :)
20 REM
30 WndProc:
40 MSG = GWBASIC_INTEROP.GETMESSAGE()
50 IF (MSG = WM_UIT) THEN
60 GOTO THE_END
70 END IF
80 IF (MSG = WM_PAINT) THEN
90 GOSUB SAY_HI
100 END IF
110 GOTO WndProc
120 SAY_HI:
130 FOR X = 1 TO 100 -
Som Shekhar wrote:
Am I missing something?
The perversion to launch a new thread from a Visual Basic 6 application, I hope :)
Som Shekhar wrote:
If a user defines a program priority as "High" or "RealTime" it only increases the share of thread time to current process. But no change in threading...
..and "realtime" isn't really realtime, but just the name of the highest level of priority.
Som Shekhar wrote:
If you look at any operating system that can support multiple processors, it automatically distributes work onto different cores.
Though it feels that way, it's an illusion. A program is made up of a logical set of commands/instructions that get executed one after another. That's reflected in our applications; we expect that the second instruction won't be execute before the first instruction. A short example;
10 A$ = "Hello"
20 B$ = "World"
30 PRINT A$ + " " + B$These three lines of code should be considered atomic, meaning that you don't want to distribute them over 2 different people to interpret. This is a task that can't be divided. One processor had access to it's own cache, and it's memory. Windows was created and started to fake multitasking. Applications would run (to line 20 in our example), get thrown into the deep-freeze, another application would be defrozen and run, ad infinitum. Do that very fast, and it seems to become a fluid movement. Threads we're already there; it was preferred to launch your own thread instead of spawning a new process if you needed to do some additional tasks. Using a thread would cost less resources and they behaved like an additional proces, owned by some other (main)-thread. Fibers were introduced also, but those never gained popularity. You wanted this processing to happen in "some other place" than the thread that ran your interface. Every Windows-application has a method that's called "WndProc", which Windows calls now and then to inform your application of mouse-movements that have occurred, or that certain parts of the form need to be repainted. Let's extend our example application;
10 REM Example :)
20 REM
30 WndProc:
40 MSG = GWBASIC_INTEROP.GETMESSAGE()
50 IF (MSG = WM_UIT) THEN
60 GOTO THE_END
70 END IF
80 IF (MSG = WM_PAINT) THEN
90 GOSUB SAY_HI
100 END IF
110 GOTO WndProc
120 SAY_HI:
130 FOR X = 1 TO 100Topic is long over, I am just loving the conversations :-D
Eddy Vluggen wrote:
and "realtime" isn't really realtime, but just the name of the highest level of priority
Yes, I know that... I guess you would know that Microsoft has told that one shouldn't use "RealTime" in their applications as it will freeze the OS. I wonder if they created a check to freeze the system if someone used "realtime" instead of letting the processor do the job. Well, you are right about the process and threading. I agree fully when it comes to the concept of threading vs processes. My question was a little different. I know that two processes are resource heave and thread do just the jobs quite well. Let me try to suggest my concept here. Lets say we got two classes "Car" and "Bike". Car has its own methods and so does the bike. We create two objects "Car1" and "Bike1" If these two are there in an application, all internal methods could be handled through a new thread and thus they will always be thread safe. Even two objects "Car1" and "Car2" will always be thread safe. Instead of programmer creating such new threads and their completion events for each of the methods, the framework could automatically run them on new threads. Is it that I have a plan for a new programming language? Am I talking weird?
-
Topic is long over, I am just loving the conversations :-D
Eddy Vluggen wrote:
and "realtime" isn't really realtime, but just the name of the highest level of priority
Yes, I know that... I guess you would know that Microsoft has told that one shouldn't use "RealTime" in their applications as it will freeze the OS. I wonder if they created a check to freeze the system if someone used "realtime" instead of letting the processor do the job. Well, you are right about the process and threading. I agree fully when it comes to the concept of threading vs processes. My question was a little different. I know that two processes are resource heave and thread do just the jobs quite well. Let me try to suggest my concept here. Lets say we got two classes "Car" and "Bike". Car has its own methods and so does the bike. We create two objects "Car1" and "Bike1" If these two are there in an application, all internal methods could be handled through a new thread and thus they will always be thread safe. Even two objects "Car1" and "Car2" will always be thread safe. Instead of programmer creating such new threads and their completion events for each of the methods, the framework could automatically run them on new threads. Is it that I have a plan for a new programming language? Am I talking weird?
Som Shekhar wrote:
Topic is long over, I am just loving the conversations
Ditto, but we'd better move to the soapbox, or email :)
Som Shekhar wrote:
Let me try to suggest my concept here. Lets say we got two classes "Car" and "Bike". Car has its own methods and so does the bike. We create two objects "Car1" and "Bike1" If these two are there in an application, all internal methods could be handled through a new thread and thus they will always be thread safe. Even two objects "Car1" and "Car2" will always be thread safe. Instead of programmer creating such new threads and their completion events for each of the methods, the framework could automatically run them on new threads. Is it that I have a plan for a new programming language? Am I talking weird?
Not at all, sounds like a convenient way to distribute the load. One way to do so would be by instantiating a BackgroundWorker and passing the
Biketo the RunWorkerAsync[^]-method. You'd only need locks where the objects need to share data. Those objects still need to be 'invoked' from somewhere, and that somewhere is most likely going to be the mainthread. It doesn't make much sense to create an Async-version for every method or class, since threads still cost performance. Creating a thread to changeForm.Visibleis going to be rather inefficient. If theBikeis a webserver-kind of class, then yes, it's the correct pattern. If theBikeis aDataGridViewColumn, then it might be wiser to keep the code rather short and simple anyway. If you need a long-running process happening in such a place, then move it to a backgroundworker and have them signalling status. There's two other interesting places to visit;- Rx extensions[^], since collections are another example where easy multhithreading makes sense
- This cheatsheet[
-
Som Shekhar wrote:
Topic is long over, I am just loving the conversations
Ditto, but we'd better move to the soapbox, or email :)
Som Shekhar wrote:
Let me try to suggest my concept here. Lets say we got two classes "Car" and "Bike". Car has its own methods and so does the bike. We create two objects "Car1" and "Bike1" If these two are there in an application, all internal methods could be handled through a new thread and thus they will always be thread safe. Even two objects "Car1" and "Car2" will always be thread safe. Instead of programmer creating such new threads and their completion events for each of the methods, the framework could automatically run them on new threads. Is it that I have a plan for a new programming language? Am I talking weird?
Not at all, sounds like a convenient way to distribute the load. One way to do so would be by instantiating a BackgroundWorker and passing the
Biketo the RunWorkerAsync[^]-method. You'd only need locks where the objects need to share data. Those objects still need to be 'invoked' from somewhere, and that somewhere is most likely going to be the mainthread. It doesn't make much sense to create an Async-version for every method or class, since threads still cost performance. Creating a thread to changeForm.Visibleis going to be rather inefficient. If theBikeis a webserver-kind of class, then yes, it's the correct pattern. If theBikeis aDataGridViewColumn, then it might be wiser to keep the code rather short and simple anyway. If you need a long-running process happening in such a place, then move it to a backgroundworker and have them signalling status. There's two other interesting places to visit;- Rx extensions[^], since collections are another example where easy multhithreading makes sense
- This cheatsheet[
I have mailed you this time instead of replying here. Given my id too.