performance or accuracy?
-
Today i came across a scene where we had a discussion on this topic. we have some work on the database side where batch files data is insert (row count is several thousand per operations )/ some text files generated by windows services / flag status maintain based on the data insert in the database per operation and many other small operations happen during this process. obviously correctness of the operation is must but at the same time performance of the operation came in to the picture. some teammates says that as there are lot of process happens during per operations it will take time to complete things while some believe that there can be something done to reduce the time. Client is okay even if that takes time as for him accuracy is more important rather than time taken. How to manage such scenarios ? after some level we can not think of optimization on query as all the things which is written is necessary. does anyone face such things ? how do you manage things to improve the performance?
Ravi Khoda
-
You lost me already at the subject line. If it's not accurate, it's buggy, and therefore wrong.
Wrong is evil and must be defeated. - Jeff Ello[^]
-
Today i came across a scene where we had a discussion on this topic. we have some work on the database side where batch files data is insert (row count is several thousand per operations )/ some text files generated by windows services / flag status maintain based on the data insert in the database per operation and many other small operations happen during this process. obviously correctness of the operation is must but at the same time performance of the operation came in to the picture. some teammates says that as there are lot of process happens during per operations it will take time to complete things while some believe that there can be something done to reduce the time. Client is okay even if that takes time as for him accuracy is more important rather than time taken. How to manage such scenarios ? after some level we can not think of optimization on query as all the things which is written is necessary. does anyone face such things ? how do you manage things to improve the performance?
Ravi Khoda
Accuracy is the only thing that matters until the client starts worrying about performance. In your case that seems to be clearly not the case so why are you worrying about it? For any algorithm there is a minimum time that it will take, and while you never know what that time is or manage to get particularly close to it, optimisation attempts do often hit a wall where any further increase in performance is not worth the effort. If you've done the quick wins (batching data at an appropriate size, indexing your tables on the right columns, using fast text writing) then you are almost certainly against that wall.
-
Today i came across a scene where we had a discussion on this topic. we have some work on the database side where batch files data is insert (row count is several thousand per operations )/ some text files generated by windows services / flag status maintain based on the data insert in the database per operation and many other small operations happen during this process. obviously correctness of the operation is must but at the same time performance of the operation came in to the picture. some teammates says that as there are lot of process happens during per operations it will take time to complete things while some believe that there can be something done to reduce the time. Client is okay even if that takes time as for him accuracy is more important rather than time taken. How to manage such scenarios ? after some level we can not think of optimization on query as all the things which is written is necessary. does anyone face such things ? how do you manage things to improve the performance?
Ravi Khoda
Probably something you already considered but which can have an enormous impact in loading your data: transactions. Grouping as much data together before committing can increase loading performance to a factor 100 (as usual, experiment with size of transaction for best result in your situation). Timing of the process is also extremely important: you might experience slow performance due to locking issues. To me, accuracy seems the most important criterium (inaccurate = wrong).
-
Today i came across a scene where we had a discussion on this topic. we have some work on the database side where batch files data is insert (row count is several thousand per operations )/ some text files generated by windows services / flag status maintain based on the data insert in the database per operation and many other small operations happen during this process. obviously correctness of the operation is must but at the same time performance of the operation came in to the picture. some teammates says that as there are lot of process happens during per operations it will take time to complete things while some believe that there can be something done to reduce the time. Client is okay even if that takes time as for him accuracy is more important rather than time taken. How to manage such scenarios ? after some level we can not think of optimization on query as all the things which is written is necessary. does anyone face such things ? how do you manage things to improve the performance?
Ravi Khoda
-
Whenever in doubt the customer is always right. The only exception to that is if you do not want to be paid for your efforts. :doh:
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopes1- The client is right about WHAT they want. Client is usually never right about HOW to do it or how much it will cost. 2- Accuracy over performance. Accuracy is usually a black or white determination (not always, but usually). Performance can be tweaked and dialed up in degrees over time and iterations. To paraphrase...first make it work, then make it work fast, then make it elegant.
-
1- The client is right about WHAT they want. Client is usually never right about HOW to do it or how much it will cost. 2- Accuracy over performance. Accuracy is usually a black or white determination (not always, but usually). Performance can be tweaked and dialed up in degrees over time and iterations. To paraphrase...first make it work, then make it work fast, then make it elegant.
jeffreystacks wrote:
first make it work
The 3 biggest lies: 1 - I love you. 2 - I will respect you in the morning. 3 - Just make it work so we can release it and we will go back and refactor it later. :-D
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopes -
Daniel Lieberwirth (BrainInBlack) wrote:
Not always, sometimes you have to convince the client to do the right thing.
In this case the client is doing the right thing. They wart accuracy over performance. How can you argue with that? :~
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopesDepends. If numerical calculations are involved, there are technical limits to the accuracy you can achieve, and trying to push down the tolerance limits below that won't achieve anything but provide a false sense of accuracy that simply isn't there.
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto) Point in case: http://www.infoq.com/news/2014/02/apple_gotofail_lessons[^]
-
Depends. If numerical calculations are involved, there are technical limits to the accuracy you can achieve, and trying to push down the tolerance limits below that won't achieve anything but provide a false sense of accuracy that simply isn't there.
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto) Point in case: http://www.infoq.com/news/2014/02/apple_gotofail_lessons[^]
Ravi Khoda wrote:
we have some work on the database side where batch files data is insert (row count is several thousand per operations )/ some text files generated by windows services / flag status maintain based on the data insert in the database per operation and many other small operations happen during this process. obviously correctness of the operation is must but at the same time performance of the operation came in to the picture. some teammates says that as there are lot of process happens during per operations it will take time to complete things while some believe that there can be something done to reduce the time. Client is okay even if that takes time as for him accuracy is more important rather than time taken. How to manage such scenarios ? after some level we can not think of optimization on query as all the things which is written is necessary. does anyone face such things ? how do you manage things to improve the performance?
Let's not lose track of the task at hand. It is database update, not calculations to put a space probe on Uranus. :-D
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopes -
Ravi Khoda wrote:
we have some work on the database side where batch files data is insert (row count is several thousand per operations )/ some text files generated by windows services / flag status maintain based on the data insert in the database per operation and many other small operations happen during this process. obviously correctness of the operation is must but at the same time performance of the operation came in to the picture. some teammates says that as there are lot of process happens during per operations it will take time to complete things while some believe that there can be something done to reduce the time. Client is okay even if that takes time as for him accuracy is more important rather than time taken. How to manage such scenarios ? after some level we can not think of optimization on query as all the things which is written is necessary. does anyone face such things ? how do you manage things to improve the performance?
Let's not lose track of the task at hand. It is database update, not calculations to put a space probe on Uranus. :-D
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopesA valid point. Then again, the only meanings of accuracy I am aware of don't relate to the only alternatives being data is there or not there! If during a transaction data is lost, then the transaction is not inaccurate, it is erraneous! ;P P.S.: I've just found that "free from errors" is actually one of the possible meanings of accurate[^]. :doh:
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto) Point in case: http://www.infoq.com/news/2014/02/apple_gotofail_lessons[^]
-
A valid point. Then again, the only meanings of accuracy I am aware of don't relate to the only alternatives being data is there or not there! If during a transaction data is lost, then the transaction is not inaccurate, it is erraneous! ;P P.S.: I've just found that "free from errors" is actually one of the possible meanings of accurate[^]. :doh:
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto) Point in case: http://www.infoq.com/news/2014/02/apple_gotofail_lessons[^]
Stefan_Lang wrote:
If during a transaction data is lost, then the transaction is not inaccurate, it is erraneous!
The normal procedure when part of a transaction fails is a roll back of the previous parts of the transaction so there is no vestiges of a partially completed transaction corrupting the database. A transaction should only be committed when all updates complete successfully.
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopes -
Stefan_Lang wrote:
If during a transaction data is lost, then the transaction is not inaccurate, it is erraneous!
The normal procedure when part of a transaction fails is a roll back of the previous parts of the transaction so there is no vestiges of a partially completed transaction corrupting the database. A transaction should only be committed when all updates complete successfully.
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopesThat is a very good point, actually: a database transaction should either work without errors, or not at all. That said, the OP implies that the 'performant' solution he has in mind would not be 'accurate', i. e. contain errors, i. e. needs to be rolled back, achieving nothing at all! Why again did he ask? ;P
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto) Point in case: http://www.infoq.com/news/2014/02/apple_gotofail_lessons[^]
-
That is a very good point, actually: a database transaction should either work without errors, or not at all. That said, the OP implies that the 'performant' solution he has in mind would not be 'accurate', i. e. contain errors, i. e. needs to be rolled back, achieving nothing at all! Why again did he ask? ;P
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto) Point in case: http://www.infoq.com/news/2014/02/apple_gotofail_lessons[^]
Stefan_Lang wrote:
i. e. needs to be rolled back, achieving nothing at all!
Maybe he is thinking about running seperate updates on individual tables rather than using proper transaction processing. That is only speculation on my part. He may have something totally different in mind.
Stefan_Lang wrote:
Why again did he ask?
I don't know. He seems to not like taking the time to insure accuracy of the process. :confused:
The report of my death was an exaggeration - Mark Twain
Simply Elegant Designs JimmyRopes Designs
I'm on-line therefore I am. JimmyRopes