How NOT to optimize a database!
-
The sad thing is that it is one column in a 300 pages specification. The guy who do the specifications often do not have the technical expertise to know that a 16 bit integer can store 32000 values in 2 bytes. The programmer usually does, but the programmer is part of the chain. Every programmer is confronted to this problem at times. Do I raise the issue? If you raise the issue it will waste a few hours and you might end up finishing your development late. Also If you raise issues a bit too often you're quickly categorized as annoying git who can't follow orders. This is not the best way to progress in a company. I am not sure if this is true in India but this is true in the countries I worked in. If you shut up and follow orders, you won't get blamed and if you're on time you'll perhaps be promoted. The blame goes on the guy who did the specs despite the fact that the programmer is the best person to notice and fix the issue. To me this is the waterfall drama.
Interesting points!
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
Pascal Ganaye wrote:
The guy who do the specifications often do not have the technical expertise to know that a 16 bit integer can store 32000 values in 2 bytes
Ahem. 65,536 values.
Somebody in an online forum wrote:
INTJs never really joke. They make a point. The joke is just a gift wrapper.
He probably kept aside the sign bit to specify gender. :-D
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
http://articles.economictimes.indiatimes.com/2011-10-23/news/30313183_1_diwali-rush-western-railway-berths[^] TLDR : The IT company that designed the online reservation system for India's western railways used a 3-character field storing numbers to hold the reservation token (001 to 999) on a per-train basis. This limit was never ever hit until last week when (due to Diwali, an Indian festival) enough people made online bookings that the database crashed and with it the reservation system. Funny side-bit : 3 characters (even non-Unicode) is still larger than a short int (memory wise). So whoever did this pseudo-optimization was terribly ignorant.
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
From the US, I think the response would be they can fit 999 people on a train? :-D On the other side of the situation. The guy in charge of the India unique ID project gave a talk at the foundation where I work. That is a mind boggling huge project, and seem to be doing good planning to prevent these kinds of issues.
Curvature of the Mind now with 3D
-
I'm sure lots of members are biting their tongues on this one.
cheers, Chris Maunder The Code Project | Co-founder Microsoft C++ MVP
Oh, you have no idea. I dare not post a single story about the outside "help" we have right now.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak -
I'm sure lots of members are biting their tongues on this one.
cheers, Chris Maunder The Code Project | Co-founder Microsoft C++ MVP
Whistles tunelessly and saunters slowly away.................
Henry Minute Do not read medical books! You could die of a misprint. - Mark Twain Girl: (staring) "Why do you need an icy cucumber?" “I want to report a fraud. The government is lying to us all.” I wouldn't let CG touch my Abacus! When you're wrestling a gorilla, you don't stop when you're tired, you stop when the gorilla is.
-
Chris Meech wrote:
Today, the moment 999 was breached, the program developed a bug
No, the bug was already there, guys.
cheers, Chris Maunder The Code Project | Co-founder Microsoft C++ MVP
Unless they created true Artificial Intelligence... Or is that called Virtual Intelligence? :D
It's an OO world.
public class Naerling : Lazy<Person>{}
-
From the article, "We could not foresee that e-tickets would ever reach 1,000. This number was beyond all our estimates when the software was created. Today, the moment 999 was breached, the program developed a bug and starting behaving abnormally. ....." :doh: :doh: :doh: When was the software created? In 666. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
There are cases where the opposite is also true... A table that will probably never get beyond a couple of 100's of records, but has a bigint as key :rolleyes:
It's an OO world.
public class Naerling : Lazy<Person>{}
-
The sad thing is that it is one column in a 300 pages specification. The guy who do the specifications often do not have the technical expertise to know that a 16 bit integer can store 32000 values in 2 bytes. The programmer usually does, but the programmer is part of the chain. Every programmer is confronted to this problem at times. Do I raise the issue? If you raise the issue it will waste a few hours and you might end up finishing your development late. Also If you raise issues a bit too often you're quickly categorized as annoying git who can't follow orders. This is not the best way to progress in a company. I am not sure if this is true in India but this is true in the countries I worked in. If you shut up and follow orders, you won't get blamed and if you're on time you'll perhaps be promoted. The blame goes on the guy who did the specs despite the fact that the programmer is the best person to notice and fix the issue. To me this is the waterfall drama.
That is exactly how our database recently got some tables with ten columns of which seven were primary key (among which a varchar(50)). The tables were closely connected, but no foreign key was made anywhere. EVERY table (out of approximately 10) had the same redundant data stored along some other data unique to that table. All in all a true DB horror. The guy who made it is supposed to be our 'new' SQL expert, but simply followed the specs... :(( Luckily another programmer noticed and raised the issue to me. That's when I did not only raise the issue, but I raised hell as well! :-O Needless to say, the tables are being re-designed :-\
It's an OO world.
public class Naerling : Lazy<Person>{}
-
Pascal Ganaye wrote:
The guy who do the specifications often do not have the technical expertise to know that a 16 bit integer can store 32000 values in 2 bytes
Ahem. 65,536 values.
Somebody in an online forum wrote:
INTJs never really joke. They make a point. The joke is just a gift wrapper.
Lol, you don't want to go negative!
-
From the article, "We could not foresee that e-tickets would ever reach 1,000. This number was beyond all our estimates when the software was created. Today, the moment 999 was breached, the program developed a bug and starting behaving abnormally. ....." :doh: :doh: :doh: When was the software created? In 666. :)
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra] posting about Crystal Reports here is like discussing gay marriage on a catholic church’s website.[Nishant Sivakumar]
You have to understand the work culture in India. You as a programmer go into a meeting to discuss the Reservation System. The manager tells you not just what the requirements are but also how exactly it must be designed. If you don't follow his instructions to the letter (this assumes you actually have the ability to think for yourself, which is a rare commodity in Indian IT graduates) but deviate even slightly from what you are instructed to do, you will be forced to re-do it the manager's way. Any suggestions from you to improve usability will be turned down. Just go to the online reservation system for Indian Railways. To get from Chennai to Thiruvananthapuram (get Nishant to say it aloud so you know how to pronounce it :laugh: ), you have to know the station codes for both stations, and the train numbers of the various trains between the two stations. You have to backtrack from the reservations screen to a different screen to get the station code, to yet another screen to get all the train numbers which you must write down on paper so that you can input it when needed. The woes go on. We may use computers in India but we will make them as difficult as a face-to-face encounter at the ticket counter! Yes, at the ticket counter, they want you write the train numbers on the ticket rquest form too and you will have no computers to look them up!
-
There are cases where the opposite is also true... A table that will probably never get beyond a couple of 100's of records, but has a bigint as key :rolleyes:
It's an OO world.
public class Naerling : Lazy<Person>{}
The company I worked for in Silicon Valley never reached beyond 1,200 customers (including one-time customers who never came back), 400 items in its product line. It was a $1.2 billion company because they shipped millions of chips (just a few kind) costing a few dollars each. A Kraut came in as VP of IT and recommended they should install SAP. The company could have been run on a network of PCs except for the security concerns. It was on a virus-proof, hacker-proof IBM AS/400. The idiots reported the loss of $20 million in unusable software in their annual report. They could not implement the Manufacturing side of SAP as it was not designed for semiconductor manufacturing. The Kraut left within 3 months of arrival to become Senior VP of IT at the company he came from so he was never there to see the fruits of his recommendation. The company went down the drain and is a mere shadow of its old self now.
-
Every bit counted when your memory was limited to 4K and disk sizes were 5 MB. That gave us the Y2K problem. With 4GB of main memory and terabyte disks aplenty, there is NO reason at all today for this kind of design.
-
Pascal Ganaye wrote:
The guy who do the specifications often do not have the technical expertise to know that a 16 bit integer can store 32000 values in 2 bytes
Ahem. 65,536 values.
Somebody in an online forum wrote:
INTJs never really joke. They make a point. The joke is just a gift wrapper.
-
http://articles.economictimes.indiatimes.com/2011-10-23/news/30313183_1_diwali-rush-western-railway-berths[^] TLDR : The IT company that designed the online reservation system for India's western railways used a 3-character field storing numbers to hold the reservation token (001 to 999) on a per-train basis. This limit was never ever hit until last week when (due to Diwali, an Indian festival) enough people made online bookings that the database crashed and with it the reservation system. Funny side-bit : 3 characters (even non-Unicode) is still larger than a short int (memory wise). So whoever did this pseudo-optimization was terribly ignorant.
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
The company I worked for in Silicon Valley never reached beyond 1,200 customers (including one-time customers who never came back), 400 items in its product line. It was a $1.2 billion company because they shipped millions of chips (just a few kind) costing a few dollars each. A Kraut came in as VP of IT and recommended they should install SAP. The company could have been run on a network of PCs except for the security concerns. It was on a virus-proof, hacker-proof IBM AS/400. The idiots reported the loss of $20 million in unusable software in their annual report. They could not implement the Manufacturing side of SAP as it was not designed for semiconductor manufacturing. The Kraut left within 3 months of arrival to become Senior VP of IT at the company he came from so he was never there to see the fruits of his recommendation. The company went down the drain and is a mere shadow of its old self now.
..And the guy went back to working for SAP? (I always wondered how they sell that piece of scheisse! Good marketting, good lobbying, hopeless software, unrealistic costs... Oh, don't get me started on SAP!! lol)
-
http://articles.economictimes.indiatimes.com/2011-10-23/news/30313183_1_diwali-rush-western-railway-berths[^] TLDR : The IT company that designed the online reservation system for India's western railways used a 3-character field storing numbers to hold the reservation token (001 to 999) on a per-train basis. This limit was never ever hit until last week when (due to Diwali, an Indian festival) enough people made online bookings that the database crashed and with it the reservation system. Funny side-bit : 3 characters (even non-Unicode) is still larger than a short int (memory wise). So whoever did this pseudo-optimization was terribly ignorant.
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
From the US, I think the response would be they can fit 999 people on a train? :-D On the other side of the situation. The guy in charge of the India unique ID project gave a talk at the foundation where I work. That is a mind boggling huge project, and seem to be doing good planning to prevent these kinds of issues.
Curvature of the Mind now with 3D
-
In India, online shopping is still limited to a minority of upper-middle and upper-class folks. Recently though the masses have started using credit cards online, so that is a factor although the whole thing is still rather stupid.
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
Another case of Famous. Last Words. When I was programming databases (actually flat files) back in '76, I encountered a date field that needed to be stored. I asked my manager whether I should store a 2 digit or 4 digit year. I was concerned about the upcoming millennium. He said, "Don't worry, by that time, the software will be replaced." Famous. Last. Words. I have had a few other times where managers would tell me limits that would not be exceeded and in the end, I learned to ignore them and build something that had excess capacity.
Psychosis at 10 Film at 11 Those who do not remember the past, are doomed to repeat it. Those who do not remember the past, cannot build upon it.
-
The sad thing is that it is one column in a 300 pages specification. The guy who do the specifications often do not have the technical expertise to know that a 16 bit integer can store 32000 values in 2 bytes. The programmer usually does, but the programmer is part of the chain. Every programmer is confronted to this problem at times. Do I raise the issue? If you raise the issue it will waste a few hours and you might end up finishing your development late. Also If you raise issues a bit too often you're quickly categorized as annoying git who can't follow orders. This is not the best way to progress in a company. I am not sure if this is true in India but this is true in the countries I worked in. If you shut up and follow orders, you won't get blamed and if you're on time you'll perhaps be promoted. The blame goes on the guy who did the specs despite the fact that the programmer is the best person to notice and fix the issue. To me this is the waterfall drama.
Pascal Ganaye wrote:
Do I raise the issue?
The answer is always yes. If the developer doesn't recognize the issue, that's one thing. (Of course, in this example, if they didn't recognize it, they should be fired. Many other times, it's not until you've done a little development that the "something doesn't feel right" notion crystallizes). The correct approach is for the developer to raise the issue, pointing out the flaws in the current approach. If he gets people to understand and accept those flaws (or not accept them and change development), then he's done his job. Anything short of that, he's failed. Should the developer fix it? That's a second question. It's o.k., from a business perspecitve, to weigh the development time versus the benefit of "doing it right" and decide that the bad scenario is so unlikely as to make the fix not worth it. We all do that. The analysis which needs to be done is (likelihood of bad thing happening) * (cost of bad thing happening) > (cost of fix), understanding that "cost" is not strictly monetary. (i.e., the cost of negative publicity on your company, getting a bad reputation, etc). In our example here, it's hard to believe that analysis was done. Certainly doesn't seem as if the cost of fix was too great, but I (of course) don't know their development.
-
There are cases where the opposite is also true... A table that will probably never get beyond a couple of 100's of records, but has a bigint as key :rolleyes:
It's an OO world.
public class Naerling : Lazy<Person>{}
That isn't as silly as it sounds - for a couple of reasons: 1) If the table is reasonably dynamic and the Key is defined as an AutoNumber, its value might get out of the int range in finite time, even if there were never more than a few hundred records in the table simultaneously. 2) If you maintain the conventions that your Primary Key is always a bigint, you always know how to declare linking fields in other tables without going back to check.
