How NOT to optimize a database!
-
Oh dear God!!! no way!!! just 999 e-tickets in a country of 1-billion + blinking people, are you serious??? even a non-programmer can tell that's downright stupid. The problem is most of the government IT projects are assigned to lowest bidding organisations whose portfolios are shady at best. IT cannot work with inefficient management, no matter how cheap you get your projects implemented for. End Of. BTW, how did this exactly happen? a regular 16/32-bit int can store more values than 999, does anyone know what they actually used? doesn't sound like an int... these days when memory comes for cheap, disasters like these just get me.
-
Every bit counted when your memory was limited to 4K and disk sizes were 5 MB. That gave us the Y2K problem. With 4GB of main memory and terabyte disks aplenty, there is NO reason at all today for this kind of design.
What? There are plenty of reasons to design for efficiency today. It's my biggest peeve with vendor software right now. They designed it as if memory and storage are infinite, and so the performance sucks because everything is bigger than it needs to be. Network is still a bottleneck, and when you're transferring 1000 rows of data, it matters if the key is 2 bytes or 4. We still need to pay attention to how big we make things.
-
..And the guy went back to working for SAP? (I always wondered how they sell that piece of scheisse! Good marketting, good lobbying, hopeless software, unrealistic costs... Oh, don't get me started on SAP!! lol)
He came from another semiconductor manufacturing company and went back to them. Inflicted SAP on them too! :laugh:
-
What? There are plenty of reasons to design for efficiency today. It's my biggest peeve with vendor software right now. They designed it as if memory and storage are infinite, and so the performance sucks because everything is bigger than it needs to be. Network is still a bottleneck, and when you're transferring 1000 rows of data, it matters if the key is 2 bytes or 4. We still need to pay attention to how big we make things.
Certainly, but there are usually a lot of other optimizations that can be made that will make much more difference. You're almost always better off trying NOT to design something that can be easily broken through usage patterns. The PITA you save may be your own.
-
That isn't as silly as it sounds - for a couple of reasons: 1) If the table is reasonably dynamic and the Key is defined as an AutoNumber, its value might get out of the int range in finite time, even if there were never more than a few hundred records in the table simultaneously. 2) If you maintain the conventions that your Primary Key is always a bigint, you always know how to declare linking fields in other tables without going back to check.
Sure, having a bigint as key is not the end of the world (certainly better than char(3) for storing numerics or (tiny)int where more is needed), but if none of your points apply it certainly deserves no beauty award ;) What I am talking about is a few hundred records ever (simultaneously or not) and every table having defined another key anyway (char, varchar, byte, tinyint, int, bigint, multiple columns etc.). Which might not deserve a beauty award at all ;p
It's an OO world.
public class Naerling : Lazy<Person>{}
-
Certainly, but there are usually a lot of other optimizations that can be made that will make much more difference. You're almost always better off trying NOT to design something that can be easily broken through usage patterns. The PITA you save may be your own.
Other optimizations exist, but making things as small as possible is often overlooked. People believe what you said - that other optimizations are better. I think this is backwards thinking. In my 25 years of doing this stuff, optimizing the size of your objects has fallen out of favor, but it still has the same performance impact that it has always had, and we see it in the results. I'm willing to go out on a limb and say, probably, most of the performance issues we face with modern software are rooted in this problem. Sure, you can take something that's too big and make the performance even worse, but if you don't start with an efficient object, you've shot yourself in the foot before you even started the race.
-
The spec is " to hold the reservation token (001 to 999) on a per-train basis", reservations per train, not tickets per country if read correctly.
aha! its 999 e-reservations per train considering that not all people book tickets online which would mean in the worst case scenario if you have 999 e-bookings and lets say even 1 non-electronic booking the system would still fail. Coz as i understand electronic reservation system is the same as the one used by staff over the counters. Weird. Also one ticket means >= 1 person (not sure what relevance that has to my statement :) ).
-
The spec is " to hold the reservation token (001 to 999) on a per-train basis", reservations per train, not tickets per country if read correctly.
In retrospect though, this is the first time I have come across the Indian railway reservation system crashing which probably could have been avoided by a software check which could have just rejected 1000th reservation. Short-sightedness on the developers' part or over-confidence? who knows....
-
Other optimizations exist, but making things as small as possible is often overlooked. People believe what you said - that other optimizations are better. I think this is backwards thinking. In my 25 years of doing this stuff, optimizing the size of your objects has fallen out of favor, but it still has the same performance impact that it has always had, and we see it in the results. I'm willing to go out on a limb and say, probably, most of the performance issues we face with modern software are rooted in this problem. Sure, you can take something that's too big and make the performance even worse, but if you don't start with an efficient object, you've shot yourself in the foot before you even started the race.
Perhaps, but I think there is a distinction to be made in WHY they are too big. One can minimize the size in ways that are more or less flexible. Obviously they chose one that was less rather then more.
-
http://articles.economictimes.indiatimes.com/2011-10-23/news/30313183_1_diwali-rush-western-railway-berths[^] TLDR : The IT company that designed the online reservation system for India's western railways used a 3-character field storing numbers to hold the reservation token (001 to 999) on a per-train basis. This limit was never ever hit until last week when (due to Diwali, an Indian festival) enough people made online bookings that the database crashed and with it the reservation system. Funny side-bit : 3 characters (even non-Unicode) is still larger than a short int (memory wise). So whoever did this pseudo-optimization was terribly ignorant.
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
The strange thing is that the 3 char field uses 3 bytes and only allows up to 999. Depending on the platform, a 2 byte int would allow at least 32767 entries.
rjemp wrote:
The strange thing is that the 3 char field uses 3 bytes and only allows up to 999. Depending on the platform, a 2 byte int would allow at least 32767 entries.
You probably missed this but the original post mentions this very thing. :rolleyes:
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
rjemp wrote:
The strange thing is that the 3 char field uses 3 bytes and only allows up to 999. Depending on the platform, a 2 byte int would allow at least 32767 entries.
You probably missed this but the original post mentions this very thing. :rolleyes:
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
If every bit helps, they could have saved 8 for every record recorded by using smallint and this would have allowed more than 32 times the 1000 limit. They weren't saving bits, they were showing ignorance. (Per another poster, the ignorance of the manager who can't code.)
-
Every bit counted when your memory was limited to 4K and disk sizes were 5 MB. That gave us the Y2K problem. With 4GB of main memory and terabyte disks aplenty, there is NO reason at all today for this kind of design.
-
rjemp wrote:
The strange thing is that the 3 char field uses 3 bytes and only allows up to 999. Depending on the platform, a 2 byte int would allow at least 32767 entries.
You probably missed this but the original post mentions this very thing. :rolleyes:
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
Heh :-)
Regards, Nish
My technology blog: voidnish.wordpress.com Part 2 in my WinRT/C++ series : Visual C++ and WinRT/Metro - Databinding Basics
-
You have to understand the work culture in India. You as a programmer go into a meeting to discuss the Reservation System. The manager tells you not just what the requirements are but also how exactly it must be designed. If you don't follow his instructions to the letter (this assumes you actually have the ability to think for yourself, which is a rare commodity in Indian IT graduates) but deviate even slightly from what you are instructed to do, you will be forced to re-do it the manager's way. Any suggestions from you to improve usability will be turned down. Just go to the online reservation system for Indian Railways. To get from Chennai to Thiruvananthapuram (get Nishant to say it aloud so you know how to pronounce it :laugh: ), you have to know the station codes for both stations, and the train numbers of the various trains between the two stations. You have to backtrack from the reservations screen to a different screen to get the station code, to yet another screen to get all the train numbers which you must write down on paper so that you can input it when needed. The woes go on. We may use computers in India but we will make them as difficult as a face-to-face encounter at the ticket counter! Yes, at the ticket counter, they want you write the train numbers on the ticket rquest form too and you will have no computers to look them up!
Err.. I think you went to some other website, because I have been using irctc website for quite sometime and I needn't do any of such things you mentioned. Infact, its a simple wizard kind of interface where you type in source & destination city, select from a list of available trains and book it - simple. Anyway, coming to topic, I think the fault lies with lack of foresightedness. One rule of thumb which I use personally is to assume Moore's Law on expected data usage -- because software generally end up using more memory than initially planned and secondly, it happen to last more than designed for.
-
Chris Meech wrote:
Today, the moment 999 was breached, the program developed a bug
No, the bug was already there, guys.
cheers, Chris Maunder The Code Project | Co-founder Microsoft C++ MVP
Yes, it just got visible. Funny old cartoons on bugs: Software Bug's Life[^]
Regards, Jwalant Natvarlal Soneji
-
The Y2K problem came from mainframes which completely pre-date micro-computers and had nothing to do with memory or disk size. Actually, they were storing 2 digit numbers in 16 bits of storage. Don't tell me it had anything to do with memory or storage.
It had every thing to do with disk or main memory sizes. Unless of course you think that programmers were stupid in the 1960s through the 1990s and Intelligent Life dawned on Earth only after the PC arrived. The largest IBM 370/168 shipped usually with 4 MB of main memory. Anything more was tremendously expensive, though it could go up to 16MB, if memory serves me right. Disk sizes were 350MB each on the IBM 3350 disk drives and a big bank could afford 20 of those drives. So programmers counted every bit and byte when they created records for storing on the disk drives.
-
If every bit helps, they could have saved 8 for every record recorded by using smallint and this would have allowed more than 32 times the 1000 limit. They weren't saving bits, they were showing ignorance. (Per another poster, the ignorance of the manager who can't code.)
The world didn't use C to write business programs but COBOL, which you would probably say is evidence of their stupidity. Programmers used Binary Coded Decimal (BCD) to represent numbers. They could have used COMPUTATIONAL-3 which would be pure binary representation but conversion from binary to BCD and back was expensive in terms of instruction execution. Computers existed before you were born as did beer and sex but I am willing to grant PC programmers the right to claim they invented all three. ;P
