Are there good discussions or books in regards to making/designing large database applications ?
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
I haven't heard of any, but you might find that different RDBMS vendors handle parallelization of large queries differently. I know how to search millions of rows in parallel on SQL Server in T-SQL, including how to configure the database system and the hardware it runs on (important!). I'd be hard pressed to tell you how using Oracle or Postgre. I don't even know if they have "distributed partitioned views" or what they would call them. My point is you might need to look at books that target a particular flavor of database engine.
Check out my IoT graphics library here: https://honeythecodewitch.com/gfx And my IoT UI/User Experience library here: https://honeythecodewitch.com/uix
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
Any book I've read would be dated by 20 years. But, I can give some points from real world experience... For a SQL/RDBMS * Do not normalize past level 3 for most things. It's not worth the performance hit. * Do not use old school Hungarian notation like
tblUsersfor names. * Learn what an index is and how to properly use it (hint, they help with reads). * Learn when to use a flattened/denormalized table (hint, when speed is a must or for a reporting db/table). * If you're in MySQL, learn the different storage types. * Learn how to use diagrams well. * For all things pure, make sure you get a proper handle foreign key constraints to help with data integrity. * Learn when or when not to use a compound key (hint, many to many). * Speaking of, learn when to use a many to many relationship (hint, clickety). * Learn the pros and cons of building a self-referential table (hint, when building a hierarchy of data). * Lean when to use views (hint, permissions and data access restrictions, read only denormalization). * Understand why concepts like code-first DB design is only purported by those who don't understand proper DB design. :) * Understand that by their design relational DBs are slower than non-relational ones. * Learn to use triggers to automate some things that would otherwise be tedious. * And for all things pure again, get a grasp on transactions and connection pools. * Learn to use SQL profiling tools, these will be your BFF. There may be a few more points I didn't think of of the top of my head, but mastering those will put you ahead of most peeps. For a NoSQL * Learn when to use one. If you need to do semi-realtime anything, NoSQL is awesome. So, stuff like logging is perfect for it. * They make a great reporting DB if you want to use this rather than a flattened table in your relational DB above. * They don't use schemas (for the most part) so the learning curve is a breeze compared to SQL. * Do not use them for mission critical data where it's ok to run a bit slower when data integrity is paramount. Don't follow the buzzword/hype train. NoSQL is awesome but there's a time and place for both relational and non-relational DBs. NoSQL is super fast and that's its main benefit, but that's not always the only concern.Jeremy Falcon
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
And probably the most important, learn to be become very, very familiar with your SQL profiling tools.
Jeremy Falcon
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
FWIW, since you mentioned UI/UX specifically and dealing with large data sets, reporting and datagrids will likely be things you will need. Pay the one-time fee for a suite of quality data-centric libraries that are built to handle large data...pagination/asynch ops/aggregates/etc. The suite we use is fantastic and well worth the expense...libs work in windows and web development. I won't 'name names' unless requested, but I believe they are a sponsor here. Also, they have tons of tutorials and demos.
"Go forth into the source" - Neal Morse "Hope is contagious"
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
for me it's like this, on a form that shows a list from a table, if this list can be longer than a certain amount of rows, I don't allow fetching the rows unless the user has entered one or more filters than I can use in the where clause, otherwise I just fetch them all
-
Any book I've read would be dated by 20 years. But, I can give some points from real world experience... For a SQL/RDBMS * Do not normalize past level 3 for most things. It's not worth the performance hit. * Do not use old school Hungarian notation like
tblUsersfor names. * Learn what an index is and how to properly use it (hint, they help with reads). * Learn when to use a flattened/denormalized table (hint, when speed is a must or for a reporting db/table). * If you're in MySQL, learn the different storage types. * Learn how to use diagrams well. * For all things pure, make sure you get a proper handle foreign key constraints to help with data integrity. * Learn when or when not to use a compound key (hint, many to many). * Speaking of, learn when to use a many to many relationship (hint, clickety). * Learn the pros and cons of building a self-referential table (hint, when building a hierarchy of data). * Lean when to use views (hint, permissions and data access restrictions, read only denormalization). * Understand why concepts like code-first DB design is only purported by those who don't understand proper DB design. :) * Understand that by their design relational DBs are slower than non-relational ones. * Learn to use triggers to automate some things that would otherwise be tedious. * And for all things pure again, get a grasp on transactions and connection pools. * Learn to use SQL profiling tools, these will be your BFF. There may be a few more points I didn't think of of the top of my head, but mastering those will put you ahead of most peeps. For a NoSQL * Learn when to use one. If you need to do semi-realtime anything, NoSQL is awesome. So, stuff like logging is perfect for it. * They make a great reporting DB if you want to use this rather than a flattened table in your relational DB above. * They don't use schemas (for the most part) so the learning curve is a breeze compared to SQL. * Do not use them for mission critical data where it's ok to run a bit slower when data integrity is paramount. Don't follow the buzzword/hype train. NoSQL is awesome but there's a time and place for both relational and non-relational DBs. NoSQL is super fast and that's its main benefit, but that's not always the only concern.Jeremy Falcon
Side note on Triggers: Know WHEN to use triggers. They are a performance hit. It may be negligible, so always load test.
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
Maximilien wrote:
books ... with large databases ?
I thought about that question maybe as long as 20 years ago. Even then it wasn't feasible. Much less so today. Consider how does one start from scratch to build Netflix? Or Amazon? What about a log aggregator. Are the problems there really going to translate into general purpose enterprise solutions? And is one book going to cover all of it? Or even consider these days is a single persistent data store going to usable for the entire enterprise? Can you cover 15 different data stores in one book? And all of that doesn't cover that a person that does understand that must be able to write about it and be willing to spend the time writing a book on it. And perhaps be surprised at how little money they are going to get for writing that book even if it does become a best seller (as a technology book.) Myself I have probably 100 books right now that cover various technologies. And I don't even do UIs. Further I have gotten rid of other books because they were just old. I will say that as general enterprise book that I have read I found the following book informative. I would also say however that it tends towards the high level and as a survey of current (relatively speaking) technologies relevant to handling back end services. But it is intended to cover larger systems. https://www.amazon.com/Building-Microservices-Sam-Newman-ebook/dp/B09B5L4NVT?ref_=ast_sto_dp[^] I will also say that I believe the author makes it clear that attempting to design a large system from day one is unlikely to do anything except lead to larger maintenance costs. But maybe I misread what he said.
-
Any book I've read would be dated by 20 years. But, I can give some points from real world experience... For a SQL/RDBMS * Do not normalize past level 3 for most things. It's not worth the performance hit. * Do not use old school Hungarian notation like
tblUsersfor names. * Learn what an index is and how to properly use it (hint, they help with reads). * Learn when to use a flattened/denormalized table (hint, when speed is a must or for a reporting db/table). * If you're in MySQL, learn the different storage types. * Learn how to use diagrams well. * For all things pure, make sure you get a proper handle foreign key constraints to help with data integrity. * Learn when or when not to use a compound key (hint, many to many). * Speaking of, learn when to use a many to many relationship (hint, clickety). * Learn the pros and cons of building a self-referential table (hint, when building a hierarchy of data). * Lean when to use views (hint, permissions and data access restrictions, read only denormalization). * Understand why concepts like code-first DB design is only purported by those who don't understand proper DB design. :) * Understand that by their design relational DBs are slower than non-relational ones. * Learn to use triggers to automate some things that would otherwise be tedious. * And for all things pure again, get a grasp on transactions and connection pools. * Learn to use SQL profiling tools, these will be your BFF. There may be a few more points I didn't think of of the top of my head, but mastering those will put you ahead of most peeps. For a NoSQL * Learn when to use one. If you need to do semi-realtime anything, NoSQL is awesome. So, stuff like logging is perfect for it. * They make a great reporting DB if you want to use this rather than a flattened table in your relational DB above. * They don't use schemas (for the most part) so the learning curve is a breeze compared to SQL. * Do not use them for mission critical data where it's ok to run a bit slower when data integrity is paramount. Don't follow the buzzword/hype train. NoSQL is awesome but there's a time and place for both relational and non-relational DBs. NoSQL is super fast and that's its main benefit, but that's not always the only concern.Jeremy Falcon
-
Maximilien wrote:
books ... with large databases ?
I thought about that question maybe as long as 20 years ago. Even then it wasn't feasible. Much less so today. Consider how does one start from scratch to build Netflix? Or Amazon? What about a log aggregator. Are the problems there really going to translate into general purpose enterprise solutions? And is one book going to cover all of it? Or even consider these days is a single persistent data store going to usable for the entire enterprise? Can you cover 15 different data stores in one book? And all of that doesn't cover that a person that does understand that must be able to write about it and be willing to spend the time writing a book on it. And perhaps be surprised at how little money they are going to get for writing that book even if it does become a best seller (as a technology book.) Myself I have probably 100 books right now that cover various technologies. And I don't even do UIs. Further I have gotten rid of other books because they were just old. I will say that as general enterprise book that I have read I found the following book informative. I would also say however that it tends towards the high level and as a survey of current (relatively speaking) technologies relevant to handling back end services. But it is intended to cover larger systems. https://www.amazon.com/Building-Microservices-Sam-Newman-ebook/dp/B09B5L4NVT?ref_=ast_sto_dp[^] I will also say that I believe the author makes it clear that attempting to design a large system from day one is unlikely to do anything except lead to larger maintenance costs. But maybe I misread what he said.
-
Especially doing front end applications that interface with large databases ? Dealing with large SQL requests ? Anything related to UI/UX ?
CI/CD = Continuous Impediment/Continuous Despair
I just deleted a lot of what I just typed. I talk to much and make it over complicated according to my wife. :) define "large" database.
Charlie Gilley “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759 Has never been more appropriate.