Unpopular opinions: LINQ
-
Now what is the big advantage of linq? It is smart to query things like one queries in SQL. But more it is smart it allows to query in a kind which is tested. Of course we rely on that test the same we rely on the compiler that it compiles correctly ;) Nothig more and nothing less [Edit] Most probably hard to understand what written above :confused:
That testing aspect is definitely one advantage, but I'd just as soon write the "functional" code long-hand and write the *tests* using LINQ to verify them. ;P
Real programmers use butterflies
-
They keep adding ketchup and yellow mustard to my language. It'll be mayonnaise next. :mad:
-
They keep adding ketchup and yellow mustard to my language. It'll be mayonnaise next. :mad:
That's like putting it on steak! C++ i have mixed feelings about. I love the flexibility. Almost every feature has an actual purpose. You'd think that wouldn't be an exceptional thing, but consider C#8 and parameter "deconstruction" and all that other nonsense we don't need. But C++ has gotten so complicated that it's about as bad C# has gotten if not worse now. So even if it is purposeful, it's hell to navigate.
Real programmers use butterflies
-
That's like putting it on steak! C++ i have mixed feelings about. I love the flexibility. Almost every feature has an actual purpose. You'd think that wouldn't be an exceptional thing, but consider C#8 and parameter "deconstruction" and all that other nonsense we don't need. But C++ has gotten so complicated that it's about as bad C# has gotten if not worse now. So even if it is purposeful, it's hell to navigate.
Real programmers use butterflies
honey the codewitch wrote:
C++ i have mixed feelings about. I love the flexibility. Almost every feature has an actual purpose. ... But C++ has gotten so complicated that it's about as bad C# has gotten if not worse now. So even if it is purposeful, it's hell to navigate.
I agree about the complexity. But the nice thing about C++ is that if you don't use it, you don't pay for it.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
honey the codewitch wrote:
C++ i have mixed feelings about. I love the flexibility. Almost every feature has an actual purpose. ... But C++ has gotten so complicated that it's about as bad C# has gotten if not worse now. So even if it is purposeful, it's hell to navigate.
I agree about the complexity. But the nice thing about C++ is that if you don't use it, you don't pay for it.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
Definitely. That's one thing I love. Until developers abuse it with cross code interdependencies, making selective linking useless. Like the ESP-IDF's bluetooth LE stack depending on its bluetooth stack despite them being *entirely* different protocols (BLE was bought from another outfit by bluetooth - it wasn't designed by them) - on an embedded device where every kilobyte of program code space counts. :mad:
Real programmers use butterflies
-
Definitely. That's one thing I love. Until developers abuse it with cross code interdependencies, making selective linking useless. Like the ESP-IDF's bluetooth LE stack depending on its bluetooth stack despite them being *entirely* different protocols (BLE was bought from another outfit by bluetooth - it wasn't designed by them) - on an embedded device where every kilobyte of program code space counts. :mad:
Real programmers use butterflies
honey the codewitch wrote:
Until developers abuse it with cross code interdependencies, making selective linking useless.
As someone said, you can write poorly-crafted code in any language. :sigh:
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
I'm not a fan of LINQ. I love functional programming but .NET's enumerator paradigm is not up to the task. It creates too many objects too quickly to be a real grown up functional language, whose iteration is highly optimized because it's a first class operation. I've benched LINQ against hand written pseudo-functional operations that do the same thing. It was not encouraging. For things that make heavy use of functional computation like parser generators, where your LINQ query might be half a page, it's a Bad Idea(TM) Worse, I think its use has been over encouraged by Microsoft. It makes green developers write even worse code, and makes it harder for a seasoned developer to understand the performance implications of the code they are writing (and I'm not talking about bit twiddling here, I'm talking about figuring out your Big O expression) I tend to avoid its use, preferring - at least in C# - to make my iteration operations explicit and long hand. If .NET had truly optimized iteration paradigm - one that didn't create new objects for every single iteration operation** - i might consider using it. ** yes i understand that LINQ combines multiple operations into a single iteration *sometimes* - in practice it's not often enough to make up for the overhead of enumerators. Now, there's a case where all of the above doesn't matter, and that's PLINQ. Theoretically, for a large enough operation, that can be highly parallelized, the overhead of enumerators suddenly isn't the biggest part of the performance equation. What I mean is it essentially pays for itself. Also, given the issues with synchronization and other cross task communication (is your operation clustered over a network?) enumerators are actually not a bad idea since you can lock behind them or RPC behind them. Contrast that with C++ iterators that are usually lightly wrapped pointer ops and you realize their limitations fast: In order to enable all of the stuff you need to make iteration operations work with each other in parallel you have to wrap every iterator operator anyway, making it as "heavy" as an enumerator in .NET, not counting the general overhead of running managed code. So basically, PLINQ is where LINQ finally covers its costs - where it reaches the point where its advantages outweigh its disadvantages. All of this of course, is one developer's opinion. And some of this doesn't necessarily apply to business software, where performance almost doesn't matter for most scenarios.
Real programmers
-
That testing aspect is definitely one advantage, but I'd just as soon write the "functional" code long-hand and write the *tests* using LINQ to verify them. ;P
Real programmers use butterflies
-
honey the codewitch wrote:
Until developers abuse it with cross code interdependencies, making selective linking useless.
As someone said, you can write poorly-crafted code in any language. :sigh:
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
It's frustrating that this is basically system level code - driver code necessary to use the hardware for the proprietary** SoC it runs on. ** how it works isn't actually a secret, but nor is it easy to dig through a mountain of technical specs written in chinese to write a custom DDK and SDK package for this hardware.
Real programmers use butterflies
-
I mostly agree, except about PLINQ. If LINQ is like shoveling money into a brazier, PLINQ is like hiring a gang of temps to shovel the money in even faster.
Hahaha, that's fair I suppose, but at least it scales out, allowing you throw hardware at the problem since the software sucks as a matter of course. :laugh:
Real programmers use butterflies
-
I'm not a fan of LINQ. I love functional programming but .NET's enumerator paradigm is not up to the task. It creates too many objects too quickly to be a real grown up functional language, whose iteration is highly optimized because it's a first class operation. I've benched LINQ against hand written pseudo-functional operations that do the same thing. It was not encouraging. For things that make heavy use of functional computation like parser generators, where your LINQ query might be half a page, it's a Bad Idea(TM) Worse, I think its use has been over encouraged by Microsoft. It makes green developers write even worse code, and makes it harder for a seasoned developer to understand the performance implications of the code they are writing (and I'm not talking about bit twiddling here, I'm talking about figuring out your Big O expression) I tend to avoid its use, preferring - at least in C# - to make my iteration operations explicit and long hand. If .NET had truly optimized iteration paradigm - one that didn't create new objects for every single iteration operation** - i might consider using it. ** yes i understand that LINQ combines multiple operations into a single iteration *sometimes* - in practice it's not often enough to make up for the overhead of enumerators. Now, there's a case where all of the above doesn't matter, and that's PLINQ. Theoretically, for a large enough operation, that can be highly parallelized, the overhead of enumerators suddenly isn't the biggest part of the performance equation. What I mean is it essentially pays for itself. Also, given the issues with synchronization and other cross task communication (is your operation clustered over a network?) enumerators are actually not a bad idea since you can lock behind them or RPC behind them. Contrast that with C++ iterators that are usually lightly wrapped pointer ops and you realize their limitations fast: In order to enable all of the stuff you need to make iteration operations work with each other in parallel you have to wrap every iterator operator anyway, making it as "heavy" as an enumerator in .NET, not counting the general overhead of running managed code. So basically, PLINQ is where LINQ finally covers its costs - where it reaches the point where its advantages outweigh its disadvantages. All of this of course, is one developer's opinion. And some of this doesn't necessarily apply to business software, where performance almost doesn't matter for most scenarios.
Real programmers
LINQ was engineered by the same person (s)? that engineered inline SQL in FoxPro. That was the part I missed the most after abandoning FP / VFP ... and rejoiced when LINQ arrived. If you write LOB apps, particularly ERP, you will understand. LINQ sucks for those that think "partitioning" a problem is for weenies. They also mangle SQL; assuming the "optimizer" will always sort out their mess. (soap box off) Paradox: sometimes you have to write "more" code to get better performance.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it. ― Confucian Analects: Rules of Confucius about his food
-
I'm not a fan of LINQ. I love functional programming but .NET's enumerator paradigm is not up to the task. It creates too many objects too quickly to be a real grown up functional language, whose iteration is highly optimized because it's a first class operation. I've benched LINQ against hand written pseudo-functional operations that do the same thing. It was not encouraging. For things that make heavy use of functional computation like parser generators, where your LINQ query might be half a page, it's a Bad Idea(TM) Worse, I think its use has been over encouraged by Microsoft. It makes green developers write even worse code, and makes it harder for a seasoned developer to understand the performance implications of the code they are writing (and I'm not talking about bit twiddling here, I'm talking about figuring out your Big O expression) I tend to avoid its use, preferring - at least in C# - to make my iteration operations explicit and long hand. If .NET had truly optimized iteration paradigm - one that didn't create new objects for every single iteration operation** - i might consider using it. ** yes i understand that LINQ combines multiple operations into a single iteration *sometimes* - in practice it's not often enough to make up for the overhead of enumerators. Now, there's a case where all of the above doesn't matter, and that's PLINQ. Theoretically, for a large enough operation, that can be highly parallelized, the overhead of enumerators suddenly isn't the biggest part of the performance equation. What I mean is it essentially pays for itself. Also, given the issues with synchronization and other cross task communication (is your operation clustered over a network?) enumerators are actually not a bad idea since you can lock behind them or RPC behind them. Contrast that with C++ iterators that are usually lightly wrapped pointer ops and you realize their limitations fast: In order to enable all of the stuff you need to make iteration operations work with each other in parallel you have to wrap every iterator operator anyway, making it as "heavy" as an enumerator in .NET, not counting the general overhead of running managed code. So basically, PLINQ is where LINQ finally covers its costs - where it reaches the point where its advantages outweigh its disadvantages. All of this of course, is one developer's opinion. And some of this doesn't necessarily apply to business software, where performance almost doesn't matter for most scenarios.
Real programmers
honey the codewitch wrote:
It creates too many objects too quickly
Can you elaborate? As far as I know you have an extra enumerator per operation, so for example:
foreach (var whatever in whatevers)
{
if (whatever.IsValid)
{
filtered.Add(whatever);
}
}Has one enumerator, while
var filtered = whatevers.Where(x => x.IsValid).ToList();
has two enumerators (the Where will call "the original" enumerator, while the ToList will call the WhereEnumerator). Other than that it's the same except that the LINQ example has an extra anonymous function (which isn't anonymous after compilation) and an extra function call for each iteration, but if the where clause is complicated enough you may do this in example 1 too. That's hardly a performance penalty, but you just won 7 lines of code and made it more readable to boot. The readability further increases when you do stuff like
whatevers.Where(x => x.IsValid)
.Select(x => new { Name = x.Name, Age = x.Age })
.OrderBy(x => x.Name)
.Take(10)
.ToList()at the expense of three extra enumerators. Object instantiation is cheap, or so I've been told. You also missed one, LINQ to Entities (or LINQ to SQL), which is also LINQ, but won't enumerate at all because the entire structure is compiled to an object structure and parsed into SQL. Let's not talk about the performance implications on that one :D For most operations it's not significantly slower though, while it saves a ton of time of SQL development and makes your database connection strong typed meaning less bugs, etc. The extra milliseconds the user is waiting is won back in hours of development time :D
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
-
honey the codewitch wrote:
It creates too many objects too quickly
Can you elaborate? As far as I know you have an extra enumerator per operation, so for example:
foreach (var whatever in whatevers)
{
if (whatever.IsValid)
{
filtered.Add(whatever);
}
}Has one enumerator, while
var filtered = whatevers.Where(x => x.IsValid).ToList();
has two enumerators (the Where will call "the original" enumerator, while the ToList will call the WhereEnumerator). Other than that it's the same except that the LINQ example has an extra anonymous function (which isn't anonymous after compilation) and an extra function call for each iteration, but if the where clause is complicated enough you may do this in example 1 too. That's hardly a performance penalty, but you just won 7 lines of code and made it more readable to boot. The readability further increases when you do stuff like
whatevers.Where(x => x.IsValid)
.Select(x => new { Name = x.Name, Age = x.Age })
.OrderBy(x => x.Name)
.Take(10)
.ToList()at the expense of three extra enumerators. Object instantiation is cheap, or so I've been told. You also missed one, LINQ to Entities (or LINQ to SQL), which is also LINQ, but won't enumerate at all because the entire structure is compiled to an object structure and parsed into SQL. Let's not talk about the performance implications on that one :D For most operations it's not significantly slower though, while it saves a ton of time of SQL development and makes your database connection strong typed meaning less bugs, etc. The extra milliseconds the user is waiting is won back in hours of development time :D
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
Great. Now try using it to generate a LALR(1) table or for that matter, even compute FIRST and FOLLOWS sets. You'll see my issue with LINQ really quickly. Especially if you benchmark it. I guess it all depends on what kind of code you're writing. These days I don't do a lot of bizdev, and I haven't touched a real database in years. Adding, I don't think you're considering all the extra overhead of LINQ not actually combining all the operations that can be combined into a single iteration eg: iterating twice where once would do. It just isn't smart enough. It's also problematic (and this isn't specific to LINQ but more of a general problem with functional programming) to do certain kinds of queries because some queries can be orders of magnitude faster if you're allowed to keep some state around. There's just no facility for that in LINQ. I don't blame LINQ for that, since it's more of a functional programming paradigm issue, but it still keeps me from being able to use it for a lot of what I would like to use functional programming constructs for.
Real programmers use butterflies
-
Great. Now try using it to generate a LALR(1) table or for that matter, even compute FIRST and FOLLOWS sets. You'll see my issue with LINQ really quickly. Especially if you benchmark it. I guess it all depends on what kind of code you're writing. These days I don't do a lot of bizdev, and I haven't touched a real database in years. Adding, I don't think you're considering all the extra overhead of LINQ not actually combining all the operations that can be combined into a single iteration eg: iterating twice where once would do. It just isn't smart enough. It's also problematic (and this isn't specific to LINQ but more of a general problem with functional programming) to do certain kinds of queries because some queries can be orders of magnitude faster if you're allowed to keep some state around. There's just no facility for that in LINQ. I don't blame LINQ for that, since it's more of a functional programming paradigm issue, but it still keeps me from being able to use it for a lot of what I would like to use functional programming constructs for.
Real programmers use butterflies
honey the codewitch wrote:
Adding, I don't think you're considering all the extra overhead of LINQ not actually combining all the operations that can be combined into a single iteration eg: iterating twice where once would do.
It's actually pretty smart about that! I've written a nice article[^] about it *pats own back* :D Anyway, I've got a hell of a lot more than 4 8 MB to spare (like at least a 1024x more I guess) :laugh: For my customers a few extra objects, iterations and even MB's are no issue at all (they don't even know it exists), but my hourly rate is :D
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
-
honey the codewitch wrote:
It creates too many objects too quickly
Can you elaborate? As far as I know you have an extra enumerator per operation, so for example:
foreach (var whatever in whatevers)
{
if (whatever.IsValid)
{
filtered.Add(whatever);
}
}Has one enumerator, while
var filtered = whatevers.Where(x => x.IsValid).ToList();
has two enumerators (the Where will call "the original" enumerator, while the ToList will call the WhereEnumerator). Other than that it's the same except that the LINQ example has an extra anonymous function (which isn't anonymous after compilation) and an extra function call for each iteration, but if the where clause is complicated enough you may do this in example 1 too. That's hardly a performance penalty, but you just won 7 lines of code and made it more readable to boot. The readability further increases when you do stuff like
whatevers.Where(x => x.IsValid)
.Select(x => new { Name = x.Name, Age = x.Age })
.OrderBy(x => x.Name)
.Take(10)
.ToList()at the expense of three extra enumerators. Object instantiation is cheap, or so I've been told. You also missed one, LINQ to Entities (or LINQ to SQL), which is also LINQ, but won't enumerate at all because the entire structure is compiled to an object structure and parsed into SQL. Let's not talk about the performance implications on that one :D For most operations it's not significantly slower though, while it saves a ton of time of SQL development and makes your database connection strong typed meaning less bugs, etc. The extra milliseconds the user is waiting is won back in hours of development time :D
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
-
honey the codewitch wrote:
Adding, I don't think you're considering all the extra overhead of LINQ not actually combining all the operations that can be combined into a single iteration eg: iterating twice where once would do.
It's actually pretty smart about that! I've written a nice article[^] about it *pats own back* :D Anyway, I've got a hell of a lot more than 4 8 MB to spare (like at least a 1024x more I guess) :laugh: For my customers a few extra objects, iterations and even MB's are no issue at all (they don't even know it exists), but my hourly rate is :D
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
Sander Rossel wrote:
For my customers a few extra objects, iterations and even MB's are no issue at all (they don't even know it exists), but my hourly rate is :-D
Yeah I can understand that, but also I'm glad that's not my situation. I like having to cram as much functionality I can into modest hardware.
Real programmers use butterflies
-
I'm not a fan of LINQ. I love functional programming but .NET's enumerator paradigm is not up to the task. It creates too many objects too quickly to be a real grown up functional language, whose iteration is highly optimized because it's a first class operation. I've benched LINQ against hand written pseudo-functional operations that do the same thing. It was not encouraging. For things that make heavy use of functional computation like parser generators, where your LINQ query might be half a page, it's a Bad Idea(TM) Worse, I think its use has been over encouraged by Microsoft. It makes green developers write even worse code, and makes it harder for a seasoned developer to understand the performance implications of the code they are writing (and I'm not talking about bit twiddling here, I'm talking about figuring out your Big O expression) I tend to avoid its use, preferring - at least in C# - to make my iteration operations explicit and long hand. If .NET had truly optimized iteration paradigm - one that didn't create new objects for every single iteration operation** - i might consider using it. ** yes i understand that LINQ combines multiple operations into a single iteration *sometimes* - in practice it's not often enough to make up for the overhead of enumerators. Now, there's a case where all of the above doesn't matter, and that's PLINQ. Theoretically, for a large enough operation, that can be highly parallelized, the overhead of enumerators suddenly isn't the biggest part of the performance equation. What I mean is it essentially pays for itself. Also, given the issues with synchronization and other cross task communication (is your operation clustered over a network?) enumerators are actually not a bad idea since you can lock behind them or RPC behind them. Contrast that with C++ iterators that are usually lightly wrapped pointer ops and you realize their limitations fast: In order to enable all of the stuff you need to make iteration operations work with each other in parallel you have to wrap every iterator operator anyway, making it as "heavy" as an enumerator in .NET, not counting the general overhead of running managed code. So basically, PLINQ is where LINQ finally covers its costs - where it reaches the point where its advantages outweigh its disadvantages. All of this of course, is one developer's opinion. And some of this doesn't necessarily apply to business software, where performance almost doesn't matter for most scenarios.
Real programmers
I don't care overmuch about eventual cost of LINQ, when I use it, it always end up making the code a lot more simple, and the (possible) performance cost of LINQ itself are peanut compared to over work I perform... However I do beg to differ with your assessment with PLINQ. For what I experimented with, and read about it, unless you have many clearly patently slow operation, plink can end up making your code run even more slowly.. It did for me.. :/ It's kind of weird, I admit.. from what I read it's when some data overlap (which is very likely), there might hidden synchronisation happening between thread slowing things down.. :( As a side note, you might be interested by this project! LinQ As Fuck.. err... :rolleyes: ;P I mean [Linq Allocation Free](https://kevinmontrose.com/2018/01/17/linqaf-replacing-linq-and-not-allocating/)!
A new .NET Serializer All in one Menu-Ribbon Bar Taking over the world since 1371!
-
I don't care overmuch about eventual cost of LINQ, when I use it, it always end up making the code a lot more simple, and the (possible) performance cost of LINQ itself are peanut compared to over work I perform... However I do beg to differ with your assessment with PLINQ. For what I experimented with, and read about it, unless you have many clearly patently slow operation, plink can end up making your code run even more slowly.. It did for me.. :/ It's kind of weird, I admit.. from what I read it's when some data overlap (which is very likely), there might hidden synchronisation happening between thread slowing things down.. :( As a side note, you might be interested by this project! LinQ As Fuck.. err... :rolleyes: ;P I mean [Linq Allocation Free](https://kevinmontrose.com/2018/01/17/linqaf-replacing-linq-and-not-allocating/)!
A new .NET Serializer All in one Menu-Ribbon Bar Taking over the world since 1371!
Assuming PLINQ's implementation is not terrible, you're probably incurring locking overhead. It doesn't make sense to try to use any kind of parallelization in the following scenarios a) your problem has interdependent components such that you can't decouple the work done by B from the result of A and C depends on the result of both, so you're elephanted. b) it doesn't do you a heck of a lot of good to query the same source in parallel with itself. It's hard to give you a good example in PLINQ but you want parallel op A to use a different datasource than B. In an RDBMS this principle is easier to understand. If I run a join across two tables, i don't have a lot i can do to make it parallel *unless* each table is on a separate drive ("spindle" in DB parlance) meaning the read operations of table A aren't dependent on waiting for read operations from B since they are different drive controllers working in parallel. The same basic idea would apply to PLINQ If a or b are an issue, you'll probably end up incurring more overhead than you gain in throughput
Real programmers use butterflies
-
> Object instantiation is cheap, or so I've been told. It can be (depends on the object of course), but the problem starts when the garbage collector comes calling
Right? I didn't want to get into it and potentially start an argument over GC arcana, but basically the concept behind a GC isn't so much that you save on allocations, but you pay for them after-the-fact. I recently had a project that needed fast pointer increment allocation like a GC has but I didn't want to pay for the object sweeping so I simply made it so my little heaps could be defined to a fixed size (of which allocations would come out of) and you couldn't delete objects at all. You could clear the entire heap in one sweep, invalidating all the data at once though. Practically free, and the use case was such that it handled everything I needed. GCs aren't all that and a bag of chips. :) But then i'm not telling you anything you don't already know. *hides from @SanderRossel*
Real programmers use butterflies
-
Assuming PLINQ's implementation is not terrible, you're probably incurring locking overhead. It doesn't make sense to try to use any kind of parallelization in the following scenarios a) your problem has interdependent components such that you can't decouple the work done by B from the result of A and C depends on the result of both, so you're elephanted. b) it doesn't do you a heck of a lot of good to query the same source in parallel with itself. It's hard to give you a good example in PLINQ but you want parallel op A to use a different datasource than B. In an RDBMS this principle is easier to understand. If I run a join across two tables, i don't have a lot i can do to make it parallel *unless* each table is on a separate drive ("spindle" in DB parlance) meaning the read operations of table A aren't dependent on waiting for read operations from B since they are different drive controllers working in parallel. The same basic idea would apply to PLINQ If a or b are an issue, you'll probably end up incurring more overhead than you gain in throughput
Real programmers use butterflies
I tried with some code I have from my long past Physics PhD that integrate some equation over time... I have a multidimensional field and each dimension was in its own thread... Mmm... come to think of it now, there is coupling between some variable I think, I wonder if it was the cause of the slow down... no matter.. not sure where this code even is now ^^
A new .NET Serializer All in one Menu-Ribbon Bar Taking over the world since 1371!
