Azure Function Hell
-
So we've built a system that uses Azure, it's a function that validates a payload and pushes it onto a queue, and then another function pulls it off to process it. It's integration, keeping two systems in sync Worked fine until the client decided to push in 2000 records at once. We find that HTTP functions blow up if they get too many calls. At one point they were sending 20, waiting 30 seconds, sending more and that still blew up. We've been chasing this for over a month. There's Azure doco on this being limiting code that you can turn off. You cannot. The doco is all over the place. We found it doesn't happen to C# script BUT C# script won't work with queues with sessions. Today we get one of my co workers to code review it all. He says it all looks good then, off the cuff, says, why does your DI code use injecttransient. Because we're injecting a few lightweight classes and I didn't trust the idea of a singleton across Azure servers being better. So I make them all singletons, and now the code works. No documentation anywhere that I can find on why that would be.
-
So we've built a system that uses Azure, it's a function that validates a payload and pushes it onto a queue, and then another function pulls it off to process it. It's integration, keeping two systems in sync Worked fine until the client decided to push in 2000 records at once. We find that HTTP functions blow up if they get too many calls. At one point they were sending 20, waiting 30 seconds, sending more and that still blew up. We've been chasing this for over a month. There's Azure doco on this being limiting code that you can turn off. You cannot. The doco is all over the place. We found it doesn't happen to C# script BUT C# script won't work with queues with sessions. Today we get one of my co workers to code review it all. He says it all looks good then, off the cuff, says, why does your DI code use injecttransient. Because we're injecting a few lightweight classes and I didn't trust the idea of a singleton across Azure servers being better. So I make them all singletons, and now the code works. No documentation anywhere that I can find on why that would be.
yikes - that's a nasty trap - kudos to co-worker btw for asking the question
-
So we've built a system that uses Azure, it's a function that validates a payload and pushes it onto a queue, and then another function pulls it off to process it. It's integration, keeping two systems in sync Worked fine until the client decided to push in 2000 records at once. We find that HTTP functions blow up if they get too many calls. At one point they were sending 20, waiting 30 seconds, sending more and that still blew up. We've been chasing this for over a month. There's Azure doco on this being limiting code that you can turn off. You cannot. The doco is all over the place. We found it doesn't happen to C# script BUT C# script won't work with queues with sessions. Today we get one of my co workers to code review it all. He says it all looks good then, off the cuff, says, why does your DI code use injecttransient. Because we're injecting a few lightweight classes and I didn't trust the idea of a singleton across Azure servers being better. So I make them all singletons, and now the code works. No documentation anywhere that I can find on why that would be.
There are tons of undocumented behaviors in Azure - it is an environment developed in production... For instance (from my personal experience) a Linux based app service is actually sits behind an IIS wrapper, where that wrapper always 'talks' to the Linux instance in HTTP, which actually makes it unusable for like 90% of the scenarios in today world, where HTTPS is a minimum requirement and checked by most...
"The only place where Success comes before Work is in the dictionary." Vidal Sassoon, 1928 - 2012
-
yikes - that's a nasty trap - kudos to co-worker btw for asking the question
Yeah, I found no doco so his suggestion was awesome. I did find other limits that I could turn off, so I assume all of those changes also worked in their own way
-
There are tons of undocumented behaviors in Azure - it is an environment developed in production... For instance (from my personal experience) a Linux based app service is actually sits behind an IIS wrapper, where that wrapper always 'talks' to the Linux instance in HTTP, which actually makes it unusable for like 90% of the scenarios in today world, where HTTPS is a minimum requirement and checked by most...
"The only place where Success comes before Work is in the dictionary." Vidal Sassoon, 1928 - 2012
*sigh* there's a lot of defunct doco still live as well. The source of truth is what your function does in real life
-
So we've built a system that uses Azure, it's a function that validates a payload and pushes it onto a queue, and then another function pulls it off to process it. It's integration, keeping two systems in sync Worked fine until the client decided to push in 2000 records at once. We find that HTTP functions blow up if they get too many calls. At one point they were sending 20, waiting 30 seconds, sending more and that still blew up. We've been chasing this for over a month. There's Azure doco on this being limiting code that you can turn off. You cannot. The doco is all over the place. We found it doesn't happen to C# script BUT C# script won't work with queues with sessions. Today we get one of my co workers to code review it all. He says it all looks good then, off the cuff, says, why does your DI code use injecttransient. Because we're injecting a few lightweight classes and I didn't trust the idea of a singleton across Azure servers being better. So I make them all singletons, and now the code works. No documentation anywhere that I can find on why that would be.
Feel free to ignore this because it's just a wild stab in the dark (and it's obvious once one knows), but are you somehow injecting an HttpClient using transient ?... because HttpClient is (as I know due to (cough) experience) "once per app" and exhibits interesting behaviour if it's continually 'new'd ... see first comment line in the c# example (link below) and text in the in the Remarks section... HttpClient Class (System.Net.Http) | Microsoft Docs[^] HTH
-
So we've built a system that uses Azure, it's a function that validates a payload and pushes it onto a queue, and then another function pulls it off to process it. It's integration, keeping two systems in sync Worked fine until the client decided to push in 2000 records at once. We find that HTTP functions blow up if they get too many calls. At one point they were sending 20, waiting 30 seconds, sending more and that still blew up. We've been chasing this for over a month. There's Azure doco on this being limiting code that you can turn off. You cannot. The doco is all over the place. We found it doesn't happen to C# script BUT C# script won't work with queues with sessions. Today we get one of my co workers to code review it all. He says it all looks good then, off the cuff, says, why does your DI code use injecttransient. Because we're injecting a few lightweight classes and I didn't trust the idea of a singleton across Azure servers being better. So I make them all singletons, and now the code works. No documentation anywhere that I can find on why that would be.
Christian Graus wrote:
No documentation anywhere that I can find
... because people who live in glass houses don't throw stones. Really? Honestly there's gotta be sample/example use somewhere. If there's no sample there's no way anyone's gonna use it. Anyone who is anybody. Now, as for Bob ...
-
Feel free to ignore this because it's just a wild stab in the dark (and it's obvious once one knows), but are you somehow injecting an HttpClient using transient ?... because HttpClient is (as I know due to (cough) experience) "once per app" and exhibits interesting behaviour if it's continually 'new'd ... see first comment line in the c# example (link below) and text in the in the Remarks section... HttpClient Class (System.Net.Http) | Microsoft Docs[^] HTH
No, in .NET Core there's a HTTP Context factory. I inject that to get a HttpClient
-
Christian Graus wrote:
No documentation anywhere that I can find
... because people who live in glass houses don't throw stones. Really? Honestly there's gotta be sample/example use somewhere. If there's no sample there's no way anyone's gonna use it. Anyone who is anybody. Now, as for Bob ...
There's tons of .NET Core and Azure functions samples. Nothing on the issue I had that I found while searching
