And when you depend on multithreading to be unpredictable, it isn't!
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
Threads are like fireflies. They tend to synchronize into repeated patterns. ;)
Latest Articles:
Client-Side Type-Based Publisher/Subscriber, Exploring Synchronous, "Event-ed", and Worker Thread Subscriptions -
Threads are like fireflies. They tend to synchronize into repeated patterns. ;)
Latest Articles:
Client-Side Type-Based Publisher/Subscriber, Exploring Synchronous, "Event-ed", and Worker Thread SubscriptionsUnless you need them to. :mad:
Real programmers use butterflies
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
Hmm. No criticism here, just old fart woolgathering. After a long time developing multithreaded applications (including UI's), I've come to some observations:
- Thinking "adding a separate thread" will fix a problem generally won't.
- If you pay attention to timeslices, you're doing it wrong.
- If threads care about sequence of execution, you are doomed to failure.
Sleep(0)to force a context switch in Windows is A Bad Idea.Sleep(_n_ > 0)is even worse.- Indiscriminately adding synchronization primitives like critical sections, mutexes, semaphores, and so on without understanding what you're doing gives you a false sense of security.
- Modifying thread priorities is bad karma. Please don't.
Software Zen:
delete this; -
Threads are like fireflies. They tend to synchronize into repeated patterns. ;)
Latest Articles:
Client-Side Type-Based Publisher/Subscriber, Exploring Synchronous, "Event-ed", and Worker Thread SubscriptionsI thought that was women and their...never mind.
Robust Services Core | Software Techniques for Lemmings | Articles
The fox knows many things, but the hedgehog knows one big thing. -
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
I honestly don't know which one to react with, so :laugh::rose: Most schedulers lack self-control and need to be disciplined!
Robust Services Core | Software Techniques for Lemmings | Articles
The fox knows many things, but the hedgehog knows one big thing. -
Hmm. No criticism here, just old fart woolgathering. After a long time developing multithreaded applications (including UI's), I've come to some observations:
- Thinking "adding a separate thread" will fix a problem generally won't.
- If you pay attention to timeslices, you're doing it wrong.
- If threads care about sequence of execution, you are doomed to failure.
Sleep(0)to force a context switch in Windows is A Bad Idea.Sleep(_n_ > 0)is even worse.- Indiscriminately adding synchronization primitives like critical sections, mutexes, semaphores, and so on without understanding what you're doing gives you a false sense of security.
- Modifying thread priorities is bad karma. Please don't.
Software Zen:
delete this;You're not wrong at least in the general case, however: 1. This isn't about adding a separate thread in my case. I'm writing a library to allow you to use threads more easily than FreeRTOS otherwise lets you 2. This isn't true if you're writing a library that includes a threadpooler on a system with a primitive scheduler that's prone to starvation. 3. Threads don't care. Hell, my code doesn't care. But it's sure hard to demonstrate out of order execution and resyncing execution order for a *demo* when I can't get the execution order to scramble in the first place 4. Yeah, but this isn't windows, see also, craptastic scheduler 5. If you're doing that to force a context switch I'm not sure what's wrong with you. =) 6. Absolutely true. To that end my library provides you access to *none* of those. :laugh: Seriously though, it offers you a message passing system in the alternative 7. See also, craptastic scheduler
Real programmers use butterflies
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
honey the codewitch wrote:
I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic
I'd say they are non-deterministically deterministic...
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support. I am certainly no FreeRTOS expert, but the docs seems to say that it can multi task using a single core well enough, but if you want to use multiple cores you in for some serious programming … Since you have been writing quite a bit about it lately, I was starting to get interested in the thing - a nice tiny core with SMP support would certainly interesting. Are you actually getting true concurrency with the thing, or is the scheduler just using one core at a time? That could certainly explain the deterministic behavior.
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support. I am certainly no FreeRTOS expert, but the docs seems to say that it can multi task using a single core well enough, but if you want to use multiple cores you in for some serious programming … Since you have been writing quite a bit about it lately, I was starting to get interested in the thing - a nice tiny core with SMP support would certainly interesting. Are you actually getting true concurrency with the thing, or is the scheduler just using one core at a time? That could certainly explain the deterministic behavior.
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
I don't know if it's any kind of true or at least standardized SMP which is why I put it in quotes, but as long as it's compiled with it there are options like xTaskCreatePinnedToCore(). That said, I don't see them in that documentation. It appears that the documentation there might be out of date. I can't profile it to determine what cores are really being utilized because while there are options where you can compile in profiling features like that, they are disabled for the ESP-IDF build, which is what I'm currently using, given my ESP_WROVER_KIT is sort of dependent on ESP-IDF, and I don't have say, an ARM based monster with JTAG debugging in the alternative. I'm left sort of having to trust the little OS more than I'd like. That was one of the reasons I was hoping I'd get better results with this sample. I *do* think it's using both cores based on my ability to create an idle priority thread on the second core and get spew back from it even when i spin a loop on the main core (causing the main core's idle thread to be starved), but I don't know how well it schedules, and I almost doubt it knows well enough to round robin threads across all cores. I'm not even sure offhand how to get the core count, and if I try to do the creation call pinned to a core that doesn't exist it crashes. :~ I *am* getting true concurrency from the looks of it. It's just badly scheduled concurrency. We'll see how it bakes out when I start doing I/O heavy stuff with it, because that's really where you'd want to use this library anyway. I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it.
Real programmers use butterflies
-
I don't know if it's any kind of true or at least standardized SMP which is why I put it in quotes, but as long as it's compiled with it there are options like xTaskCreatePinnedToCore(). That said, I don't see them in that documentation. It appears that the documentation there might be out of date. I can't profile it to determine what cores are really being utilized because while there are options where you can compile in profiling features like that, they are disabled for the ESP-IDF build, which is what I'm currently using, given my ESP_WROVER_KIT is sort of dependent on ESP-IDF, and I don't have say, an ARM based monster with JTAG debugging in the alternative. I'm left sort of having to trust the little OS more than I'd like. That was one of the reasons I was hoping I'd get better results with this sample. I *do* think it's using both cores based on my ability to create an idle priority thread on the second core and get spew back from it even when i spin a loop on the main core (causing the main core's idle thread to be starved), but I don't know how well it schedules, and I almost doubt it knows well enough to round robin threads across all cores. I'm not even sure offhand how to get the core count, and if I try to do the creation call pinned to a core that doesn't exist it crashes. :~ I *am* getting true concurrency from the looks of it. It's just badly scheduled concurrency. We'll see how it bakes out when I start doing I/O heavy stuff with it, because that's really where you'd want to use this library anyway. I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it.
Real programmers use butterflies
Quote:
I *am* getting true concurrency from the looks of it.
Cool :thumbsup:
Quote:
I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it
I'll read it with great interest :-D
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
Quote:
I *am* getting true concurrency from the looks of it.
Cool :thumbsup:
Quote:
I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it
I'll read it with great interest :-D
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
Here's that article. FreeRTOS Thread Pack: Create Multithreaded IoT Code The Easy Way[^]
Real programmers use butterflies
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
Look up "[lock convoying](https://en.wikipedia.org/wiki/Lock\_convoy)". It's a well-known problem with lock-based multithreading.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support. I am certainly no FreeRTOS expert, but the docs seems to say that it can multi task using a single core well enough, but if you want to use multiple cores you in for some serious programming … Since you have been writing quite a bit about it lately, I was starting to get interested in the thing - a nice tiny core with SMP support would certainly interesting. Are you actually getting true concurrency with the thing, or is the scheduler just using one core at a time? That could certainly explain the deterministic behavior.
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
Use the overload that let's you specify a bool to indicate whether execution should be non-deterministic (default is false) ;p
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
-
Look up "[lock convoying](https://en.wikipedia.org/wiki/Lock\_convoy)". It's a well-known problem with lock-based multithreading.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
I'm not explicitly using locks in my own test code, though I did notice that Serial.println() appears to be atomic.
Real programmers use butterflies
-
Espen Harlinn wrote:
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support
ESP32 uses a modified version of FreeRTOS[^] with added symmetric multiprocessing support.
Oh, is that what that is? I better put some conditional compiles in my code.
Real programmers use butterflies
-
I'm not explicitly using locks in my own test code, though I did notice that Serial.println() appears to be atomic.
Real programmers use butterflies
You may find that other parts of the O/S use locks. For example, I/O in blocks larger than the maximum supported by the hardware may be divided into blocks which are serialised using some sort of queue or lock. It's not the way you'do do it in Windows or Linux, but it works. :)
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
You may find that other parts of the O/S use locks. For example, I/O in blocks larger than the maximum supported by the hardware may be divided into blocks which are serialised using some sort of queue or lock. It's not the way you'do do it in Windows or Linux, but it works. :)
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
Yeah. That's what I was implying when I said Serial.print/println seemed atomic - other stuff locks. :)
Real programmers use butterflies
-
Here's that article. FreeRTOS Thread Pack: Create Multithreaded IoT Code The Easy Way[^]
Real programmers use butterflies
Quote:
Here's that article.
Thanks, I will read it :)
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
This is the predictability equivalent of a "Heisenbug".
