And when you depend on multithreading to be unpredictable, it isn't!
-
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support. I am certainly no FreeRTOS expert, but the docs seems to say that it can multi task using a single core well enough, but if you want to use multiple cores you in for some serious programming … Since you have been writing quite a bit about it lately, I was starting to get interested in the thing - a nice tiny core with SMP support would certainly interesting. Are you actually getting true concurrency with the thing, or is the scheduler just using one core at a time? That could certainly explain the deterministic behavior.
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
I don't know if it's any kind of true or at least standardized SMP which is why I put it in quotes, but as long as it's compiled with it there are options like xTaskCreatePinnedToCore(). That said, I don't see them in that documentation. It appears that the documentation there might be out of date. I can't profile it to determine what cores are really being utilized because while there are options where you can compile in profiling features like that, they are disabled for the ESP-IDF build, which is what I'm currently using, given my ESP_WROVER_KIT is sort of dependent on ESP-IDF, and I don't have say, an ARM based monster with JTAG debugging in the alternative. I'm left sort of having to trust the little OS more than I'd like. That was one of the reasons I was hoping I'd get better results with this sample. I *do* think it's using both cores based on my ability to create an idle priority thread on the second core and get spew back from it even when i spin a loop on the main core (causing the main core's idle thread to be starved), but I don't know how well it schedules, and I almost doubt it knows well enough to round robin threads across all cores. I'm not even sure offhand how to get the core count, and if I try to do the creation call pinned to a core that doesn't exist it crashes. :~ I *am* getting true concurrency from the looks of it. It's just badly scheduled concurrency. We'll see how it bakes out when I start doing I/O heavy stuff with it, because that's really where you'd want to use this library anyway. I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it.
Real programmers use butterflies
-
I don't know if it's any kind of true or at least standardized SMP which is why I put it in quotes, but as long as it's compiled with it there are options like xTaskCreatePinnedToCore(). That said, I don't see them in that documentation. It appears that the documentation there might be out of date. I can't profile it to determine what cores are really being utilized because while there are options where you can compile in profiling features like that, they are disabled for the ESP-IDF build, which is what I'm currently using, given my ESP_WROVER_KIT is sort of dependent on ESP-IDF, and I don't have say, an ARM based monster with JTAG debugging in the alternative. I'm left sort of having to trust the little OS more than I'd like. That was one of the reasons I was hoping I'd get better results with this sample. I *do* think it's using both cores based on my ability to create an idle priority thread on the second core and get spew back from it even when i spin a loop on the main core (causing the main core's idle thread to be starved), but I don't know how well it schedules, and I almost doubt it knows well enough to round robin threads across all cores. I'm not even sure offhand how to get the core count, and if I try to do the creation call pinned to a core that doesn't exist it crashes. :~ I *am* getting true concurrency from the looks of it. It's just badly scheduled concurrency. We'll see how it bakes out when I start doing I/O heavy stuff with it, because that's really where you'd want to use this library anyway. I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it.
Real programmers use butterflies
Quote:
I *am* getting true concurrency from the looks of it.
Cool :thumbsup:
Quote:
I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it
I'll read it with great interest :-D
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
Quote:
I *am* getting true concurrency from the looks of it.
Cool :thumbsup:
Quote:
I'm about to release a new article that builds on the stuff i recently wrote, only (hopefully) not specific to the ESP32 this time, and has more stuff like threads and thread pooling added to it
I'll read it with great interest :-D
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
Here's that article. FreeRTOS Thread Pack: Create Multithreaded IoT Code The Easy Way[^]
Real programmers use butterflies
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
Look up "[lock convoying](https://en.wikipedia.org/wiki/Lock\_convoy)". It's a well-known problem with lock-based multithreading.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support. I am certainly no FreeRTOS expert, but the docs seems to say that it can multi task using a single core well enough, but if you want to use multiple cores you in for some serious programming … Since you have been writing quite a bit about it lately, I was starting to get interested in the thing - a nice tiny core with SMP support would certainly interesting. Are you actually getting true concurrency with the thing, or is the scheduler just using one core at a time? That could certainly explain the deterministic behavior.
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
Use the overload that let's you specify a bool to indicate whether execution should be non-deterministic (default is false) ;p
Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript
-
Look up "[lock convoying](https://en.wikipedia.org/wiki/Lock\_convoy)". It's a well-known problem with lock-based multithreading.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
I'm not explicitly using locks in my own test code, though I did notice that Serial.println() appears to be atomic.
Real programmers use butterflies
-
Espen Harlinn wrote:
If I understand the docs[^] correctly, FreeRTOS is quite limited with regard to multi-core support
ESP32 uses a modified version of FreeRTOS[^] with added symmetric multiprocessing support.
Oh, is that what that is? I better put some conditional compiles in my code.
Real programmers use butterflies
-
I'm not explicitly using locks in my own test code, though I did notice that Serial.println() appears to be atomic.
Real programmers use butterflies
You may find that other parts of the O/S use locks. For example, I/O in blocks larger than the maximum supported by the hardware may be divided into blocks which are serialised using some sort of queue or lock. It's not the way you'do do it in Windows or Linux, but it works. :)
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
You may find that other parts of the O/S use locks. For example, I/O in blocks larger than the maximum supported by the hardware may be divided into blocks which are serialised using some sort of queue or lock. It's not the way you'do do it in Windows or Linux, but it works. :)
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
Yeah. That's what I was implying when I said Serial.print/println seemed atomic - other stuff locks. :)
Real programmers use butterflies
-
Here's that article. FreeRTOS Thread Pack: Create Multithreaded IoT Code The Easy Way[^]
Real programmers use butterflies
Quote:
Here's that article.
Thanks, I will read it :)
Espen Harlinn Senior Architect - Ulriken Consulting AS The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.Edsger W.Dijkstra
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
This is the predictability equivalent of a "Heisenbug".
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
How could instructions be non-deterministic when they share the same clock (even on different cores) ? Even random numbers generators are determinitic. I think that the only way to introduce some "chance" in a piece of code is to get information from "outside" : wait for something from a mechanical disk, a keyboard, an other computer...
-
How could instructions be non-deterministic when they share the same clock (even on different cores) ? Even random numbers generators are determinitic. I think that the only way to introduce some "chance" in a piece of code is to get information from "outside" : wait for something from a mechanical disk, a keyboard, an other computer...
I'm writing to an external serial port, and it waits for the writes to complete. That should give it some amount of non-determinism.
Real programmers use butterflies
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
-
It's bad enough that multithreaded code is nondeterministic. I propose that it is also meta-nondeterministic: You can not even count on it to be non-deterministic :-\ When you need it to be unpredictable, the scheduler will inexplicably run your timeslices the exact same way, even when threads are executing on different cores, and even reboot to reboot. I'm stuck on creating an *example* simply because I cannot create a situation wherein two secondary threads appear to be in competition (with the third thread being the main application thread) on a dual core ESP32 running FreeRTOS. I can do it where one thread is in competition with the primary thread, but it's as if the scheduler is just a dog when it comes to scheduling between two threads on the same core or something. Grrr. It's bizarre.
Real programmers use butterflies
I'm an admittedly rookie programmer who hasn't done much with multithreading. But wouldn't a Real Time Operating system always do things in the same order? Much like a processor on a PLC? I'm here to learn, so if you have the time to answer that would be great. :)
-
I'm an admittedly rookie programmer who hasn't done much with multithreading. But wouldn't a Real Time Operating system always do things in the same order? Much like a processor on a PLC? I'm here to learn, so if you have the time to answer that would be great. :)
Not if there's I/O or something involved, but otherwise, yes, it "probably" will. The reason I say probably is because when you're dealing with multiple cores, they don't run in lockstep with one another. There's a small amount of non-determinism just in the fact that the cores keep their own schedulers and may not have started at precisely the same moment nor even work together*** *** they might synchronize with each other - it's a FreeRTOS-ESP32variant implementation detail I haven't looked into. But that aside, there's also the issue of I/O, which when dealing with an external device, can introduce non-determinism. In my code, I'm outputting to a serial UART, that's connected through an FTDI built bus/USB-bridge controller to a windows PC. Any latency introduced by the PC will ripple back to the thread that's running waiting on I/O.
Real programmers use butterflies
-
You're not wrong at least in the general case, however: 1. This isn't about adding a separate thread in my case. I'm writing a library to allow you to use threads more easily than FreeRTOS otherwise lets you 2. This isn't true if you're writing a library that includes a threadpooler on a system with a primitive scheduler that's prone to starvation. 3. Threads don't care. Hell, my code doesn't care. But it's sure hard to demonstrate out of order execution and resyncing execution order for a *demo* when I can't get the execution order to scramble in the first place 4. Yeah, but this isn't windows, see also, craptastic scheduler 5. If you're doing that to force a context switch I'm not sure what's wrong with you. =) 6. Absolutely true. To that end my library provides you access to *none* of those. :laugh: Seriously though, it offers you a message passing system in the alternative 7. See also, craptastic scheduler
Real programmers use butterflies
Quote:
Absolutely true. To that end my library provides you access to *none* of those. Laugh | :laugh: Seriously though, it offers you a message passing system in the alternative
Message passing is probably the only way to rein in the complexity of threads. Message-queues is the easiest message-passing interface you will find. I recently did a simple message queueing library (based on pthreads) for a personal project and still managed to get the system to deadlock eventually (only happened on Windows due to different scheduling algorithm[1]). After fixing it I realised that there was no value in a linked-list queue. I implemented my message-queue library as a double-linked list, so that any thread taking a message off of the queue does not block any thread trying to put a message onto the queue. My intention was that threads removing messages from the queue would never hold a lock that threads posting messages to the queue would need (and vice versa). Unfortunately all threads still have to lock the entire queue just to check if
(head==tail)in case there is only one item in the queue (then that item is both the head and the tail). This is the stupid way of doing this. Don't do what I did. Instead, do one of the following: 1. Use a fixed-length message queue (either fixed at runtime or fixed at compile-time). This removes quite a lot of the unnecessary complexity; you're going to lock the entire queue for any posting or removal, but you're going to do that anyway with linked-lists too, so no big deal. 2. Address the fixed-length queue using modulus of the length (with appropriate locks); this gives you a circular buffer with noifstatements.message_t messages[BUFLEN];
...
messages[index % BUFLEN] = new_message; // Posting a message
...
message_t mymessage = messages[index % BUFLEN]; // taking a message off the queueThe problem with doing this is that it would automatically drop old messages (which, strategically, may be something you want, actually). Also, if you're not using C++ (no smart pointers) that's going to be a memory leak. 3. If your target platform and implementation allows (which it will), use #defines to define a CMPXCHG macro that expands to the assembly of the cmpxchg opcode. You can then use that for a superfast single lock with a sleep in nanoseconds or milliseconds that gradually decrements by a fixe
-
Quote:
Absolutely true. To that end my library provides you access to *none* of those. Laugh | :laugh: Seriously though, it offers you a message passing system in the alternative
Message passing is probably the only way to rein in the complexity of threads. Message-queues is the easiest message-passing interface you will find. I recently did a simple message queueing library (based on pthreads) for a personal project and still managed to get the system to deadlock eventually (only happened on Windows due to different scheduling algorithm[1]). After fixing it I realised that there was no value in a linked-list queue. I implemented my message-queue library as a double-linked list, so that any thread taking a message off of the queue does not block any thread trying to put a message onto the queue. My intention was that threads removing messages from the queue would never hold a lock that threads posting messages to the queue would need (and vice versa). Unfortunately all threads still have to lock the entire queue just to check if
(head==tail)in case there is only one item in the queue (then that item is both the head and the tail). This is the stupid way of doing this. Don't do what I did. Instead, do one of the following: 1. Use a fixed-length message queue (either fixed at runtime or fixed at compile-time). This removes quite a lot of the unnecessary complexity; you're going to lock the entire queue for any posting or removal, but you're going to do that anyway with linked-lists too, so no big deal. 2. Address the fixed-length queue using modulus of the length (with appropriate locks); this gives you a circular buffer with noifstatements.message_t messages[BUFLEN];
...
messages[index % BUFLEN] = new_message; // Posting a message
...
message_t mymessage = messages[index % BUFLEN]; // taking a message off the queueThe problem with doing this is that it would automatically drop old messages (which, strategically, may be something you want, actually). Also, if you're not using C++ (no smart pointers) that's going to be a memory leak. 3. If your target platform and implementation allows (which it will), use #defines to define a CMPXCHG macro that expands to the assembly of the cmpxchg opcode. You can then use that for a superfast single lock with a sleep in nanoseconds or milliseconds that gradually decrements by a fixe
1. I don't write my own concurrency safe queues because FreeRTOS has one and so does .NET so I've not had the need. 2. Yeah, when I wrote a ring buffer in C# I did that 3. I don't know a good reason to use that over say, std::atomic. In my experience, anything that won't support std::atomic won't support atomic CMPXCHG operations at the CPU level anyway, at least not that way. With the atMega2560 for example, IIRC it doesn't have one, forcing you to disable interrupts and then reenable them after the operation is complete. Don't quote me on the mega's capabilities, I'm not an AVR expert. It might be a bad example. Particularly, #3 is curious to me. Why wouldn't you use for example, std::atomic_int? Is it because it's a C++ thing? I use C++ even on 8-bit machines with 4kb of RAM. I just severely limit my use of things like The STL to the bare minimum. std::atomic is one area I use. std::chrono is another. Why? Because writing cross platform CMPXCHNG and timer code is error prone and i don't have access to all that hardware.
Real programmers use butterflies
-
1. I don't write my own concurrency safe queues because FreeRTOS has one and so does .NET so I've not had the need. 2. Yeah, when I wrote a ring buffer in C# I did that 3. I don't know a good reason to use that over say, std::atomic. In my experience, anything that won't support std::atomic won't support atomic CMPXCHG operations at the CPU level anyway, at least not that way. With the atMega2560 for example, IIRC it doesn't have one, forcing you to disable interrupts and then reenable them after the operation is complete. Don't quote me on the mega's capabilities, I'm not an AVR expert. It might be a bad example. Particularly, #3 is curious to me. Why wouldn't you use for example, std::atomic_int? Is it because it's a C++ thing? I use C++ even on 8-bit machines with 4kb of RAM. I just severely limit my use of things like The STL to the bare minimum. std::atomic is one area I use. std::chrono is another. Why? Because writing cross platform CMPXCHNG and timer code is error prone and i don't have access to all that hardware.
Real programmers use butterflies
Yes, it's because it's a C++ thing, and my queueing library is a C thing (My mention of pthreads should have given it away :-))
