Help on Multi-Threaded applicaiton [modified]
-
Hi, I am having an application where we mainly do analysis on images being grabbed in real-time. The current version basically works in the following way- Loop[^] the critical section code within writing into shared buffer looks as follows
// Critical section of code CCriticalSection CritSect; CritSect.Lock(); // Copy into shared memory memcpy(SharedBuffer, cap_buffer, FrameSize); Timestamp_FrameWrite = LoopStart->GetTimeDiff(); //"frame write" CritSect.Unlock();and the code within reading frame is as follows// Critical section of code CCriticalSection CritSect; CritSect.Lock(); // Copy from shared memory memcpy(proc_buffer, SharedBuffer, FrameSize); Timestamp_FrameRead = LoopStart->GetTimeDiff(); //"frame read" CritSect.Unlock();From the block diagram you will notice that camera thread will call the function to write the newly captured frame into shared buffer for every 20ms (since the exposure time is 20ms) and this is also verified by finding the difference between two successive "frame write" timestamps (~18ms to ~22ms). This is all fine. Now when I find the difference between two successive "frame read" timestamps, it always gives me either ~15ms or ~31ms as shown below15.8758 15.6622 31.008 15.845 15.605 31.062 15.819 15.663 15.748but when you average these over total number of frames read, it give ~20ms which shows that it is not dropping frames while reading (in other words its analyzing all frames being captured). So, the question is why is the "frame read" timestamp doesnt correspond to frame rate (i.e ~20ms or ~49Hz), this is the same case even when I am not doing any analysis (which hardly takes more than 10msec)?? I have to say that the loop function runs on a TIMER which has 5msec timeout value. -thanksPKNT
-
Hi, I am having an application where we mainly do analysis on images being grabbed in real-time. The current version basically works in the following way- Loop[^] the critical section code within writing into shared buffer looks as follows
// Critical section of code CCriticalSection CritSect; CritSect.Lock(); // Copy into shared memory memcpy(SharedBuffer, cap_buffer, FrameSize); Timestamp_FrameWrite = LoopStart->GetTimeDiff(); //"frame write" CritSect.Unlock();and the code within reading frame is as follows// Critical section of code CCriticalSection CritSect; CritSect.Lock(); // Copy from shared memory memcpy(proc_buffer, SharedBuffer, FrameSize); Timestamp_FrameRead = LoopStart->GetTimeDiff(); //"frame read" CritSect.Unlock();From the block diagram you will notice that camera thread will call the function to write the newly captured frame into shared buffer for every 20ms (since the exposure time is 20ms) and this is also verified by finding the difference between two successive "frame write" timestamps (~18ms to ~22ms). This is all fine. Now when I find the difference between two successive "frame read" timestamps, it always gives me either ~15ms or ~31ms as shown below15.8758 15.6622 31.008 15.845 15.605 31.062 15.819 15.663 15.748but when you average these over total number of frames read, it give ~20ms which shows that it is not dropping frames while reading (in other words its analyzing all frames being captured). So, the question is why is the "frame read" timestamp doesnt correspond to frame rate (i.e ~20ms or ~49Hz), this is the same case even when I am not doing any analysis (which hardly takes more than 10msec)?? I have to say that the loop function runs on a TIMER which has 5msec timeout value. -thanksPKNT
Can you share more about how do you measure time difference two frames? -Saurabh
-
Can you share more about how do you measure time difference two frames? -Saurabh
the time between two successive frames is just a simple subtraction between two successive
TimeStamp_FrameWrites. Same applies withTimeStamp_FrameReads. For example, lets say theLoopStartwhen we start the loop is 0ms then theTimeStamp_FrameWriteis always ~20ms, ~40ms,~60ms etc, which is same as camera exposure time. TheTimeStamp_FrameReadshould also be multiples of ~20ms , but its always either ~15ms or ~30ms as I mentioned earlier. Hope I am clear now.PKNT
-
Hi, I am having an application where we mainly do analysis on images being grabbed in real-time. The current version basically works in the following way- Loop[^] the critical section code within writing into shared buffer looks as follows
// Critical section of code CCriticalSection CritSect; CritSect.Lock(); // Copy into shared memory memcpy(SharedBuffer, cap_buffer, FrameSize); Timestamp_FrameWrite = LoopStart->GetTimeDiff(); //"frame write" CritSect.Unlock();and the code within reading frame is as follows// Critical section of code CCriticalSection CritSect; CritSect.Lock(); // Copy from shared memory memcpy(proc_buffer, SharedBuffer, FrameSize); Timestamp_FrameRead = LoopStart->GetTimeDiff(); //"frame read" CritSect.Unlock();From the block diagram you will notice that camera thread will call the function to write the newly captured frame into shared buffer for every 20ms (since the exposure time is 20ms) and this is also verified by finding the difference between two successive "frame write" timestamps (~18ms to ~22ms). This is all fine. Now when I find the difference between two successive "frame read" timestamps, it always gives me either ~15ms or ~31ms as shown below15.8758 15.6622 31.008 15.845 15.605 31.062 15.819 15.663 15.748but when you average these over total number of frames read, it give ~20ms which shows that it is not dropping frames while reading (in other words its analyzing all frames being captured). So, the question is why is the "frame read" timestamp doesnt correspond to frame rate (i.e ~20ms or ~49Hz), this is the same case even when I am not doing any analysis (which hardly takes more than 10msec)?? I have to say that the loop function runs on a TIMER which has 5msec timeout value. -thanksPKNT
What is the source of these timestamps? Aside from the high-resolution performance counter, there's no timers available on Windows with a resolution better than ~15ms. Also, your critical sections are useless as shown. A critical section will only work if threads lock on the SAME critical section object. If your frame read and write code shown is executed on separate threads, you effectively have NO syncronization. Mark
-
What is the source of these timestamps? Aside from the high-resolution performance counter, there's no timers available on Windows with a resolution better than ~15ms. Also, your critical sections are useless as shown. A critical section will only work if threads lock on the SAME critical section object. If your frame read and write code shown is executed on separate threads, you effectively have NO syncronization. Mark
The source of timestamps is a small custom defined class with calls to QueryPerformanceCounter/frequency. Yes, those critical sections have no real use as far as I tested and we dont use in our application. Actually another lab took our code and modified it for their application and their programmer left without actually resolving these issues and they want me to check it. The timing is pretty much accurate for 'frame write' around ~20ms, its the 'frame read' that are not what we expect.
PKNT
-
The source of timestamps is a small custom defined class with calls to QueryPerformanceCounter/frequency. Yes, those critical sections have no real use as far as I tested and we dont use in our application. Actually another lab took our code and modified it for their application and their programmer left without actually resolving these issues and they want me to check it. The timing is pretty much accurate for 'frame write' around ~20ms, its the 'frame read' that are not what we expect.
PKNT
Interesting. The timestamps look like something somewhere is relying on a low-res timer... suspiciously with an accuracy very close to the tickcount in Windows :) Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Interesting. The timestamps look like something somewhere is relying on a low-res timer... suspiciously with an accuracy very close to the tickcount in Windows :) Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
But its not either ~15ms or ~31ms all the time, when he displays some real-time graph from analysis of each frame, the time diff between two successive frame reads is all over the place from ~2ms to ~42ms. Maybe I have to provide with more clear code to understand. But anyways the basic idea of their application is, camera frame grabbing is always ~20ms so, as soon as a frame is grabbed and while we are doing analysis of the current frame, we have the camera to capture the next frame which will take another 20ms and since we know analysis doesnt take more than 10ms, theoretically the loop has to wait for the camera to get the next frame for the rest of 10ms while the camera is capturing next frame. What would be the best way to accomplish this, I have some ideas that I would like to try on Monday, but meanwhile if you come up with something, please lemme know.
PKNT
-
But its not either ~15ms or ~31ms all the time, when he displays some real-time graph from analysis of each frame, the time diff between two successive frame reads is all over the place from ~2ms to ~42ms. Maybe I have to provide with more clear code to understand. But anyways the basic idea of their application is, camera frame grabbing is always ~20ms so, as soon as a frame is grabbed and while we are doing analysis of the current frame, we have the camera to capture the next frame which will take another 20ms and since we know analysis doesnt take more than 10ms, theoretically the loop has to wait for the camera to get the next frame for the rest of 10ms while the camera is capturing next frame. What would be the best way to accomplish this, I have some ideas that I would like to try on Monday, but meanwhile if you come up with something, please lemme know.
PKNT
Kiran Satish wrote:
I have to say that the loop function runs on a TIMER which has 5msec timeout value
What kind of timer?
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Kiran Satish wrote:
I have to say that the loop function runs on a TIMER which has 5msec timeout value
What kind of timer?
Mark Salsbery Microsoft MVP - Visual C++ :java:
Its the default MFC Timer that we initialize using CWnd::SetTimer function, but the camera grabbing runs in a different thread.
PKNT
-
Its the default MFC Timer that we initialize using CWnd::SetTimer function, but the camera grabbing runs in a different thread.
PKNT
Kiran Satish wrote:
ts the default MFC Timer that we initialize using CWnd::SetTimer function
Not good. I recommend multimedia timers (after all, this is what they're for!). IME, on XP+ they are pretty precise to 1ms :)
Kiran Satish wrote:
but the camera grabbing runs in a different thread
Are you responsible for grabbing frames at the interval you want or does the hardware/driver signal you somehow when a frame is available? IME, the ideal situation is when the driver can signal an event that wakes a thread for each frame. If that's not possible and you have to grab frames at a given interval, the next best performance is with multimedia timers. I personally wouldn't use Windows timers except for situations where timer resolution doesn't matter at all. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Kiran Satish wrote:
ts the default MFC Timer that we initialize using CWnd::SetTimer function
Not good. I recommend multimedia timers (after all, this is what they're for!). IME, on XP+ they are pretty precise to 1ms :)
Kiran Satish wrote:
but the camera grabbing runs in a different thread
Are you responsible for grabbing frames at the interval you want or does the hardware/driver signal you somehow when a frame is available? IME, the ideal situation is when the driver can signal an event that wakes a thread for each frame. If that's not possible and you have to grab frames at a given interval, the next best performance is with multimedia timers. I personally wouldn't use Windows timers except for situations where timer resolution doesn't matter at all. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
I never used those multimedia timers, maybe its time for me to use them :) . Coming to camera grabbing, we use a command from the frame grabber's library and whenever you call that command, it grabs a frame with a set predefined exposure time which, in our case is 20ms. So unless you call that command it wont capture a frame.
PKNT
-
I never used those multimedia timers, maybe its time for me to use them :) . Coming to camera grabbing, we use a command from the frame grabber's library and whenever you call that command, it grabs a frame with a set predefined exposure time which, in our case is 20ms. So unless you call that command it wont capture a frame.
PKNT
In that case, it seems like you don't need a timer. The grabber thread can loop grabbing frames - the exposure time will throttle the loop to 50 fps. After each frame is grabbed, set an event that wakes the processing thread. The processing thread can loop waiting on the grabber event. When the event is signaled, it can process the frame and wait again. If your processing time is ~10ms your frame-data critical section should never have to wait/block. Ideally, you'd want to eliminate the need for the critical section altogether if possible, even if you have to copy the frame data to another buffer on the processing thread. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
I never used those multimedia timers, maybe its time for me to use them :) . Coming to camera grabbing, we use a command from the frame grabber's library and whenever you call that command, it grabs a frame with a set predefined exposure time which, in our case is 20ms. So unless you call that command it wont capture a frame.
PKNT
Here's what I meant in the last post:
// Camera Thread
While capturing
{
Capture frame (20ms exposure)
Add frame data to shared buffer
Set frame-grabbed event
}// Processing Thread
While processing
{
Wait on frame-grabbed event
Copy data from shared buffer
Process the data
}You still need to synchronize access to the shared buffer. Only lock long enough to copy data to and from the buffer, then release the lock immediately. That's all based on ideal situation where processing time is always ~10ms. In reality, your threads can (and will be) interrupted by other threads in the system. If it's critical to process every frame, then change the shared buffer to a FIFO queue of some kind. The grabber thread can queue each frame and the processing thread can dequeue frames. If the queue holds pointers to frame buffers, then accessing the queue is fast, which makes lock time shorter. That's pretty much how I handle audio capture, where I want every sample regardless of thread timing fluctuations. Does that make sense? :) Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Here's what I meant in the last post:
// Camera Thread
While capturing
{
Capture frame (20ms exposure)
Add frame data to shared buffer
Set frame-grabbed event
}// Processing Thread
While processing
{
Wait on frame-grabbed event
Copy data from shared buffer
Process the data
}You still need to synchronize access to the shared buffer. Only lock long enough to copy data to and from the buffer, then release the lock immediately. That's all based on ideal situation where processing time is always ~10ms. In reality, your threads can (and will be) interrupted by other threads in the system. If it's critical to process every frame, then change the shared buffer to a FIFO queue of some kind. The grabber thread can queue each frame and the processing thread can dequeue frames. If the queue holds pointers to frame buffers, then accessing the queue is fast, which makes lock time shorter. That's pretty much how I handle audio capture, where I want every sample regardless of thread timing fluctuations. Does that make sense? :) Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
Yeh, it makes sense, thanks a lot. Actually thats what I like to implement , but I am not thinking of implementing locking of shared buffer earlier. But before implementing this, I would like to time exactly how much time each function in the processing thread is taking in worst case. Based on those timings, I would implement queuing on the captured frames. I have to change a lot of code thats currently been implemented. Since, this is not our lab's work, I might be working on and off on this. So, will keep here posted on how it goes :) .
PKNT
-
Yeh, it makes sense, thanks a lot. Actually thats what I like to implement , but I am not thinking of implementing locking of shared buffer earlier. But before implementing this, I would like to time exactly how much time each function in the processing thread is taking in worst case. Based on those timings, I would implement queuing on the captured frames. I have to change a lot of code thats currently been implemented. Since, this is not our lab's work, I might be working on and off on this. So, will keep here posted on how it goes :) .
PKNT
Cool good luck! Keep in mind: Locking is bad - eliminate it if possible - only lock for as long as necessary, and there's lots of other threads running in the system to mess with your timing :) Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Cool good luck! Keep in mind: Locking is bad - eliminate it if possible - only lock for as long as necessary, and there's lots of other threads running in the system to mess with your timing :) Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
I have a question, I tried to implement the same thing in my application (with some difference, in our application we can not grab frame while processing the current image as we have to update other hardware before grabbing next frame). Here is what I tried to do, I have a camera thread and my loop thread. The loop thread waits for the Camera thread to get a new frame and once the camera gets new frame, it waits for the loop thread until it process and signals it to get next frame. Thats the basic idea. After doing the current image analysis and changing the hardware status based upon current image analysis results, I spend around 6-8ms to update some displays and do some logs, so I thought I can signal the camera meanwhile to get new frame. But I am stuck with deadlocks with signal flags between threads.
Loop thread
do{
if (newframe){
do analysis on the frame
signal = true;//signal the camera to get new frame
update displays and do logs for the current frame
}
}while(looprunning);camera thread
do{
if (signal){
newframe = false;
get new frame - takes 20ms
newframe = true;
signal = false;
}
}while(camrunning);I can see an obvious deadlock, how can I get around this problem :confused: ? -thanks
PKNT
-
I have a question, I tried to implement the same thing in my application (with some difference, in our application we can not grab frame while processing the current image as we have to update other hardware before grabbing next frame). Here is what I tried to do, I have a camera thread and my loop thread. The loop thread waits for the Camera thread to get a new frame and once the camera gets new frame, it waits for the loop thread until it process and signals it to get next frame. Thats the basic idea. After doing the current image analysis and changing the hardware status based upon current image analysis results, I spend around 6-8ms to update some displays and do some logs, so I thought I can signal the camera meanwhile to get new frame. But I am stuck with deadlocks with signal flags between threads.
Loop thread
do{
if (newframe){
do analysis on the frame
signal = true;//signal the camera to get new frame
update displays and do logs for the current frame
}
}while(looprunning);camera thread
do{
if (signal){
newframe = false;
get new frame - takes 20ms
newframe = true;
signal = false;
}
}while(camrunning);I can see an obvious deadlock, how can I get around this problem :confused: ? -thanks
PKNT
Flags? no no no :) There's handy synchronization objects for that, something like:
autoreset events:
newframeevent
signaleventLoop thread
do
{
waitForSingleObject(newframeevent)
do analysis on the frame
SetEvent(signalevent) //signal the camera to get new frame
update displays and do logs for the current frame
}
while(looprunning);camera thread
do
{
waitForSingleObject(signalevent)
get new frame - takes 20ms
SetEvent(newframeevent)
}
while(camrunning);The worst thing I see in the flags implementation is two threads modifying the signal flag. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Flags? no no no :) There's handy synchronization objects for that, something like:
autoreset events:
newframeevent
signaleventLoop thread
do
{
waitForSingleObject(newframeevent)
do analysis on the frame
SetEvent(signalevent) //signal the camera to get new frame
update displays and do logs for the current frame
}
while(looprunning);camera thread
do
{
waitForSingleObject(signalevent)
get new frame - takes 20ms
SetEvent(newframeevent)
}
while(camrunning);The worst thing I see in the flags implementation is two threads modifying the signal flag. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
Thanks, I feel like a complete dumbo. I always use CEvent::WaitForSingleObject(,) in many threads so that they wont lock the CPU and never thought of using them here. Maybe I need to start thinking slowly ;P
PKNT
-
Thanks, I feel like a complete dumbo. I always use CEvent::WaitForSingleObject(,) in many threads so that they wont lock the CPU and never thought of using them here. Maybe I need to start thinking slowly ;P
PKNT
:)
Kiran Satish wrote:
I always use CEvent::WaitForSingleObject(,) in many threads so that they wont lock the CPU
Yes!. It also throttles your thread loops so they don't sit there spinning while waiting for the flag to be set, eating all your CPU for 1 processor (each). Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Thanks, I feel like a complete dumbo. I always use CEvent::WaitForSingleObject(,) in many threads so that they wont lock the CPU and never thought of using them here. Maybe I need to start thinking slowly ;P
PKNT
plus...you can add a terminate event if needed so the threads can be shutdown gracefully. Then you'd wait on multiple objects. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
