threads and their time slots. [modified]

Souldrift

Good morning :), just a brief question. I got a multi-threaded program. Now is it possible to ensure that a certain thread at a certain point won´t lose its 'working permission'. Hm, don´t know if I expressed that correctly. I have a thread going into a function where it has to wait for a previously defined time, then go on. The thing is, it waits and waits and at some point its time slot is taken away. When it is given the next slot, though, the time it has waited for has 'long' passed. And thus it´s always too late. The code in question is:

int RTPEngine::SendRTPPacket( BYTE* data, int size )
{
if( data && size > 0 && m_pHeader )
{
m_pLogger->Out(Logger::DEBUG, "RTPEngine: Sending data package with size %d (excluding header).\n", size);

	BYTE\* packet = new BYTE\[size+12\];
	BYTE\* headerBytes = m\_pHeader->GetInc();

	for( int i = 0; i < 12; i++ )
	{
		packet\[i\] = headerBytes\[i\];
	}
	for( int i = 0; i < size; i++ )
	{
		packet\[i+12\] = data\[i\];
	}

	double now = 0.0; // milliseconds
	if( m\_dPacketSendTime != 0.0 )
	{
		now = ((double) clock() / CLK\_TCK) \* 1000; // milliseconds
		int tmp = 0;
		//m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Time now = %f.\\n", now );
		while( now <= m\_dPacketSendTime )
		{
			if( tmp >= 20 )
			{
				m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Time now = %f, send time = %f.\\n", now, m\_dPacketSendTime );
				tmp = 0;
			}
			tmp++;
			now = ((double) clock() / CLK\_TCK) \* 1000;
		}
	}

	now = ((double) clock() / CLK\_TCK) \* 1000;
	int rc=sendto(m\_oUDPSocket,(char\*)packet,size+12,0,(SOCKADDR\*)&m\_oUDPAddress,sizeof(SOCKADDR\_IN));

	m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Packet sent at = %f.\\n", now );

	m\_dPacketSendTime = ((double) clock() / CLK\_TCK) \* 1000; // milliseconds
	m\_dPacketSendTime += (m\_dPacketLengthMillis\*(1.0 - m\_dRTPOverlap));
	m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Next send time = %f.\\n", m\_dPacketSendTime );

	return size;
}
else
	return 0;

}

And one more thing is, on my Vista machine it works fine. The logger prints this: RTPEngine: Next send time = 53637.795918. RTPEngine: Sending data package with size 480 (excluding header). RTPEngine: Time now = 53628.000000, send time = 53637.795918. . . . RTPEngine: Time now = 53636.000000, send time = 53637.795918. RTPEngine: Time now = 53636.000000, send time = 53637.795918. RTPEngine: Time now = 53637.000000, send time = 53637.795918. RTPEngine: Time now = 536

Stuart Dootson

Let me get my traditional admonition about Windows + real-time behaviour out of the way first - Windows is not anywhere near being a real-time OS, there's too much stuff going on, it has high interrupt latency etc etc. Ha, that's better. Now - the one thing that may improve your situation is to increase the priority of your sending thread. Alternatively, you could use a different strategy - use a waitable timer[^], with a completion routine that sends the packet. Put the thread into an alertable wait state when you set the timer and the completion routine should get called by the timer. The advantage of doing this is that the more your code is running, the more Windows will give a dynamic priority boost to other, similarly prioritised threads. If your code sleeps, that won't happen, and also, your system as a whole will be slightly more responsive.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Souldrift

Thanks for the idea. I tried it (very exitedly). But I´m running into the same problem. My new code is:

int RTPEngine::SendRTPPacket( BYTE* data, int size )
{
if( data && size > 0 && m_pHeader )
{
m_pLogger->Out(Logger::DEBUG, "RTPEngine: Sending data package with size %d (excluding header).\n", size);

	BYTE\* packet = new BYTE\[size+12\];
	BYTE\* headerBytes = m\_pHeader->GetInc();

	for( int i = 0; i < 12; i++ )
	{
		packet\[i\] = headerBytes\[i\];
	}
	for( int i = 0; i < size; i++ )
	{
		packet\[i+12\] = data\[i\];
	}

	double now = 0.0; // milliseconds
	if( m\_dPacketSendTime != 0.0 )
	{
		now = ((double) clock() / CLK\_TCK) \* 1000; // milliseconds
		m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Time now = %f.\\n", now );
		double dueMS = m\_dPacketSendTime-now; // relative time to wait in ms
		if( dueMS > 0 )
		{
			HANDLE hTimer = 0;
			LARGE\_INTEGER liDueTime;
			hTimer = ::CreateWaitableTimer(0, TRUE, L"WaitableTimer");
			liDueTime.QuadPart = -((int)dueMS \* 10000); // relative waitin time converted to '100-nanoseconds'
			SetWaitableTimer(hTimer, &liDueTime, 0, NULL, NULL, FALSE);
			::WaitForSingleObject(hTimer, INFINITE);
		}
	}

	now = ((double) clock() / CLK\_TCK) \* 1000;
	int rc=sendto(m\_oUDPSocket,(char\*)packet,size+12,0,(SOCKADDR\*)&m\_oUDPAddress,sizeof(SOCKADDR\_IN));

	m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Packet sent at = %f.\\n", now );

	m\_dPacketSendTime = ((double) clock() / CLK\_TCK) \* 1000; // milliseconds
	m\_dPacketSendTime += (m\_dPacketLengthMillis\*(1.0 - m\_dRTPOverlap));
	m\_pLogger->Out( Logger::DEBUG, "RTPEngine: Next send time = %f.\\n", m\_dPacketSendTime );

	return size;
}
else
	return 0;

}

I´m at a loss ... Souldrift Edit: Actually I just found that this doesn´t work as expected. The output is: RTPEngine: Next send time = 3009. RTPEngine: Sending data package with size 480 (excluding header). RTPEngine: Time now = 3000. RTPEngine: Packet sent at = 3000. RTPEngine: Next send time = 3009. RTPEngine: Sending data package with size 480 (excluding header). RTPEngine: Time now = 3000. RTPEngine: Packet sent at = 3000. RTPEngine: Next send time = 3009. RTPEngine: Sending data package with size 480 (excluding header). RTPEngine: Time now = 3000. RTPEngine: Packet sent at = 3000. It doesn´t wait at all ??

modified on Friday, July 3, 2009 7:34 AM

Stuart Dootson · modified on Friday, July 3, 2009 7:34 AM

The documentation for WaitForSingleObject[^] says this:

To enter an alertable wait state, use the WaitForSingleObjectEx[^] function

As I said, you need to be in an alertable wait state for your APC to be called.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Roger Stoltz

As Stuart already stated - windows is not a real-time operating system. You might find this article[^] on the subject interesting. However, I'm curious about why you need to send data with such accuracy you seem to be aiming for. Since you're using UDP sockets there's always going to be some kind of network latency that you cannot do anything about and it may not be the same for each packet sent. It seems to me as if this is about RTP, i.e. Real-time Transport Protocol, since the name of your class is RTPEngine. If this is the case then you shouldn't have to worry about timing when sending the data. Every RTP-packet has a timestamp that the receiver should use in order to figure out when the media data in the packet should be played. Read more about RTP here[^]. If you cannot make use of the timestamp in the RTP-packets, or if this is really not about RTP, I'd try the following:

Create a sending worker thread that reads data packets from a queue, e.g. std::queue; the packets are added to the queue by another thread, probably the main thread.
The sending thread waits on a semaphore, with e.g. ::WaitForMultipleObjects(), that is released once for each packet added to the queue. The trick here is that a thread waiting on a synchronization object, such as a semaphore, gets a temporary priority boost when the object is signalled which means that it will be scheduled to run ASAP.
Each data packet has a waitable timer created and set by the thread that adds the packets to the queue. The sending thread then waits on the timer and sends the data when that the timer is signalled.

I would create a class that represents a packet with a waitable timer as a member. I would also use reference counting smart pointers that reflects the data packets as elements of the queue; this way I don't have to worry about writing cleaning up code in my sending thread. When the object goes out of scope everything is released - heap and waitable timers. For more information on worker threads - read this[

Souldrift

Unfortunately that changes nothing :(.

Souldrift

The reason is not so much the exact timing but the available memory on the other side. The receiving end is not another computer but a phone. Its an audio stream from a tts-server. And the phone has only limited capacity so we cannot simply flood it since the sending end works much much faster than the processing of the stream at the receiving end. Therefor we wanted to build in something like a brake. That´s what the variable RTPOverlap is for. At the moment at a value of 10% (0.1). So the sending time of each RTP packet only overlaps with the previous one by about 10%. And these 10 percent are calculated from the playback time that one packet actually needs. Which is about 10ms for an RTP packet of 480 bytes. Meaning we send one packet and then the next comes after 9ms and so on. Only ... it doesn´t work. And of course I already have a 'sending worker'. The synthesizer of the tts-server works very fast. So I store the synthesized packages in one class like a scheduler and send them piece by piece away. By the way, I am familiar with (no specialist, but I´ll find my way around) worker threads and smart pointers and RTP. I´ll look into your links now, anyway .. thanks so far. Souldrift

Stuart Dootson

I put together a wee program demonstrating waitable timer usage:

#include "stdafx.h"
#include <iostream>
#include <Windows.h>

void ReportTime(const char* message, LONGLONG const& when)
{
std::cout << message << double(when)/10000.0 << std::endl;
}

VOID CALLBACK DoSendHere(__in_opt LPVOID lpArgToCompletionRoutine,
__in DWORD dwTimerLowValue,
__in DWORD dwTimerHighValue)
{
LARGE_INTEGER const& ftSleepStart = *(LARGE_INTEGER*)lpArgToCompletionRoutine;
LARGE_INTEGER liTimer;
liTimer.LowPart = dwTimerLowValue;
liTimer.HighPart = dwTimerHighValue;
ReportTime("DoSendHere::Timer slept for ", liTimer.QuadPart-ftSleepStart.QuadPart);
delete lpArgToCompletionRoutine;
}

int SendLoop()
{
LARGE_INTEGER ftStart;
GetSystemTimeAsFileTime((FILETIME*)&ftStart);
HANDLE hTimer = 0;
hTimer = ::CreateWaitableTimer(0, TRUE, L"WaitableTimer");

LARGE_INTEGER ftNow;
LARGE_INTEGER ftWaitTill;
GetSystemTimeAsFileTime((FILETIME*)&ftNow);
while(1)
{
ReportTime("SendLoop::Now = ", ftNow.QuadPart-ftStart.QuadPart);

  ftWaitTill.QuadPart = ftNow.QuadPart + 5000000I64;
  SetWaitableTimer(hTimer, &ftWaitTill, 0, &DoSendHere, new LARGE\_INTEGER(ftNow), FALSE);
  ftNow = ftWaitTill;
  ::SleepEx(INFINITE, TRUE);

}
}

int _tmain(int argc, _TCHAR* argv[])
{
SendLoop();
return 0;
}

This is intended to set a timer for 500ms in the future and sleep until the timer's triggered, at which point a) the completion routine is called, and b) the SleepEx call exits. I get this output (on Windows XP SP 3):

SendLoop::Now = 0
DoSendHere::Timer slept for 499.978
SendLoop::Now = 500
DoSendHere::Timer slept for 499.955
SendLoop::Now = 1000
DoSendHere::Timer slept for 499.933
SendLoop::Now = 1500
DoSendHere::Timer slept for 499.91
SendLoop::Now = 2000
DoSendHere::Timer slept for 499.888
SendLoop::Now = 2500
DoSendHere::Timer slept for 499.866
SendLoop::Now = 3000
DoSendHere::Timer slept for 499.843
SendLoop::Now = 3500
DoSendHere::Timer slept for 499.821
SendLoop::Now = 4000
DoSendHere::Timer slept for 499.798
SendLoop::Now = 4500
DoSendHere::Timer slept for 499.776
SendLoop::Now = 5000
DoSendHere::Timer slept for 499.754
SendLoop::Now = 5500
DoSendHere::Timer slept for 499.731
SendLoop::Now = 6000
DoSendHere::Timer slept for 499.709
SendLoop::Now = 6500
DoSendHere::Timer slept for

Keith Worden

It could be the use of the clock() function; on your XP machine it seems to have a much coarser granularity as the time isn't changing at all in the loop. I always use GetTickCount() for millisecond timing - you can work with integers and don't have to bother doing any division. It's never going to be accurate to a millisecond though; I tried a similar loop on my system and only got a different value about every 15 ms even though I was continually calling the function.

Souldrift

Thanks for that. I highly appreciate you getting into this so much. I all but copied your code and it works on my machine. But as soon as I put it on our server machine, the ol´ troublemakers show up again. I set a sleep time of appx. 10ms and usually the thing sleeps the day away ... well, 15 ms :). The output in my case is

RTPEngine: Next send time = 12891106961478.547000.
RTPEngine: Sending data package with size 480 (excluding header).
RTPEngine: Time now = 12891106961468.750000.
DoSendHere::Timer slept for 15.625000
RTPEngine: Packet sent at = 12891106961484.375000.
RTPEngine: Next send time = 12891106961494.172000.
RTPEngine: Sending data package with size 480 (excluding header).
RTPEngine: Time now = 12891106961484.375000.
DoSendHere::Timer slept for 0.000000
RTPEngine: Packet sent at = 12891106961484.375000.
RTPEngine: Next send time = 12891106961494.172000.
RTPEngine: Sending data package with size 480 (excluding header).
RTPEngine: Time now = 12891106961484.375000.
DoSendHere::Timer slept for 15.625000
RTPEngine: Packet sent at = 12891106961500.000000.
RTPEngine: Next send time = 12891106961509.797000.
JobManager: Job 0_0 processed packet with size 2048. Packets available: 67
RTPEngine: Preparing to send data bundle with size of 1520 bytes.
RTPEngine: Adding leftover (352 bytes).
RTPEngine: Sending data package with size 480 (excluding header).
RTPEngine: Time now = 12891106961500.000000.
DoSendHere::Timer slept for 15.625000
RTPEngine: Packet sent at = 12891106961515.625000.
RTPEngine: Next send time = 12891106961525.422000.
RTPEngine: Sending data package with size 480 (excluding header).
RTPEngine: Time now = 12891106961515.625000.
DoSendHere::Timer slept for 15.625000
RTPEngine: Packet sent at = 12891106961531.250000.

What bugs me even more than that (as seen in the output) is that the Timer sometimes sleeps for 0 ms even though it should wait. Looking into it ... might be something with threads and time-slices, after all. Souldrift

Souldrift

I don´t think so anymore. I now work with FILETIME (see code from Stuart Dootson). And I still got the same trouble. Souldrift

Souldrift

I just found something else. A little disturbing. On our server machine (WIN XP) the Timer ALWAYS sleeps for 15.625ms. If it goes to sleep at all, that is. So this seems to be some kind of minimum time-slice (or a multiplied version of it). On my own developing computer (Vista) I don´t have theis kind of time-slices. The Timer sleeps for 9ms, then for 10ms and thereafter for 8ms. Does anybody now if one can manually control these time-slices? Souldrift

Keith Worden

There's some references on the web to the minimum timer period being 10 ms for uniprocessor systems and 15 ms for multiprocessor. It could also be different depending on whether it's 32 or 64 bit Windows. The minimum on my system seems to be 15 ms - that's on Vista 32-bit with a dual-core processor.

Souldrift

Yeah, this would make our efforts kind of hopeless. But I just found out another little something. It doesn´t seem to be a Vista-XP-thing. But a network-thing. Whatever computer I use as a server, as long as the client comes in via network, the timer sleeps like 15.something millis. If server and client are on the same machine, there doesn´t seem to be a minimum time-slice (or it´s below what I saw so far). The timer sleeps as long (or brief) as I want. Souldrift

Souldrift

By the way. I want to thank all who tried and helped me (not only in this thread) so far. This forum has brought me forward quite a bit. I love the hivemind :). Cheers Souldrift

Souldrift

One last thought. I just tried to not let the thread sleep or create a wait object or anything. I simply went into a while-loop. Endlessly. Where I fetched and printed the system time. This was the result

RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781458000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000, send time = 12891115378155.598000.
RTPEngine: Time now = 128911153781614000.000000

Stuart Dootson

Souldrift wrote:

And so on ... it always jumps 15,6ms forward

Here's what Mark Russinovitch and David Solomon say in Windows Internals

_

On Windows 2000 Professional and Windows XP, threads run by default for 2 clock intervals; on Windows Server systems, by default, a thread runs for 12 clock intervals. The rationale for the longer default value on server systems is to minimize context switching. By having a longer quantum, server applications that wake up as the result of a client request have a better chance of completing the request and going back into a wait state before their quantum ends

The length of the clock interval varies according to the hardware platform. The frequency of the clock interrupts is up to the HAL, not the kernel. For example, the clock interval for most x86 uniprocessors is about 10 milliseconds and for most x86 multiprocessors it is about 15 milliseconds. (The actual clock rate is not exactly a round number of milliseconds—see the following experiment for a way to check the actual clock interval.)

EXPERIMENT: Determining the Clock Interval Frequency

The Windows GetSystemTimeAdjustment function returns the clock interval. To determine the clock interval, download and run the Clockres program[^] from http://www.sysinternals.com.

_

That kind of confirms what you're seeing (and what I see - which is 15.625, or 500/32). HOWEVER!!!! I steered you wrong with waitable timers. I mistook them for a TimerQueueTimer in my memory. Try this:

#include <Windows.h>
#include <tchar.h>
#include <iostream>

void ReportTime(const char* message, LONGLONG const& when)
{
std::cout << message << double(when)/10000.0 << std::endl;
}

LONG count;
LARGE_INTEGER liStart, liEnd, liFreq;

VOID CALLBACK DoSendHere(__in_opt LPVOID lpArgToCompletionRoutine,
__in DWORD dwTimerLowValue,
__in DWORD dwTimerHighValue)
{
QueryPerformanceCounter(&liEnd);
count++;
}

void SendLoop()
{
HANDLE hTimerQueue = CreateTimerQueue();
HANDLE hTimer;
count = 0;
CreateTimerQueueTimer(&hTimer, hTimerQueue, (WAITORTIMERCALLBACK)&DoSendHere, 0 , 1, 1, WT_EXECU

Roger Stoltz

Souldrift wrote:

it always jumps 15,6ms forward

I doubt the granularity and accuracy of the time value returned by ::GetSystemTimeAsFileTime(). Even though the FILETIME structure is a time value where the least significant bit represents 100 nanoseconds, it doesn't mean that it has an accuracy of 100 nanosecs. It's like looking at your watch, measuring a time with the needle showing seconds and then multiplying the value by 1,000,000 and claim you measured with microsecond accuracy. For time measurements of this kind you should you the performance timer; ::QueryPerformanceCounter() and ::QueryPerformanceFrequency(). It will give you the best accuracy available on your hardware. Read here[^] for more info on the high resolution performance counter.

"It's supposed to be hard, otherwise anybody could do it!" - selfquote
"High speed never compensates for wrong direction!" - unknown

Roger Stoltz

Bah, you beat me to it... :-\

"It's supposed to be hard, otherwise anybody could do it!" - selfquote
"High speed never compensates for wrong direction!" - unknown

Stuart Dootson

Sorrreeeee :-O

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p