Throttling web service calls on different threads

Gareth H

I have a web service that recieves calls from a number of clients (600 and growing). This calls a 3rd party web service to return some image data, which is then sent to the client. If i spam the 3rd party service with several calls at once (around 20+), it grinds to a halt, and takes over 2 seconds per call to return, when it normally takes 0.3 seconds on a single call. I noticed that if i throttle the calls (added a sleep of 500ms between each web request), the 3rd party runs fine. The question is, whats the best way/how can i implement a throttling system in my code, so if i receive several calls on different threads, i dont send them all at once, but que them up, so the 3rd party server has time to respond.

Regards, Gareth. (FKA gareth111)

Ian Shlasko

A throttle is generally a looping background thread, combined with a queue and a signaling mechanism... Here's one way to do it (Pseudocode):

public SendItem()
{
Add request to queue
Send a new data signal (Perhaps a System.Threading.EventWaitHandle)
}

private Throttle() (Running on background thread)
{
Loop indefinitely
{
If queue is empty
Wait for signal
Else
{
Process item from queue
Wait X milliseconds
}
}
}

Proud to have finally moved to the A-Ark. Which one are you in?
Author of the Guardians Saga (Sci-Fi/Fantasy novels)

Luc Pattyn

I agree with Ian's approach, which basically is tunneling all 3rd party accesses through a single thread. However, it may not scale well, as in the end either the single thread tunnel or the web service itself will become a bottleneck. If the 3rd party package is slowing down under extra load, my first impression is it is running out of memory and starts to page to disk all the time. Depending on what exactly causes the phenomenon, you could limit the number of outstanding requests to N (N to be determined and apparently less than 20). Either use a single thread and some logic to achieve this; or just launch N threads, each trying to find and launch one job at a time, all from a single queue (with appropriate lock while accessing the queue). OTOH, if the web service itself is basically single-threaded (e.g. because a COM component is involved), then whatever multi-threaded stuff you add to feed it may not achieve anything. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

Abhinav S

Remember that browsers to have their limitations - as in IE 8 (for e.g.) can open only 8 (or was it 6) asynchronous connections at a time. So even if your web services can handle tons of data (you are going to run into problems on the client side). Firefox allows you to configure this limit.

The funniest thing about this particular signature is that by the time you realise it doesn't say anything it's too late to stop reading it. My latest tip/trick Visit the Hindi forum here.

Gareth H

The client is not a web browser, its a C# windows service. Sorry i did not make that clear.

Regards, Gareth. (FKA gareth111)

Gareth H

Below is the code i've done from your example. The only bit i am unsure about is the waithandle. I know what/how to use them, but am unsure how it works in this scenario. Why would i want to hold the thread at a point?, is it just to save cpu cycles so it doesnt keep checking if the queue is empty?

    public void AddImageRequestToQueue(string isbn)
    {
        Contract.Requires(!string.IsNullOrEmpty(isbn));

        lock (\_queuePadLock)
        {
            \_imageRequests.Enqueue(isbn);
        }
    }

    private void ProcessImageQueue()
    {
        do
        {
            if (\_imageRequests.Count > 0)
            {
                string isbn = string.Empty;
                lock (\_queuePadLock)
                {
                    isbn = \_imageRequests.Dequeue();
                }

                IImageResult imageResult = GetImage(isbn, ImageSize.Small);

                OnProcessImageCompleted(new ProcessImageCompletedEventArgs(imageResult.Image, imageResult.ISBN));

                Thread.Sleep(SleepTime);
            }
        }
        while (true);
    }

Regards, Gareth. (FKA gareth111)

Gareth H

Cheers for your reply Luc, informative as always :) I am going to contact the 3rd party as well, as you say, i might be doing all this work for nothing if their end has limitations.

Regards, Gareth. (FKA gareth111)

Ian Shlasko

Yep, that's exactly what the wait handle is for. Makes your app more efficient by not having the thread endlessly looping.

Proud to have finally moved to the A-Ark. Which one are you in?
Author of the Guardians Saga (Sci-Fi/Fantasy novels)

Gareth H

Cool cheers!

Regards, Gareth. (FKA gareth111)