Event Barrier

In the above example, a number of threads start at different times with the arrival of each clock event and wait till all threads finish running before the next clock event arrives.

Such barrier can be implemented using various techniques with different level of difficulties. One possible implementation can be found in Joseph Albahari’s Threading in C# where a monitor construct is used to wait on a synchronization object in the main thread after the worker threads are queued.

    lock (workerLocker)
    {
      while (runningWorkers > 0)
        Monitor.Wait (workerLocker);
    }

In each of the worker thread, the number of running workers is decremented when the thread finishes:

    lock (workerLocker)
    {
      runningWorkers--;
      Monitor.Pulse (workerLocker);
    }

And when the running number of workers reaches zero, the monitor exist the wait condition and the main thread continues.

The wait on the locker object forms a barrier in the example above. The method illustrated here is simple and elegant. It can be used to create the barriers needed to synchronize the clock event mentioned earlier.

Threadpooling might not always be the optimal solution though. While the number of threads in a thread pool is typically bounded to a few dozens, it may incur excessive context switching penalties if the majority of the threads are CPU bound. In a computation intensive situation, a fixed number of threads approximating the number of computational cores would be more appropriate (for instance, four threads on a four core CPU).

Here I will show another relatively simple way to construct a barrier where only a predetermined maximum number of threads can reach at a given time. To achieve such a ceiling, a global semaphore is used (in this example three concurrent threads can enter the semaphore):

public class Global
{
    public static Semaphore sp = new Semaphore(3, 3);
}

To simplify coding and shield the programmer from the synchronization mechanism details, I created an IEvent interface and an abstract EventBase as follows:

interface IEvent
{
    void OnEvent(int arg);
}

public abstract class EventBase : IEvent
{
    #region IEvent Members
    public void OnEvent(int arg)
    {
        Global.sp.WaitOne();
        Run(arg);
        Global.sp.Release();
    }
    #endregion

    public virtual void Run(int arg)
    {
    }
}

Users can simply implement their code that needs to block at the barrier by inheriting from EventBase and overridding the Run() method:

public class Event1 : EventBase
{
    public override void Run(int arg)
    {
        Console.WriteLine("Thread {0}, arg {1}", Thread.CurrentThread.ManagedThreadId, arg);
        Thread.Sleep(1000);
    }
}

To illustrate how this barrier implementation can be used considering the following code:

class Program
{
    delegate void MyEvent(int arg);

    static void Main(string[] args)
    {
        IEvent e = new Event1();
        MyEvent m = new MyEvent(e.OnEvent);

        List<IAsyncResult> l = new List<IAsyncResult>();

        for (int i = 0 ; i < 6 ; i++)
        {
          IAsyncResult r = m.BeginInvoke(i, null, null);
          l.Add(r);
        }

        foreach (IAsyncResult r in l)
        {
            m.EndInvoke(r);
        }
    }
}

In the code above, we simulate six concurrent threads that need to be synchronized. The multi-threading is achieved via asynchronous invocation. Because the capacity of the semaphore is set to be 3 in this example and if this code is run on a quad-core machine, you will see that the six asynchronous threads are run in two batches, three at a time. And all the threads are synchronized at the end. Here is the sample output:

Thread 10, arg 1
Thread 11, arg 2
Thread 6, arg 0
Thread 13, arg 4
Thread 10, arg 5
Thread 12, arg 3

Using this approach, the number of concurrently running threads can be precisely controlled by the maximum threads allowed in the semaphore.

For the type of applications that perform the same task over a large dataset for example,

for (int i = 0 ; i < num ; i++)
{
    //lengthy tasks
}

we could easily break down the for loop evenly into chunks that fit into the number of processors given. And for each of the subsets we could create a new process to perform the desired tasks. For instance, if we were to execute the above for loop in parallel on a duo-core processor, we could re-write each of the loops like the following:

//core 1
for (int i = 0 ; i < num ; i+=2)
{
    //lengthy tasks
}

//core 2
for (int i = 1 ; i < num ; i+=2)
{
    //lengthy tasks
}

By executing the application in separate processes, we eliminated certain race conditions that must be properly handled in a multi-threaded process and thus it is far easier to execute multiple processes concurrently when there is no inter-process communications involved than to handle multi-processing within a process via multi-threading. In a multi-process scenario, the operating system handles the concurrency issue behind the scene whereas in the multi-threading case the application developer must ensure the code regions are thread safe.

The following C# code shows how to dispatch tasks among different processes using asynchronous method invocations. Each process being invoked can identify its order by the extra integer parameter (line 36) passed to it. Thus we only needed to modify the target application to take this extra parameter into account (for instance, in a quad-core system, the first process would process items 1,5,9… and the second process would run items 2,6,10, and so on)

using System;
using System.Diagnostics;
using System.Threading;

namespace MPDispatcher
{
    class Program
    {
        public delegate bool CmdDlg(string fn, string args);

        private bool RunCommand(string fn, string args)
        {
            bool r = true;
            Process p = new Process();

            p.StartInfo.FileName = fn;
            p.StartInfo.Arguments = args;

			Console.WriteLine(String.Format("Starting {0} with arguments {1}. Thread ID: {2}", fn, args, Thread.CurrentThread.ManagedThreadId));
            r = p.Start();
            p.WaitForExit();

            return r;
        }

		public bool DispatchCommand(int numOfThreads, string args)
        {
            bool r = true;

            IAsyncResult[] ars = new IAsyncResult[numOfThreads];
            CmdDlg[] dlgs = new CmdDlg[numOfThreads];

            for (int i = 0; i < numOfThreads; i++)
            {
                dlgs[i] = new CmdDlg(RunCommand);
                ars[i] = dlgs[i].BeginInvoke(args, i.ToString(), null, null);
				Thread.Sleep(100);
            }

            bool finished = false;

			//wait till all processes are finished.
            while (!finished)
            {
                bool t = true;

                for (int i = 0; i < numOfThreads; i++)
                    t = t && ars[i].IsCompleted;   

                if (t) finished = true;

                Thread.Sleep(100);
            }

            for (int i = 0; i < numOfThreads; i++)
            {
                r = r && dlgs[i].EndInvoke(ars[i]);
            }

            return r;
        }

        static void Main(string[] args)
        {
            Program p = new Program();

			if (args.Length < 1 || args[0].ToUpper() == "-H") {
				Console.WriteLine("Usage: MPDispatcher {app name} {app args}");
			} else {
				int numThreads = 4; //use 4 threads
				string argStr = string.Empty;

				for (int i = 1 ; i < args.Length; i++) {
					argStr = String.Format("{0} {1}", argStr, args[i]);
				}

            	p.DispatchCommand(numThreads, argStr);
			}
        }
    }
}

The above code works under Windows and Linux (Mono).

Since the implementation of many of the functions are threaded, it makes the task of writing high performance applications much easier. Since it is a set of “performance primitives” as the name suggests, it uses its own data structures (e.g. Ipp8u) and does not provide functions to directly inter-operate with data coming from other sources (e.g. image files).

Fortunately, such data conversion is pretty straight forward. In this post, I will illustrate how to convert an image file (e.g. .jpg, .png, .gif) to the data IPP uses, and how to save the result into a standard image file once the processing is done.

Magick++ is a very comprehensive image-processing C++ library and the Image class it provides handles image files quite well. So I chose to use Magic++’s API to convert image files to the data structure IPP uses. In this particular example, I will use a gray level image. But in practice, color images can be easily handled in a similar fashion.

For the code mentioned below, the following headers and namespaces are used:

#include <Magick++/Image.h>
#include <Magick++.h>
#include <ipp.h>

using namespace std;
using namespace Magick;

And the following code shows how to convert a standard image file data into format that is suitable for IPP.

    IppStatus sts;
    Image img("{Image File Name}");
    Geometry g = img.size();

    unsigned int width = g.width();
    unsigned int height= g.height();

    Pixels view(img);
    PixelPacket *pixels = view.get(0,0,width,height);

    int stepByte = 0;
    //allocating a buffer of unsigned char (Ipp8u) for the image.
    Ipp8u *imgCache = ippiMalloc_8u_C1(width,height, &stepByte);

    for (unsigned int row = 0; row < height ; row++)
    {
        for (unsigned int column = 0; column < width ; column++)
        {
            PixelPacket *p = &pixels[column + row * width];
            Color c = Color(p->red, p->green, p->blue);
            double i = c.scaleQuantumToDouble(c.intensity()) * 255;
            imgCache[column + row * width] = (char) i;
        }
    }

The above code snippet first reads the image data into *PixelPacket and the pixel buffer is then converted into a one channel buffer of chars (color value 0-255). Note that the range of the image data Magick++ reads in is between 0 and QuantumRange, which needs to be converted back to the range 0-255 accepted by the Ipp8u buffer. If other types of IPP image buffers are used (e.g. Ipp32f), this scaleQuantumToDouble() conversion may not be necessary. At the end, the image data is converted into the one dimensional array imgCache which can be used by IPP procedures.

Once we are in the IPP data domain, we can proceed with whatever processing we had in mind. Here I will show the code for edge detection using Canny algorithm.

    IppiSize orgImgSize = {width, height};
    IppiSize newImgSize = {width + 2, height + 2};

    Ipp8u *imgCache1 = ippiMalloc_8u_C1(width + 2,height + 2, &stepByte);
    sts = ippiCopyReplicateBorder_8u_C1R(imgCache, width, orgImgSize,  imgCache1,
            width + 2 , newImgSize, 2, 2);
    IppiSize roi = {width, height};

    Ipp32f low=30.0f, high=100.0f;
    int size, size1, srcStep, dxStep, dyStep, dstStep;

    Ipp8u *src, *dst, *buffer;
    Ipp16s *dx, *dy;

    sts = ippiFilterSobelNegVertGetBufferSize_8u16s_C1R(
            roi, ippMskSize3x3, &size);
    sts = ippiFilterSobelHorizGetBufferSize_8u16s_C1R(
            roi, ippMskSize3x3, &size1);

    if (size<size1) size=size1;
    ippiCannyGetSize(roi, &size1);
    if (size<size1) size=size1;

    buffer = ippsMalloc_8u(size);
    dx = ippsMalloc_16s(size);
    dy = ippsMalloc_16s(size);
    dst = ippsMalloc_8u(size);

    sts = ippiFilterSobelNegVertBorder_8u16s_C1R (
            imgCache1, width + 2 , dx, (width + 2) * 2 ,
            roi, ippMskSize3x3, ippBorderRepl, 0, buffer);
    sts = ippiFilterSobelHorizBorder_8u16s_C1R(
            imgCache1, width + 2, dy, (width + 2) *2 ,
            roi, ippMskSize3x3, ippBorderRepl, 0, buffer);
    sts = ippiCanny_16s8u_C1R(dx,
            (width + 2) * 2, dy, (width + 2) * 2,
            dst, width, roi, low, high, buffer);

The code shown above is adopted from Intel’s IPP manual for image and video processing (by default it is located at /opt/intel/ipp/{version number}/em64t/doc/ippiman.pdf).

Please pay special attention to how the original image is extended via ippiCopyReplicateBorder_8u_C1R. The image is extended by 2 pixels in each direction because the 3×3 mask used for the filtering operation. I omitted error checking code for simplicity, but in general you need to check the return status of each ippi function call. When the call is successful, the status is ippStsNoErr. If you receive a value other than ippStsNoErr (e.g. ippStsStepErr) you will need to check your buffer size to make sure that they are adjusted according to the data type. For instance, in the code above dx and dy are both 16bit signed integers and thus they both occupy two bytes each.

To save the result image, we convert the pixel buffer back to PixelPacket type. Again we need to convert the data in the buffer (Ipp8u) to the range accepted by Magick++ API.

    for (unsigned int y = 0; y< height ; y++)
    {
        for (unsigned int x = 0; x < width; x++)
        {
            Color c;
            float clr = (float) dst[x + y * width] /255;
            float q = c.scaleDoubleToQuantum(clr);
            pixels[x+ y * width] = Color(q, q, q);
        }
    }

    view.sync();
    img.syncPixels();

    img.write("{Image File Name}");

We can also save the result data into a new image (instead of syncing the data back to the image object holding the original image data) using the code below:

    Image img1(Geometry(width, height),"white");

    for (unsigned int y = 0; y< height ; y++)
    {
        for (unsigned int x = 0; x < width; x++)
        {
            Color c;
            float clr = (float) dst[x + y * width] /255;
            float q = c.scaleDoubleToQuantum(clr);
            img1.pixelColor(x,y, Color(q, q, q));
        }
    }

    img1.write("{Image File Name}");

And finally the buffers used are freed.

    ippiFree(imgCache);
    ippsFree(buffer);

The following images show Canny edge detector in action using the code in this article:

Original

Canny Edge Detector applied

The standard algorithm for generating Mandelbrot Set is extremely easy to adapt to using TBB. In fact the loops look almost identical to those in the single-threaded approach, except that the iterations are calculated within a 2D range block (blocked_range2d) instead of the entire two dimensional space.

Because xlib by itself is not thread-safe, special attention must be made when trying to update the display concurrently. One way to address this issue is to employee a shared memory region (X11/extensions/XShm.h and sys/shm.h), the display is first built in memory and then the shared memory is attached to the display. In my examples above I used code (video.h, xvideo.cpp) from the sample code that come with the TBB library, which uses the shared memory method I mentioned earlier to make the X11 calls thread-safe.

Many optimization methods can be used to further enhance the performance of the algorithm. One of the most efficient methods is to utilize SSE instructions found on all modern Intel processors (examples can be found here: Using SSE3 Technology in Algorithms with Complex Arithmetic). This approach however might be difficult to implement and debug since parallel data structures must be used in order to benefit from SSE instructions. Also, explicit assembly level coding makes porting code to other machine architectures a daunting task. Modern compilers can already take full advantage of the underlying machine architecture. For example, the gcc compiler (4.2.3) already generates SSE instructions for the code snippet above. While hand tweaking using SSE instructions might further improve the performance, we would certainly sacrifice code simplicity and portability.

The approach I am going to take to further optimize the code is to use loop unrolling. Since the inner loop of the standard algorithm is pretty short, unrolling the inner loop should lessen the burden of loop overhead and decrease the chances of stalling the pipeline (when branching must be predicted). So a high-level loop unrolling should be able to improve the performance.

Here is the code after the inner loop is unrolled:

This code generates identical results as the code mentioned previously. As you can see, the inner loop is not unrolled to handle two data points at a time.

As it turned out, this algorithm runs almost twice as fast as the code mentioned earlier(280ms versus 510ms on Intel Q9450 @ 3.4GHz).

Source code for this article can be downloaded here.

For a very large interval (e.g. 3~1,000,000), the TBB version of the prime program achieved a 4x speed up given a reasonably large grain size (e.g. 100). Smaller grain size resulted in slightly more overhead. On a quad-core machine (Q9450 @ 3.2GHz), it took 217.83 ms for the single threaded routine to find all the prime numbers within 1,000,000, whereas it only took 58.32 ms for the TBB version, which runs roughly four times as fast. The TBB framework took care of dividing the task according to the number of processors automatically.

As new processors roll out, new benchmarks tend to only capture the performance of that specific processor family and rarely compare the performance between different generations of processors. From many sources and the readily available data, we know that in terms of multi-media and gaming application which heavily rely on SSE* instructions, the new Core family processors out performs the old Pentium processors by leaps and bounds (The new Core architecture has made significant improvement over its Pentium predecessors). But there is very little information on how scientific applications might have benefited from the enhanced architecture.

I recently just got a new ThinkPad T60 laptop at work (it has a Core Duo T2400 1.83 GHz processor), so I decided to run the same benchmark program I created in my earlier post and see how the numbers turn out.

And here are the results for the total run time when running 1 to 4 threads:

Time (1 thread) = 1548.6 ms Time (2 threads)= 784.8 ms Time (3 threads)= 785.3 ms Time (4 threads)= 788.1 ms

In comparison, here are the results for the same benchmark program when running on Pentium 4 3GHz processor:

Time (1 thread) = 756.2 ms Time (2 threads)= 754.7 ms Time (3 threads)= 769.3 ms Time (4 threads)= 772.5 ms

And Pentium D 2.8GHz processor:

Time (1 thread) = 931.6 ms Time (2 threads)= 469.4 ms Time (3 threads)= 466.9 ms Time (4 threads)= 485.8 ms

As you can see, the performance of Core Duo is not that impressive for scientific calculations even if the clock speeds were the same. And the Core Duo T2400 mobile processor is probably a little slower than the Pentium D processor with the same clock speed.

According to an article Mobile CPU Wars: Core 2 Duo vs. Core Duo on AnandTech, the new Core 2 Duo processor is roughly 10% faster than Core Duo processor under the same clock speed. Of course the mobile version of Core Duo or Core 2 Duo are running at a lower bus speed (667MHz) compared to their desktop counterparts (1066MHz) so from what we’ve seen the performance for scientific calculations on Core 2 Duo processors are probably only marginally better then the current Pentium D (in multi-threaded scenario) or Pentium (in single-threaded scenario) processors.

This is not all that surprising as the simple benchmark application I created is so small in size that the working set is likely to fit in Pentium’s much smaller on-die cache, so the extra L1/L2 cache memory found in Core series does not provide any significant advantage here. Also, the calculation is somewhat predictable in our simple program so the inefficiencies in the deep pipe-lined Pentium processors do not signify. Further, we did not utilize the new SSE* instructions either.

Nevertheless, it is clear that unless we consciously utilize the new instruction sets found in Core serious processors and have a large working set to work with, the speed advantage of the new processors are not significant compared to Pentium processors.

There is no doubt that Hyper-Threading can, under certain circumstances, boost application performance. The performance gain is highly dependent on application type, and according to Intel, this performance gain is at an average of 15-30%.

But for computational intensive multi-threaded applications, Hyper-Threading does not provide much benefit however. Here I will show some benchmarks that support this assertion.

The Benchmark Program

I created a very simple benchmark program using C# which generates prime numbers for a given range and for a user specified number of threads. The main program creates as many threads as the user specified and each threads computes prime numbers within the given range. The calculation for each thread is interleaved so that each thread generates a unique subset of the prime numbers within the given interval.

The main function is listed here:

public double Run(int numOfThreads, int ubound){

MPThreadingTest[] threads = new MPThreadingTest[numOfThreads];

int startNum = 3;

for (int i = 0; i < numOfThreads; i++)

threads[i] = new MPThreadingTest(startNum + i * 2, numOfThreads * 2, ubound - (numOfThreads - i) * 2);

perfCounter.Start();

for (int i = 0; i < numOfThreads; i++) threads[i].Start();

for (int i = 0; i < numOfThreads; i++) threads[i].Wait();

perfCounter.End();

return perfCounter.TimeElapsed("MS"); }

The high precision timing unit was described in one of my earlier post.

Results

The following are the results from two CPUs, one is a Pentium 4 3.0 GHz HT (Northwood), the other is a Pentium D 2.8 GHz (Smithfield). All the calculations are repeated for 10 times and the results are averaged. The time obtained is the time it takes to enumerate the prime numbers within 1,000,000.

As can be seen, using 2 threads instead of 1 on the Hyper-Threaded CPU, we only achieved roughly 0.2%. While on Pentium D we are able to achieve 98.5% speed gain.

Intuitively, adding more threads than the actual core slows down the operation. But as the benchmarks suggested, the slowdown might not be as dramatic as you might have imagined. Scaling to 10 threads from 2 only slowed down the calculation speed by approximately 10% for both CPUs.

In the article WinForms UI Thread Invokes: An In-Depth Review of Invoke/BeginInvoke/InvokeRequred on asp.net , Justin Rogers described in detail how to make cross thread usage of UI code in a thread-safe manner.

Threading problems are typically very hard to debug without the clear understanding of the problem domain. And the thread safety of UI components is vastly ignored by many developers. If you want to have an in-depth look of how the UI thread should be used please take a look at Justin’s article. What I wanted to show you is that the enhanced debugger in Visual Studio 2005 actually can detect some invalid cross-thread operations.

Consider the following code snippet for a windows form application:

private void BtnTest_Click(object sender, EventArgs e) {

TimerCallback timerCallBackDelegate =

new TimerCallback(TimerCallBackHandler);

System.Threading.Timer timer =

new System.Threading.Timer(timerCallBackDelegate, null, 100, 100);

}

private void TimerCallBackHandler(object state) {

i += 1;

Label1.Text = i.ToString();

}

The button click event creates a timer and on the timer callback a label is updated. If you run this code in Visual Studio 2003, it would appear that everything runs fine. But there is a potential bug in TimerCallBackHandler, namely updating the Label control on a non-UI thread (spawned by the timer).

If you run the above code in Vistual Studio 2005, you will get the following message:

InvalidOperationexception was unhandled

Cross-thread operation not valid: Control ‘Label1’ accessed from a thread other than the tread it was created on.

To fix this problem we need to replace the TimerCallBackHandler with the code below:

private void TimerCallBackHandler(object state) {

if (Label1.InvokeRequired) {

TimerCallback callBackDelegage =

new TimerCallback(TimerCallBackHandler);

BeginInvoke(callBackDelegage, new object[] { state });

} else {

i += 1;

Label1.Text = i.ToString();

}

}

Where the call to update Label1 is marshalled back to the UI thread if Invoke is required.

Clearly, such coding errors will be caught more easily with the new Visual Studio 2005 IDE. Note that this is a feature of the new development environment not the new version of the CLR. If you compile and run the first code snippet using 2.0 framework and run it without starting from the IDE, you will not get the exception message either.

As a side note, it seems that the internal error reporting mechanism has changed between framework 1.1 and 2.0. If we change the TimerCallBackHandler to the snippet below (note, here we create the label control on a thread other than the UI thread):

private void TimerCallBackHandler(object state) {

Label l = new Label();

l.Left = 10;

l.Top = 10;

l.Text = “Test “;

this.Controls.Add(l);

}

We would get the following exception message in Visual Studio 2003:

An unhandled exception of type ‘System.ArgumentException’ occurred in system.windows.forms.dll

Additional information: Controls created on one thread cannot be parented to a control on a different thread.

But if we run the same code in Visual Studio 2005: the following exception message appears:

InvalidOperationexception was unhandled

Cross-thread operation not valid: Control ‘Label1’ accessed from a thread other than the tread it was created on.

Which essentially is the same as the message from the first exception.