Without using any shift registers, a single ATmega328p chip can handle a around 20 I/O ports. I decided to use 16 LEDs, twice the number of LEDs as the ones you commonly see. This gives me at least four free pins for other purposes. You can probably build a POV with around 20 LEDs, but If you need more than that, you will have to use shift registers to expand the available IO pins or change to a chip that has more built-in IO pins (i.e. ATMega1280). But as you shall see later, the image generation process will be considerably more complex and the required memory storage will be much larger as well.

POV Display and Its Synchronization

In order to achieve a stable image output, the image frames generated by the POV device must be synchronized. A POV display can operate as a hand device (e.g. a wand) or be driven by a motor. When the POV device is built for hand use, it usually has a motion detector (from simple spring-in-cylinder switch to the more exotic accelerometer) so that the device can start displaying when the hand is waved. I decided to go with a motor driven design so that I can have more precise control over the output image. In this approach, image frames must be synchronized for each full rotation. A simple way to detect the motor rotation is to use a IR transmitter and receiver.

The following is the full design of my POV display (the left side shows the optical triggering mechanism mentioned above):

16 LED POV Schematics

And here is the finished circuit mounted on a motorized rotating platform:

16 LED POV Display (top view)

Here is a picture of the side view, you can see how the IR transmitter and receiver circuits are mounted. Whenever the transmitter and the receiver line up (it happens every rotation), the transistor would output a LOW signal to the ATMega328p (pin 28, Arduino analog pin 5) which indicates the beginning of a new frame. To reduce the light interference from the environment, the receiver is mounted on the rotor so it is facing downwards.

16 LED POV Display (side view)

Image Encoder

Building the circuit is only half of the game. In order to provide the most flexible and the richest displays, I decided to build a image encoder which can be used to automatically translates a binary image into the appropriate code that is needed for the POV display. For simple POV displays, the display patterns can be mapped directly into a binary two dimensional array, with 1 represents LED on and 0 represents LED off. But this approach is not practical when displaying larger images. For an image the size of 16×100, 1600 bytes would be required to store the image information. This might not sound like much but that is almost all of ATMega328′s usable SRAM. Of course, you can always store the data structure off in Flash memory using PROGMEM, but it is not the most efficient way to do so. A better approach would be to pack these pixels into bytes so that each byte can represent 8 pixel values.

This code is shown as below. This C# code works with both Microsoft’s .Net platform or Mono.Net and thus it can be run on either Windows or Linux:

using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using System.Runtime.InteropServices;

namespace convertImg
{
	class Program
	{
		static void Main (string[] args)
		{
			if (args.Length != 1) return;

			Image img = Image.FromFile (args[0]);			

			Bitmap bm = new Bitmap (img);
			BitmapData bmd = bm.LockBits (new Rectangle (0, 0, img.Width, img.Height),
			                              ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);

			byte[,] ay = new byte[img.Width, img.Height];
			byte[] compactedAy = new byte[2 * img.Width];

			for (int y = 0; y < img.Height; y++) {
				for (int x = 0; x < img.Width; x++) {
					byte b = Marshal.ReadByte (bmd.Scan0, y * bmd.Stride + x * 4);

					if (b == 0)
						ay[x, y] = 1;
					else
						ay[x, y] = 0;

					Console.Write (ay[x, y]);
				}

				Console.WriteLine ();
			}

			Console.Write ("byte dispArray[] = {");

			for (int x = 0; x < img.Width; x++) {
				if (x % 8 == 0) {
					Console.WriteLine ();
					Console.Write ("\t");
				}

				compactedAy[2 * x] = (byte)(ay[x, 0] << 7 |
				                            ay[x, 1] << 6 |
				                            ay[x, 2] << 5 |
				                            ay[x, 3] << 4 |
				                            ay[x, 4] << 3 |
				                            ay[x, 5] << 2 |
				                            ay[x, 6] << 1 |
				                            ay[x, 7]);			

				compactedAy[2 * x + 1] = (byte)(ay[x, 8] << 7 |
				                                ay[x, 9] << 6 |
				                                ay[x, 10] << 5 |
				                                ay[x, 11] << 4 |
				                                ay[x, 12] << 3 |
				                                ay[x, 13] << 2 |
				                                ay[x, 14] << 1 |
				                                ay[x, 15]);

				if (x < img.Width - 1) {
					Console.Write ("{0,3} ", compactedAy[2 * x]);
					Console.Write (", ");
					Console.Write ("{0,3} ", compactedAy[2 * x + 1]);
					Console.Write (", ");
				} else {
					Console.Write ("{0,3} ", compactedAy[2 * x]);
					Console.Write (", ");
					Console.Write ("{0,3} ", compactedAy[2 * x + 1]);
				}
			}

			Console.WriteLine ();
			Console.WriteLine ("};");
		}
	}
}

The code above converts a binary image into an array of packed bytes. Every vertical line consists of 16 pixels (since we are using 16 LEDs) which is translated into two consecutive bytes in the resulting array. In general, the more LEDs you have, the more space it would take for each encoded line. As you can see, making the number of LEDs divisible by eight enables us to use full bytes for each encoded line. It is possible to use number of LEDs that do not fall into this pattern, but the resulting code would be either more difficult if full byte utilization is desired or leave some unused space between each line if lines need to start on byte boundaries.

Here is our test image.

Using the image above (in PNG format) as the input, the encoded image is outputted to the terminal (formatted as a C++ array)

byte dispArray[] = {
    0, 2, 0, 30, 0, 252, 7, 240, 31, 48, 252, 48, 224, 48, 252, 48,
    31, 48, 7, 240, 0, 252, 0, 30, 0, 2, 0, 0, 0, 0, 0, 0,
    15, 254, 15, 254, 6, 0, 12, 0, 12, 0, 12, 0, 0, 0, 1, 240,
    7, 252, 14, 14, 12, 6, 12, 6, 12, 6, 12, 6, 6, 12, 255, 254,
    255, 254, 0, 0, 0, 0, 0, 0, 0, 0, 15, 248, 15, 252, 0, 14,
    0, 6, 0, 6, 0, 6, 0, 12, 15, 254, 15, 254, 0, 0, 0, 0,
    0, 0, 0, 0, 207, 254, 207, 254, 0, 0, 0, 0, 0, 0, 0, 0,
    15, 254, 15, 254, 6, 0, 12, 0, 12, 0, 12, 0, 14, 0, 7, 254,
    3, 254, 0, 0, 0, 0, 0, 0, 1, 240, 7, 252, 6, 12, 12, 6,
    12, 6, 12, 6, 12, 6, 6, 12, 7, 252, 1, 240, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 3, 192, 12, 48, 48, 12, 32, 4, 76, 2,
    76, 66, 128, 33, 128, 17, 128, 17, 128, 17, 76, 34, 76, 66, 32, 4,
    48, 12, 12, 48, 3, 192, 0, 0
};

and is ready to be consumed by the Arduino side of the code.

Arduino Code

And here is the code on the Arduino side (I used Netbeans as my IDE, and you may need to change a couple of the includes if you are using the IDE bundled with Arduino. For more information, please see my previous article on this.)

As you can see, the dispArray is plugged in directly from the output of the encoder shown above. Note that here I assumed the image size is 16×100, if your image widths are different, then you will need to adjust the range of curPos accordingly.

The analogRead() statement in the main loop detects whether the IR transmitter and receiver are aligned. If so, the analog pin output would become lower as the transistor on the receiver side becomes saturated and when a pre-defined threshold is crossed, the decoding process begins. Depending on how fast the display spins, colDelay may need to be adjusted accordingly.

#include <binary.h>
#include <HardwareSerial.h>
#include <pins_arduino.h>
#include <WConstants.h>
#include <wiring.h>
#include <wiring_private.h>
#include <WProgram.h>
#include <EEPROM/EEPROM.h>

const int p1 = 0; //2;
const int p2 = 1; //3;
const int p3 = 2; //4;
const int p4 = 3; //5;
const int p5 = 4; //6;
const int p6 = 5; //11;
const int p7 = 6; //12;
const int p8 = 7; //13;
const int p9 = 8; //14;
const int p10 = 9; //15;
const int p11 = 10; //16;
const int p12 = 14; //23;
const int p13 = 15; //24;
const int p14 = 16; //25;
const int p15 = 17; //26;
const int p16 = 18; //27;

const int triggerPin = 5; //28;

const int colDelay = 8;

int ledPins[] = {p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13, p14, p15, p16};
int curPos = 0;

/** Image generator output */
byte dispArray[] = {
    0, 2, 0, 30, 0, 252, 7, 240, 31, 48, 252, 48, 224, 48, 252, 48,
    31, 48, 7, 240, 0, 252, 0, 30, 0, 2, 0, 0, 0, 0, 0, 0,
    15, 254, 15, 254, 6, 0, 12, 0, 12, 0, 12, 0, 0, 0, 1, 240,
    7, 252, 14, 14, 12, 6, 12, 6, 12, 6, 12, 6, 6, 12, 255, 254,
    255, 254, 0, 0, 0, 0, 0, 0, 0, 0, 15, 248, 15, 252, 0, 14,
    0, 6, 0, 6, 0, 6, 0, 12, 15, 254, 15, 254, 0, 0, 0, 0,
    0, 0, 0, 0, 207, 254, 207, 254, 0, 0, 0, 0, 0, 0, 0, 0,
    15, 254, 15, 254, 6, 0, 12, 0, 12, 0, 12, 0, 14, 0, 7, 254,
    3, 254, 0, 0, 0, 0, 0, 0, 1, 240, 7, 252, 6, 12, 12, 6,
    12, 6, 12, 6, 12, 6, 6, 12, 7, 252, 1, 240, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 3, 192, 12, 48, 48, 12, 32, 4, 76, 2,
    76, 66, 128, 33, 128, 17, 128, 17, 128, 17, 76, 34, 76, 66, 32, 4 of,
    48, 12, 12, 48, 3, 192, 0, 0
};

/**
 * delay ms * 0.1 milliseconds
 */
void delayMilliseconds(int ms) {
    for (int i = 0; i < ms; i++) {
        delayMicroseconds(100);
    }
}

void setup() {
    for (int i = 0; i < 16; i++) {
        pinMode(ledPins[i], OUTPUT);
        digitalWrite(ledPins[i], LOW);
    }
}

void loop() {
    if (analogRead(triggerPin) < 800) {
        for (int curPos = 99; curPos >= 0; curPos--) {
            byte b1 = dispArray[curPos * 2];
            byte b2 = dispArray[curPos * 2 + 1];

            for (int i = 0; i < 8; i++) {
                digitalWrite(ledPins[i + 8], (b1 & (1 << i)) >> i);
                digitalWrite(ledPins[i], (b2 & (1 << i)) >> i);
            }

            delayMilliseconds(colDelay);
        }

        for (int i = 0; i < 16; i++) {
            digitalWrite(ledPins[i], LOW);
        }
    }
}

POV Display in Action

Here’s the POV display in action. Using the programs in this article what you can show on this POV display is not just limited to characters, any binary image design that fits the image dimension (16×100 in this example) can be used. If you use different images for different frames you can even create your own POV animation.

16 LED POV Display in Action

Event Barrier

In the above example, a number of threads start at different times with the arrival of each clock event and wait till all threads finish running before the next clock event arrives.

Such barrier can be implemented using various techniques with different level of difficulties. One possible implementation can be found in Joseph Albahari’s Threading in C# where a monitor construct is used to wait on a synchronization object in the main thread after the worker threads are queued.

    lock (workerLocker)
    {
      while (runningWorkers > 0)
        Monitor.Wait (workerLocker);
    }

In each of the worker thread, the number of running workers is decremented when the thread finishes:

    lock (workerLocker)
    {
      runningWorkers--;
      Monitor.Pulse (workerLocker);
    }

And when the running number of workers reaches zero, the monitor exist the wait condition and the main thread continues.

The wait on the locker object forms a barrier in the example above. The method illustrated here is simple and elegant. It can be used to create the barriers needed to synchronize the clock event mentioned earlier.

Threadpooling might not always be the optimal solution though. While the number of threads in a thread pool is typically bounded to a few dozens, it may incur excessive context switching penalties if the majority of the threads are CPU bound. In a computation intensive situation, a fixed number of threads approximating the number of computational cores would be more appropriate (for instance, four threads on a four core CPU).

Here I will show another relatively simple way to construct a barrier where only a predetermined maximum number of threads can reach at a given time. To achieve such a ceiling, a global semaphore is used (in this example three concurrent threads can enter the semaphore):

public class Global
{
    public static Semaphore sp = new Semaphore(3, 3);
}

To simplify coding and shield the programmer from the synchronization mechanism details, I created an IEvent interface and an abstract EventBase as follows:

interface IEvent
{
    void OnEvent(int arg);
}

public abstract class EventBase : IEvent
{
    #region IEvent Members
    public void OnEvent(int arg)
    {
        Global.sp.WaitOne();
        Run(arg);
        Global.sp.Release();
    }
    #endregion

    public virtual void Run(int arg)
    {
    }
}

Users can simply implement their code that needs to block at the barrier by inheriting from EventBase and overridding the Run() method:

public class Event1 : EventBase
{
    public override void Run(int arg)
    {
        Console.WriteLine("Thread {0}, arg {1}", Thread.CurrentThread.ManagedThreadId, arg);
        Thread.Sleep(1000);
    }
}

To illustrate how this barrier implementation can be used considering the following code:

class Program
{
    delegate void MyEvent(int arg);

    static void Main(string[] args)
    {
        IEvent e = new Event1();
        MyEvent m = new MyEvent(e.OnEvent);

        List<IAsyncResult> l = new List<IAsyncResult>();

        for (int i = 0 ; i < 6 ; i++)
        {
          IAsyncResult r = m.BeginInvoke(i, null, null);
          l.Add(r);
        }

        foreach (IAsyncResult r in l)
        {
            m.EndInvoke(r);
        }
    }
}

In the code above, we simulate six concurrent threads that need to be synchronized. The multi-threading is achieved via asynchronous invocation. Because the capacity of the semaphore is set to be 3 in this example and if this code is run on a quad-core machine, you will see that the six asynchronous threads are run in two batches, three at a time. And all the threads are synchronized at the end. Here is the sample output:

Thread 10, arg 1
Thread 11, arg 2
Thread 6, arg 0
Thread 13, arg 4
Thread 10, arg 5
Thread 12, arg 3

Using this approach, the number of concurrently running threads can be precisely controlled by the maximum threads allowed in the semaphore.

So I followed the instructions given in these two blogs: Install Mono 2.4 on Ubuntu,
Building Mono and MonoDevelop from source on Ubuntu 9.04 64bit to re-install Monodevelop 2.0 and its dependencies from the source.

One thing you will need to pay special attention to is that for the monodevelop-debugger-mdb-2.0 to compile successfully, you will need mono-debugger version 2.4, instead of the latest version 2.4.1. Otherwise you will receive an error on Thread.AbortInvocation as there is a breaking change in mono-debugger 2.4.1 which added a parameter to the method (see mono debugger 2.4.1 change log).

For the type of applications that perform the same task over a large dataset for example,

for (int i = 0 ; i < num ; i++)
{
    //lengthy tasks
}

we could easily break down the for loop evenly into chunks that fit into the number of processors given. And for each of the subsets we could create a new process to perform the desired tasks. For instance, if we were to execute the above for loop in parallel on a duo-core processor, we could re-write each of the loops like the following:

//core 1
for (int i = 0 ; i < num ; i+=2)
{
    //lengthy tasks
}

//core 2
for (int i = 1 ; i < num ; i+=2)
{
    //lengthy tasks
}

By executing the application in separate processes, we eliminated certain race conditions that must be properly handled in a multi-threaded process and thus it is far easier to execute multiple processes concurrently when there is no inter-process communications involved than to handle multi-processing within a process via multi-threading. In a multi-process scenario, the operating system handles the concurrency issue behind the scene whereas in the multi-threading case the application developer must ensure the code regions are thread safe.

The following C# code shows how to dispatch tasks among different processes using asynchronous method invocations. Each process being invoked can identify its order by the extra integer parameter (line 36) passed to it. Thus we only needed to modify the target application to take this extra parameter into account (for instance, in a quad-core system, the first process would process items 1,5,9… and the second process would run items 2,6,10, and so on)

using System;
using System.Diagnostics;
using System.Threading;

namespace MPDispatcher
{
    class Program
    {
        public delegate bool CmdDlg(string fn, string args);

        private bool RunCommand(string fn, string args)
        {
            bool r = true;
            Process p = new Process();

            p.StartInfo.FileName = fn;
            p.StartInfo.Arguments = args;

			Console.WriteLine(String.Format("Starting {0} with arguments {1}. Thread ID: {2}", fn, args, Thread.CurrentThread.ManagedThreadId));
            r = p.Start();
            p.WaitForExit();

            return r;
        }

		public bool DispatchCommand(int numOfThreads, string args)
        {
            bool r = true;

            IAsyncResult[] ars = new IAsyncResult[numOfThreads];
            CmdDlg[] dlgs = new CmdDlg[numOfThreads];

            for (int i = 0; i < numOfThreads; i++)
            {
                dlgs[i] = new CmdDlg(RunCommand);
                ars[i] = dlgs[i].BeginInvoke(args, i.ToString(), null, null);
				Thread.Sleep(100);
            }

            bool finished = false;

			//wait till all processes are finished.
            while (!finished)
            {
                bool t = true;

                for (int i = 0; i < numOfThreads; i++)
                    t = t && ars[i].IsCompleted;   

                if (t) finished = true;

                Thread.Sleep(100);
            }

            for (int i = 0; i < numOfThreads; i++)
            {
                r = r && dlgs[i].EndInvoke(ars[i]);
            }

            return r;
        }

        static void Main(string[] args)
        {
            Program p = new Program();

			if (args.Length < 1 || args[0].ToUpper() == "-H") {
				Console.WriteLine("Usage: MPDispatcher {app name} {app args}");
			} else {
				int numThreads = 4; //use 4 threads
				string argStr = string.Empty;

				for (int i = 1 ; i < args.Length; i++) {
					argStr = String.Format("{0} {1}", argStr, args[i]);
				}

            	p.DispatchCommand(numThreads, argStr);
			}
        }
    }
}

The above code works under Windows and Linux (Mono).

Here is the output from the main routine:

(t1);t2;t3;
(t4);t5;t6;
(t1);t2;t3;t4;t5;t6;
(t1);t2;t3;t5;t6;
(t2);t3;t5;t6;

As you can see, tree t1 is formed by adding nodes t2 and t3 and the resulting tree is a two-level tree with t1 being the root. Similarly, t4 is formed by adding nodes t5 and t6. We then combined this two tree we just created, note how node t4 changed to a leaf in order to satisfy the two level property. In the next two output lines, we removed nodes from the two level tree. Note the last line: when we removed the root node, a leaf node (t2) is chosen as the new root of the tree.

Two level tree is extremely useful in capturing relationships in real world applications. For instance, if a is related to b and c, and d is related to e and f. Now we have two sets of distinct relationships (both are two level trees):

(a);b;c

(d);e;f

Now suppose that we also know c is related to e, what would the new relationship be? We know that a, b, c, d, e, f are then all related. But how do we capture this kind of relationships programmatically? As it turned out, we can use two level trees. In the example we just discussed, the relationship between e and c are captured via combining the two trees. And thus we get the following two level tree:

(a);b;c;d;e;f

Since related items are not necessarily limited to just tree roots (e.g. c and e are both leafs), we needed to be able to refer to a tree by either its root or its nodes and this is why in my previous post I mentioned that it would be a lot easier if we allowed referencing a tree by either the root node or its leafs.

One benefit of using a two level tree to capture relationships is that finding out the related items is a constant time operation since all of the related items are in a two level tree and no further tree traversal is required. In a large data set, this algorithm becomes very effective since no recursion is required.

A two level tree

There is no internal nodes in a two level tree. A two level tree has only a root node and a set of leaf nodes.

There are two types basic of operations we can perform on a two level tree: we can Add a two level tree to another two level tree and thus form a new two level tree or we can remove a node or a set of nodes from a two level tree.

To add a two level tree to another two level tree, we need to ensure that the combined tree remains a two level tree. when we say adding a tree, it means adding the "whole tree". But in practice, we really do not care which part of the tree we refer to. For instance, the above tree can be referred to by A (the root node), or by any of the four children (e.g. C). Thus there are four scenarios when adding two trees together, depending on how the tree is referenced. The following graph illustrate this:

Merging Two Two Level Trees

The above four different scenarios depend on how the tree is referenced (the shaded letters). We could mandate that a tree referenced by its root node and thus simplify the merging algorithm, but you will see in a later post that this approach adds more flexibility. The following code (C#) shows how the above steps are accomplished: 

The "remove" operation is pretty simple as well. Bascically, when the node to be removed is leaf, we simply remove it. When it is root, we select one of the leave nodes to be root and forming a new two level tree.

In the next post, I will show you one of the applications of a two level tree.

This change affected one of the applications I wrote a while ago. Since I stored all the configuration information inside the web.config file, I thought it was just a simple matter of changing the server from ‘server1′ to ‘server2′ and then I was done. Well, as it turned out, the result was totally unexpected.

Upon finishing the configuration change, I decided to test the application even though I was sure that everything would be working just fine. I was wrong, instead of getting the expected results, I got the following error message while trying to send email through the application:

Exception:
Could not access ‘CDO.Message’ object.

InnerException:
Exception has been thrown by the target of an invocation.InnerException:
at System.RuntimeType.InvokeDispMethod(String name, BindingFlags invokeAttr, Object target, Object[] args, Boolean[] byrefModifiers, Int32 culture, String[] namedParameters)
at System.RuntimeType.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParameters)
at System.Type.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args)
at System.Web.Mail.LateBoundAccessHelper.CallMethod(Type type, Object obj, String methodName, Object[] args)
at System.Web.Mail.LateBoundAccessHelper.CallMethod(Object obj, String methodName, Object[] args)

Since the only thing changed was the server configuration, I thought that it was definitely a configuration problem of the new mail server. But that could not have been the problem as many applications had already been migrated over and they were working just fine.

After a Google search, it became apparent to me that many people out there are having the same issue. But no information seemed to point to a definitive answer. And the tricks mentioned in various articles did not solve my particular problem. After some struggle, I found out that the culprit was actually my MailMessage.From field. Instead of an actual email address, I had always used some random names (e.g. system notification, etc.). And apparently, the spaces in the From field string would cause the error I mentioned above. And if I remove all the spaces in the From field, the exception goes away. Since the SmtpMail object is simply a wrapper for the Win32 CDO (Collaborative Data Objects), the error was poorly propagated back (like many of the errors in .Net as I mentioned previously. See 1, 2, 3).

Indeed, had the error message been something more clear like “Error: From field must not contain any space“, I would not have spent the good portion of the afternoon hunting for bugs (I still think it must have something to do with the new mail server’s configuration, since this code has been working with other mail servers).

For tasks that contain both intensive IO operations and CPU usages however it is desirable to parallelize the IO bound portion and the CPU bound portion of code so that maximum parallelism can be achieved.

An example of such IO and CPU intensive tasks is finding duplicate files in a file system. To find duplicate files, files or their hashes must be compared. In this post, I will discuss a couple of different approaches to achieve this using threading.

1. Sequential find
First let us take a look at how to find duplicate files sequentially (all the sample code can be downloaded from here):

Suppose we have a function that returns a list of files within a directory (see GetFileList in FileUtils). In function FindDuplicates, we loop through the list of files and calculate the hash of each file. The file name and hash are used together as a key to a hash table. Whenever the same key is indexed the count is incremented by one indicating a duplicate file is found. Here is what the code looks like:

2. ThreadPool
We could improve the performance by moving the hash calculation to separate threads.

The first approach to parallelize the task is to use a thread pool to queue the work of calculating file hashes. Since calculating file hash involves reading in the whole file (IO bound) and some calculation on the content, there is some inherent parallelism so using multiple threads could potentially improve performance.

The code below shows this approach:

3. Producer and Consumer
Even through the previous approach improved the duplicated file finding performance, it is still not optimum. For example, even though multiple threads are used to calculate file hashes, they are joined at the end before another set of calculation is queued. Also, the IO is operations are distributed across multiple threads and on some older hard drives handling IO in multiple threads can actually slow things down. An ideal solution would be to use a single thread for IO operation since IO is inherently sequential.

As it turned out, this problem is a classic producer-consumer problem. In this third approach, we will examine how to use the producer consumer model to solve the problem of finding duplicates. In this approach, the producer thread searches for files within a given directory. It opens each file as it encounters and read the content into a ProducerConsuerQueue. At the same time, the consumer computes the hash from the bytes stored in the queue and determines whether a file is duplicated or not. The code is slightly different than what you saw above as files are being added to the queue and consumed from the queue simultaneously.

If you run each versions of the code, you will see that this approach has most of the performance gains. It has better performance than ThreadPool approach since the IO thread always tries to keep the queue. Further, concentrate IO operations into a single thread helps the system optimize IO requests and thus increases the IO subsystem performance.

4. Putting it all together

In fact, using an extended control isn’t that much different than using the stock control thanks to inheritance.  For the most part, the extended control works exactly the same as the standard GridView control. I wanted to just touch base on a couple of places that a lot of people are having problem with.

First is the asp tag prefix issue. By default, all the custom controls (*.ascx) will automatically show up in the Toolbox strip within Vistuo Studio. So using the extended GridView is a simple matter of drag and drop. However, the default prefix for custom control is cc1, and depending on how many pre-existing custom controls there are on a page, it could be cc{num}. This becomes hard for replacing existing GridViews with the extended GridViewEx. To resolve this naming issue, we can register the assembly which contains the extended control within system.web->pages->controls tag in web.config:

With this prefix defined, we can use the tag ExtendedGridView wherever GridView is expected. By registering the prefix, we have also provided the solution to a second problem people often run into when using extended web controls–namely the ability to "skin" the control. With the extended control prefix registered, we can now create skins the same way as we do with other controls. For instance, my default.skin might look like:

In this way, the theming issue of the extended GridView is addressed.

The code for the code behind page to display the extended GridView is trivial:

As you can see,  the code is pretty much in line with what we would use for a typical GridView control. When the page is posted back, we set the DataSource and bind the data to the grid. Everything else happens transparently behind the scene. To illustrate the capability of GridViewEx, I included a CheckBox at the end of each row, whenever the check status is changed (CBSelected_CheckedChanged) we dynamically obtain the reference to the GridViewEx object (as an alternative, we could have used the actual control (GridViewEx1) instead of going through the trouble of dynamically converting if from the sender object since GridViewEx1 exists at compile time. Should the control be created dynamically, the code here will come in handy). Please pay special attention to how the checked state is determined. Because paging might be involved when the bound list has more entries than can be displayed, the current row index on the screen might not be the same as the row index in the list. To simplify coding, I created an Index function within GridViewEx to translate the selected row index into what is stored in the list.

The code samples provided in this series can be downloaded here.

Of course, there are a lot more fancy things you can do with the grid or with any objects in that matter. The only limit is your imagination…

See also

Code Download: ExtendingTheGridView.zip

3. GridViewPager

 Here is the code for GridViewPager:

As you can see, the pager control is quite simple. It communicates with GridViewEx control by the SelectedIndexChanged event. Depending on whether the state of number of items per page needs to persist between post backs, either ViewState or SessionState can be used.

See also

Code Download: ExtendingTheGridView.zip

2. GridViewEx
To make the binding as generic as possible, there are pretty much two ways. One way is to bind to an object collection of typed value objects. Since all value objects can be cast into objects, we could manipulate them generically. And because value objects are typically consisted of simple types, we could encapsulate all the operations needed for the GridView within the extended GridView itself.

Another solution is to utilize generics in .Net 2.0 and above. One obvious advantage is that the type information is not lost at any point of the object manipulation and thus a lot of small mistakes can be caught at compile time. This approach does have its draw backs. As you will see, because the purpose of generic types are to eliminate casting introduced by object containers, we have to change the extended GridView to accommodate the type of objects we are binding to the list whenever a new generic type is introduced. This of course requires the changing of the extended GridView control itself whenever we add a new generic type and thus might not be as convenient as the pure object oriented approach mentioned previously. However, weighing both the advantages and disadvantages I personally think this approach is more attractive.

So here, I will show you how to extend the GridView to handle generic lists.

GridViewEx inherits from GridView. We override the DataSource property so that we get a chance to inspect it before the databinding actually taking place. This gives us an opportunity to test whether the data source to be bound is empty. We use this information to decide whether to show the pager or not as you will see in a later post.

The initialization routine (OnInit) is overridden to setup some property values and hook up some events so that we provide row editing functionalities.

The RowEditing, PageSelectedIndexChanged and some of the other overridden functions are handled in a pretty standard way. To demonstrate how easy it is to add new functionality to the GridView, I added an event handler for RowDataBound. The purpose of the code for this RowDataBound event is to check if the CheckBox on a particular row is checked and if so, the background color of that row is changed. This behavior is particularly helpful for giving a visual cue to the end user which row of data he/she is currently dealing with.

When each row is created, OnRowCreated method is called, here we determine whether the row being created is the header or not and if so, we add an arrow to indicate the current sorting option depending on how the grid is currently sorted. If the row being created is a data row, we add some nice touch so that the data row becomes highlighted when mouse moves pass it.

We also have to override the OnSorting function to handle our generic lists. This is where we need to add the supported list types (see comments in code). For additional generic list types, we simply need to add three lines for each type of the list. While this might not be as convenient as using generic objects, all the changes are kept in a central place so that adding/removing types should not be an issue.

Because we are using customized objects as data bound objects, we also need to provider our own comparer so that the grid knows how to sort each column. This task is fairly easy and I modified the implementation from GridViewGuy. One thing you need to pay special attention is that since the comparison is done using property reflections, the property names are by default case sensitive (e.g. if you have a value property named Item, the binding in ASP.Net must be "Item" as well. "item" will give a run time error).  The code for the comparer is as follows:

See also

Code Download: ExtendingTheGridView.zip

For instance, the default data binding is limited to SqlDataSource, ObjectDataSource or a DataSet object (including DataView, DataTable, etc.). When building a large scale business solution, it is generally desirable to complete separate the data manipulation and business logic from the presentation layer and the most common solution is to employee a Model-View-Controller pattern.  This requirement makes SqlDataSource and DataSet bindings unsuitable. While ObjectDataSource does provide some level of isolation, the way business objects can be constructed is limited in order for these objects to be used as ObjectDataSource.

Most business objects incorporate the so-called Value Object data structure. Basically, it is a serializable class with the sole purpose of holding values and possibly other functionalities like data validation. Unfortunately, by default, GridView’s support for such value objects is poor (i.e. you can bind a value object to a grid view, but sorting is not supported by default). Over the years, I have seen many ad-hoc implementations to make GridView suitable for binding with customized value objects. Unfortunately, many of the solutions were implemented poorly and the implementation techniques vary from person to person, making it difficult if not impossible to maintain.

In the next few blog posts, I will show you how to extend the stock GridView in ASP.Net to make it suitable for using with value objects. The goal of extending the standard GridView is not only to add sorting capabilities, but to make the new component compatible with the standard GridView. I also intended to hide as much implementation details as possible so that the end users do not need to worry about the syntactical nuances as do with the standard GridView when binding to objects.

I assume that the development environment is Visual Studio 2005 with AJAX extension installed. The samples I show here can run within either Visual Studio 2005 (with AJAX extension) or Visual Studio 2008 without any modifications.

1.Defining the Value Object

As mentioned before, the reason we would like to define our own so-called value objects instead of using a DataSet to hold data is to ensure type safety. Since DataSet can hold any kind of objects, debugging becomes more difficult as error checking happens at run time not compile time. It also makes it easy to make mistakes as columns must be accessed by their names and any changes in column names could result in code broken at runtime somewhere in the UI.  Most importantly, binding a dataset directly to the UI violates the layered design principal, making the data and the presentation tightly coupled and difficult to adapt. The solution is to create a level of indirection by utilizing the so called value objects. Value objects are nothing fancy, they are merely a serializable class that provides the properties that represent the data (e.g. a row in a table) we are dealing with. For instance, here is what we might create as a value object for a simple phone book:

By creating a collection of such value object we just defined, we have a full representation of the result set returned from the database via a query or stored procedure. And via value objects, all the accesses are guaranteed to be type safe.

See also

Code Download: ExtendingTheGridView.zip

Recently, while working on some projects, I accidentally found out how easy it is to crash the Visual Studio .Net IDE and all it takes is literately one line of code.

Apparently, it seems that some developers at Microsoft must have forgotten the old design principal in software engineering: Never assume that a user won’t do certain things. From the surface Visual studio 2005 and 2008 look much fancier than their old cousin Visual Studio .Net 2003.  But neither of them can  handle the following line of code correctly without crashing:

s = s.Trim().Trim()….. Trim();  //will crash if the line length is too long

If you try to move the cursor to the end of the line in either VB or C#, the IDE will crash (here is the sample code for VB.Net and C#). And it seems that there’s little consistency between the different .Net language compilers (vbc and csc) in terms of what is the maximum line length it can handle. It seems that for C#, the IDE can handle line width (in terms of the code above) of roughly 20,000 before crashing while VB.Net could not handle even half of that. It is also interesting that the code line length limitation seemed to be more restrictive in Visual Studio 2008 than in Visual Studio 2005 (note, it could as well be the OS limitation since in my experiment, the Visual Studio 2008 instance was running under Windows Vista whereas the Visual Studio 2005 instance was running under Windows Server 2003).

Also, the error messages you received when running the same program within VS 2005 and VS2008 were different as well. For instance, if you are compiling the sample programs under Visual Studio 2008, the C# program gives a pretty accurate error message:

An expression is too long or complex to compile near ”

But the VB program could not be compiled at all, and you will get an internal error from the VB compiler:

The "Vbc" task failed unexpectedly.
System.Runtime.InteropServices.SEHException: External component has thrown an exception.
   at Microsoft.Build.Tasks.Hosting.IVbcHostObject.EndInitialization()
   at Microsoft.Build.Tasks.Vbc.InitializeHostCompiler(IVbcHostObject vbcHostObject)
   at Microsoft.Build.Tasks.Vbc.InitializeHostObject()
   at Microsoft.Build.Utilities.ToolTask.Execute()
   at Microsoft.Build.BuildEngine.TaskEngine.ExecuteInstantiatedTask(EngineProxy engineProxy, ItemBucket bucket, TaskExecutionMode howToExecuteTask, ITask task, Boolean& taskResult)  

Sometimes, you will get this error when trying to compile the VB program:

And here are the corresponding application events in event viewer (corresponding to the VB environment crash):

When trying to move to the end of the line, both VB and C# application would crash the IDE and give the following error:

And the Visual Studio 2008 IDE would then close.

Within Visual Studio .Net 2005, if you try to move the cursor to the end of the line in the C# version, the application simply closes (crashed) but no error message is given. But if you try to compile the C# code, you actually get the following error indicating that the line is too long:

An expression is too long or complex to compile near ‘VSCrash.Program.Main(string[])’  

The same code in VB would not even load without giving the following error:

And if you close the error dialog and try to compile the VB.Net code, you get this error message:

Visual Basic compiler is unable to recover from the following error:
System Error &Hc00000fd&
Save your work and restart Visual Studio.

Which is the same error message as in Visual Studio 2008. And the event viewer records the following information:

The VB.Net environment’s crashes are more troublesome. As indicated by the event viewer, it seems that the the exception chain went all the way to the kernel32.dll (note, if you choose the debug option when the VS 2005/2008 environment crashes, you can actually see the full exception chain along the execution path).

In my opinion it is one thing that the IDE has some design or practical limitations and it is totally another if the IDE simply crashes. Alas,
the only IDE that can handle this code correctly is Visual Studio .Net 2003.

This problem can be easily circumvented by using reflections. The following code snippet demonstrates how the current function’s fully qualified name can be obtained using reflection:

And reflection can also be used to obtain the values of member fields (e.g. public and private, etc.):

This is particularly helpful when logging exceptions, as the member variables’ states would be preserved and it might provide valuable information in determining the causes of the exceptions.

We need to be careful where we use the techniques mentioned above however, as reflection typically comes with pretty hefty performance penalties and thus must be used sparsely where logging is absolutely needed. When reflection is used within the exception catching block, performance is usually not a concern. But we must pay attention to ensure not to cause further exceptions within the exception handling code.

 A Google search on that error would turn out many hits. The official explanation can be found in this Microsoft knowledge base article. In my case, I have a solution file that contains multiple projects. Among them, one is a web project and one is a console application project. And the web project has a reference to the console application since it shares some functionality with the console app.

This setting has been working well until I added a web deployment project. After adding the web deployment project, I would get

Could not Find File ‘Microsoft.Windows.CommonLanguageRuntime, Version=2.0.50727.0

error at compile time. The easiest way to get around this limitation is to change the console application into a class library and create another project (in this case a console application) that simply instantiate the class and cal the original console app’s corresponding function. Using this technique, the web application no longer references to the console application but to a class library and thus the above compile time error would not occur.

After you fix this error, you might get a "could not find certificate error" when you try to compile. This is because when the deployment project was built the very first time (when the web project had a reference to the console application), the deployment was trying to build the console application portion as a ClickOnce application. If you look at the project settings (Project->Properties) for the console app (or class library if you had changed it to a class library project), you will see that the "Sign the ClickOnce manifests" option is checked. Uncheck this option will eliminate the certificate error. The check box might be checked but grayed out (at least this was what I saw), and if this is the case, you will need to manually remove the configuration from the .proj file (search for ClickOnce).

When a control resides within an AJAX update panel which in turn resides in a Content control on a page that uses a master page however, changing the control ID will result in the attached event not being fired (please note, if the update panel is placed in a page directly, e.g. no master page is used, the code iluustrated below works correctly). At least this is what I have observed in AJAX extension 1.0 for Visual Studio 2005. 

The the following simple user control for example  

When the above control (a dynamic button and a label) is embedded within a normal page (e.g. a non-AJAX page), the CheckedChanged event is fired whenever the check box is clicked. But when the same control is put into an update panel, whith the highlighted line of code included, the event will no longer fire.

This strange behavior had caused me some headaches while debugging through a rather complex web user control. So you will need to be extra careful when controls are placed in update panels. Due to limitations like the one I just mentioned a control that works in a normal page might not work properly in an AJAX enabled web page. The solution to this problem is simple, just remove the highlighted line and the event will fire again.

The symptom: while using a custom object data source (e.g. as simple as a value object, I have not tested other data sources yet, but a quick search on this issue on the Internet seemed to suggest that the issue I will be talking here happens with other data sources as well), it seems that changing SelectedValue after the control is bound would sometimes trigger the infamous ‘DropDownList1′ has a SelectedValue which is invalid because it does not exist in the list of items. exception.

This problem is rather frustrating as it does not always happen. It seems to me though that whenever the data source has non-consecutive values bound to SelectedValue, this exception will occur if trying to change the SelectedValue property after the data has already been bound.

After some experiment, I think I found a workaround. While changing SelectedValue triggers the exception, changing SelectedIndex does not. So, if the selected value of a dropdown list must be changed after data bound, change the SelectedIndex instead.

This workaround of course, adds a little bit complexity to the code as you must use some routine to determine the index of th value that is being selected. In most cases though, the added code is no more than a for loop that inspects the data-bound object.

I was doing some ASP.NET work and was using customized objects as data sources to bind to DataGrid. For sorting, I used the technique similar to what described in Sorting GridView Using IComparer. When I was testing my grids, however, I noticed that for most of the columns it worked perfectly, but for a few I somehow was greeted with the null reference error.

At first I thought that this was caused by the null items in those columns. After spending quite sometime digging, I realized that this was due to the case sensitivity issue in reflection. I had used the wrong case in the DataField of  the BoundField. And after I changed the spelling to match that defined in the value object, the problem was solved.

VB developers should pay special attention when using reflections, as in VB.Net the language itself is not case sensitive.

As a side note, even though it is possible to set the binding flag to make the member lookup case insensitive, it is not recommended as such approach tend to make the operation less efficient.

Like their WinForm counterparts, controls on a webform can be and often is hierarchical and thus requires recursion to traverse all the controls. But unlike WinForm controls, web controls do not uniformly share the Enabled property. This means that we would have to cast each control back to their own types before we could set their Enabled property. But in reality, this should not be a big concern as there are only a few primitive types of controls that are editable.

The following code snippet shows how to achieve this in C# (here we set the TextBox and Button controls to be disabled): 

In ASP.Net 2.0 and above, this code can be placed within the MasterPage and thus makes enabling and disabling controls at runtime even more trivial.

There are many articles out there discussing problems developers have encountered with File.Copy. While most of those problems are user problems, a few of those problems were simply mysterious to say the least. Here is the problem I encountered recently:

I have written a C# application that moves files from one NAS to local machine. The user under which the application is launched belongs to an AD group which has Administrative rights on both locations. The strange thing is that once a while (for example, one out of a hundred files) I would receive an Access denied exception saying that I do not have the rights to copy a particular file from the UNC location. With the same user credential as the the application however, I could manually copy the problematic file without any issues.

At first, I thought that it might have just been an transient error, but multiple tries would error out on the exact same file (all together, there were hundreds of files). Interestingly enough, this file has exactly the same permissions as other files that were successfully copied. Even though this file is a bit larger than the other files, my applications had successfully copied many larger files before…

One discussion thread I found suggested that it might be a delegation issue. But given that the application was able to move many other files except the once-awhile hitch-ups, I doubt that it was the permission issue.

The only thing I can think of is that the copy might simply have encountered a problem (e.g. disk problem, maybe), but somehow the wrong error message got returned. While, I wouldn’t be surprised if this is indeed the case, as I have encountered similar problems before.

Anyway, I will see if I can figure this out…