At ARC I recently got to work on a multi touch screen application for a Smart Board. The application is for when guests come to visit the office, they use the touch screen to lookup the person they are here to see, then create a visitor pass for their visit. The application was built in Flash, which is a technology I have almost no experience with. However, they Flash guys were having some trouble getting the multi-touch piece working on the board. That’s when I came in.

I ended up building a TUIO bridge that is an overlay on top of their application. When a touch is recorded I am building up TUIO packets and flushing it to all connected clients on TCP port 3000. We received a version of a TUIO overlay from the guys at SMART but it didn’t work and the code was a procedural mess. I was committed to writing an OO friendly version of the overlay, and so far so good. There were some tiny things that we had to change to get this working. For instance our target machine was a 64 bit copy of Windows 7. Because of this we had to change some registry settings for the SMART board software. This was a pain to figure out but we got some decent help from a developer at SMART Technologies. Here’s a snippet of an email he from the SMART guy.

We replicated the problem on this end with my Windows 7 machine, so I have a fix for you. What was happening is that when you did a second contact, the software was feeding both contacts to the Windows 7 Gesture recognition engine. To bypass this.

FIRST:

Regedit:

HKEY_CURRENT_USER\Software\Classes\Virtual Store\MACHINE\SOFTWARE\Wow6432Node\SMART Technologies\SMART Board Drivers\Board1\IsDoGesture to 0

The key should already exist.

SECOND:

Run SMART Board Control Panel
- Under SMART Hardware Settings\Mouse and Gesture TURN OFF Enable Multitouch Gestures and Enable Single Touch Gestures.

THIRD:
- Restart SMART Board Service so it picks up the new settings. Under SMART Board Control Panel -> About Software and.. -> Tools -> Diagnostics -> Service -> Stop.

And then start it again.

The registry keys for other versions of Windows are:

Where X is a board number >= 1.

In order to make sure the overlay was functioning properly, I built a test tool that listens for all the xml that got flushed to TCP port 3000 and displays it in a console application. When building applications like this, it’s much more important to have useful logging rather then depending on a debugger.

The SMART Board API required me to listen to messages on the windows message pump then funnel those message up in to the SMART board sdk, which then gets processed and pumped back to me via event handlers. When a touch is received it gets a unique id assigned to it. The Smart board is only capable of handling two touches at a time, so only 2 id’s ever appear. The recorded touches are flushed down a TCP socket 30 frames per second. This means that as a unique touch is moving across the board, the last known x and y coordinates for that touch is what should be flushed down.

The key to capturing touches and drags was wiring up event handlers for the OnXYDown, OnXYUp, and OnXYMove events.

public partial class Shell
{
  [DllImport("user32.dll")]
  static public extern int RegisterWindowMessageA([MarshalAs(UnmanagedType.LPStr)] string lpString);

  int SBSDKMessageID = RegisterWindowMessageA("SBSDK_NEW_MESSAGE");
  ISBSDKBaseClass2 Sbsdk;

  public Shell()
  {
    InitializeComponent();
    Loaded += (o, e) =>
    {
      Sbsdk = new SBSDKBaseClass2();
      ((_ISBSDKBaseClass2Events_Event) Sbsdk).OnXYDown += (x, y, z, pointer_id) =>
      {
        TouchTrigger.fire(new Down(pointer_id, x, y));
      };
      ((_ISBSDKBaseClass2Events_Event) Sbsdk).OnXYMove += (x, y, z, pointer_id) =>
      {
        TouchTrigger.fire(new Down(pointer_id, x, y));
      };
      ((_ISBSDKBaseClass2Events_Event) Sbsdk).OnXYUp += (x, y, z, pointer_id) =>
      {
        TouchTrigger.fire(new Up(pointer_id));
      };

      var handle = new WindowInteropHelper(this).Handle;
      var int_handle = handle.ToInt32();
      Sbsdk.SBSDKAttachWithMsgWnd(int_handle, false, int_handle);
      Sbsdk.SBSDKSetSendMouseEvents(int_handle, _SBCSDK_MOUSE_EVENT_FLAG.SBCME_NEVER, -1);
      HwndSource.FromHwnd(handle).AddHook(new_message);
    };
  }

  IntPtr new_message(IntPtr hWnd, int Msg, IntPtr wParam, IntPtr lParam, ref bool Handled)
  {
    if (Msg == SBSDKMessageID && Sbsdk != null) Sbsdk.SBSDKProcessData();
    return IntPtr.Zero;
  }
}

The TouchTrigger will fire off the touch to any listening observers.

public class TUIOProtocol : Protocol
{
  public TUIOProtocol(double screen_width, double screen_height)
  {
    this.screen_width = screen_width;
    this.screen_height = screen_height;
  }

  public void record(Touch touch)
  {
    touches[touch.id] = touch;
  }

  public void publish_to(Connection connection)
  {
    connection.send(build_for(connection));
  }

  Serializable build_for(Connection connection)
  {
    var xml = new Xml();
    xml.add("<OSCPACKET ADDRESS='{0}' PORT='{1}' TIME='{2}'>", connection.ip, connection.port, create_time_stamp());
    foreach (var touch in touches.Values)
    {
      touch.append_header(xml, screen_width, screen_height);
    }
    xml.add("<MESSAGE NAME='/tuio/2Dcur'>");
    xml.add("<ARGUMENT TYPE='s' VALUE='alive' />");
    foreach (var touch in touches.Values)
    {
      touch.append_footer(xml);
    }
    xml.add("</MESSAGE>");
    xml.add("<MESSAGE NAME='/tuio/2DCur'>");
    xml.add("<ARGUMENT TYPE='s' VALUE='fseq'/>");
    xml.add("<ARGUMENT TYPE='i' VALUE='{0}'/>, sequence.next());
    xml.add("</MESSAGE>");
    xml.add("</OSCPACKET>");
    return xml;
  }

  double create_time_stamp()
  {
    return DateTime.Now.Subtract(reference_time).TotalMilliseconds/1000000000;
  }
}

The TUIOProtocol stores updates the recorded touch for each unique touch id. When the timer elapses it tells the TUIOProtocol to publish the changes to a TCP Connection. As the xml is built it tells each touch to append it’s own header and footer to the xml. If it’s Down touch then something gets appended. If it’s an UP touch then nothing gets added.

public class Down : Touch
{
  public Down(long id, double x, double y)
  {
    this.id = id;
    this.x = x;
    this.y = y;
  }

  public long id { get; private set; }

  public void append_header(Xml xml, double screen_width, double screen_height)
  {
    xml.add("<MESSAGE NAME='/tuio/2Dcur\'>");
    xml.add("<ARGUMENT Type='s' VALUE='set' />");
    xml.add("<ARGUMENT Type='i' VALUE='{0}' />", id);
    xml.add("<ARGUMENT Type='f' VALUE='{0}' />", plot_x(screen_width));
    xml.add("<ARGUMENT Type='f' VALUE='{0}' />", plot_y(screen_height));
    xml.add("<ARGUMENT Type='f' VALUE='0.0000000' />");
    xml.add("<ARGUMENT Type='f' VALUE='0.0000000' />");
    xml.add("<ARGUMENT Type='f' VALUE='0.0000000' />");
    xml.add("</MESSAGE>");
  }

  public void append_footer(Xml xml)
  {
    xml.add("<ARGUMENT TYPE='i' VALUE='{0}' />", id);
  }

  double plot_x(double screen_width)
  {
    return x/screen_width;
  }

  double plot_y(double screen_height)
  {
    return x/screen_height;
  }

  double x;
  double y;
}

The UP touch does not append anything. It just symbolizes a gesture where the user has lifted their finger off of the touch surface.

public class Up : Touch
{
  public Up(long id)
  {
    this.id = id;
  }

  public long id { get; private set; }

  public void append_header(Xml xml, double screen_width, double screen_height){}

  public void append_footer(Xml xml){}
}

And that’s all folks. Developing an application for the SMART board has been fun. I would love to get an opportunity to build a full blown WPF app on either the SMART board or the SMART table. For more info on the SMART Developer Network.

Download Source

comments powered by Disqus