The Back Room

We have a Flash application that takes an input from a bar code scanner via serial keys in windows. Each time a ticket is read by the scanner, it’s translated into a keyboard input, and that is sent to the Flash application. The application can then analyze the string of characters, validate them through php with a Microsoft SQL database, and return whatever info we need. It’s a good example of integrating Flash with an external input device – the scanner. Although the scanner we’re using requires a few pieces of software to send keyboard command (because of the way it works with the touchscreen), most just emulate a keyboard input when an item is scanned.

Because what we’re making has to work hundreds of times a day, every day, we needed a way to do some solid stress testing. It wasn’t practical to have one of us sit in front of a computer with a ticket and scan it every 5 seconds to make sure that our application and the hardware were functioning properly. So, Charles came up with the idea to trigger a small servo motor to scan a ticket every XX number of seconds. In about two hours, he had the device you see below. It is pretty utilitarian, but  the main function is to just make sure the ticket scans properly, so it doesn’t have to be too sexy. The program is actually a modified version of the servo motor example that ships with the Make Controller that we have used in a few exhibits. The controller is pretty handy for connecting Flash to all sorts of external devices.

We’re not showing the actual program we’re testing, but instead you can see the input from the scanner appear on notepad. It helped us assure that our software could take the continuous input of thousands of scans a day and still be functioning well.

If you’ve never developed a touch screen application before, there’s a lot to think about that makes it different from a browser based or native application. It’s impossible to go through the full breadth of things to consider, but I figured posting a few helpful tips might be good.

Well Spaced and Big Button Sensing Areas
So, this does not necessarily mean giant buttons. Often that’s a good solution, since museum visitors are not just those familiar with computers but include many folks who don’t interact daily with computers. Really though, it’s about making sure that the sensing areas of buttons are forgiving enough to work with clumsy fingers, and to account for the different alignment people will get from looking at the screen from different angles. If you touch a screen from a low angle on what you see as the button area, and then change your perspective, that button area can be drastically different from above. A mouse is absolute. The button is the button. But with a fingers on a large screen, it can actually chance. So, large sensing areas for the buttons, and don’t put them too close together to account for different perspectives and clumsy fingers.

Consider Hand Position
When using a touch screen, a person uses their hand, thus creating a natural block for some of the screen area. With a mouse pointer, it’s small enough that unless an interface is very tiny, it won’t block anything when using it to select menu items, etc. But, with a finger, you’ve got a whole hand attached. So, imagine you have a screen where you have 5 buttons. Each button makes a change to a large image in the center of the screen. Although many web pages have navigation and other items at the top, this makes little sense in our scenario, because you want to see how your selection is changing the screen. If the buttons are at the top, you’ll have to move your hand away each time you press one to see the results. It may make more sense to place buttons along the bottom, or the right side. Although it may seem biased, you have to consider that most people are right handed, and if you have to favor one or the other, then choose right. The main thing to get from this one is just to think about where people’s hands are going to be and if they’ll be blocking items you need them to see when operating the screen.

Choose the Best Viewing Angle
This has more to do with fabrication than the software programming. You should make sure you consider who the audience is when choosing what angle and height to put the screen at. It seems like a basic enough idea, but there are many times when we haven’t been involved in the hardware or installation decisions of an institution, and they’ve installed a touch screen for children flat on a table that’s 36 inches high. How can a kid see that properly without standing on something? Or, a kiosk is supposed to take a photo of a person to put on postcard and email home, yet the camera is pointing almost at the floor, so if you’re over 3 feet tall, you’re bending awkwardly and contorting to get in the picture. Little things like this really affect the usability of a screen.

Hide the Mouse
I really don’t understand museum exhibits that have mice showing. I think people leave it there so that you see where you’re clicking and navigating. The thing is, you don’t want people to walk up to a touchscreen and see it as just a computer program running. You want them to walk up and see it as an experience that is something they can’t get at home. You can click on buttons at home, but you can’t touch a screen and move elements around the same way you can on a touch screen. Having the mouse pointer on screen breaks up your interface and reminds people they’re just looking at a computer screen.

Give Clear Direction
An interface must give users clear direction about what to do. Children will walk up to an exhibit and start banging away on the screen and they’ll figure it out. It’s the visitors that are new to touch screens and computer interactive programs that will just stand and stare at the screen, not sure what to do next. Things need to be clearly defined, and text must be readable. Sure, many people won’t read the directions, not matter how much you highlight them, but they need to be able to understand what to do by visual clues. There is a lot that can be written about creating touch screen interfaces, but my overall comment here is make sure people understand what to do.

I’m open to suggestions here, and I am not saying my comments are the be-all and end-all, but we’ve certainly learned a lot from working in this area for years. Some other miscellaneous items to consider:

- There are no rollOver or rollOut commands. Tooltips, highlights, etc. won’t work since it’s just about single touches.
- It’s good to use high contrast colors to make sure it’s visible to as many people as possible
- Choose a type of touch screen technology that works for what you need. (Capacitive, Resistive, SAW, etc.)

Anyone else have any tips?

If you are streaming mp4 files with Flash or using IIS with mp4 files, it is likely that you may have had a few issues.  The problems normally range from irregular video playback to no playback at all.

Mp4 Files and IIS

IIS 7 (and I assume 6) by default does not have mp4 (or even .f4v) files added to its known MIME types.  This will not allow Mp4 files to playback in your flash applications even if you have implemented them correctly in your code.  In order to fix this, you simply have to add a handler mapping for it.  I will demonstrate how to do this in IIS.

Assuming you have it installed, in Administrative tools -> Internet Information Services (IIS) Manager you will find an icon for MIME Types on the default website selection menu.  Click Add in the top left.  For File name Extension: enter .mp4 and MIME type enter video/mpeg.  After doing this, you should be able to playback the video assuming you don’t have a metadata issue, which I will describe below.

Adding an mp4 MIME Type in IIS7

The IIS7 Default Website Icons

Mp4 Files and Metadata With Flash

When doing video playback in Flash I have always been a huge fan of using Netconnections in conjunction with the Netstream class.  Although this is not as easy as using the FLVPlayback component, I feel like it gives me a lot more control of the video playback.  If you are familiar with using Netstream, you will know that there is a continual event called onMetadata that gets called when starting playback of a video.  I normally use this as a trigger to do things like resize the video.

With mp4 files generated by most video encoding and editing software, you will not receive the onMetadata event until the end of the file!  That means in order to even start playback, your users would have to download the entire video!  This is because the metadata of an MP4 file typically gets stored at the end of the file.  There are a free software tools that you can use to move this metadata to the front and allow the mp4 file to be streamed.

I am not going to make this a huge run-down of how to use XML in Flash, but instead share a few tips I’ve come across while incorporating it into my applications. If you’re looking for a place to start from the beginning and get the basics of using XML with Flash, then I’d go here to kirupa.com

Use a Configuration File

Since we mostly develop stand-alone exhibits for public spaces, there is some basic configuration information that we want in all of our programs and so each application has its own config XML with some standard parameters, and some that are specific to the application. The file we load at the start of each program has a config node, which will hold much of the items we use for every application. It might look like this …

<config>
<timeout>90</timeout>
<attractScreen>attract.swf</attractScreen>
<showMouse>0</showMouse>
</config>

It loads values that the application will use to determine the time the program waits to go to the attract loop, the location of the attract loop file, and whether or not the mouse should be shown at program startup. Another common thing that is done with our config XML is loading any dynamic text into the program, so it’s easy to change this text and it won’t have to be done by a programmer. There’s a text node and it holds a few different items that we may use.

<text>
<intro>Please Choose a category</intro>
<menuInstructions>Now, select a submenu below to start exploring.</menuInstructions>
<outro>Thanks for visiting, we hope you enjoy the rest of your day!</outro>
</text>

It’s important to pull that stuff out of the code and into easily configurable external files. It’s very common to have a client call and let you know that they think wording should be changed, or that the timeout happens too fast on a particular exhibit, and the ability for that change to be made by them is valuable.

Deleting an XML Node

So, to add a node to an existing XML instance in Flash, the most basic function is appendChild(). If you have two variables holding XML, and want to add one to the other, it could be as simple as …

myXML.appendChild(myXML2.copy());

This will vary depending on what you want to do specifically, but that’s the basic idea. So, if you want to delete a child from XML, you’d think you would just use a function like removeChild() or deleteChild(). Sorry, that’s not how it works. You actually have to use the fairly odd delete command. And, it doesn’t even get values fed like a regular function. If you had your XML that held photo nodes and wanted to delete the second item, it would look like …

delete myXML.photo[1];

It is not intuitive at all, and I am sure there is some reason it’s done that way based on the object model and available functions in the different classes, but it’s still a bit difficult to figure out.

Creating XML in Flash

For some applications, you may want to create a variable that holds XML for whatever reason. Now, if you’re just making a copy of or adding to some existing XML, it’s pretty straightforward. But, if you create the XML from scratch, there’s one important step that if you skip, will cause your XML not to work properly. You have to create a top level <data> node in it to hold your child nodes. If you don’t do that, it may not act as it should. So, if I am doing some sort of searching or sorting, and want to create a brand new XML that holds photo nodes from another XML document that fit certain search criteria, I can’t just say …

var newXML:XML = new XML();
newXML.appendChild(photoXML.photo[2]);
newXML.appendChild(photoXML.photo[5]);
//try to trace out the new XML, but you’ll see nothing
trace(newXML);

What you have to do is create a <data> tag when you make that new XML, so that it will have a node to append the children to. So, instead try …

var newXML:XML = <data></data>
newXML.appendChild(photoXML.photo[2]);
newXML.appendChild(photoXML.photo[5]);
//Now you’ll see the new XML nodes reflected
trace(newXML);

So, make sure you create that top level to allow items to be manipulated with code.

On more than one occasion, I’ve needed to allow smoothing of video in my Flash projects. Of course, it’s always best to compress video to being the size that you need, but if you want to allow any zoom effects on the video, allow the user to resize it, or maybe create a low-res version for use on the web, you’ll need to enable smoothing on the video. The performance takes a hit, since the computer playing back the content needs to do some math to make the video look good, but if you’re in a controlled environment, or have a reasonable idea of the audience’s install base, then smoothing is a viable option. I’ll mention two methods of playing back video – using the NetStream class, and using the FLVPlayback class. They each require a different method to enable the smoothing.

The NetStream is streaming at its most basic. You create a connection and stream the content. It takes more code to create the netstream (rather than using an FLVPlayback component that’s on the stage already), but it’s a little more clear how to smooth the video, since you just set the smoothing property of whatever your video is true. Here’s how you might stream something with NetStream, with the smoothing part added.


var video:Video = new Video();
var nc:NetConnection = new NetConnection();
var ns:NetStream;
//
//Call the function below
startVideo();
//
//This function adds the video elements
function startVideo(){
//Add the video to the stage
addChild(video);
//
//connect the netStream/netConnection with the video
nc.connect(null);
ns = new NetStream(nc);
ns.client = {onMetaData:ns_onMetaData, NetStatusEvent:ns_onPlayStatus};
video.attachNetStream(ns);
ns.play("YOUR_VIDEO.flv");
//
//Start with video being smoothed
video.smoothing = true;
}
//
function ns_onMetaData(_data:Object){
//put stuff here if you want.
}
//
function ns_onPlayStatus(e:NetStatusEvent){
//put other stuff here if you want.
}
//
//See the difference between video being smoothed and not smoothed by
//pressing "s"
stage.addEventListener(KeyboardEvent.KEY_DOWN, checkKey);
function checkKey(e:KeyboardEvent):void{
if (e.keyCode == 83){
video.smoothing = !video.smoothing
}
}

So, that’s if you use NetStream to add your video. If you have an FLVPlayback component on your stage, then you have to first extract the video stream from it, and then you can apply smoothing to that. You can’t just set a property of the FLVPlayback, since none exists for smoothing.

function smoothVideo(){
var myVideo = FLVPlayback_Instance.getVideoPlayer(0);
myVideo.smoothing = true;
}