PDC2008.Last;

I reported on the first half of PDC2008 the other day. Today I reflect on the last half of the conference.

Wednesday's keynote reviewed work being done by Microsoft Research, a division that's involved in pure research. They're an impressive group. Started in 1991, they have grown to over 850 PhD researchers in six locations around the world. Each summer they have 1000 graduate student interns working on projects. 25% of Computer Science PhD graduates in the United States have worked for Microsoft Research by the time they finish their degree. 15% of the division's budget is granted to universities. 30% of papers read at peer-reviewed conferences are submitted by Microsoft Research fellows.

One of the most interesting items they presented was a highly modified surface device. (A surface device is a multi-touch sensitive table top display.) This particular device was configured such that if you put another semi-opaque surface over it, something like a piece of tissue paper or frosted plastic, then a different image would be seen on the secondary screen. An example was a picture of an animal on the primary display. Hold a piece of paper over it and text about the animal was seen. Pretty cool stuff!

Another really cool piece of software technology presented in one of the sessions on Wednesday was the Concurrency Analysis Platform, or CAP for short. This is a low-level library used during the testing cycle that inserts itself between the application and the operating environment and replaces the thread scheduler. There are implementations for both Win32 and managed code. This provides a means for testing tools be built that explicitly control the scheduler. Think about this for a second. The scheduler is no longer non-deterministically interrupting your code, but rather can be controlled in a deterministic manner. Very, very cool.

CHESS is an automated tool built on this framework that analyzes a project and finds the points where schedule interleaving may make a difference. Without this analysis, it would have to run each thread with each line interrupted by the scheduler. This problem's scale is n^nk where n is the number of threads and k is the number of lines. With the double exponent, relatively small numbers cause this to explode to more permutations than there are estimated atoms in the universe. By analyzing the code, the problem space can be reduced to (n²*k)^c*nⁿ where c is a small number, like 2 or 3, and n and k are as defined previously. This becomes a much more tractable problem space.

Within this reduced space, the tool runs the code with each permutation of schedule interleaving to detect assertions, dead-locks, live-locks and data race conditions. When a failure is found, the problematic interleaving can be captured to directly, consistently reproduce the problem. Over my career, this could have saved me literally weeks of debugging time. During the session, they presented a couple real life case studies where this tool was used to find and fix very obscure, rare threading bugs in non-trivial code bases. They have also used it on several projects to find and fix bugs that have not yet been experienced. Obviously, using a tool like this throughout the development cycle has the potential to significantly increase the reliability of threaded applications.

Thursday marked the end of PDC2008. We had the same number of break-out sessions but ended early since there was no keynote speaker and only a short break for lunch. As I attended the sessions, it seemed like they had saved some of the best material for last.

First up are two new things coming out the research arm: CodeContracts and Pex. These two independent but complementary technologies are designed to improve code quality.

Inspired by Bertrand Meyer's Design by Contract work and rooted in the earlier Microsoft Research project Spec#, CodeContract is a .NET library to enforce method and class contracts. Calls to the CodeContract class are put at the beginning of methods to indicate requirements (pre-conditions) and expectations (post-conditions). As far as the build process is concerned, these become part of the method signature, so warnings and errors are emitted during build for cases where static analysis can detect calling code that violates the contract. Further, parts of classes can be marked as invariant, allowing the same checks on derived classes without re-declaring the constraints. Lastly, these same conditions can be placed on interfaces. This allows the checks to automatically be applied to all implementing classes without the class having to explicitly declare them.

Pex is a static code analyzer which works without CodeContracts but will use the additional information contained in them if they exist. It provides additional information about the warnings emitted by the CodeContract as well as the facility to automatically generate test cases based on method signatures and branches within the code that it has analyzed. Basically, it maximizes code coverage in the target class while minimizing the number of test cases needed for that coverage.

The last session I attended was a well presented introduction to the newish language F#. It is a functional language^[1] in the same way C++ is object oriented. The purpose of C++ is to write object oriented code while not excluding procedural code. Similarly, F#'s purpose is to write functional code while not excluding an object oriented paradigm. F# has been around for a couple years as a research language and has matured to the point where it's going to become a fully supported language in the near future.

The presenter started with a common, simple math problem and implemented it in F# using a typical procedural approach. It was basically line for line how it'd be written in C# with F# syntax. He then rewrote it in a slightly more mathematical way and then again using a fully functional approach.

Following this he presented a more realistic problem of downloading a CSV file from Yahoo's financial web site, slicing the data apart, running it through a function and emitting some results. He leveraged existing classes of the .NET framework for the data gathering and used a bit of F# code to do the data manipulation and glue things together. The first implementation had a single thread of execution. He finished with a highly impressive finale: with very minor changes to about three lines of code, he transformed this into a multi-threaded app that gave the same results in significantly less time.

All in all, this was a great conference with a lot of really interesting information. Videos of the sessions are available at microsoftpdc.com/.

1. One of the major tenets of functional languages is side-effect free methods. This eliminates many dependencies on order of execution, in turn making threading a much easier problem.

PDC2008.Length mod 4 = 2

I'm now at the half way point of my first Microsoft conference. I've attended many conferences in the past, but this is my first sponsored by the Redmond giant. For a Monday morning start, I flew into LA Sunday afternoon on a CRJ-700. I must say it was the quietest plane I've ever flown on. Sunday evening I had a delightful dinner with some family who live in the area.

Overall, the conference itself has been very good. Good presenters. Great information. Super food. The purpose of this conference is for Microsoft to cast their vision of the computing future. It is mostly about new technology that has not been released yet. This spans the gamut from next versions of current products to things in research to directions they think the industry is headed.

In Monday's keynote they talked about a new product called Windows Azure that they are positioning to be the OS for the web. It's made up of a number of coordinating services such as .Net Services, SQL Services and Live Services that they host in their data centers. The idea is they take care of the infrastructure and you use it to put your applications in. It reminds me of Amazon's offering on a larger, more ambitious scale. As they talked about it, I kept having two thoughts: really cool technology! And, how do they pay for it? I thought they were going to finish without answering the question. But at the very end, they finally claimed they haven't worked out all the details, but it'll probably be a subscription service based on traffic and needed reliability.

In Tuesday's keynote, Windows 7 was presented. It's the next version of the desktop client to replace Vista in 18 or so months. Each attendee was given a hard drive, as well as a CD, with the install for Windows 7 and a host of other software to support it. I tried installing it on a virtual machine from the CD but got errors about a file not being found. It installed flawlessly, and much faster, from the HD. So far I haven't had a chance to use it other than playing around with a few settings, but as a pre-beta release, it seems pretty mature. Someone claimed everyone at Microsoft is using it for their day to day work.

So far there have only been two down sides to the conference: the bumper music has been awful; it makes me want to do violence to some sound equipment. And secondly, the special event last night was at Universal Studios. I was there once before and sort of ambivalent about going, but the food was free and I figured I'd check out the changes they've made in the last 20 years. It was terrible. None of the rides held any interest to me whatsoever and the lines were long enough I didn't feel like wasting any of my life on them. The worse part though was they were prepared for Halloween. This meant they had people in costume that would try to scare people, get in their face and generally be highly annoying. In addition, smoke machines limited visibility to 4 feet in places and the music (using the term loosly) was loud enough I couldn't hear the people taking my order for food. Needless to say, I didn't spend more time there than needed to get dinner and leave. They did a great job at changing my ambivalence about going to outright refusal for future visits.

Never write software that fails silently!!

One of the tenets of software development is to fail early. The earlier something fails the less costly it is since you don't have as much invested as if it failed later. Visual Studio needs to learn this lesson.

I just spent the better part of an hour trying to figure out why a DeploymentItem attribute on a test didn't work. I had created some new files for the test, added them to the project and created an attribute that should have copied them over. The test didn't fail in the expected manner, because the code hadn't been updated to handle multiple files, but rather because there were no files at all. Setting a break point indicated that in fact the files were not being copied even though the were right there in the project.

After much frustration, I found through cygwin's grep the missing element: the files themselves needed to have their Copy to Output Directory property changed from Never to Every time. Why couldn't anything in the five steps leading up to the test execution have told me there might be a problem?

(And why did Explorer's Search fail to find the files that grep did?!?)

Adventures in firmware development: You learn something new everyday

Ethernet and TCP/IP, the core technologies for the internet, have been around long enough that typically you plug things in and they "just work." So I have been pretty puzzled the last couple weeks trying to figure out why this little Ethernet development kit I'm working with had a 30 second delay on initial start up. Internally, I could see the controller thought everything was up and running, it just didn't see any packets even though the link and data lights would light up.

I contacted technical support for the board and they couldn't reproduce the problem. Hmm. I hate those kinds of issues. After a number of messages back and forth, we found that I was going through the corporate router and they used direct connections to their test PCs. I changed my configuration to talk directly to my PC and the problem went away. Well, this was a step in the right direction.

So, the next step was to talk to our network admin to see if he had any ideas. He immediately remembered a configuration parameter called PortFast on our switches that by default was disabled.

PortFast is an option on Cisco switches to turn STP, short for Spanning Tree Protocol, on and off. I'd never heard of it before but this is what I learned: when you have multiple switches, routers and hubs on a network, it is possible to create a physical loop. Without STP, a physical loop can cause each router to forward packets to the other, ad infinitum, flooding a network with an increasing number of identical packets leading to a non-responsive network. It's basically a DOS attack on yourself. STP is something implemented by switches to detect physical loops so they can filter problematic packets, eliminating the cascades that can take down a network.

STP has five steps it goes through, two of which take approximately 15 seconds each. This is where my delay came from. When you have something plugged into a switch that you know cannot create a physical loop, you can enable PortFast. This bypasses these two steps, changing the start up time from around 30 seconds to almost instant.

The network admin hasn't had a chance to change this setting, but in a number of tests, I'm almost certain this is the problem I was having. I can reliably set things up to keep the router from knowing the board has reset. When I do this, it restarts in a few seconds; the expected behavior. In the end, the problem had nothing to do with the new board and everything to do with the existing router. It happens all the time with my PC too, it's just that the computer takes long enough to boot that the network has finished its protocol negotiation by the time the computer is ready to go.

For more detailed information:

• Cisco documentation: Understanding Spanning-Tree Protocol


• Cisco documentation: Using PortFast and Other Commands to Fix Workstation Startup Connectivity Delays


• Wikipedia's entry on Spanning Tree Protocol

Adventures in firmware development: New task, new tools

Idaho Technology, the company I work for, is experimenting with the idea of using ethernet for communicating with future instruments. Since our hardware engineers have zero experience with networking, and I have some from the PC side, I've recently been tasked with helping them out. I must say, it's been a delightful change of pace from the Windows desktop app development I've been doing of late.

Last time I worked on firmware was 20 years ago. Way back then, to change the the code on the device, I had to use a command line C compiler to generate .hex files on a PC. Then I used an EPROM burner to put it on a memory chip, physically take the memory out of the burner and put it in the device and finally turn it on. Debugging was a matter of examining the source code (and sometimes the generated assembly code) to figure out why it didn't work as expected. Changes to the code involved taking the memory out of the device, erasing the memory using UV light for 30 minutes (I think that's how long it took) and doing the whole, compile/burn cycle over again. The insides of the chip were completely opaque; there was no way to examine what was going on inside the device.

Now, the tools are so much cooler. I have a ethernet development system with an 8051-based micro-controller and programmer sitting on my desk. The programmer is plugged into the PC through a USB port on one side and to the 8051 controller through a ribbon connector on the other. There's an IDE I can use to program, compile and download the new firmware to the chip through. No power cycles on the device, no memory chips to erase and program. And the really, really cool thing: I can set breakpoints, evaluate registers and step through the firmware all from my PC. It's not much different than writing software for the PC itself.

Yeah, I know, these tools have been around for awhile. It's just this is the first time I've had a chance to play with them. Pretty fun stuff.

Welcome to the new home of Skylark Software

We're in the process of changing our servers. The old articles will be posted here as I have time to move them over and I have plans, time permitting, for new articles.

How can I tell the difference between a text file and a binary file?

Short answer: You can't. The question does not make sense. A "text" file type is a subset of all binary file types. Let's make the question more generic: how do I tell the difference between file type X and a binary file (where X is any chosen file type (text, MP3, wav, etc.)? Or to put it in a different domain, how do I tell the difference between oak and wood? Now does it make a bit more sense why you can't ask this question? If not, read on...

A binary file contains binary data. Binary data is composed of bytes with values ranging from 0 to 255. Therefore, every file is a binary file. A file of any given type is a binary file which has a specific structure. It is simply a matter of convention and the interpretation of the structure of the data which differentiates one type of file from another. Many file types have specific values which are expected at particular byte offsets within the file. If these values are not found, then the file is not of that type. Of course, just because the file contains those bytes at the expected locations does not ensure the file is of that type, it just gives an indication that it might be.

The text file type typcially does not have this type of structure, an exception being some of the unicode standards. Some of these have an expected byte value in the first two bytes of the file. If these bytes exist, then it's assumed the file is a unicode text file.

All this said, for some uses, it might be possible to define, in a limited way, what it means to be a text file. One definition would be if the file contains anything other than byte values 9 (tab), 10 (new line), 13 (carriage return) or within the range of 32 to 127, then it is not a text file. The downside to this is that it eliminates the use of accented characters and does not include other control characters which might be included in some applications. The definition could be expanded to include the accented characters in the range 129 through 255. However, this now includes most of the range of bytes and might cause some false positives.

The bottom line is every file is a binary file. Every other type of file is a matter of interpretation of the binary data.