Slowly moving things together – the long tail of Beyond the Keyboard

Posted on March 14, 2011


This blog started as a alternative to my usual work: programming stuff in Flash. The first post covered “Building a pressure sensitive Touch Surface” and successive posts investigated QR codes, Android phones, Android hardware and hacking hardware with the Android OS.

Previous to that I looked at using AR toolkit in Flash (FLARtoolkit) to build tables like the Reactable (as the software made open to public uses OSC – data streams for musical purposes and delivering a lot of overhead – and apparently still makes things crash).

Right now I am building several prototypes of multi-screen and tablet- and touch table surfaces to test their use in real life.

My main belief is that our environments will become smarter and smarter. The main goal and purpose is to look at the many different ways we can interact in a simple and direct manner with that environment and pick out the ones that work best in a specific situation.

In the next six months I will have more and more time to finalize the concepts I have been writing about until now. Below you will find a short summary that mostly covers the hardware parts.

The long tail of Beyond the Keyboard

The following things are part of the long tail of my work on Beyond the Keyboard:

  1. Embedded systems and low cost, low power domestic servers – using whatever is there to build low power systems to store data and respond to users and events in an (domestic) environment
  2. Use of alternative input using AR Markers, QR codes and printed material – for quick, direct and tangible interfacing with home systems
  3. Use of touch screens and pen input – for direct (eye to action) communication with systems
  4. Smart phones and Tablet computers – for “direct” and “remote controlling” systems and environments
  5. XBee, Arduino, WiFi modules – allowing for wired and wireless sensors- and systems setup through an environment
  6. Post PC computing – moving away from the holy quartet of: “computer, keyboard, mouse, screen” and offering alternative ways to working with and communicate to software
  7. A possible new personal route to follow – It might be that I find an entire new calling in building stuff with hardware, provided I find the people and partners who have room for someone like me and projects that really make a difference to something and someone. Hacking today’s hardware (webcams, Netbooks, Kinect cameras, ARM boards, Touch panels) is very rewarding due to both the low prices and high tech stuff you can get for that.

One of the things your will not find in this list is Augmented Reality. Regardless of the sweetness of the concept, it is not in my personal list of relevant things.

May 1st, 2011 – the end of a 10 month Flex project for Reed Elsevier

With the end of April also comes the end of a 10 month non-stop project as one of the lead-developers for an application that – as a spin-off effect helped me clean up all the final ropes and attachments from the 2006 start-up.

The route I am considering – back to hardware

I am considering taking a full year dive into one of the established industries regarding either Climate Control Systems or domotica as a systems designer to increase my expertise.

Climate Control using embedded systems systems go back at least 30 years, so a lot has been established already, but no doubt a lot can be done still.

Domotica is a bit younger and – according to my feeling – currently is most successful from companies who started with home security systems. It is a second choice to consider, unless it is for people who are benefiting and real problems are solved, like designing smart houses for people with a physical disability.

Roomware Project – The Netherlands

Moving back to hardware started with the Roomware Project. The Roomware Project, The Netherlands started somewhere around 2005 / 2006 anmd was hosted first in my office. The Roomware project is no more since May/June 2010 – at least in the shape it was. After an emotional response and misfired post from my side it became clear that the goals and purposes the original three founders (James, Robert and Tijs) had were not met and the project has been terminated.

As I still think the name rocks I hope someone will pick it up again, create a clear vision around it and trademark it.


I have been considering continuing something like the Roomware workshops, but other priorities took over. A second factor is that I suck at organizing that kind of stuff when I have to do it alone. Right now it is not an option until I find more traction for the things I am doing and building.

The commercial side of hardware hacking

I think hardware hacking is becoming more and more relevant. I think it will take about 5 years before we see the Internet bleed into reality through displays and public spaces. Not only via services like and Foursquare but also via systems and venues responding to your presence, using whatever you carry on you to identify you.

Smart and responsive environments

With hardware getting cheaper, less hungry for energy and more powerful each new year, it is becoming more and more doable to build smart environments with home brew solutions. Besides being “smart” by sensors and response patterns (“Its getting cold, nobody is in the house, lets not heat the room”) we also want to be able to connect to it in a simple and direct manner.

And – in my case – preferably without having to use the old-school PC-type of interface with mouse and keyboard.

Scrablet, modified Netbooks, projectors and the Scrable

The Scrablet is one of the interfaces I have in mind. Modified Netbooks, using projectors and big screens used as table surfaces (Screen + Table = Scrable) another.

The main goal is to see how I can merge different things together so that hooking into- and interacting with your domestic environment becomes a natural thing.

Post PC computing

Apple coined it with the launch of the iPad 2. Who-ever and whatever before and after, I do not care. It is a good term to coin. Let me define my point of view on “Post PC”:

  1. Getting rid of the concept of Windows – The concept of Windows sucks. Especially when you move to Touch computing. Application Windows are like clutter floating around on your desk. The only moment they are useful is when you want to drag and drop something from “A” to “B”.
  2. Getting rid of the mouse – The mouse is OK. It is quite a precise tool to point at things on your dekstop and combined with the keyboard it rocks. For dragging and dropping, the mouse rules over touch. For quick interactions, making drawings and clicking on stuff, touch defeats the mouse. In a touch based environment without windows, the mouse is not needed.
  3. Getting rid of the keyboard – Like with the mouse, the keyboard has its use, but when most of the stuff you do is manipulating stuff on screen, the keyboard is a secondary item. The concepts of the Scrablet and Scrable do have a keyboard as typing still kicks ass, but it is a removable bluetooth keyboard.
  4. Making the screen the center point for interaction – That is where it all happens. To allow people to touch it, draw on it, interact through it with their fingers is the new shit. (Although touch-interfaces are already on the market since the 1980’s).
  5. Connecting to devices in rooms via tablets and smart phones – Using WiFi to connect to local servers, tablets, smart phones and laptops can connect to the services being exposed to check on what is going on and set and reset specific aspects (like lighting and temperature).

Low power ARM based processors

Silently – with over 90% market share – ARM is the dominant factor in mobile phones. The low power, and increasingly more powerful Systems on a Chip are now slowly bleading into the consumer market with low power Do It Yourself Boards for prices below 200 euro, capable of running Linux and outputting info to 2 screens in HD.

The main issue on Domotica and smart environments is the power consumption of the “brains” of it all. Where a normal desktop easily consumes 200 Watt, your energy bill will cancel out any benefit you might have from a “smart house”.

Systems based on ARM processors consume about 3 to 7 Watt when active. The energy footprint is hardly noticeable and with Linux they can operate as full blown Home Servers.

XBee, WiFi modules and Arduino

To connect to physical devices, three main options are available now:

  1. XBee – Offering a “closed” wireless network to send data to and from XBee devices. The main benefits are: low price per module, relatively easy configuration, ready to go for sensors and devices and low power consumption
  2. WiFfi Modules – higher prices per unit, up to 90 euro. Similar to XBee. Main benefits: can send data directly to any IP-number and receive data from any location without a “man in the middle”
  3. Arduino – is in the basis a PLC made for the consumer market. You can program the Arduino to do autark actions or connect it to a wireless solution or the USB port of your computer.

You also have USB devices simply offering ports for analog and digital measuring and sending signals out to flip switches and instruct other hardware.

Android, Linux and hardware

Android is not what I expected it to be for hardware hacking. Serial ports and basic USB peripherals like Arduino and XBee are not supported yet. So – for hardware hacking – the choice will move towards ARM + Linux and towards low end Netbooks. I like the ARM solution best for its solid state / no moving parts required aspects. It means that nothing can burn out due to dirt and failing mechanical parts (the cooling fan).

For ad-hoc installations, modified Netbooks rule at this point. The hardware is dirt cheap, also due to the insanely low retailer margins (below 10% of the factory price) and with the screen and all ports is quite a complete solution. When keeping a keen eye on special offers you will find Netbooks below 200 euro. And if you like to- or want to modify stuff, Netbooks are quite forgiving.

The world of domotica and embedded stuff

What I am writing about and what Roomware partially covered is nothing new in the world of climate control. Think flower bulbs, cheese factories and greenhouses here, houses of disabled people and people of age. For anything organic, temperature and humidity are two key factors for stuff to rot down or become the beautiful product you can plant in your garden or put on your sandwich. For disabled and old people having a panic-switch (“Help! I feel down and can not get up again”) and movement sensors (“House 235 to Central: no movement since 8:30 and subject still in house, please take action”) can be the difference between a horror story and something you can joke and / or brag about at birthday parties.