A Word About Cell Phones

A word about cell phone game development…

Ah, the joys of working on another device that requires more low level knowledge of the underlying hardware then you care to know about. Qualcomm tried to ease things by coming out with their massive solution, called BREW, but this API leaves a lot to be desired.

A bit of a note before I start slamming BREW. I was using BREW version 2.1.3. There are severe differences between 2.1.3 and 3.1. Most notably is the image processing section, and the addition of several exposed features that, put simply, weren’t available in Brew 2.1.3. Additions like multiple output displays (for phones with more then one screen, which I believe is all phones now) and the ability to stream audio.

I finished working on cell phones over a month ago. I wanted to wait a while before posting anything because I wanted to clear my mind of all the bad taste left in my mouth due to the BREW SDK. Not to mention all the difficulties with the ARM9 compiler (RealView Compilation Tools directly from the ARM guys themselves)

The task was simple, at first. Build a basic sound processing system that we could use to animate characters on screen. Basically, imagine all those nifty visualizations in Windows Media Player, and put them on a screen less then 2 inches wide. In theory, it sounds good!

The first thing we had to do was to parse out the sound and get the frequency data. You would think that would be simple. All you have to do is decode the sound file if it is in anything other then pure PCM, run a simple fixed point version of a Fast Fourier Transform (FFT) and then viola; you have the frequency info right there. It doesn’t take a ton of processing, and it’s a nice feature to have!

The first thing I did was generate a fixed point FFT. This involves fairly remedial math involving taking Tailor series numbers and converting them to their fixed point equivalent and then doing all the calculations in fixed point space. Using 32 bit signed integers, you can easily compute the fixed point space using whatever granularity you chose for precision. I used 16.

This worked like a charm. I was able to get the frequency data directly from the PCM data and display that on screen. Then came the next part, and all hell broke loose!

So the first step is to take the encoded sound data and decode that to pure PCM. This turned out to be impossible! Brew simply doesn’t allow that in Brew 2.1.3. All they say is that this feature is purely provider dependant. Grrr… back to the basics.

Okay, so we can’t decode in hardware. Can we do it in software? Nope, because then we’d have to build our own sound streaming function. Sound streaming is something I’ll get to later, perhaps in another post, but the basics of sound streaming means that you take a signal and you output it to speaker.

Turns out this feature is not available in Brew 2.1.3 either.

All we can do is play sounds one at a time. In order to get the frequency data we have to open the file, decode the sound in software, extract the frequency info and store that on the phone, then close the file and reopen it using their media interface and then we can output the sound to speakers meanwhile loading the frequency info on the fly.

Project complete!

Then, the next project we wanted to do was to build a nifty little streaming video, camera to camera, cell phone to cell phone and PC, or any other device that can connect to the internet.

At last, we have all the tools and functions necessary for doing this. We have a sockets that we can use to connect to the internet and we have direct access to the camera data. Although rather slow access, we have access! Those were the only two ingredients necessary to complete this task. The compression and decompression were to be based on simple HAAR wavelets, then bitwise compressed, then zero length encoded, and then sent along their merry little way.

Sounds complicated? I know, I know. You must be asking yourself what the hell all this means. Put simply, it worked. You can download the video of it in action at Whatif’s website.

I did, however, run into one small stumbling block that was actually easy enough to get around. Turns out the under TCP/IP phones can’t “listen” for connections. This means your tiny little phone is incapable of being a server. This is easy to get around though, just use UDP!

I built up this whole UDP system based around a state machine of sorts. It would “listen” for other UDP packets coming in and then trade messages back and forth. It would attempt to TCP connect, mainly just for determining when a connection has been broken, and if not then it would continue to run.

Alas! It all worked. Everything from the compression to the UDP system, and even the camera data all worked as it should’ve!

In conclusion, I started typing this up to give a sense of where cell phone gaming is, in its current state. However, despite all my luck and charm, I’m simply not convinced that the cell phone will be replacing the PSP or the GBA any time soon. Sure, some phones are hardware accelerated, phones like the nifty, and now overly distributed, Razr V3 all have ATI chips in them. However, while working on the cell phones I simply couldn’t bring myself to try to code something 3D on a 2 inch screen. It just doesn’t make sense.

Yeah, okay, sure, streaming video on a 2 inch screen equates to streaming a very small image on a PC. We’ve had the tech to do that for years. Being able to view, real time, what your buddies are up to when they go to the strip club, well, that’s priceless.

Games, however, on a 2 inch screen is just stupid!

Have you ever tried to play something real time with that tiny little keyboard? Even games like Pac-man become increasingly difficult when trying to navigate around the maze and you repeatedly hit the wrong button.

While I was working on the cell phones, I did think of a user input system that would be kinda cool. If you took the orientation of the phone, using the camera and syncing frame by frame, you could, in essence, have a controller similar to the Wii controller with its multi axis design.

Now, all of a sudden games could become more interesting. Can you imagine riding on the subway with someone playing a game using this style of user input and watching them ducking and diving, swerving their phone to avoid the oncoming monster. Oh, that’d be sweet!

Next week… sound processing
-Ken Noland