Leap Motion Impressions, Input Sanitation, and 3D Gesture Ideas

Pros:

• For the most part, does what it claims it does, and gives you extremely precise, fast tracking of fingers.

Cons:

• Really hates thumbs for some reason.
• Has a lot of trouble with pens or other implements.
• Fingers must be separated.
• Fairly easy to get positions that break the camera because it can't see the fingers.
• No one has any idea how to write software for it.

I just got my Leap Motion device today, and for the most part, I like it. It does most of the things I expected it to do. It seems like something that could become very useful down the line. Unfortunately, right now, I wouldn't recommend getting one if you wanted something that's more than a toy.

This isn't entirely the fault of the device itself, but more a combination of driver limitations and boneheaded software that doesn't know what it's doing. The device is so new, no one knows how to properly utilize it, so almost all the apps that exist right now are either garbage or ridiculously sensitive to your setup. There are, however, a few more serious problems that should be addressed by the Leap Motion team.

First of all, the driver hates thumbs. It seems like it's trying to edit them out when they aren't positioned like a finger, which makes any sort of pinching gesture impossible to do. In addition, it doesn't like it when you try to use a pen, despite the fact that it's specifically designed to let you do so. Both of these problems seem more like driver analysis problems, so I expect they'll be corrected soon. I know these aren't hardware limitations, because the camera can see the stupid pen, the driver just has to recognize it as a tool and disregard the hand holding it, which is really doesn't want to do - It's obsessed with hands. I should call it the Lyra driver from now on.

A more fundamental and concerning issue is the ease in which I can assume a position that blocks the camera's view of some or most of my fingers. Pretty much anything that involves my fingers being vertically aligned breaks the driver, along with everything from a roughly 90 degree to 135 degree angle, pointed away from the device. This appears to be an inherent issue with the device itself, and its the same problem the kinect would have, because the camera's view of the fingers gets blocked. How much more effective would a leap motion device with two cameras, one on either side of the monitor, be? Or just a wider camera angle (since it's using dual emitters anyway to get depth information). As it is right now, it's cute and futuristic looking, but any minority-report-esque gestures completely break it.

A much more unexpected issue is the fact that the device makes no attempt to deal with fingers that are held together, despite the fact that the shape of two touching fingers is fairly easy to figure out. If you've been tracking one finger and suddenly it meets up with another one and you end up with one really wide "finger", it should be trivial to send this information as a "double finger" pointing, because this is an incredibly useful gesture (see below for details). It's not like the finger is going to change into a hippopotamus all of a sudden.

3D Gestures
Most of the other problems are software oriented, not hardware or driver related. The simple fact is that no one has any idea how to properly utilize the device. Most apps seem to assume there's only going to be one finger pointed at the screen, and will get confused if you have any other fingers hanging off even if they're not pointed at the screen. The Leap Motion driver gives you a rough estimate of what it thinks the hand orientation is, and using this information its fairly trivial to figure out which finger someone is pointing with, even if their other fingers are only slightly below that finger. The camera is on a flat surface at a 90 degree angle from the screen, which means the finger you want is almost always going to be the finger that is the "highest" in relation to the hand orientation.

The rest of the apps either don't do much processing on the raw data, or do an incredibly bad job of it. The touch screen recreation had this horrible tendency to start sliding off, and the cursor would sometimes jump around. I knew it shouldn't be doing that because I've been watching the diagnostic information and it's just getting confused by ghost fingers popping in and out. Guess what? A finger can't teleport. If you've been tracking one finger, chances are you should keep tracking it no matter what else happens.

In addition, no one's really come up with effective 3D gestures. All the apps try to use the 3D plane by turning it into a virtual touchscreen and it doesn't work very well. Instead, we should be using gestures that are independent of a 2D plane. The apps need to be paying attention to the angle the finger is pointing in instead of trying to rely on moving your entire hand around. It seems like everyone is using the position information instead of the angle information, which the driver provides. If I'm pointing up and to the right, my cursor should be in the corner of the screen no matter where my hand is.

Furthermore, no one seems to be trying very hard to deal with noise. When the driver starts losing precision, you can tell because the angle information or position information will start to jitter. The more jittering, the less accurate the reading is. A simple way to quantify this is by calculating the variance of the recent data. The higher the variance, the less accurate your reading is. You should then scale how heavily you average out the points based on how much variance there is in the data. This allows you to avoid destroying precision while preventing the cursor from exploding when you lose accuracy. There's a lot of algorithms dedicated to solving this exact problem, so it's kind of ridiculous that no one is using them. In a more specific scenario, humans do precise manipulations with their fingers, not their hands. Small pertubations in the input data that occur in the hands should be removed from the finger positions, because it's just the hand trembling, not the actual finger.

Instead of trying to recreate "pushing a button" in 3D, we should use a gesture interface that lets you move the cursor around with your index finger, and then when you bring up your middle finger against your index finger (so you are now pointing at the screen with two fingers), this counts as a "click" or a mouse-down. You can then drag things around with two fingers, or separate them to generate a mouse-up event. To right-click, bring your two fingers together again, and then rotate your hand clockwise. This could be combined with a more traditional "pushing a button" gesture for easier clicks, but provides a much more robust way of moving things around that isn't awkward or difficult to use.

Since the driver currently doesn't let you track two fingers held together, this same behavior can be emulated by simply keeping track of two fingers and detecting when both of them accelerate towards each other until one of them vanishes. On this same note, because fingers can drop out of tracking, programs can't rely on the driver to be perfect. If you lose track of a finger, it's not that hard to guess where it might be based on the orientation of the hand, which is necessary if we're going to have good gesture recognition. The recognizer needs to be able to interpolate likely positions of lost fingers if it's going to make an effective guess about what gesture was intended.

Leap Motion is really cool. Leap Motion is definitely the future. Unfortunately, we haven't gotten to the future yet. We need a better software framework to build gesture interfaces on top of, and we need a new standard set of gestures for 3D that aren't simply 2D ripoffs. Incidentally, I'd love to try building an open-source library that addresses a lot of the input sanitation problems I outlined in this article, but I'm a bit busy with the whole "finding a job so I don't starve to death" thing. Hopefully, someone else will take up the cause, but until they do, don't expect to be very productive with the Leap Motion.

Aurora Theory Released!

Aurora Theory has been released! Buy it on bandcamp for $9, or $10 on iTunes, Amazon, and Google Play. The album is also available on Spotify, last.fm, other online radios, and can be previewed on YouTube.

Aurora Theory has been 4 years in the making, a compilation of all the songs I managed to make in the middle of attending university. The earlier songs have been extensively improved, and all songs have been remastered for the album's release. This album represents nothing less than the highest quality work I can possibly manage at this point in my career. I've acquired a fair number of fans over the years, and I want to thank you for supporting my musical endeavors so far.

Track List:
1. Soar (Original Mix) [5:49]
2. Aurora Theory (Redux) [4:10]
3. Cosminox [5:41]
4. Tendril [5:32]
5. Birefringence [3:57]
6. Critical Damage (Extended Mix) [3:45]
7. Starstruck [4:22]
8. Dream Among The Stars [4:06]
9. At The Nationals [4:02]
10. The Cloud River [5:38]
11. The Piano And The Cello [3:53]
12. Snowflower [5:53]
13. Spiral Nebula [5:44]
14. Next Year [5:31]

Help Order My Album!

And the experiment is over! Here's the final album ordering.

What I Learned In College

"In times of change, learners inherit the earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists." ― Eric Hoffer

Yesterday, the University of Washington finally mailed me my diploma. A Bachelor of Science in Applied Computational Math and Science: Discrete Math and Algorithms. I learned a lot of things in college. I learned how to take tests and how to pinpoint exactly what useless crap a particular final needed me to memorize. I learned that math is an incredibly beautiful thing that has been butchered so badly I hated it all the way until my second year of college. I learned that creativity is useless and all problems have one specific right answer you can find in the back of a textbook somewhere, because that's all I was ever graded on. I learned that getting into the CSE major is more about fighting an enormous, broken bureaucratic mess than actually being good at computer science. But most of all, I learned that our educational system is so obsessed with itself it can't even recognize it's own shortcomings.

The first accelerated program I was accepted into was the Gifted program in middle school. I went from getting As in everything to failing every single one of my core classes. Determined to prove myself, I managed to recover my grades to Bs and Cs by the end of 7th grade, and by the end of 8th grade I was back up to As and Bs. I didn't do this by getting smarter, I did it by getting better at following directions. I got better at taking tests. I became adept at figuring out precisely what the teacher wanted me to do, and then doing only that, so I could maximize both my free time and my grades. By the time I reached high school, I would always meticulously go over the project requirements, systematically satisfying each bullet point in order to maximize my score. During tests, I not only skipped over difficult questions, I would actively seek out hints in the later questions to help me narrow down possible answers. My ability to squeeze out high grades had more to do with my aptitude at filling in the right bubbles on a piece of paper then actually understanding the material.

I fantasized about attending college, where I would be judged on my intellectual prowess, and not on my test taking skills. I longed for the pursuit of knowledge in it's purest form, only for this dream to be completely and utterly crushed. Instead of a place free from the endless battery of tests I had been subjected to during high school, I quickly realized that college was nothing but tests. I once had a math course where 95% of my grade was split between a first midterm, a second midterm, and a final. By the end of my second year of college, I simply stopped attending lectures. I could teach myself the material out of the textbook, and went to class only to take a test or turn in homework. I earned my degree by becoming incredibly adept at memorizing precisely which useless, specific facts were needed to properly answer questions. I was never asked nor told how to apply these to real world scenarios.

Thankfully, in one of the last classes I actually attended lecture in, the TA teaching the class said something that sparked a epiphany in me: "Math is simply repeated abstraction and generalization." Suddenly, I was able to connect math and programming, and began to realize that I had loved math all my life. What I hated about math was the trivial nonsense they taught in middle school. I signed up for the most advanced math classes I could get away with, even when I could barely pass them. I began to realize that the most important thing these classes taught me was what I didn't know. Once I knew what I didn't know, I could teach it to myself, but only after I found the holes in my knowledge. You can't fill a hole if you don't know where it is. I didn't know what combinatorics was until it was mentioned to me by that TA; Chrome still doesn't think combinatorics is even a word.

Everyone finds the beauty of math in their own way, but we teach it like an automated assembly line of cars. Math is taught as some kind of rigid tool, when it is really a language for expressing logic, one with multiple dialects, each with their own personality. We invented music to express emotions that cannot be described; we invented math to express logical abstractions that defy explanation. Every tool in math is like another instrument in a grand orchestra, each note echoing off the others, reflecting a whole that is greater than the sum of its parts. Some composers prefer the string section, others prefer the woodwinds. There is no single right answer, only different ones. Instead of giving our children a brush and telling them to use their imagination, we give them a coloring book and grade them on how well they stay inside the lines.

I mean, we all know creativity is overrated. It must be, since we systematically destroy it even when we try to encourage it. It doesn't matter how many programs we fund for encouraging things like art and music when the kids are still ultimately judged on how well they follow instructions and fill in little scantron bubbles. Kids are not stupid. I cannot believe how many adults still think they can get away with telling kids one thing and then doing another. They know what you're up to. They know the only thing the school system cares about is their grades, and that their grades are based entirely on how well they follow directions. They know that answering a question "almost right" doesn't matter. They know all problems have one answer in the back of the teacher's textbook, and their job is to figure out what it is. The vast majority of them have absolutely no idea how to approach a problem that has no correct answer. They don't know how to judge the correctness of a solution, because in school, everything is either right or wrong. All they know how to do is guess how likely it is that their solution is the solution the teacher wants, not how well the solution would actually work.

This is, of course, completely contradictory to everything in life. Life does not have answers in the back of the book. Life does not have a single correct answer to any problem. There is no right way to do anything, there are simply pros and cons. The fact that many people continue to delude themselves into thinking otherwise is a sad symptom of this issue. Our obsession with tests has trained a generation of robots, not engineers. They're more skilled at working their way through a bureaucracy than designing rockets. Then again, considering that colleges have now turned into enormous, obstructive bureaucracies, perhaps this isn't entirely a bad thing.

After all, with only 160 (now 200) spots open in its CSE major program each year when it has over 27000 undergraduate students enrolled^[1], the University of Washington gets mighty picky about who they let in. After getting a 3.7 and 3.4 in my first two calculus classes, I slipped and got a 2.8 in my third math class, so despite the fact that I got a perfect 5 on the AP Computer Science AB exam and was qualified to skip both introductory programming courses, they rejected my application and demanded I take Matrix Algebra before letting me in. So I got a 3.9 in Matrix Algebra (a grade that was exceptionally good, according to one professor), and then... they still didn't let me in. They complained that my entrance essay sounded "too cocky" and had me take the second introductory programming course even though I already had credit for it. When I failed to get an exceptionally good grade in that class for all the wrong reasons (like being graded down for having both too few comments and too many comments in my code), I simply could not bring myself to compete in a hyper-competitive environment where the only thing I was judged on was how many irrelevant details I could regurgitate. So, I majored in Applied Mathematics and simply took all the condensed, non-major CSE courses instead.

This obsession with tests extends into the evaluation of the educational system itself. One of the reasons nothing is getting better is because we use the very thing that is wrong with the educational system to judge it. We fill out ridiculous polls made out of those same interminable bubbles that are destroying the curriculum. We introduce more standardized testing. The entire system is completely obsessed with tests, and yet the only place that tests actually exist is... inside the system itself. Education has become so enraptured with this imaginary world it has constructed, it's completely forgotten about the reality it's supposed to be teaching kids about.

Kids know this imaginary world has nothing to do with reality. We lament about how to teach kids math when they refuse to understand it, without realizing that they are simply applying the same method of learning they use in everything else - memorize useless facts, then regurgitate them on a test. The reason our math curriculum is failing so badly is because in math, you can't simply memorize things, you need to understand them. Consequently, Math acts as a canary in the coal mine for our entire educational system. Kids make no effort to understand anything, because they aren't graded on how well they understand concepts, they are graded on how well they memorize random, useless details and follow directions.

We live in a world being overrun by automation. Any task that can be reduced to simply following a set of instructions over and over is being done by robots and software. This constant attrition of jobs involving menial work and physical labor will continue at a steady pace for the foreseeable future. We are teaching our kids skills that are being made irrelevant in a modern economy. We are preparing our children for a world that no longer exists. At the same time, while I could write a series of blog posts outlining an effective educational system, it will never be implemented in public schools. The establishment is too deeply entrenched. Foolish startups repeatedly and continually attempt to "disrupt" the educational system without realizing just how laughably outmatched they are. This is not something you can fix with a cute website. People complain about global warming, space travel, all sorts of adorable little problems, but miss the elephant in the room.

The greatest challenge our species has ever faced is the educational system itself.

How To Complain About Men And Be Sexist At The Same Time

What if I told you about an article that complained about how social media and instant gratification has eroded away at our social fabric? How we don't take the time to pause and reflect on how we live our lives? How we fail to really work at making the world a better place, and instead waste time building websites to share cat photos on? I'd think that it raised several important issues about modern society.

What if I then told you that the entire article only applied this to men? What if it was titled, "Why Men Aren't Really Men Anymore"? Suddenly, things just got a lot more sexist. In fact, that entire article is built almost entirely out of gender stereotypes. It's that subtle, classy kind of sexism; bigotry that hides behind delicate prose, hiding it's true nature. If the article were rewritten so that it described these issues in gender-nuetral terms, I'd be liable to agree with it. Trolling, aggression, lack of human interaction, billions of dollars being spent on worthless startups that solve first world problems - these are all real issues. Yet, to imply that men are the ones with these problems, to imply that any class of problems belong to only one gender, is sexist.

The article is wonderfully subtle in its sexism. Just look at how it claims "real" men should treat women:

Real men are not selfish. Real men are just as concerned for the feelings, needs and minds of women as they are for their own — not just women’s bodies and their sexual usefulness. Real men have a well-defined code of ethics and respect that they follow.

Isn't that sweet? Except, oh wait, we have this disturbing sentence in the middle of the article:

Men have become lazy pussies. I don’t even want to use the word pussy because it brings to mind women, who nowadays have much more character than men.

Not to be outdone, the article's last paragraph contains this gem:

Some great women are settling for these fools and then finding that they themselves have no choice but to wear the pants in the family because their “man” is PMSing.

This is horribly sexist. These three sentences enforce multiple gender stereotypes and tell us a number of things:
  1. Women want manly men. If they think they don't want a manly man, they just haven't found one manly enough yet.
  2. Men are always the ones who should be wearing the pants in the family, because men always have to be the manly ones.
  3. Women shouldn't have more character than men.
  4. Women are allowed to be PMSing because they're women, and everyone knows women get emotional, but a man should never be emotional, because he's a man.

See, the reasoning behind all these is just "because men are men and women are women." That isn't a reason, it's sexism. It's bigotry. It's enforcing stereotypes and trying to tell people how they should live their lives based on a set of gender roles that society arbitrarily decided on. It simply assumes it knows what women want, and goes so far to imply that the only reason a women wouldn't want a "real" man is because they haven't seen one yet. This is exactly like those annoying little assholes who tell you "Oh you just haven't heard good dubstep! You'll like it then!" Inevitably, after you still don't like it, they just tell you that something is clearly wrong with you and you have no taste.

At no point do they entertain the notion that, maybe, just maybe, you actually don't like dubstep.

Our society suffers from the same tunnel vision. We assume that when a women is working overtime and the man is doing laundry that it's the man's fault for not being manly enough and the women has been forced to become the head of the household. If she had just married a man manly enough, she wouldn't have had to do that!

It never crosses their minds that maybe, just maybe, the women actually likes it this way. Maybe some men just don't want to be manly. Maybe some women like men who aren't manly. Maybe you can't fit every single human being into nice neat little gender boxes.

It is not a man's job to be manly simply because they are men. It is not a women's job to be PMSing or making you a sandwich. It is not society's place to tell anyone how they should live their lives. You do not know what they want and you should never pretend you do. We can make certain observations about genders, like men tend to be more aggressive, and women tend to be more emotional, but we should never assume that men should be more aggressive, or women should be more emotional. That is for the individual to decide, not society.

A human being is something precious, something complicated, something that can't be easily categorized. Stop trying to stuff them into square boxes.