DanKam: Augmented Reality For Color Blindness
So, for the past year or so, I’ve had a secret side project.
Technically, that shouldn’t be surprising. This is security. Everybody’s got a secret side project.
Except my side project has had nothing to do with DNS, or the the web, or TCP/IP. In fact, this project has nothing to do with security at all.
Instead, I’ve been working on correcting color blindness. Check it out:
These are the Ishihara test plates. You might be familiar with them.
If you can read the numbers on the left, you’re not color blind.
You can almost certainly read the numbers on the right. That’s because DanKam has changed the colors into something that’s easier for normal viewers to read, but actually possible for the color blind to read as well. The goggles, they do something!
Welcome to DanKam, a $3 app being released today on iPhone and Android (ISSUES WITH CHECKOUT RESOLVED! THIS CODE IS LIVE!). DanKam is an augmented reality application, designed to one of several unique and configurable filters to images and video such that colors — and differences between colors — are more visible to the color blind.
[EDIT: Some reviews. Wow. Wow.
@waxpancake: Dan Kaminsky made an augmented-reality iPhone app for the colorblind. And it *works*.
Oh, and I used it today in the real world. It was amazing! I was at Target with my girlfriend and saw a blue plaid shirt that I liked. She asked me what color it was so I pulled up DanKam and said “purple.” I actually could see the real color, through my iPhone! Thanks so much, I know I’ve said it 100x already, but I can’t say it another 100x and feel like I’ve said it enough. Really amazing job you did.
I went to a website that had about 6 of those charts and screamed the number off every one of those discs! I was at work so it was a little awkward.
–JJ Balla, Reddit
Thank you so much for this app. It’s like an early Christmas present! I, too, am a color blind Designer/Webmaster type with long-ago-dashed pilot dreams. I saw the story on Boing Boing, and immediately downloaded the app. My rods and cones are high-fiving each other. I ran into the other room to show a fellow designer, who just happened to be wearing the same “I heart Color” t-shirt that you wore for the Forbes photo. How coincidental was that? Anyway, THANKS for the vision! Major kudos to you…
Just found this on BoingBoing, fired up iTunes, installed and tried it. Perfectly done!
That will save me a lot of trouble with those goddamn multicolor LEDs! Blinking green means this, flashing red means that… Being color blind, it is all the same. But now (having DanKam in my pocket), I have a secret weapon and might become the master of my gadgetry again…
Can’t thank you enough…!
Holy shit. It works.
It WORKS. Not perfectly for my eyes, but still, it works pretty damned well.
I need a version of this filter I can install on my monitors and laptop, asap. And my tv. Wow.
That is so cool. Thanks for sharing this. Wish I could favorite it 1000 more times.
–zarq on Metafilter
As a colorblind person I just downloaded this app. How can I f*cking nominate you to the Nobel Prize committee? I am literally, almost in tears writing this, I CAN FRICKIN’ SEE PROPER COLORS!!!! PLEASE KEEP UP THIS WORK! AND CONTINUE TO REFINE THIS APP!
At this point, you’re probably asking:
- Why is some hacker working on vision?
- How could this possibly work?
Well, “why” has a fairly epic answer.
So, we’re in Taiwan, building something…amusing…when we go out to see the Star Trek movie. Afterwords, one of the engineers mentions he’s color blind. I say to him, these fateful words:
“What did you think about the green girl?”
He responds, with shock: “There was a green girl??? I thought she was just tan!”
But I was not yet done with him. See, before I got into security, I was a total graphics nerd. (See: The Imagery archive on this website.) Graphics nerds know about things like colorspaces. Your monitor projects RGB — Red, Green, Blue. Printouts reflect CYMK — Cyan, Yellow, Magenta, blacK. And then there’s the very nice colorspace, YUV — Black vs. White, Orange vs. Blue, and Red vs. Green. YUV is actually how signals are sent back to the brain through the optic nerve. More importantly, YUV theoretically channelizes the precise area in which the color blind are deficient: Red vs. Green.
Supposedly, especially according to web sites that purport to simulate color blindness for online content, simply setting the Red vs. Green channel to a flat 50% grey would simulate nicely the effects of color blindness.
Alas, H.L. Mencken: “For every complex problem there is an answer that is clear, simple, and wrong.” My color blind friend took one look at the above simulation and said:
“Heh, that’s not right. Everything’s muddy. And the jacket’s no longer red!”
A color blind person telling me about red? That would be like a deaf person describing the finer points of a violin vs. a cello. I was flabbergasted (a word not said nearly enough, I’ll have you know).
Ah well. As I keep telling myself — I love being wrong. It means the world is more interesting than I thought it was.
There’s actually a lot of color blind people — about 10% of the population. And they aren’t all guys, either — about 20% of the color blind are female (it totally runs in families too, as I discovered during testing). But most color blind people are neither monochromats (seeing everything in black and white) or dichromats (seeing only the difference between orange and blue). No, the vast majority of color blind people are in fact what are known as anomalous trichromats. They still have three photoreceptors, but the ‘green’ receptor is shifted a bit towards red. The effect is subtle: Certain reds might look like they were green, and certain greens might look like they were red.
Thus the question: Was it possible to convert all reds to a one true red, and all greens to a one true green?
The answer: Yes, given an appropriate colorspace.
HSV, for Hue, Saturation, Value, is fairly obscure. It basically refers to a pattern of computing a “true color” (the hue), the relative proportion of that color to every other color (saturation), and the overall difference from darkness for the color (value). I started running experiments where I’d leave Saturation and Value alone, and merely quantize Hue.
This actually worked well — see the image at the top of the post! DanKam actually has quite a few features; here’s some tips for getting the most out of it:
- Suppose you’re in a dark environment, or even one with tinted lights. You’ll notice a major tint to what’s coming in on the camera — this is not the fault of DanKam, it’s just something your eyes are filtering! There are two fixes. The golden fix is to create your own, white balanced Light. But the other fix is to try to fix the white Balance in software. Depending on the environment, either can work.
- Use Source to see the Ishihara plates (to validate that the code works for you), or to turn on Color Wheel to see how the filters actually work
- Feel free to move the slider left and right, to “tweak” the values for your own personal eyeballs.
- There are multiple filters — try ’em! HueWindow is sort of the brute forcer, it’ll only show you one color at a time.
- Size matters in color blindness — so feel free to put the phone near something, pause it, and then bring the phone close. Also, double clicking anywhere on the screen will zoom.
- The gear in the upper left hand corner is the Advanced gear. It allows all sorts of internal variables to be tweaked and optimized. They allowed at least one female blue/green color blind person to see correctly for the first time in her life, so these are special.
Really though, not being color blind I really can’t imagine how this technology will be used. I’m pretty sure it won’t be used to break the Internet though, and for once, that’s fine by me.
Ultimately, why do I think DanKam is working? The basic theory runs as so:
- The visual system is trying to assign one of a small number of hues to every surface
- Color blindness, as a shift from the green receptor towards red, is confusing this assignment
- It is possible to emit a “cleaner signal”, such that even colorblind viewers can see colors, and the differences between colors, accurately.
- It has nothing to do with DNS (I kid! I kid! But no really. Nothing.)
If there’s interest, I’ll write up another post containing some unexpected evidence for the above theory.