Emma Smith

Hello Beautiful

January 22, 2019

Hello Kitty gets a mouth, knows your name, and tells you you’re beautiful. I’m using the OpenCV computer vision library to do facial recognition, Koa for a server, and the poet Anne Waldman for the voice.

Hello Kitty

Why am I doing this? This project is a kind of magpie’s nest, a single place to put all the things that interest me. At the moment this includes state machines, conversational UIs, and, especially, computer vision.

Computer vision

We know that machine learning is being put to nefarious ends. I am using it to create something that is fun, useless, and delightfully weird. (Delightful to me, at least.)

Hello Kitty

I wanted to learn a bit about computer vision, and in so doing be a little more informed about the challenges and opportunites it presents. There is power in making something that is utterly useless. I am free to play, discovering the true limits of this stuff. And, hopefully, Hello Beautiful can become a basis for sharing what I learn.

State machines

One of the challenges of this project is managing the state of the UI. (Actually, I think that might be the challenge of every UI.) Hello Kitty starts in a “sleeping” state. When someone approaches, she notices and wakes up.

Statechart for waking and sleeping

This state change shows up on the screen like this:

Hello Kitty waking and dreaming

When she sees someone, she doesn’t just rise from her glitchy slumber. She says hello, if it’s someone she already know. If it’s someone new, she introduces herself. The “Awake” state just got a bit more complex:

Statechart for waking and sleeping

It would be weird if she just kept saying hello over and over.

Hello Kitty waking and dreaming

Talking isn’t the only thing she does while awake. After she’s talked for a bit, she goes into a state of admiration:

Statechart for waking and sleeping

Modelling this sort of human-ish behavior gets tricky pretty fast. In my first prototype I found myself with a bunch of nested if statements. These conditionals were scattered across the presentational code. Every time I touched one it extolled a little mental tax. It became really tough to figure out which bit of logic was driving any particular bahavior. And I’m the one who wrote it! Not good.

I’m now using statecharts to model the behavior of the UI. A statechart is kind of like a super-powered state machine. You define the various states of the system, and then you describe the paths by which the system can transition from one state to another.

The diagrams above are statecharts. But statecharts can also be written directly into your code. (Check out xstate library, which complies with the W3C’s statechart spec. Oh yeah — there’s a spec!) Once you wrap your head around the formal syntax, it’s not much more difficult than drawing circles and arrows. And it’s really nice to have all of that logic in one place.

The future

One of my goals is to run this on a Raspberry Pi and hook it up to an old Hello Kitty TV via the Pi’s analog video output. I’ll be writing about how it goes… more to come.

I work in New York City building invisible machines for the arts. Say hi on Twitter, or, if you're the sort, check out my Github.