"Gimme, gimme" is not feedback

Koala's prototype was so good creators said it was perfect. Here's how we turned that into real product decisions.

By Miles Sellyn, VP of Strategy

Hill I am currently ded on: the most frustrating sentence in user testing is "I have no notes." It’s worse than "I don't get it.”

"I don't know what to tell you, bud" reaction GIF

That is roughly what creators told Koala when they got their hands on the prototype. In Justin's words: "It was so good that creators said, 'I have no issues at all. I need this today.' And we were like, okay, that doesn't help us."

I know this sounds like a humblebrag dressed up as a problem so that I can write a newsletter about it.

And yes, it is partly a humblebrag, but not only a humblebrag. It's also a real problem, and today is about what to do with it, plus what you should be asking any design partner (us included) to hand you before engineering starts.

What you’ll find below:

Why some users can't give feedback on a deck
Prototypes you can hold
How "I need this today" turned into an actual product decision
Three things to demand before production code

Some users can't react to slides

Justin comes from Silicon Valley, where you can often get genuine feedback on a couple of slides. But many audiences, including creators, are a different audience. As he put it: "The user base we're targeting, they need to look, feel, and touch something to give feedback."

That makes total sense when you think about it. Creators spend all day in some of the best consumer products ever made. Instagram, TikTok, YouTube. Show them a static mockup and they compare it, mostly subconsciously, to something with a billion dollars of polish. Show them a clickable Figma flow on a laptop and they're picturing an app while looking at a laptop.

So when we took Koala to testing, the bar for "real enough to react to" was high. Higher than most products need, honestly. But every audience has a version of this: a thing it can't evaluate in the abstract.

Prototypes you can hold

So our team built the prototypes in React, so they could be tested directly on phones, in the hand, during design. The onboarding prototype incldued parallax and gyroscope effects, so tilting the phone actually moved the scene. They were working tools for the entire design phase, rather than a validation step tacked on at the end.

(Side note - it’s crazy that the tools are advancing so quickly now that I think we’d have a completely different approach to building a prototype like this today than we did a few months ago. What a time to be alive.)

Koala prototype running on a phone in hand — a React gif (it’s a dev joke)

That level of fidelity changed how the team thinks about what a "design deliverable" is. A Figma file describes a product, albeit in really high fidelity, but what we can do with prototyping now is genuinely insane. These prototypes on a phone are products, just ones without backends yet.

Justin didn't mince words on why it mattered: "Without that, there's zero chance we get the information back from creators."

How "it's perfect" became a decision

So the prototype was good enough that creators mostly just wanted it. Great for morale, useless for a roadmap 😂 The team had to push past the praise.

What came out once they did was groups. Creators kept asking for a group they could get to at the click of a button. That one piece of info did a lot of work:

Groups became a featured surface instead of a seconary one
It shaped the algorithm: how soon group recommendations show up in a new user's home feed, and how heavily groups are weighted
Justin says plenty of the decisions they're making now, in the middle of the build, trace back to that same prototyping feedback and decision

That's the main difference between testing and validating. Validating asks "do you like it?" and creators said yes. Testing has to dig out what would make it worth opening on a Tuesday (see my previous sick graphic about the heartbeat of a daily use product), and groups was the answer.

Three things to demand before production code

If you're about to sign off on a build, here are the criteria I'd hold any partner to, including us - and it is the same criteria our internal engineering team holds us to:

1. A prototype your actual users can evaluate in their actual context. For Koala that meant a phone in a creator's hand. For an enterprise tool it might be a working flow in the browser with realistic data. For a voice product, something you can actually talk to. The test: could your least imaginative user react to it honestly? If they'd need to "picture it," it's not ready.

2. A feedback plan that assumes praise. People are nice, especially to founders. Build your sessions around questions that force a tradeoff rather than invite compliments. Open ended is fine, but the more pointed, the better info you’re going to get. Some that work:

"What would you open first tomorrow morning?"
"If we could only ship three of these, which three?"
"What would make you stop using this after a week?"
"Who else would you send this to, and what would you say in the DM when you did?"

"Do you like it?" is not on the list. You already know the answer, and it's yes. Especially if we designed it, heyoooooooo

3. A visible line from feedback to decision. Every important design choice should be able to point back to something a user did or said. If you can't draw those lines, your product probably has opinions with good typography.

If you do this properly, engineering builds from decisions instead of guesses. That's what you're paying for when you test properly. Fewer arguments in month four, and less uncertainty at launch.

That's it for today. If you found this useful, forward it to whoever is about to run user testing and is planning to ask people if they like it 😬

Next Thursday is the finale: what actually shipped, and the part of the Koala story I think gets told wrong most often - PLUS I may have an early access invite code for ya!!