Articles, Discovery, Lean Startup, Quotes

How to decide which product concepts to test

Mike Fishbein wrote about my approach to determining what to test in “Concept Testing: 6 Lessons From Product Leaders” on This is Product Management. Here’s an excerpt:

Holly Hester-Reilly, a product consultant and former Product Owner at Shutterstock, shared her framework for determining what concepts needs to be tested in the first place. Holly plots each test her team is considering running on a four-quadrant diagram. The diagram maps “the negative impact if the concept isn’t viable” on the Y-axis, and “the likelihood that the concept isn’t viable” on the X-axis.

2x2 graph for what to test
What to test?

Holly then tests the concepts in the upper right-hand quadrant – concepts that have a high likelihood of not being viable and would have a major negative impact if they aren’t viable. This framework helps her get answers to her most critical product questions.

Read the full article on the Alpha blog for more insights:

Concept Testing: 6 Lessons From Product Leaders

Articles, Design Thinking, Discovery

How to Use Product Discovery Skills to Build Alignment, At Work or At Home

This article was originally published in Product Management Insider on Medium.

Me, fielding yet another concern about our vacation planning

Maybe agreeing to this wasn’t such a good idea, I thought.

I was reading the latest message in a fraught conversation with my family about something seemingly simple: where to go for an upcoming group vacation. The trip was supposed to be in just 2 weeks, and we hadn’t booked anything yet. Despite the fact that I already often felt stretched thin between working with multiple clients and projects at once and managing family life with an active toddler and baby, I’d been spending an inordinate amount of my time fielding concerns from my family about how this trip would go.

So how had I gotten myself into this situation?

One day a few weeks earlier, my brother asked if we’d like to get our families together for a vacation. I hadn’t seen my brother and his family since the summer before, and since then we’d both added second children to our families. He’d moved to London and I’d been beginning to wonder if, despite the fact that we had once been so close that I asked him to be best man at my wedding, the distance between us would keep growing greater. His family was going to be within a day’s trip of us and he suggested that we could join them at a beach.

I wanted to believe that our families could be closer, so I quickly said yes. Nevermind that we’d never taken a group vacation together before. Nevermind that neither my mom nor my own family had funds set aside for summer travels. Nevermind that we had different travel habits, with my brother’s family enjoying active adventures, air travel, and Airbnb while my family preferred to stay local or stay with family.

Soon we found ourselves disagreeing over all the details. We had lots of ideas on where to go and what to do, but there were also some strong reactions, even the possibility of aborting the whole plan.

We all had something different to say about the potential vacation.

After a decade working alongside passionate people in tech startups, I’m used to facing a plethora of strong opinions on the best thing to do. I’ve found that while people often tend to focus on solutions, when we step back to build agreement on what problem we are solving and which constraints are important, we find it much easier to agree on which solution to pick.

So, like any good product manager, I set out to understand the people and their problems better, hoping to build agreement even when it seemed impossible.

Here’s how you can use product discovery skills to build agreement with others in your own life

The next time you face disagreeing decision makers, follow these product discovery steps to define the problem in a way that leads to easy agreement.

1. Capture the solution ideas

We humans don’t generally let go of our ideas easily. If we’re told to step back from ideas to look at the problem, we often hang on tight to the ideas in our head. I’ve found it helps to get all those ideas out, so that we can feel safe in the knowledge that they are waiting for us when we are ready to return to evaluating solutions.

Our proposed vacations had differences in cost, environment, amenities, and travel requirements.

2. Prioritize your research goals

Now, you’re going to step back from the ideas to learn more about the problem space. But what’s most important to know? Try starting with a hypothesis about which differences are important to your target audience. What are the differences in the solution ideas? How do these differences affect the way the target audience would use the solution? Which differences in the solution options will impact their behaviors and decisions most strongly?

For each difference, think about the cost of choosing one choice over another at this stage. Do any set you down a path that’s costly to change? Those are the things that you want to make sure you cover in your early research, because they are the riskiest to get wrong. You also want to empathize with each person’s point of view and find out about the drivers of their decisions and behaviors.

For our vacation, the transportation options and environment would be largely fixed with this decision, so those were critical areas to learn about.

3. Do the research

Once you have prioritized your research goals, you can conduct quality research that stays focused on getting the information that you will need to drive smart decisions. One on one interviews are most effective for this kind of research, because you need to establish a rapport to dig deep enough to uncover the hidden motivations, desires, and constraints.

Digging deeper through one-on-one conversation uncovered what each person was thinking.

4. Synthesize Findings

Once you’ve conducted the research, you’ll need to share the results with your stakeholders. Did you invalidate any assumptions? What was most surprising or unexpected? Were there themes or common ground that emerged? What constraints will be most important to meet in order to deliver a compelling solution? At this point in time you should have validated that you have a valuable problem to solve and be able to say with confidence what the desired outcome of a solution is.

We all genuinely wanted to spend more time together.

5. Generate and evaluate solutions

Now that you have a clearer picture of the problem to be solved, you should give the team a chance to come up with new ideas or revisit their original ones. How do each of the options meet the constraints you identified? How well do they support the desired outcome?

6. Make a decision

Now that you and your stakeholders have a solid understanding of how well each of the solutions supports the desired outcome, don’t hesitate — make the call!

At this stage, when we found a beach house in Old Lyme, Connecticut, that was in our budget, it was easy for everyone to agree that it was the best choice.

We had twice as many happy vacationers, all feeling closer together after our trip.

So the next time you find yourself struggling to rally disagreeing stakeholders, try using these product discovery steps to build alignment.

Agile, Articles, Discovery, Method, Stakeholders

Product leaders, educate your stakeholders by adding this one thing to your sprints

This post originally appeared in Product Management Insider on Medium.

The BLP Demo helps user-focused discovery grow in both old and new organizations

Many product teams struggle to dedicate enough time and resources to discovery and experimentation amidst the pressure to ship new features. I’ve found that a key to developing continuous discovery and delivery practices— sometimes called dual-track agile — is to master the simple practice of hosting a “Built-Learned-Planning Demo” (BLP Demo).

In short, the BLP is a demonstration that gets integrated into your sprint cycle and emphasizes the value and takeaways from doing customer development.

It’s designed for your agile cadence and can help you nurture your user-focused practices to grow strong amongst the challenges of a growing company or the organizational inertia of a large organization.

How do you do a Built-Learned-Planning Demo?

A good BLP Demo includes three sections: what we built, what we learned, and what we’re planning. I’ll review each briefly.

What we built: Most teams already do this as part of their agile process. But in case your company’s sprint reviews need some new life, I suggest that teams pick a few of their most interesting developments and be sure to describe what user problem they were trying to solve in addition to what was built.

What we learned: This section can include updates from any team member. Engineers might describe results of a research spike or proof of concept prototype. Designers might share learnings such as usability test results or user interviews. Product managers might share product insights such as updates on the competition, new data on product usage, or results of customer discovery research. Having a place in the demo for updates from non-engineering roles also builds the team’s understanding of each other’s contributions, helping to develop a cohesive team of diverse perspectives.

What we’re planning: In this section, we share some things we’re planning, connecting it to our learnings. This might include architecture plans, UX designs, product or UX research, or roadmap updates. As we share next steps at the same cadence that our engineers deliver software, we build understanding of how product decisions are made and what valuable product discovery is happening. This also helps us develop trust with our stakeholders, so that they can attach to something other than a feature schedule.

Why is a Built-Learned-Planning demo a good way to drive change?

Because the principles behind the ceremonies used in agile software development can be used for agile discovery too. Even better, expanding familiar practices to include the agile discovery work can make these changes easier for agile development organizations to adopt. As Ken Rubin, author of Essential Scrum, explains it,

“The goal of the sprint review is to inspect and adapt the product that is being built.”

By covering not just what we built but also what we are planning, we are ensuring that both tracks of our dual-track agile software development are inspected and adapted regularly. By sharing what we learned and how we’ve updated our plans accordingly, we can keep our stakeholders up to date, coaching them to value learning as well as delivery and to trust our teams to decide their most impactful next steps. Over time, they will come to look forward to the insights as much or more than the demos of built software.

The Built-Learned-Planning Demo at work for the Shutterstock Editor team

For example, consider some highlights from a BLP Demo we did when I led the Workflow Group at Shutterstock.

The Shutterstock Editor team showed text outlining during the Built section of a BLP Demo

During the Built section, the Shutterstock Editor team showed the addition of the outline functionality for the text tool. Instead of just showing that we built an outline tool, we described the problem — when users add text on top of an image, it can be hard to read the text if the image has lots of color variation. Because we were developing iteratively, we wanted to develop the simplest solution. After exploring the feasibility of both outline and dropshadow effects, we started with the solution that we could get to market most quickly. After this explanation, our engineers demo’ed the outline functionality. Instead of getting a “that’s neat, but so what?” reaction, our stakeholders were able to understand what value the functionality delivered and why we’d chosen the scope that we did.

During the Learned section, one update was usage data on a feature over time, highlighting how the usage changed after we rolled out a UX update. Sharing lessons such as this on how a UX change drove increased usage of a feature helped to shift stakeholder’s conversations from output to outcomes.

For the Planning section, the workflow team, which was in charge of features on the core site that support and enable customer workflow, shared a high level look at ethnographic research plans to dive into the experiences of a target set of customers. This moved the conversation to how we would flesh out details on the initiatives we were exploring for future sprints and gave us a chance for early feedback on our plans.

These highlights of a BLP Demo from my time at Shutterstock illustrates how the simple practice of adding product discovery to the sprint review pushes the team and stakeholders to add a focus on measurable outcomes and lessons learned to the usual cycle of shipping code.

Get your own Built-Learned-Planning Demo started quickly

Here’s a template that I use to get a team started on these Built-Learned-Planning Demos. Feel free to copy and edit it for your own uses.

The three parts of the BLP Demo

If you are a team member, try working this into your team’s practice. If you are a leader, try putting support in place to encourage this on your teams, and hold them accountable to produce and share learnings every sprint. This will plant the seeds for your organization’s continuous discovery practice to grow.

At young companies, practicing this will help you scale without losing your customer focus. At existing companies, these teams can become the impetus for cultural change that drives innovation.

As with any process change, you need to try it for a few cycles and iterate to figure out what works best in your organization. Then please let me know how it’s working for you and what challenges you’ve encountered along the way!