Articles, Product Management, Stakeholders

Being Right is Not Enough

This article was originally published in Product Management Insider on Medium.

“This isn’t what we asked for,” the director said, indignance in his voice. “They have to fix this. I want our team to make a list of all the changes we need.”

I was sitting in a conference room, discussing the state of a million dollar software project which had just been delivered for acceptance testing. The project had started with hundreds of pages of documentation, followed by nearly a year of “on track” status updates. Once we could finally test the software, the director was livid to see that it didn’t match his vision. And he wanted his team (a team of environmental engineers, not designers) to redesign it — after it had been built and the budget was nearly spent. Talk about scope creep!

I tried to tell the director, the assistant director, the contract manager — anyone really — that this was a bad idea. We should get it in front of real users first, I argued. But no one wanted to listen to me. Instead, the director put other associates on the task of building their own wireframes from scratch. They spent endless hours on learning to use wireframe software and creating new designs, which resulted in change agreements and more money spent. And did I mention that this was a government project? These were taxpayer dollars!

The first time I tried to convince a boss to change his mind, it went something like this.

I started looking for a new job not long after that.

I also did a lot of soul-searching. I had learned through years of academic achievement that being correct was what mattered. This experience brought home for me something I now tell product managers regularly:

It’s not enough to be right. You also have to be persuasive.

But how? What if I told you that I’ve been testing and refining a step by step process for driving evidence-based product decisions that set products and product managers up for successful growth?

You might not believe me. So I’d like to tell you another story.

This one takes place years later. I’d just joined Shutterstock as the product manager for Skillfeed, an online education platform. As I asked more of my new colleagues for their perspectives, I discovered that before I was hired, some of them had argued, unsuccessfully, for shutting it down — and they may have been right to do so. But they hadn’t been persuasive.

If I approached this the same way, I would have failed this time too. So I had to do something different. I had to put energy towards not just the problem, but also the people around the problem.

Do you focus on the problem, or the people?

It was a slow and gradual process. I started with empathy for the decision makers. I put myself in their shoes and learned as much as I could about why they supported this project, what they thought it could do for them and the company, and what objections they had when others had tried to change their mind. I went out and gathered evidence, shared new insights, and asked follow-ups. I didn’t deliver fully-baked thoughts to them from on high; I guided their thinking, let them develop their own thoughts, come to their own conclusions, and convince themselves.

By doing that, I ultimately did something no one else had. I persuaded the billionaire CEO of a public company to kill his pet project.

Since then I’ve found that while the specifics won’t transfer to every new situation, there is a core process to it that will — a process that has worked dozens of times to lead to product and career growth for my teams and clients.

I can’t claim that I’ve never failed to bring others along with my vision and strategy, because I have learned these skills the hard way. But what I can do is tell you what went wrong, what I tried differently the next time, and what challenges to expect along the way.

I want to share what I’ve learned with you so that you can set yourself up for career success. And the next time that you’re asked to execute a feature or product idea that you aren’t excited to add to your resume, you’ll know just what to do.

Articles, Discovery, Lean Startup, Quotes

How to decide which product concepts to test

Mike Fishbein wrote about my approach to determining what to test in “Concept Testing: 6 Lessons From Product Leaders” on This is Product Management. Here’s an excerpt:

Holly Hester-Reilly, a product consultant and former Product Owner at Shutterstock, shared her framework for determining what concepts needs to be tested in the first place. Holly plots each test her team is considering running on a four-quadrant diagram. The diagram maps “the negative impact if the concept isn’t viable” on the Y-axis, and “the likelihood that the concept isn’t viable” on the X-axis.

2x2 graph for what to test
What to test?

Holly then tests the concepts in the upper right-hand quadrant – concepts that have a high likelihood of not being viable and would have a major negative impact if they aren’t viable. This framework helps her get answers to her most critical product questions.

Read the full article on the Alpha blog for more insights:

Concept Testing: 6 Lessons From Product Leaders

Articles, Design Thinking, Discovery

How to Use Product Discovery Skills to Build Alignment, At Work or At Home

This article was originally published in Product Management Insider on Medium.

Me, fielding yet another concern about our vacation planning

Maybe agreeing to this wasn’t such a good idea, I thought.

I was reading the latest message in a fraught conversation with my family about something seemingly simple: where to go for an upcoming group vacation. The trip was supposed to be in just 2 weeks, and we hadn’t booked anything yet. Despite the fact that I already often felt stretched thin between working with multiple clients and projects at once and managing family life with an active toddler and baby, I’d been spending an inordinate amount of my time fielding concerns from my family about how this trip would go.

So how had I gotten myself into this situation?

One day a few weeks earlier, my brother asked if we’d like to get our families together for a vacation. I hadn’t seen my brother and his family since the summer before, and since then we’d both added second children to our families. He’d moved to London and I’d been beginning to wonder if, despite the fact that we had once been so close that I asked him to be best man at my wedding, the distance between us would keep growing greater. His family was going to be within a day’s trip of us and he suggested that we could join them at a beach.

I wanted to believe that our families could be closer, so I quickly said yes. Nevermind that we’d never taken a group vacation together before. Nevermind that neither my mom nor my own family had funds set aside for summer travels. Nevermind that we had different travel habits, with my brother’s family enjoying active adventures, air travel, and Airbnb while my family preferred to stay local or stay with family.

Soon we found ourselves disagreeing over all the details. We had lots of ideas on where to go and what to do, but there were also some strong reactions, even the possibility of aborting the whole plan.

We all had something different to say about the potential vacation.

After a decade working alongside passionate people in tech startups, I’m used to facing a plethora of strong opinions on the best thing to do. I’ve found that while people often tend to focus on solutions, when we step back to build agreement on what problem we are solving and which constraints are important, we find it much easier to agree on which solution to pick.

So, like any good product manager, I set out to understand the people and their problems better, hoping to build agreement even when it seemed impossible.

Here’s how you can use product discovery skills to build agreement with others in your own life

The next time you face disagreeing decision makers, follow these product discovery steps to define the problem in a way that leads to easy agreement.

1. Capture the solution ideas

We humans don’t generally let go of our ideas easily. If we’re told to step back from ideas to look at the problem, we often hang on tight to the ideas in our head. I’ve found it helps to get all those ideas out, so that we can feel safe in the knowledge that they are waiting for us when we are ready to return to evaluating solutions.

Our proposed vacations had differences in cost, environment, amenities, and travel requirements.

2. Prioritize your research goals

Now, you’re going to step back from the ideas to learn more about the problem space. But what’s most important to know? Try starting with a hypothesis about which differences are important to your target audience. What are the differences in the solution ideas? How do these differences affect the way the target audience would use the solution? Which differences in the solution options will impact their behaviors and decisions most strongly?

For each difference, think about the cost of choosing one choice over another at this stage. Do any set you down a path that’s costly to change? Those are the things that you want to make sure you cover in your early research, because they are the riskiest to get wrong. You also want to empathize with each person’s point of view and find out about the drivers of their decisions and behaviors.

For our vacation, the transportation options and environment would be largely fixed with this decision, so those were critical areas to learn about.

3. Do the research

Once you have prioritized your research goals, you can conduct quality research that stays focused on getting the information that you will need to drive smart decisions. One on one interviews are most effective for this kind of research, because you need to establish a rapport to dig deep enough to uncover the hidden motivations, desires, and constraints.

Digging deeper through one-on-one conversation uncovered what each person was thinking.

4. Synthesize Findings

Once you’ve conducted the research, you’ll need to share the results with your stakeholders. Did you invalidate any assumptions? What was most surprising or unexpected? Were there themes or common ground that emerged? What constraints will be most important to meet in order to deliver a compelling solution? At this point in time you should have validated that you have a valuable problem to solve and be able to say with confidence what the desired outcome of a solution is.

We all genuinely wanted to spend more time together.

5. Generate and evaluate solutions

Now that you have a clearer picture of the problem to be solved, you should give the team a chance to come up with new ideas or revisit their original ones. How do each of the options meet the constraints you identified? How well do they support the desired outcome?

6. Make a decision

Now that you and your stakeholders have a solid understanding of how well each of the solutions supports the desired outcome, don’t hesitate — make the call!

At this stage, when we found a beach house in Old Lyme, Connecticut, that was in our budget, it was easy for everyone to agree that it was the best choice.

We had twice as many happy vacationers, all feeling closer together after our trip.

So the next time you find yourself struggling to rally disagreeing stakeholders, try using these product discovery steps to build alignment.

Discovery, Interviews, Podcasts

High Impact Experimentation is Product Management

I spoke with Mike Fishbein about making evidence based product decisions through high impact experimentation for episode 113 of the This is Product Management podcast. In it, I shared:

We don’t need to test everything. We need to test the things that are risky.


Check out the post about it to read more or find iTunes and Stitcher links for the podcast.

High Impact Experimentation is Product Management

Data Science, Strategy

Strategy for machine learning products in CMS Wire

I spoke with Kaya Ismail of CMS Wire about the need for large data sets and real customer problems to drive successful machine learning (ML) products for his coverage of Hubspot’s acquisition of Kemvi. Here’s an excerpt:

Holly Hester-Reilly of H2R Product Science believes HubSpot’s acquisition of Kemvi is a, “smart move for both teams.”

Hester-Reilly continued, “[By] combining DeepGraph’s algorithm and process for processing publicly available data with HubSpot’s data on real sales looks like a great opportunity to develop real-world machine learning solutions to solve real problems for HubSpot’s sales customers.

Check out the full piece over at CMS Wire for more details on why Hubspot’s acquisition of Kemvi is exciting news for artificial intelligence, machine learning, and big data product practitioners.

HubSpot Broadens AI Reach With Kemvi Acquisition

Articles, Data Science, Product Management, Quotes, Stakeholders

How to kill a bad product idea on Alpha blog

I talked with Mike Fishbein about techniques to identify bad product ideas and convince stakeholders when it’s time to end an initiative. Here’s an excerpt:

When working on a product that’s already launched, Holly emphasized the importance of looking beyond “vanity metrics”, such as page views and user counts, to truly understand if the product is valuable to customers.

Read the full article on the Alpha blog for more insights, including advice from Tami Reiss and Beth Temple:

How do you kill a bad product idea?

Agile, Articles, Discovery, Method, Stakeholders

Product leaders, educate your stakeholders by adding this one thing to your sprints

This post originally appeared in Product Management Insider on Medium.

The BLP Demo helps user-focused discovery grow in both old and new organizations

Many product teams struggle to dedicate enough time and resources to discovery and experimentation amidst the pressure to ship new features. I’ve found that a key to developing continuous discovery and delivery practices— sometimes called dual-track agile — is to master the simple practice of hosting a “Built-Learned-Planning Demo” (BLP Demo).

In short, the BLP is a demonstration that gets integrated into your sprint cycle and emphasizes the value and takeaways from doing customer development.

It’s designed for your agile cadence and can help you nurture your user-focused practices to grow strong amongst the challenges of a growing company or the organizational inertia of a large organization.

How do you do a Built-Learned-Planning Demo?

A good BLP Demo includes three sections: what we built, what we learned, and what we’re planning. I’ll review each briefly.

What we built: Most teams already do this as part of their agile process. But in case your company’s sprint reviews need some new life, I suggest that teams pick a few of their most interesting developments and be sure to describe what user problem they were trying to solve in addition to what was built.

What we learned: This section can include updates from any team member. Engineers might describe results of a research spike or proof of concept prototype. Designers might share learnings such as usability test results or user interviews. Product managers might share product insights such as updates on the competition, new data on product usage, or results of customer discovery research. Having a place in the demo for updates from non-engineering roles also builds the team’s understanding of each other’s contributions, helping to develop a cohesive team of diverse perspectives.

What we’re planning: In this section, we share some things we’re planning, connecting it to our learnings. This might include architecture plans, UX designs, product or UX research, or roadmap updates. As we share next steps at the same cadence that our engineers deliver software, we build understanding of how product decisions are made and what valuable product discovery is happening. This also helps us develop trust with our stakeholders, so that they can attach to something other than a feature schedule.

Why is a Built-Learned-Planning demo a good way to drive change?

Because the principles behind the ceremonies used in agile software development can be used for agile discovery too. Even better, expanding familiar practices to include the agile discovery work can make these changes easier for agile development organizations to adopt. As Ken Rubin, author of Essential Scrum, explains it,

“The goal of the sprint review is to inspect and adapt the product that is being built.”

By covering not just what we built but also what we are planning, we are ensuring that both tracks of our dual-track agile software development are inspected and adapted regularly. By sharing what we learned and how we’ve updated our plans accordingly, we can keep our stakeholders up to date, coaching them to value learning as well as delivery and to trust our teams to decide their most impactful next steps. Over time, they will come to look forward to the insights as much or more than the demos of built software.

The Built-Learned-Planning Demo at work for the Shutterstock Editor team

For example, consider some highlights from a BLP Demo we did when I led the Workflow Group at Shutterstock.

The Shutterstock Editor team showed text outlining during the Built section of a BLP Demo

During the Built section, the Shutterstock Editor team showed the addition of the outline functionality for the text tool. Instead of just showing that we built an outline tool, we described the problem — when users add text on top of an image, it can be hard to read the text if the image has lots of color variation. Because we were developing iteratively, we wanted to develop the simplest solution. After exploring the feasibility of both outline and dropshadow effects, we started with the solution that we could get to market most quickly. After this explanation, our engineers demo’ed the outline functionality. Instead of getting a “that’s neat, but so what?” reaction, our stakeholders were able to understand what value the functionality delivered and why we’d chosen the scope that we did.

During the Learned section, one update was usage data on a feature over time, highlighting how the usage changed after we rolled out a UX update. Sharing lessons such as this on how a UX change drove increased usage of a feature helped to shift stakeholder’s conversations from output to outcomes.

For the Planning section, the workflow team, which was in charge of features on the core site that support and enable customer workflow, shared a high level look at ethnographic research plans to dive into the experiences of a target set of customers. This moved the conversation to how we would flesh out details on the initiatives we were exploring for future sprints and gave us a chance for early feedback on our plans.

These highlights of a BLP Demo from my time at Shutterstock illustrates how the simple practice of adding product discovery to the sprint review pushes the team and stakeholders to add a focus on measurable outcomes and lessons learned to the usual cycle of shipping code.

Get your own Built-Learned-Planning Demo started quickly

Here’s a template that I use to get a team started on these Built-Learned-Planning Demos. Feel free to copy and edit it for your own uses.

The three parts of the BLP Demo

If you are a team member, try working this into your team’s practice. If you are a leader, try putting support in place to encourage this on your teams, and hold them accountable to produce and share learnings every sprint. This will plant the seeds for your organization’s continuous discovery practice to grow.

At young companies, practicing this will help you scale without losing your customer focus. At existing companies, these teams can become the impetus for cultural change that drives innovation.

As with any process change, you need to try it for a few cycles and iterate to figure out what works best in your organization. Then please let me know how it’s working for you and what challenges you’ve encountered along the way!

Roadmaps, Stakeholders

Techniques for managing timelines on Tech Republic

I talked with Moira Alexander about using long term agile planning exercises to set teams up for success and communicating candidly with stakeholders to manage expectations.  Here’s an excerpt:

​When it comes to addressing expectations, Hester-Reilly says H2R Product Science focuses on regular communication with stakeholders about what’s known and unknown. She utilizes visuals to show the complexities in the problem and in the technology that her team builds. In the planning stage, Hester-Reilly walks her teams through a technique similar to agile planning poker, to improve work estimates. The goal is to help her teams identify various projects they’ve done and assign a value, then they look at the work planned and do a comparison.

Check out the full article on Tech Republic to learn more tips for scheduling large initiatives:

How to address project scheduling difficulties

Building Teams, Culture

How to find great women in tech in Stack Overflow Talent

I shared thoughts with Rachel Ferrigno for her piece about finding great female talent in tech. Here’s an excerpt:

Holly Hester-Reilly, Founder of H2R Product Science, prefers interviews that mirror the actual job functions of the position she is applying for. She says, “In terms of process and structure of interviews, I love it when employers focus more on asking for real demonstrations of my skill in the activities involved in the job, instead of focusing on the titles or roles I’ve played in the past. Since advancement within organizations is often affected by implicit biases or the impact of relationships that can be harder for women to build up in tech workplaces, de-emphasizing past roles and advancement in the interview process is a great way to identify female talent that will thrive on the job.”

Check out the full piece for more tips from women in tech on what employers can do to attract and retain a diverse workforce.

What Women in Tech Wish Employers Knew