Season 2 Highlights: The Product Science Method in Practice
We look back on highlights from Season 2 to illustrate the Product Science Method and how to use it to build sustained success.
.png)
Holly Hester-Reilly
Before we finish Season Two of the Product Science Podcast, we wanted to take a moment to reflect back on all of the great tidbits we’ve heard and pull out some key takeaways.
At H2R Product Science, we use the Product Science Method to help you turn user research into an actionable plan to guide long term growth. In this episode, we look at the three steps: understand your customers, know your market, and lay the foundations, and bring together quotes from our guests who have lived that journey for themselves.
Season Two has been a great ride, and we hope you’ll come along for Season Three.
Subscribe for the full episode on Apple, Google Play, Spotify, Stitcher, and more. Love what you hear? Leave us a review, it means a lot.
Resources
- The 2nd Nir Eyal Hypothesis: When We Understand our Triggers and Plan Our Time, We Can Become Indistractible
- The Kate Rutter Hypothesis: Things Can Seem Simple and Still Be Very Hard
- The Audrey Crane Hypothesis: If You Don’t Hire Enough Designers, You Don’t Get No Design, You Get Bad Design
- The Jim Morris Hypothesis: Product Teams Do Best When They Build Just Enough to Learn
- The David Bland Hypothesis: Assumption Mapping Before Testing Business Ideas Facilitates Better Product Decisions
- The Heather Browning Hypothesis: Great Product Design Can Make Healthy Behaviors Easier
- The Tim O’Reilly Hypothesis: Build a Market by Building an Ecosystem
- The Susan Goebel Hypothesis: Bringing Structure to Startup Chaos Helps Teams Develop Breakthrough Growth
- The Janna Bastow Hypothesis: True Product Companies Step Back, Focus, Measure, and Iterate
- The Dan Melinger Hypothesis: Product-Led Growth Leaders Align Companies and Teams on the Fundamentals
- The John Cutler Hypothesis: Great Product Leaders Foster an Environment Where the Best Decisions Can Happen
- The Babur Habib Hypothesis: Rapid Iterations Drive the Slow Growth That Overcomes Inertia
Quotes from Holly Hester-Reilly in this episode
An outcomes-based roadmap focuses on a sequence of user outcomes you’re going to drive in each stage in your work, and makes sure each user outcome has a key clear metric for whether you’re getting there.
Great products and great companies are customer-obsessed. If you want to build something truly innovative, start by empathizing with your customer. The best way to do this is to talk to them – lots of them.
When it comes down to it, getting that balance right is the key. Knowing you need to make a change is important, but there are some changes you don’t want to make overnight, or even in a year.
Don’t be afraid to admit when you don’t know something, but make sure you figure out how to fill in that gap with actual user and market research.
Transcription
Nir Eyal: Yeah, so let’s start with just an overview of this model of becoming indistractable. There are only four steps. The first one is to master these internal triggers, but in order to really understand how do we avoid distraction, let’s make sure we define the term, so we understand what we’re talking about.
To understand what distraction is, we have to understand what it is not. So the opposite of distraction is not focus. The opposite of distraction is traction. Both words come from the same Latin root, trahere, which means to pull. Both words end in the same five letters, A-C-T-I-O-N, that spells action. So traction is any action that pulls you towards what you want to do with intent. The opposite of traction is distraction. Any action you do that pulls you away from what you want to do with intent. So that’s a really important point. You can’t call something a distraction unless you know what it distracted you from. The second step after we’ve mastered the internal triggers, the second step is to make time for traction. That means we have to plan out our day, and we can talk about those strategies as well. The third step is to hack back the external triggers. We know that what pulls us towards either traction or distraction, what prompts us are either these internal triggers or external triggers.
So internal triggers we have to start within, we have to figure out strategies to cope with this discomfort, but we can also do all kinds of things to hack back the external triggers that lead us to traction or distraction. Now the external triggers are the usual suspects. The pings, dings, rings, all of these things in our environment that prompt us either to traction or distraction. What many folks don’t realize is that these external triggers go well beyond just our devices. Open floor plan office, huge source of distraction. Of course, email, group chat, meetings. Oh my God, what a huge distraction meetings can be for the average knowledge worker. So there are all these environments in our life that we have to hack back these external triggers. Then finally, the fourth step is that we can prevent distraction with pacts. So the antidote for impulsiveness is forethought.
Holly Hester-Reilly: That was Nir Eyal, our second guest this season on the Product Science Podcast and author of the book, Indistractible. We had a great conversation about how to minimize the distractions in your life and get that traction Nir talks about.
We’re wrapping up Season Two of the Product Science Podcast, so I wanted to take this final episode to highlight some of the amazing insights our guests have shared over the past six months.
At H2R Product Science, we use the Product Science Method to help organizations develop an evidence-based growth plan. What it means is working through three main steps to better understand your customers, understand the market you’re working, and then lay the foundations for long term growth. Throughout the process, you’re continuously doing customer research, gathering data to gain new insights, and iterating and testing to challenge your assumptions.
We’ll start with step 1: understand your customers.
Great products and great companies are customer-obsessed. If you want to build something truly innovative, start by empathizing with your customer. The best way to do this is to talk to them – lots of them.
It goes beyond user interviews, though. In order to build something truly useful for them, you’ll need user science – the application of psychological and behavior science principles to understand and predict user behavior.
Kate Rutter: What was so interesting is that all of the techniques and approaches in Build Better Products, yes they’re just good methods to build products but they’re also much more deeply integrated with what I would call modern software, modern product creation, including agile behaviors over waterfall and not a lot of big upfront research and design, but integrating that research and design through the process. And being an absolutely user research as a core fundamental part of great product. What I felt was interesting is for people who were coming into the product world now, Lean is so pervasive that it’s kind of like, “Well of course. This is not a big deal. Stop fighting the battles, the war has been won,” kind of thing.
Because it makes sense. It makes sense that you would ask a question and see to what degree you could prove that it’s true, or see evidence that it’s true. That just makes sense. People have been learning the scientific method for years and years. So for younger designers or newly minted designers coming into the world, they’re very comfortable with this sense of validation using research, using usability testing to gain insight to how they can improve their designs.
Holly Hester-Reilly: That was Katie Rutter, adjunct professor of design at the California College of the Arts and our guest on episode seven. But it’s one thing to take to users, and another thing to make sure you’re taking the right things away from those conversations.
In Episode 19, we spoke with Audrey Crane, author of What CEOs Need to Know About Design. One thing that came up was how important it is to be cautious with how you approach user research. She talks about this idea of “the crystal goblet” that I think is so important when it comes to getting everything you can out of your conversations with users.
Audrey Crane: And I use a metaphor with them. There's a great essay by a typographer Beatrice Warde called The Crystal Goblet and she's writing about typography. But very briefly what she says is, if as a typographer you do your job right, it's like being a great wine glass. You are thin, you are transparent, you're creating space, nobody really notices you. Nobody notices the glass, right? Because the whole reason for the glass to be there is to make the wine the best that it can be so that you can smell it better and see the color better and taste it better. And so she makes this parallel to topography that nobody should notice the size of the gutters or the font you picked or the line spacing. They should just be able to take in the content that you've typed that at its best.
I take that analogy and apply it to conducting research with users. And I say, "You guys need to be the crystals goblets. You need to be present as little as possible." And there's all these specific tips about how to do that. Like be quiet, count to five in your head. If there's silence to see if they'll fill that space in. Don't use your own words. Use their words. Don't ask leading questions. In fact, don't ask questions. They ask these trailing questions. "So would you say that experience of using this product was..." See, that's awkward and you want to say something right now instead of saying, "Was it good? Was it bad? Was it good or bad?" Right? And so I have seen, every time I run this workshop, the first thing somebody says is, "That was way harder than I thought it was going to be." Which is great because it's not rocket science, but understanding that you really need to apply yourself to take off your charismatic hat and be as boring as you can possibly be, as not there as possible, and make space for that person to fill. And you will learn so much.
Holly Hester-Reilly: At the end of the day, your focus needs to be on identifying their pains and ideal outcomes. Once you start to see consistency and patterns, you can draft a customer problem statement to put your plan on paper. What problem does the customer have that you are best positioned to solve?
In episode 21, we talked to product coach Jim Morris about how transitioning to this sort of process can be transformational for a development process, and how he saw it for himself.
Jim Morris: It was educational on the product side because that's when I went to my first really only Marty Cagan seminar. And that was the moment I realized I had been doing everything the hard way. Because I was building all the software. I was an engineer, I hired engineers. My tool of choice was engineering. You gave me a problem, I just wrote software. I wrote software, I had teams write software. I didn't know how to solve it.
Remember I didn't know how to manage product managers. And so at that moment, that aha moment, which many people have at these seminars, I realized, okay, there's a different way to do this that's faster than coding. And you basically use design and what we now call product discovery to figure these things out with consumers, with your customer, yeah, before you make it. And to me it became really viable actually about a year or so later. Because it's hard to actually adopt these techniques when you're in a company that has its own way of doing things. Right. One of the challenges of being a product discovery coach.
Holly Hester-Reilly: In step 2 of the Product Science Method, once you’ve developed a deep empathy for your customer, you need to Know Your Market.
That means taking your research and quantifying the most interesting areas. Looking at the data for what I call “golden insights” with segmentation to compare behaviors and metrics across groups.
In Episode 13, we caught up with David Bland, founder of Precoil and author of Testing Business Ideas, who understandably had a lot to say on this topic.
David Bland: So it’s really interesting for me to see crowdfunding kind of play a bigger role now. But you could also do pre sales, mock sales, there are a lot of things you could do on landing pages where you’re price testing different product tiers, like some different service tiers of, Hey, we have this tier, this tier and this tier, and when they click on one, it doesn’t mean you build them right away. But there could be we’re still working on this, we’re not ready. Do you want to give us your email for when we’re going to roll out this tier? And there’s some case studies in the book of companies that did that too, in the past, but I think where it gets a little, I was a little awkward, or maybe uncomfortable for product managers is that it becomes really obvious that you’re influencing the business there. And I think product managers should embrace that because I feel that is part of the role is product and business are very integrated.
You know, it’s almost like a system and so you can’t have this amazing product in your business model because you’ll fail. And you also can’t have an amazing business model and a terrible product and you’re going to fail. So I think as product managers kind of grow and embrace experimentation and mature, I think they’re going to find that they have this responsibility to do more of this testing to help educate business, because I’ve seen this happen in the past where people build an amazing product, and they kind of throw it over the wall and say, okay, figure out how to charge for this.
And that never ends well, by the way, because it’s like, you’re asking somebody with almost no context of all the experiments you read and how you’ve grown and developed this product to say, Okay, now just come up with like, a model for it and just to sell it and so I do think earlier on we need to be testing for viability. Certainly like the R&D groups, innovation labs, I deal with a big companies, they’re being asked to come up with business models now, which terrifies them by the way because it was pretty much a business model free zone in the past and so they need to level up. And then certainly there’s some tools and everything out there that they can use now, but yeah, I think is just anxiety around viability. But I am a big believer in testing that earlier on if possible, and then helping that inform the conversation. When you go bigger later, you want to scale something you want to know you’ve kind of tested the model out.
Holly Hester-Reilly: A key here is product market fit. That’s not just about understanding your product, but understanding the market itself and your product’s place within it. What are you disrupting? What are you doing differently? What are you doing that nobody else is doing? And is this all sustainable?
Heather Browning, VP of Product at Ria Health, had some special insights about this related to her experience in educational game development that she shared with us in episode 16.
Heather Browning: So as I worked for the early stage startups, I initially went into UX design and then product mostly from the advantage of being on small teams and wearing many hats and finding sort of the level of question that most interested me. And the idea of what is actually going to be a good market fit is the type of question that I was most interested in solving, because again, I had been feeling very frustrated for making these amazing experiences and amazing products that then maybe didn’t quite have enough of a market fit to take off. And that was sort of my move into product, is really wanting to answer and understand those product market fit questions better.
Holly Hester-Reilly: Those are some of the biggest questions that come up at early stage startups and I certainly talked to a lot of people who have the same frustration you just described. So I’m curious if you could tell us a little bit, I mean, if you’re able to share a story about what were the things that you saw at the company you were at. I guess, did you get enough access to be able to do some of the research that would help you see why it didn’t reach product market fit, or what it was like being there at that time?
Yeah. Well, I think actually in the educational game space, the problem that I still think is yet to be cracked in educational technology and in general is sort of how to actually get into the hands of students, and it’s just the pipeline into schools has not yet been disrupted. And that was also a big reason why I didn’t go ahead and start an educational company myself, was the question that if you are an educational tech fit that you have to have the answer to is how do you disrupt the pipeline into schools? And I just didn’t have an answer to that question. And so without an answer to that question, it doesn’t matter how amazing, how engaging, how educational the software you build is because your end user and your buyer is not the same person. You have to solve the, how is the person who’s going to buy this going to do that? And I think that’s a really still an open question in educational software.
Also, because it was such a great team and I really solidly stand by the educational efficacy and also the enjoyment of the games we made. There’s still some of the proudest things I’ve ever made, but it was a really great learning for me, especially as my first job in software and my first venture-backed startup. It really helped me understand I think product in a way that I maybe wouldn’t have otherwise because if that had worked out, I would sort of just have this idea that, “Oh, if you make a good enough product, that’s all you need.” And that’s not the reality of what makes a startup work. So I’m really grateful for that experience and learned a lot out of it, and I think it really set me on the path I’m on now.
Holly Hester-Reilly: This also makes me think of Tim O’Reilly of O’Reilly Media and our first guest of the season. We had a wide-ranging conversation that is definitely worth listening to in its entirety, but one thing that stuck with me was this idea that maybe we’re thinking about markets the wrong way...
Tim O’Reilly: And of course you see this in the political debate today. Where people are starting to say, again, windmill. We’ve built an economy where some people win really, really big and other people don’t. And again, it’s not that simple because even in the parallel between say an Uber in which passengers really win and Uber really wins but drivers get screwed. It’s kind of analogous to the free trade regime that we have in today where giant corporations really, really win, people in developing countries really, really win. And American you know, workers were left out in the cold.
And so the question is how do we build better markets? There’s a book I love. It was actually given to me by Uber’s chief economist is called Who Gets What-and Why and it’s by Alvin E. Roth, is a Stanford economics professor who basically founded a field called market design. And he got a Nobel prize for his work on redesigning a kidney transplant marketplaces so they could be more efficient. And this idea has obsessed me over the last four or five years has been the great opportunity of AI and big data systems is actually to design better markets.
And we think that we have this optimal market where it’s just the invisible hand and people competing and so on. But we’re really moving into a world of algorithmically controlled markets and have the best intentions, we’re going to bring the world together by sharing and discover that the market you’ve created is actually a very bad market. And then you have to redesign and intervene. And so this, what I’m really spending a lot of time doing is trying to wake up Silicon Valley to the idea that the market is not a given, that it’s something you have to design and you have to design it with everyone in mind.
Holly Hester-Reilly: So that’s understanding your market. Once you’ve done the work and got a better grip on your product market fit, it’s time to Lay the Foundations.
This third step in the Product Science Method is where you actually create the structures that set you up for long term success and make a plan for growth. There might be designs or features you should start including to help further on down the road.
In Episode 10, bioscience researcher and product development expert Susan Goebel explains the balance you need to strike between your big goals and the metrics it takes to get you there.
Susan Goebel: You also have to have the component of the vision. So if you take the vision and you really focus it down on core values, and when I sit there and I say 10 years out, three years out, one year out, "Where am I going to be?" You have to be able to feel that in your core. If you can't feel it in your core after going through all the processes and the exercises that the EOS would take you through, you haven't got it yet. Right?
So the visionary comes in and he's got a thousand great ideas, and he comes over to the bomb squad and the bomb squad sifts and sorts with the vision and the document of what the values, and this idea when we measure it up, this works and this one we really can't resource it yet. It's a good idea, but we're going to put it on the shelf for just a minute, and we're going to focus in on these ideas when we go through growth hacking and when we go through product iteration. So let's make sure we've done a really good job at evaluating our resources. Let's make sure that we have absolutely everything that we need in order to successfully iterate. Right?
I find that that is one of the big challenges with the chaotic startup, is everybody's so busy doing, they forget to just take their time. It really is valuable to take the step back and go, "Do I have the data? Do I have KPIs? Do I have metrics? Am I using the right metrics, the leading metrics, the lagging metrics? Am I using vanity metrics?" That's really where EOS starts to come in and go, "People, vision, data, processes," and when that all hits the road, now you've got traction.
Holly Hester-Reilly: Like Susan talks about, this stage of the Product Science Method is where the rubber truly meets the road. It’s also where you want to start thinking about a roadmap. How are you going to answer executives’ requests when they ask for one? Or stakeholders? Or salespeople? And how do you set up enough guideposts for the team to understand where they’re going?
A lot of people, when they hear a “roadmap,” they think about this feature-based roadmap where the executive team asks you to say what you’re going to build and when you’re going to build it before you’ve even finished doing the work. Or doing the discovery research, or figuring what technical hurdles you’re going to have to overcome. A lot of us have been there and know how difficult and painful that is, and I really love what Marty Cagan has to say about the problem with roadmaps.
Instead we do an outcomes-based roadmap at H2R Product Science, which focuses on setting up a sequence of what user outcomes you’re going to drive in each stage in your work, and making sure that each of those user outcomes has a key clear metric that measures whether you’re getting there. Using that to keep your team aligned and to communicate to your stakeholders and to your executives what it is that you’re going to be doing in each phase of work. That’ll help keep your team aligned and that let you know you’re on the right track.
For Janna Bastow, Founder and CEO of ProdPad and our guest on episode 3, it came down to identifying user behaviors that impacted their conversion rate.
Janna Bastow: So we had a pretty high drop-off right from the trial sign up. And so we decided to really, really focus down, we had to drop everything we were doing, all development work, all work that we're doing and focus on improving this one conversion rate. And at the expense of all of our work, we cut down all of our cost base, everything and focused on this one conversion rate. And it worked, we managed to do a few things. One of them was, we actually focused on changing the trial length. So we used to have a 30-day trial, but we looked at the numbers and we did an analysis. We looked at; what kind of activities people would do in their trial? And we realized that we could tell with 85% certainty by day nine who was likely to buy or not.
And so, if we've realized who was going to buy by day nine, why did they have an extra 21 days to make up their mind or not? Why don't we just ask them for their credit card on day nine or some other day? Why 30 days. 30 days was just an arbitrary thing. So we actually chopped the trial time in half. And actually, at that point in time, our trial conversion rate went up because the new cohort had less time, and because they had less time, they actually felt more pressured to use more of the features. And by using more of the features, they actually got more of the value. Because if somebody uses the features, they're like, "Oh, now I, I've used the features, I now know how to use the features and therefore, I see more value, and I'm more willing to give you my credit card."
So now, when you join ProdPad, you actually get seven days, but if you do key actions like fill in your company name, you get two days free. Tell us the name of your product, you get a couple of days. Add your first idea, get a day free. Set up a JIRA Integration, get four days free. Invite a colleague, get a few days free. Add your billing information, get some extra days free. So you actually earn extra trial time based on how you're actually using the app itself. And so what this is actually doing, each time we do these actions, you get a little demo of how to do it and that sort of thing.
So people are learning how to use the app as they're going, they're getting that demo, they're learning how to use each of the pieces of the app. And they're unlocking pieces of the app, and they're also getting the additional trial time that they wanted. And this itself actually bumped our conversion rate up hugely and bumped our numbers up and got us out of that slump that we had in 2015. So instead of solving it by throwing money at it and throwing investors at it, we solved it by throwing more of a UX and product mind at it.
Holly Hester-Reilly: On episode 15, we spoke with Dan Melinger, Principal Consultant at Realtime Lab, who also emphasized why organizing your team around key metrics is so important for sustained growth.
Dan Melinger: When you can have very clearly a stated objective for your team that everyone understands, and part of that is understanding how you measure success. That's the key results, the KPIs or metrics, that are ideally numbers, and you define what looks like success, you can point to that and say, is this working or is this not? And you all generally come up with the same answer or can at least get to the same answer. And that was a great tool for aligning us and also for ensuring that people understood at any given moment, whatever they're doing, am I helping us work towards the achievement of these objectives? And that really helped us as an organization.
And since then I've helped other organizations kind of implement OKRs, and I found it to be... It can be tricky, especially if the organization already has that philosophically like, "Oh yes, we understand we're objective focused," many don't, and that can be tricky because it's a cultural shift. But I think it's incredibly powerful cultural shift and often one that it's worth investing some time and even some pain in making.
Holly Hester-Reilly: What’s important to remember here is that there is no magic formula, there is no silver bullet. There’s no world where you set something up and then just sit back and watch it happen. Continuous growth is about continuous practice, and so you’re always striking a balance between what changes you need to make and what will work with your team and organization. This came up in my conversation with John Cutler for episode 9. He’s a Product Evangelist at Amplitude, but has seen a lot over his long and wide-ranging career in product—not to mention the time he spent driving a rickshaw and developing indie games in New York City.
John Culter: You know, I wrote this post called The Way of Ways, and everything is just like this. And once you see it, and this goes back though to … There’s so much talking, how are you doing … Like, what are you doing? What are you doing week in, week out. And what I’ve noticed at the end of the day is a lot of these … This is where technical practices, and I have so much admiration for the engineering leaders who kind of create the momentum here. Half the anxiety with product managers and all these things is that the ability to have a resilient technology that can accommodate some of these changes in learning and not shipping every day or every hour, but just at the right cadence to learn that you need to learn without overwhelming that team.
So, what you see is this wicked cycle, like when you don’t have love on that side to create a resilient architecture and resilient things, you find all these silver bullets which then make it even harder for them to do what they want, which then causes more silver bullets, which means you play more Tetris. And then before you know it, no amount of reading Marty Cagan posts or talking about MVPs or doing whatever will work. Yet, we’re all sitting around with our thumbs under our knees because we’ve created the mess that now is causing nothing to be able to happen. So, product is often unconsciously complicit in the degradation or entropy of their whole system that they have at their companies.
And then there’s this finger pointing game that goes on. But the reason I’m bringing that up is that there’s a lot of talk stemming from just impatience and anxiety about our jobs. But when you’re on a team that’s just going to work and kicking butt, you spend a lot less time talking about theoreticals and you talk about what’s happening right now. What does the information say? What does the data say? What did that customer say? What did we learn this week? That’s why we don’t see as many blog posts from the companies that are kicking ass.
Holly Hester-Reilly: When it comes down to it, getting that balance right is the key. Knowing you need to make a change is important, but there are some changes you don’t want to make overnight, or even in a year.
This also came up when I spoke with Babur Habib, who has an extensive background in EdTech but has now found himself trying to create organizational change and innovation in a different way as Founder and CEO of the Portfolio School.
Babur Habib: You can't change things too quickly, especially in a school kind of setting, but if you're changing it, you have to go to their core principles as well to understand, hey, this is why we believe we started this. What are the values there that we can't just give up on? And also, I think there's ... In that vein, one of the things that ... When you're looking at sort of making a ... you're looking at a really big product, like okay, we want to change the game for schools, that's not sort of a small idea. It's a big idea, right? And how do you tackle something that big and figure out how you're going to break it down and work on things that are most important in the beginning stages. And how are you going to wedge yourself into an already very big ecosystem and then be able to expand from there, as opposed to building products which may be improving on a couple of features or are sort of smaller in scope.
If there are entrepreneurs, product designers who are thinking about ... I hate using this word, but that's disrupting certain industries, right, I mean I think you have to always keep an eye on that big idea, but then be very sort of focused on okay, how are you going to get into it? What are you going to do in your first few years? How are you going to then grow from there? Those are important things to ... and what are the things you're going to keep changing, keep iterating on that'll actually take you there to that big idea, the big goal that you have in mind?
Holly Hester-Reilly: What comes up again and again with all of our guests on the Product Science Podcast are a few simple takeaways. Don’t be afraid to admit when you don’t know something, but make sure you figure out how to fill in that gap with actual user and market research. Don’t be afraid of failure, because in every failure is an opportunity to learn how to do things better. Even better, plan a system of working that expects and celebrates learning over planning, and everyone will get comfortable with the failures along the way.
Season two has been filled with great conversations, so be sure to check them all out at our website, h2rproductscience.com, or subscribe on whatever platform you get your podcasts on if you haven’t already.
I’m looking forward to new conversations and insights in Season 3. For now, keep learning, keep improving, and I’ll catch you soon on the Product Science Podcast.
