An interview with John Cutler, senior product manager at Zendesk. John has written extensively on different product management topics. In one of his most widely read Hacker Noon posts, John discusses what he calls Feature Factories. In this post we explore how to avoid the feature factory approach to product management.
How would you describe a feature factory?
A feature factory is a product or company that measures output over outcomes. It’s a storypoint machine where the company only cares about is how much is being shipped. The company celebrates the work done and it feels like a conveyor belt going in one direction. No one is thinking about the question: is any of this really working!. No one is thinking if any of this complexity is adding business value.
There is a fundamental difference between knowledge work and the manufacturing industry. Output cannot be measured in the same way. You have to get that experience right for the customers and focus on the right outcomes.
Some engineers can be motivated by a decently challenging technical problem. There are aspects of building a product which is challenging regardless of whether it has impact. For product people, if there isn’t anything compelling about the product you can’t do your job.You just check out or leave so the impact matters.
What do you do to start focusing on outcomes and not output?
We have to start by measuring product differently.Leadership can measure every other department pretty easily in the company like operations and sales by thinking about the bottomline of the business. With disciplines like product and UX it’s much harder.
The next thing is to look at the problem and figure out how to create outcomes in that area. One thing I always say is if the product is not doing their job they are outsourcing product strategy to the rest of the company. Put the pieces of the puzzle together in a way which people can relate to and understand. Most people get that that product can be destroyed by unvalidated complexity. You have to figure out how to create a system of value delivery.
Avoiding focusing on output also means that it’s not that you are not working enough or working hard enough. It’s that you are working more thoughtfully.
Tip: One book I would recommend reading is The Phoenix Project which illustrates this well.
Let’s go through 10 ways in which we can do this:
1. Measure the results and learn from them:
Companies spend a huge amount of time making sure they are tracking every action of the user. The main problem I’ve seen is not lack of access of data but they don’t measure if any of the features actually work.
To start measuring the impact of releases it’s also important for a company to have a learning mindset. You need trust in your organization to measure impact because there will be failure along the way and some releases may not work as intended. The learning aspect of measurement is also different than core level KPIs — it’s how can we measure things to get better than just coming up with KPIs and OKRs. Differentiate measuring outcomes from measuring people and teams.
It’s also important to establish health metrics for the product to make sure user experience is maintained at all times as well.
2. Storytelling and rituals make difference
One small way to create a product mindset around outcomes is to create small rituals around reflection. I did this in the form of precap design studio where when you kick off an initiative you give a presentation with blank numbers. And we would reflect on the before and after regularly once we ship it.
Having a ritual of reflection is thinking about the before and after. By doing this you get the team focused on the artifact. Every two weeks do a retrospective on the metrics. Doing a little over time is better than doing a whole lot at once and over committing.
Another simple tactic is emailing out your metrics dashboard to the organization or sharing the retrospective meeting notes. These little things have an ability to become viral in an organization. Small rituals help build the habits.
People usually over commit in terms of the methodology. Small companies invest massive resources to try and do some form of predictive analytics which takes at least an entire day to even report on. Start with what’s sustainable and what you can keep up.
3. Avoid shuffling of teams frequently and the project culture:
Some companies get the product and engineering people in a room quarterly, and have engineers self select the initiatives they want to work with.
Sometimes shaking off teams can be advantageous but in this case the company gets obsessed with 150 things they have to do in the quarter and starts playing team tetris.
Product teams should be focused around a mission. The release of a feature is just the start of it and teams should get the satisfaction of seeing it through. The moment you start calling features “projects” then it signals that when it’s released that’s the end, when in most products it’s just the beginning.
4. Pay attention to how you celebrate product
Make sure you celebrate outcomes not just outputs. Some companies have demos where once in a week or two weeks the product team demos all the new features, which is the signal of being part of a feature factory.
There is a huge impact of how you celebrate as a company. You can talk about different things like we talked to so many customers and what we’ve learned, what happened with the feature we shipped and how the customers are reacting to it. Little words matter.
Support and marketing have the need to know what is being shipped — usually these shipping success theatres start when someone says “I don’t know what’s coming out”. That’s a different problem than knowing if the stuff you ship is working. Be good to other teams and relay features but don’t solve that problem by celebrating shipping only as a company.
5. Iterate immediately after you ship to achieve better outcomes
Some PMs tell engineers that improvements will come “later” and move on the moment the feature is shipped. Pulling a team to work immediately on something else and getting back to it after 6 months doesn’t work in practice.
Companies also don’t acknowledge what the opportunity cost is. Sometimes teams take weeks to understand what the other team has built to iterate on it. One way to convince your company (assuming your dev team costs $50k per week) is to ask management if they’ll be willing to spend $150k to come back to the item later. The best teams usually iterate on it to reach a better outcome.
6. Get into the discipline of removing complexity and features which don’t work
One good exercise is to match customer value streams to areas of complexity. There will be areas of the product that have a large set of features but don’t map to a lot of complexity. Killing features require multiple dialogues in organizations.
Once you start mapping your features to customer usage you can start digging into why and what you need to kill. In some cases you will see features being used for the way it’s not intended .
7. Make sure PMs do retrospectives regularly
Products are a series of decisions. Humans are terrible at talking about decisions retrospectively.
Usually product management meeting are about what we ship and status updates on ships that have sailed, they tend to be very reactive and there isn’t really any time for reflection.
No PM says they made a wrong decision a few months ago. It’s important for product people in the company to get together and discuss how the craft can be better as a whole and reflect on the decisions they have taken.
8. Get in the habit of iterating on features to achieve the right outcomes
In feature factories product roadmaps have no room for iteration and it’s all about shipping new features.
The trick to get the company to think around iterating is to craft stories everyone can relate to around adoption. For example, you can say we have 1250 customers right now who meet the category for which a feature is valuable and right now 100 of them use it, so we want to focus on getting adoption up.
You should be able to make assumptions for your people who will be adopters. Say you are getting 500 beta testers on a new feature every week, one thing you can also say is that we will keep iterating with successive cohorts till we we get a high satisfaction score from most of them. Tie it to metrics and customers and tell a good story.
Sometimes the people who get nervous with this approach are your engineering management who are highly incentivized by a feature factory and their measure of success is being responsive to product needs and keeping churn on the team low. They don’t want to be blamed for being slow in any way. Usually what happens in that case is that engineers resign to the feature factory model and say we did our part and it was a poor product decision. This just drives more output and not good outcomes which are perverse incentives for engineers as well. Make sure you align the iterations with good incentives for engineers as well. Incentives are a difficult topic, but if people are incentivized for the fast completion of projects you will get a different culture than if teams are incentivized and rewarded for outcomes and learning.
9. Don’t just focus on reactive prioritization
All product managers focus on prioritization — in many cases it’s the wrong type, it’s reactive prioritization all the time. PMs are always stressed out with so many input channels and are trying to balance all of them to ship things and getting sign-offs from the company. Much of prioritization is about making educated guesses and making sure all stakeholders are happy (even if there is very little data backing it up). These guesses are important and it’s good have a good set of heuristics but don’t stop there.
Have conversations about the cost of delay — what will people pay to have a feature out today and not delay it. Have the conversation about whether is really worth it to us. Are we shipping value every sprint and by value it’s not just shipping, it’s learning and risk reduction as well. That’s more important to focus on. Force different types of conversation in the organization.
What people are missing is talking about if the product delivers continuous value and continuous learning. You want to bet on your horses when they are moving not before it and spending too much time on the reactive prioritization part.
10. Avoid handoffs and tackle bottlenecks
In a lot of companies it’s all about being more efficient and product management teams are outsourcing to other teams to get work done efficiently. So people start focusing on local maxima and not global maxima and managers end up playing team tetris. Customer feedback is a great example of a hand-off gone wrong. Your team ships a feature and you wait patiently for support to tell you when things have gone wrong.
The culture of handoffs also means the culture of working around people. Instead of tackling bottlenecks you start working around them.
The key to solve it is to visualize the system and show what steps things go through before it gets to the right people who can really act on it. Periodic reflection helps as well. To deliver 3x value from your product you have to make big shifts in the way you operate and chasing local maxima won’t do it.
Finally John, what’s your product management process?
I don’t use scrum and don’t do story points. It’s important to ask oneself why are we hiring the tools we are using and then use them. Scrum in a toxic and non trusting environment just becomes more toxic. If scrum is a good template for you go ahead and use it. But if people say they want to use scrum to just measure velocity or holding engineers accountable you want to ask why.
For example, we are good at writing stories and decomposing problems in general we take 3–5 days for every story so you can easily calculate velocity, we don’t need scrum for that.
We use JIRA but it has it’s limitations, it’s built on the philosophy that there is only one owner so you start to develop a shadow economy if engineers need help from each other. Know the limitations of the tools you are using. Kanban is an amazing visualization tool which I like.
The more important question to focus on is what’s the outcome and are we delivering continuous value every sprint which is what we try to do.