Leave a comment
Get the GH Bookmarklet

AMAs

Chris is the Head of Growth at Mozilla for Firefox on desktop, iOS and Android. For the past five and a half years, Chris has driven initiatives that has helped take the non-profit open source Mozilla from being data averse to data-driven. These initiatives were based on a foundation that Chris built from scratch and were based on insights. The path to this new future of data was not straightforward given there were not always robust scalable enterprise solutions that would meet Mozilla’s open source preferences.

After selecting vendors and doing custom installations, Chris was able to perform Growth experiments resulting in new insights that were previously unknown. Throughout the experiments they were able to keep users anonymous and their information private. What was once a future plan to be able to A/B test experiences, Chris turned it into a reality and was able to perform experiments to improve acquisition, activation and retention rates, and ultimately grow Firefox’s user base. Even though most professionals in modern digital marketing had access to dozens of applications to manage and understand their user base, the Growth team was able to successfully perform tests with just a single few tools.

Prior to Mozilla, Chris was Director of Web communications at Penn State University and served as a chair on the University’s web council. At the University he transformed a college’s web presence from hundreds of separate websites to a centralized web service running open source technology which became the most progressive and advanced college web team in the United States. The SEO and web analytics on the college’s websites proved to be valuable information to the faculty which eventually used the data as inputs to their research projects.

Before leadership roles in marketing and IT, Chris was a full-stack engineer for a decade. He started his career as a web engineer during his sophomore year in College at one of the first Internet providers in Pennsylvania. He did everything from web development, Linux server management, network monitoring, and building databases. With Chris’ diverse background in engineering, marketing, and business, his t-shaped skills made him the perfect candidate for a future in Growth.

Chris is a Computer Science graduate from Edinboro University of Pennsylvania and a first-generation college student from a small town in Pennsylvania where he spent his youth taking things apart and making them better. Outside of Growth, Chris is a semi-professional racecar driver and races sports and Formula cars in sprint and endurance events. He and his family live in the heart of Silicon Valley and swear they are not anything like the hit HBO show.

If you want to know more, talk to Chris on Twitter @chrismore and connected with him on LinkedIn.

He will be live on September 13 starting at 930 AM PT for one and a half hours during which he will answer as many questions as possible.

  • HQ

    Hila Qu

    about 1 year ago #

    Hi Chris,

    Such an amazing opportunity to learn from an expert like you. Thank you for doing this!

    I stumble upon this Mozilla's growth team wiki online, which looks super interesting, is this the place the growth team collect ideas and share learning?
    https://wiki.mozilla.org/Growth_Team

    1) I am curious how does the growth team get started? How does the team find the position within Mozilla and get support from the broader organization? What’s the process the growth team follow today to run experiments, and share learning?

    2) In your bio you mentioned about setting up infrastructure for data and testing so that your team can A/B test experience, what’s the most challenging thing about that for Mozilla? How did you overcome that?

    3) Lastly, can you share some of the most successful activation and retention experiments the team did, and what you learned from it?

    I have so many other things I want to ask, but will stop here :) Look forward to your insight!

    Hila

    • CM

      Chris More

      about 1 year ago #

      Hi Hila.

      Thanks for submitting lots of questions, so let’s jump right in!

      1) How does the Growth team get started

      We’re a curious bunch and we are always asking questions. We also get a lot of feedback from other employees, our volunteer community, and our users. We take all of that feedback, quantitative data, and qualitative insights, and then we develop a hypothesis. We put that hypothesis on our backlog spreadsheet, categorize it, and doing a prioritization based on the traffic level, potential impact (to acquisition, activation or retention), and the level of effort to perform the experiment. Then we pull the top priorities off of the list, develop a recipe that includes the hypothesis, insights, test plan, and success metrics. After we have it all documented, we determine who we will need to help us perform the experiment and file any required bugs/issues to ensure we are unblocked. After the test is all set up and we have any needed approvals, we kick off the experiment, let it run for the estimated time, and then finally shut it off. Once the experiment is over, we go back to the recipe wiki page, and update all of the funnel and success metrics. The last two steps are the most important. We determine if we can accept or reject the hypothesis and then determine next steps. Without next steps, an experiment could be a complete waste of time. Even if an hypothesis is false, determining what you should do about that is critical.

      2) How do we get support

      Some experiments we can do completely by ourselves and others are much more complex and require coordination of multiple teams. We have designed a workflow process that is pretty comprehensive that covers most of it. Since Mozilla is nearly 100 major websites that are all separate and owned by different teams, opportunities exist all over. The internal process generally states that is the Growth team is performing an experiment in a place that is owned by another team, that we will give a heads up to that team on what we are doing and when we are going to do it by. If they don’t have any issues, we just proceed. Also, if our experiments are injecting any code into another websites or product and haven’t gone through the typical development cycles, we get a develop from the touchpoint or product to review the new code. The code review is mainly just there from a sanity perspective to ensure we are not breaking anything. For the most part, it works well, and doing experiments in other people’s “sandboxes” has brought attention to the entire Growth mindset and process and this has caused other teams to adopt similar practices. Thus, there are mini growth teams all over the organization now that have grown out of the seeds we’ve planted over the years. Pretty exciting!

      3) How we share learnings

      This is a good question. We have an internal mailing list that we share upcoming hypotheses that we are going to test and the results of recent experiment. We also have company-wide Firefox meetings that we share our learnings. I like to do them in a format of “Did you know?” so that the audience can walk away with something they hopefully didn’t know before and can apply to their work. People love the learnings and they are always asking for more. We also want to find a way to share learnings externally without having to worry about sharing any sensitive product metrics that could be misinterpreted.

      4) A/B testing infrastructure challenges at Mozilla

      I think the most challenging part of using any analytics or A/B testing tool is that there are not many great open-source options available. Also, given that Mozilla is very strong on privacy and user choice, performing an A/B test on user anonymously can sometimes feel anti-open. The way that we handle this is by working with our vendors and having them improve their products so that meet open web standards that that users have the ability to opt-out of future experiments. One example is Google Analytics. In Google Analytics’ account settings you have the ability to opt-out of any of Google’s uses or 3rd party uses of the data that is collected in Google Analytics. Those settings are because of Mozilla working with the Google Analytics team to build in features to give Mozilla more choice and control over the data. We also wrap all of our A/B testing and analytics products in a Do Not Track (DNT) conditional so that if users have that setting enabled in their browser, none of our Growth tools will fire off and a user has an choice to not be involved in testing. Before any tool is used, we go through a rigorous legal and privacy review of the vendor’s data practices and to ensure it meets our strict policies. While this process slows down the implementation of tools that can be used in Growth and dramatically reduces the tools available to us, it is for the right reasons. We feel good about that and how our mission comes through what we build and how we build.

      5) Successful experiments

      Here’s a few good ones.

      Onboarding: We have found recent successes with build a segmented onboarding experience that doesn’t get in the user’s way and they can re-engage on an as-needed basis. For the segments, we have found new vs returning users to be a strong segment to provide unique experiences given their needs are quite different.

      Activation: We have found the conversion rate to activate increases dramatically when we say this is the final step or that they are almost done. People generally like progress and especially progress that is about to be over. When people choose a browser, they probably have some task in mind and if you can reassure a user that whatever setup steps they are doing will be completed very soon, they are more likely to finish the setup then skip it. Especially, if the activation provides them clear value.

      Self identification: We were trying to determine what is the best way we can segment users into anonymous buckets so that we can provide content that is best suited for them. We tried something that we haven’t done before and that was simply asking them them a very simple question. We expected about 10-20% to self-identify themselves for segmentation and it turned out to be 50% of all users! We could have spent months trying to come up with a really complex method and just asking users for us most of what we needed. Plus, the user has just made a really simple investment in the product and that investment turns into improved retention.

      Thanks for the questions!

      6 Share
      • HQ

        Hila Qu

        about 1 year ago #

        Wow Chris, this is a gold mine here in your answers! Thank you for much for sharing!

  • ES

    Edward Stephens

    about 1 year ago #

    Hi Chris,

    Cannot wait for this AMA - have been a big fan of Mozilla.

    1) When it says Mozilla was data adverse, what legacy issues did you inherit and how did you overcome the lack of data drive within the culture?

    2) On the subject of data again what do you see the limitations for products like Duck Duck Go are given that they neglect data capture? What technologies are Mozilla focusing on to encrypt personal data in the future?

    3) What web technologies excite you and what technologies will be critical to Mozilla's continued growth?

    I can't wait for your answers.

    Best wishes,

    Ed

    • CM

      Chris More

      about 1 year ago #

      Hi Ed!

      Thanks for being a fan of Mozilla and the open web!

      Let’s get rolling.

      1) Data and culture

      This is probably the biggest challenge I’ve faced since I’ve been at Mozilla, but it’s a challenge that makes Mozilla who we are. Without that challenge, we probably wouldn’t have a mission and we wouldn’t build the products that we do.

      The challenge is simply stated as I’ve heard it said “If we don’t have any data on anything, we have nothing to worry about”. That was the status quo for quite a while, and the only data that we had was to help basic questions like “Are people downloading Firefox?”, “Are people installing Firefox?”, “Is Firefox crashing?”. The only data we really had was the most basic steps of the funnel and almost nothing on the user experience. There was some data available with user experiences, but it was all opt-in and not many people understood what opting-in meant, thus there were few actionable insights.

      To change the culture, you first have to pick a tool or build something that the organization can get behind for an experiment. That requires reviewing the organizations behind the tools that we are interested in using to ensure their data and privacy policies work with Mozilla’s mission. After passing all of the internal reviews, we had to perform experiments to validate that the tool or new data source would be something useful to do on a regular basis. That means, you are looking for wins, which do not initially come quickly in growth. We try to iterate and move as quickly as possible to validate not only the hypotheses, but the tool itself. If we find wins and it looks like we can build upon those insights, the tool gets added to our list of approved tools. Given that we strive to be as transparent as possible, we make sure our privacy policies reflect the tools that we use and that there are understood ways to opt-out in the future. It may seem like a lot of hoops to jump through to ultimately make a better product, but that is what makes Mozilla unique in the world.

      2) Not capturing data

      Similar to as I mentioned above, if you don’t capture any data, you just have to be honest on how much you will learn and how you will improve your product. It is possible to do it without data and it just means you need to find other ways to develop insights. For example, you may offset a lack of product or experience data with doing more user research studies and interviews. Not all data is bad and the way that we like to look at it is as follows: “Will this data benefit the user themselves?” If the data won’t help improve the product or provide direct value to the end user, we just don’t include it in the product or experiences. Given that we are a non-profit mission-driven organization, this seems like a good way to evaluate data and always putting the interests of the users first.

      3) What technologies excite me

      I am very excited about the future of Firefox and thinking about Firefox as being more than a means to search and browse. We are attempting to reinvent how users discover content online as most of that happens now through social networks and search engines that have interests beyond their end users. I’m excited about how Mozilla can unlock discovery on the Web and revolutionize what a browser can be in ways similar to how Firefox 1.0 changed everything. There are scaling and technical abilities now on the Web that didn’t exist in 2004 which means there are new growth opportunities and possibilities.

      Thanks for the questions and let us know what you think about the latest version of Firefox. A new version is baked every 6 weeks. :-)

      3 Share
  • AS

    Alex Sherstinsky

    about 1 year ago #

    Hi Chris, thank you so much for doing an AMA with us! At Mozilla, how is the Growth Team placed organizationally? Is it a part of the Marketing function? Or is it its own independent unit? What were the reasons for this arrangement, and how did it become that way?

    • CM

      Chris More

      about 1 year ago #

      Hi Alex.

      The history of growth at Mozilla is a really interesting story and I will break it down by year.

      2011:

      I started with Mozilla to build their full stack web development team that we later called “Web Productions”. The team would later have front/back-end web engineers, UX, project management and analytics. We wanted to build a team that could do everything from concept, to deployment, and optimization. Later in 2011, we moved the Web Productions team to the marketing organization since most of the requests were coming from them. Moving the Web Productions team into marketing also made it easier to align on priorities and goals.

      2012:

      I wasn’t happy with how we were making data-driven decisions on our web properties and I rolled out Google Analytics Premium (with additional privacy measures in place) to nearly all of Mozilla’s 100 websites. The driver for rolling out analytics was so that we had a baseline understanding of how our experiences were performing.

      2013:

      The Web Productions team started to perform Mozilla’s first A/B tests and found wins/saves (loses) with our download page. We knew that downloads were not the only KPI that was important, but it was something we could do, so we developed a new muscle on hypothesis based experimentation. At this point, we weren’t calling it growth, we were simply validating assumptions made years ago by various teams and trying to improve our websites.

      2014:

      This is the year everything started to change. With a solid foundation in analytics and experience performing basic A/B tests, the idea of growth would soon take off. It started with a conversation I had with our CEO, where I told him about hypothesis-based testing, and some of the wins we found in our spare time. Then, it became a reality. He gave me six weeks to drop all my current responsibilities, and focus on experiments, take risks and see if we can move any levers that would make an impact on Firefox’s future growth. The six weeks scratched the surface on what was possible, and was a success in experimenting with the idea of growth. Immediately after the six weeks, Chris and Gareth Cull (https://twitter.com/garethcull) left the marketing group and joined the Firefox product team to create the first official growth team. The first thing the team set up was a new process and documentation on how the team would evaluate new ideas and perform experiments. The team of two needed a bigger team and thus identified 50 people that either worked with data or had the mindset for experimentation. The growth team ran hundreds of experiments and shared the learnings throughout the organization. If someone had an idea within the organization, they would turn to the growth team to run experiments. Outside of running experiments, Chris spent most of his time educating people hypothesis-based testing and how these insights would be useful in improving our products and align with our mission.

      2015:

      A few key hires happened this year. We hired Alex Davis (https://twitter.com/davismtl) to focus on mobile growth and Francesco Polizzi (https://twitter.com/francescostl) to be a Growth Engineer intern for the summer. Chris also brought Mozillian Justin Crawford (https://twitter.com/hoosteeno) from another part of the organization to join the growth team to focus on developer growth. The team was more than double in size and worked across many platforms, and as a result the team was able to run experiment at higher rates and help spread the growth mindset across the entire organization. Our documentation got better, our hypotheses improved, and we found lots of small to medium size wins across our user touchpoints that had gone unnoticed for years. Francesco ended his 2015 summer internship with a great presentation on growth and how the new approach changed the organization: https://air.mozilla.org/francesco-polizzi-marrying-growth-data-and-privacy-on-the-web/

      2016:

      In late 2015, Mozilla hired a new CMO, Jascha Kaykas-Wolff (https://twitter.com/kaykas) and his background in Agile Marketing and Growth made it a good opportunity to move the growth team back over to marketing. Jascha’s proposal was to set up small dedicated teams in marketing focused on moving a very specific KPI and build t-shape skills in people while using the Agile framework for getting work done. These small teams were called “durable teams”, but they were very similar to what the growth team was already doing and it was the perfect opportunity to create many new growth teams. Durable teams were created to focus on retention, acquisition, developer marketing, Mozilla Internet issues and more. Each team had a prioritized backlog, worked in two week sprints, identified clear KPIs, and the teams were made up of people who could do at least 80% of the work themselves, thus eliminating the need for external resources. Chris lead an additional team called “Firefox Desktop Retention” (FDR) with team members who had skills in product marketing, email, design, engineering, analytics, and project management. The FDR team’s KPI was moving user retention and they were going to get there be creating a completely new onboarding experience. Francesco Polizzi also came back as a Growth Engineer the summer of 2016. He helped engineer this onboarding experiment that was deployed in July. The results demonstrated that the first v1.0 of the onboarding experience improved user retention above their initial target. Now, the team is focused now on the next iterations to see if they can beat the retention improvement from the first version.

      The Future of Growth

      Like anything, change is inevitable. We are always evaluating the role of growth and where we go from here. Currently, the seeds about data-driven insights have made an impact in the organization. Teams across the organization have incorporated the growth processes and documentation we created, and are using it to manage their team to determine what they will build. Growth is now more than just a team, it’s an idea that has helped change Mozilla and Firefox into being more data-driven while maintaining our core principles and mission. Before Francesco Polizzi left his internship, he did another video presentation on the future of growth at Mozilla and it summarized the progress we made in just a few short years: https://growthhackers.com/videos/firefox-growth-mozilla-s-comeback-story-1f0e2172-30ca-4022-952a-3802911cba1f/

      Thanks for the question Alex and hope everyone enjoyed the story!

      6 Share
      • AA

        Anuj Adhiya

        about 1 year ago #

        This was amazing to read (which is saying quite a bit given the rest of your responses) :)

  • CM

    Chris More

    about 1 year ago #

    Hello everyone! Thanks for joining my AMA today. It looks like we are getting some great questions and I am working on responses as I type! Talk in a few minutes. -Chris

  • EW

    Eric Willner

    about 1 year ago #

    Would be great to hear about Mozilla's growth goals with regards to desktop vs. mobile and domestic vs. international. What are some ways Mozilla is planning to achieve those goals while competing with the likes of Google, Microsoft and without alienating users who strongly believe in the organic open source community?

    • CM

      Chris More

      about 1 year ago #

      Hi Eric.

      Great questions!

      I’ll take each question one at a time.

      1) Desktop vs mobile.

      The goals for these products are not as straightforward as it would seem. Both products goals are about growth in general, but as we all know growth can be many things. In previous years, we have had acquisition goals for each of the products in terms of Year-over-Year growth of downloads or installs. Basically, very top-of-funnel KPIs, but we have moved away from those being the top-line KPIs. We are focus more around keeping a healthy user base.

      Specifically for desktop, retention a specific weeks (W1, W4, W6) and engagement ratio (DAU / MAU) as being two top KPIs. The reason that we do weeks for retention instead of days is that a utility like a browser is quite different of a choice than installing the latest bird game. Also, we have seen that desktop browsers are used at somewhat random intervals. You may install a desktop browser on a specific day, use it 3 days later once, and then a third time a week later. If we did retention by day, the volatile usage of desktop would make daily retention look pretty confusing. Thus, we aggregate retention at the week level since most users users a desktop browser at least once a week.

      For mobile, the KPIs are mostly around acquisition and Year-over-Year install growth. While we don’t want to forget about retention and engagement ratios on mobile, we are simply trying to get the most amount of users to try the mobile browser. Our mobile Firefox is available on both Android and iOS and is a very strong product with a rich feature base and customizations also seen on desktop.

      Regardless of desktop vs mobile, both products are alternative browsers on all platforms (except from Linux distributions) thus, it is an active choice users are taking to not accept the default browser. That choice, makes most sense when you are using both our desktop and mobile products. When you do use Firefox on all platforms, you can create a Firefox Account, which automatically synchronizes your bookmarks, passwords, history, and tabs across all of your devices. If you just use Firefox for mobile, you’re not taking advantage of Firefox Accounts and Sync, thus the value proposition isn’t as strong since it is likely that your default browser has those abilities. This is why we try to lead with desktop and then value proposition mobile as an extension of your data in the Firefox platform. The best retention we’ve seen is when users are using Firefox on desktop and mobile and are using a Firefox account to sync their experience.

      2) Market strategies

      All of our products and websites are translated in over 70 languages, but as you know, just translating words is just part of the story. We have found that the words we use to promote Firefox resonates very differently from market to market. One market may be more interested in privacy and other markets may be more interested in customizations. We work with Mozilla’s 10,000+ volunteer contributors around the world to translate our experiences into words that resonate with that audience. The words are still all centric to the product and organization behind Firefox, but they are tweaked by market and it works well. We are also going to take the same approach to user onboarding in the future.

      3) How are we getting to these goals

      A few areas have been working out well:

      SEM ads for Firefox desktop. The channel has worked really well for us and especially since people (hijackers) like to take Firefox open-source software, build a version with malware inside, build a web page, and then put up SEM ads for the “official” Firefox version. While advertisers say this is against their policy, it doesn’t detract the hijackers from doing this with Firefox. So the SEM ads are about taking over the #1 SEM ad spot to reduce the chance of hijackers from distributing Firefox with malware inside.
      Onboarding. We’ve been experimenting in lots of different areas to improve retention and outside of product performance, improved onboarding (or even having effective onboarding) seems to be the ticket on our products. We are also experimenting with segmented onboarding where there isn’t a one-fit-all approach for every user in world. The results of the onboarding retention tests have been pretty promising, thus we are continuing to iterate on them.

      4) Not alienating the open source community

      Given that Mozilla is a non-profit open-source organization, our products are built in the open on publicly available code repositories. We use tools like Github issues and Bugzillla to manage the creation new features, track bugs, and to deploy our products to the world. Since most of what we do is in the open, nearly anyone can join in and contribute, which is why we have over 10,000 volunteer contributors all over the world. The most common way people contribute is to localize copy from a marketing campaign to an in-product feature to their native language. Mozilla’s contributors is what keeps us humming. The two most common ways that people contribute to Firefox is localizing copy and writing code. Now when it comes to Growth, this is where it gets more challenging. The way we run growth is that we have a backlog of prioritized opportunities (traffic level + impact potential + level of effort) that includes a link to a recipe. The recipe states the hypothesis, current insights, and a test plan on how the null hypothesis could be rejected or accepted. These recipes usually have data about our acquisition, activation and retention rates. This core product and funnel data is fairly sensitive since it can easily be interpreted the wrong way without all the context. Thus, we keep those product metrics internal, but whatever code we write to perform the experiment is done in the open. We also want to eventually start to blog publicly about our insights and what is working and not working, so that others can learn from what we have tried. We haven’t seen any strong push back from the community and we have even had some community members express desire to become involved in our Growth experiments. The only comments we’ve seen is the community wishes that our Growth stack and tools were completely open-source tools. We would love to have a completely open-source Growth stack and we will continue to push for more transparency and available open tools on the market.

      3 Share
  • DH

    Dani Hart

    about 1 year ago #

    Hi Chris,

    Thanks so much for taking the time to participate today. Here are my questions...

    1. What do you look for when hiring for your growth team? Are there any specific skills or attributes of team members that make them successful?
    2. What tools are in your team's "Growth Stack"? Any tools you use on a daily basis that you'd recommend to others?

    Can't wait to hear what you have to say!

    -Dani

    • CM

      Chris More

      about 1 year ago #

      Hi Dani!

      I love questions about team members.

      1) Growth hiring

      First, I have a strong feeling that people who are successful in Growth have a Growth mindset (http://carriekepple.com/wp-content/uploads/2015/04/Growth-v-Fixed.jpg). I have found people who are successful in growth appear to use that same mindset all throughout their professional and personal lives. It is just how you approach anything and being totally comfortable with failures. Outside of the mindset, I think people with strong t-shape skills (https://en.wikipedia.org/wiki/T-shaped_skills) is critical to growth. Why do growth people need to have diverse background? Because growth isn’t just something found in code, or in marketing, or in onboarding, or the product itself. Growth isn’t just one thing, but it is all of those things. To be strong in growth, you need to evaluate all potential opportunities and there are rarely one-fit-all strategies. Growing your product really comes from a unique combination of strategies and tactics that can’t be done by only one channel or one experiment. Thus, you have to have a bit of everything in your background or enough to be able to evaluate the ROI of a new idea. If all you know is marketing, there could be major product opportunities that could be ignored. If all you know is development, there could be big marketing opportunities that could be not taken advantage of. These people are challenging to find, but once you find them, don’t let them go because they can have impact in so many areas of product, marketing and growth.

      2) Our growth stack

      Here’s the list of what we use regularly, which is a fairly small list compared to most in Growth.

      * Google Analytics (for general web analytics)
      * Optimizely (for web a/b testing and launching cohort tests)
      * Re:dash (analyzing product metrics)
      * Tableau (analyzing cohort experiments)
      * Spreadsheets (one-off analysis)
      * SurveyGizmo (getting feedback from users)
      * Storemaven (mobile A/B testing)
      * SensorTower (mobile app store optimization)
      * App Annie (mobile market data)
      * Custom builds of Firefox for cohort testing (code named funnelcakes)
      * Firefox in-product features (Heartbeat for satisfaction surveys, Test Pilot opt-in experiments, Telemetry data)

      All of those applications have worked for us and each has a unique purpose it is filing. Like any tool that Mozilla uses, the tool and the organization goes through an exhaustive privacy and legal review to ensure it aligned to our principles and mission.

      Thanks for the questions!

      5 Share
  • AA

    Aldin A

    about 1 year ago #

    Hi Chris,

    Thanks for doing this AMA!

    1)There are a million things you could be working on growth at any one time. Can you talk about your process for
    figuring out the MOST important thing you have work on RIGHT for growth? How do you make and prioritize your growth road map?

    2)How do think about retaining users, if your user only need to use your app occasionally by nature (ex shopping app)?
    If your app isn't used frequently building up the habit is hard, which makes it even harder to retain the user.
    How do you go about trying to stay top of mind so when the user has a need that your app solves they think of you?

    3)Can you name some resource you use to learn from about growth and business in general? Books, podcasts, blogs, courses etc?

    Thanks

    • CM

      Chris More

      about 1 year ago #

      Hi Aldin!

      Let’s jump into your questions and let me know if you have any follow-ups.

      1) How to prioritize growth opportunities

      This is a good one and I answered something similar in a previous question. Basically, we prioritize based on 3 factors:

      a) Traffic - how many eyeballs (or pairs of eyes) could potentially see this experiment if it is successful and rolled out later? (1 = low, 5 = high)

      b) Impact - If this experiment is successful, how much could it potentially impact product growth and top-level KPIs? (1= low impact, 5 = high impact)

      c) Level of effort or cost - How difficult will it be to perform this experiment and roll out a win? (1= easy, 5=hard)

      Then, we simply add all of the 1-5 scales up to get a growth score. We then sort the growth score in ascending order. The ones at the top of the list are usually the quick wins, the ones in the middle are experiments that will consume most of the team’s bandwidth and the bottom batch may stay in the hopper forever.

      2) Retaining users

      Good question. We have seen usage patterns on mobile and desktop to be dramatically different. Usage for desktop is fairly random where someone may use it once or twice a week and then not use it for another week. On mobile, usage is much more consistent. We have found that the best way of retaining users is to get them to find value and have the “ah ha” moment as quick as possible after choosing Firefox. That is hopefully on day 1, but ideally in the first few days to weeks. Ultimately retention is when users are choosing Firefox across platforms and that means they are active Firefox users on both desktop and mobile. We have a feature called Firefox Accounts and Firefox Sync that allows you to sync your passwords, bookmarks, history, tabs and more across devices and once a user sets up Sync across devices, their retention jumps up dramatically. If you don’t sync your data and are not using it on multiple devices, you are probably using another browser that does that already. Thus, for the complete Firefox experience, we want users to try and find value of using Firefox on multiple devices. This is why we have the Firefox Accounts activation early in the onboarding experience because it is one step closer to using Firefox across devices and seeing the value it provides.

      Onboarding is really important to us now and I am leading a team that has been experimenting with developing new onboarding experiences. We believe that onboarding is the key to get users to find value sooner in their experience with the product and ideally choose Firefox for all of their devices.

      3) Growth resources

      This website is surely one I would recommend! :-) Following #Growth and #GrowthHacking on Twitter is a good source for insights, but I generally have found that most tactics are specific to every product. Thus, you need to first start with the Growth Mindset, question everything, get good at the scientific method, build your t-shape skills, and start to perform basic experiments with product and user experiences. Use a very clearly defined and documented process on how you will develop and evaluate a hypothesis and the most important thing I would say is to determine “next steps” after you wrap up an experiment. If you “rinse and repeat” this enough, you will start to build the growth muscle and all of the wins and saves (loses) will build up learnings that will hopefully turn into tangible wins. It takes time, but first start with developing your growth mindset and apply it to everything in life. Once you have the mindset, performing experiments is just a means to an end to grow your product.

      Thanks for the questions, Aldin.

      2 Share
  • SA

    Shaker A

    about 1 year ago #

    Hey Chris,

    Great to have you here.

    1) How do you define short term and long term growth goals? How do you differentiate which growth goals should be long term vs. short term? Lastly, how do you balance your priorities between the two types of goals?

    2) What are the most valuable lessons you've learned in your career about growth and business in general?

    3) In your opinion what are things early stage startups have to do to not only survive, but thrive? Conversely what do you see startups messing up that they can't afford to, and how do they fix them?

    Thanks

    • CM

      Chris More

      about 1 year ago #

      Hi Shaker.

      Thanks for joining my AMA today! Let’s get rolling.

      1) Growth goals

      We try to set annual growth goals related to retention and engagement ratio (DAU / MAU), but we set milestones on a quarterly basis to get us to the annual growth targets. Some of the initial milestones could be binary like “Develop an onboarding experiment” for a specific quarter, because there is just not enough time in a quarter to develop a test plan, create a complex retention experiment, deploy it, and wait for the cohorts to churn and get in the results. Though, we don’t want to have many binary KPIs for a quarter and try to move as quick as possible into iterative experiments that ultimately add up to meeting or exceeding the annual targets. We then break down quarters into 2-week sprints where we define user stories and acceptance criteria that will help again get closer to the quarterly targets. The cadence is 2-week sprint goals that add up to quarterly deliverables that support annual targets.

      2) What are the most valuable lessons I’ve learned

      Probably the most valuable lesson I’ve learned is that I am surprised how many people are not comfortable with failure. For myself personally, failure motivates me to do better. I look at failures as a KPI on how far away I am from the goal. It helps tune future experiences. The more that I know, the more that I realize that I don’t know and that is completely fine. You have to accept that intuition and what you have learned from the past doesn’t always apply to the future. Even past growth experiences may not dictate the future, thus you should try to always validate. Since growth is not just one thing and requires the coordination of various teams, you have to prepare them for those failures knowing that they will happen. Spinning the failures as “saves” helps motivate the team to keep trying and not give up. For example, last week, I was in a motorsports race and my goal was to come in 8th in the United States. I came in 7th and I was less than a second off of the national champion. I could look at the 7th place as a loss, but I look at it as a save because I learned lots about the car that needs to change and I identified places in my driving that I need to improve. I now have a baseline and I can perform experiments on the car and myself to improve. The failures are what motivate me to do better and that can be applied to anything.

      3) Surviving the startup phase

      Good question.

      Startups have a tough challenge. They need some user base and they are usually starting from zero. If you don’t have many users, it is even harder to perform experiments and reach statistical significance. My recommendation is to develop a growth model on how your product should grow, identify all of the channels, determine what triggers you will use to engage users, and make a few initial big bets. Ideally, each experiment will have a single independent variable that you can A/B test your way to growth from early stages, but that is difficult without having much history and an existing user base. Be willing to take risks early on, but have enough metrics and insights to course-correct as soon as possible. Once you have a user base and you have some insights on your product, start to break down those assumptions and identify the wins and saves (loses).

      Thanks for the questions, Shaker!

      2 Share
  • LS

    Logan Stoneman

    about 1 year ago #

    Hey Chris! Thanks for hopping on here today - as stated in your bio, someone experienced in engineering, marketing and business has the necessary skills for working in growth. I believe this makes it appear that working in growth requires a high barrier to entry. If you had to focus on only one of these skills (engineering, marketing, business) - which would you cultivate to begin a career in growth?

    • CM

      Chris More

      about 1 year ago #

      Hi Logan.

      Great question and I am glad you asked.

      Even though having a diverse set of skills is important for growth roles, they don’t all have to be equal. This is generally known as t-shaped skills where you are deep in one area and you know a decent amount in many other areas. If I had to pick one area to start, it would probably be engineering. It doesn’t mean that you have to work as a dedicated engineer or have a Computer Science major, it just means that having some foundation in the building blocks of technology (web development, scripting, apps, HTML, SQL, etc.) helps. I have found it is easier for someone with some engineering experience and the right mindset to learn marketing and business than vice versa. You can still learn engineering later in your career, but once you put on an engineering hat briefly, it helps you evaluate opportunities and guide you from a concept to a deployed experiment.

      Though, before you develop your expertise whether its engineering, marketing or business, you need to first start with the growth mindset. Without the right mindset, you cannot simply take an engineer or marketing person and turn them into a growth person. It may be possible, but you will often be up against their own mental model of the world. You can find these people all over and in many different disciples and they simply may have never heard about growth or know it’s actually a thing. If you hear someone (or yourself) saying “What if….” in many situations, you are looking at someone ripe for growth.

      Thanks for the questions Logan and joining the AMA today!

      3 Share
  • JQ

    Jari Qudrat

    about 1 year ago #

    What is your strategy to make Mozilla top of mind when it comes to browsing vs. Google and Bing?

    • CM

      Chris More

      about 1 year ago #

      Hi Jari.

      This single question is pretty complex, but I will try to keep it brief. :-)

      Each of the markets around the world have Internet users with unique needs and interests online. For example, Germans may be bigger on privacy than other markets.

      Regardless of market, our main message is about being “free”. Free to choose, free to search, and free from corporate control. Given that Mozilla’s a non-profit and independent organization that builds in the open and puts users in control of their online life, that becomes what we want people to think about when they are considering making a choice about a web browser. Modern web browsers on desktop and mobile are very feature rich already and Mozilla differentiates ourselves from any other major web browser by our mission, who we are as an organization, and how that comes out in the products we build. Ultimately, that what we mean by being “free”.

      Let me know if that is what you are looking for and thanks for the question!

      2 Share
  • RB

    Ry B

    about 1 year ago #

    Chris,

    Excited for this AMA! I'm a regular user of fire fox!

    1)Can you talk about some of the challenges of scaling the Mozilla and how you've overcome them?

    2)How do you look at competition, specifically when your going up against bigger, and better funded competitors that have a larger market share than you (Chrome, etc)? How does that affect your strategic plan, if it does at all? What is your mindset when you go to compete with these other browsers that are ahead of you in the proverbial race?

    3) How do you look at hiring? Can you talk about some of the mistakes you've made hiring (and also seen others make)? What have you learned about hiring A+ talents?

    4)How do you look at balance in your professional and personal life? When it comes to work, how do you decide what you have to work on today(i'm sure you have alot of fires to put out everyday)?

    Thanks

    • CM

      Chris More

      6 months ago #

      Good questions and sorry for the late reply. :)

      > 1)Can you talk about some of the challenges of scaling the Mozilla and how you've overcome them?

      Wow, that's a loaded question, but it is also something I talked about often. The most difficult part of scaling Firefox at Mozilla is that at the end of the day Mozilla is a non-profit organization and Firefox is open-source and free. We don't see anything and we don't monetize users. Thus, at the top of our pyramid we don't have money/revenue driving the equation. When the reason why you do nothing is not to make money, the reasons can get fuzzy.

      Our mission is:

      "Our mission is to ensure the Internet is a global public resource, open and accessible to all. An Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent."

      So, "doing good" is sometimes hard to quantify and what if you grow the product in a way that is "doing bad", is that good for growth and bad for the mission and non-profit organization. These are conversations and challenges that we have on a regular basis. Also, given that we are mission driven and put users first, we have very little data in our products and it is mostly anonymous, some options and users can opt-out to whatever they want. That means that it makes performing experiments and understanding the results very challenging. This challenge is what has kept me at Mozilla for over 6 years now. I have about the hardest growth job out there because I am try to grow a product that doesn't monetize, has very little data, and "doing good" can often feel like the north star. We have lots of crazy ideas to try, but you have to balance the mission with something that actually stands to grow the product. It is not easy, but that's who we are and there's almost no one else out there like Mozilla and Firefox.

      > 2)How do you look at competition, specifically when your going up against bigger, and better funded competitors that have a larger market share than you (Chrome, etc)? How does that affect your strategic plan, if it does at all? What is your mindset when you go to compete with these other browsers that are ahead of you in the proverbial race?

      Again, good question. We surely think about our competition and there's no way we can out-spend Google or Microsoft. They have bigger budgets and more humans (and non humans) to build software very quickly. What we do think about often is what makes us different. Mozilla is a non-profit open-source company that does not sell anything or directly monetizes users, so that is something that Google or Microsoft cannot be. We can also make decisions and do things differently in our products that puts users first and in control that our competition cannot given their business models. As mentioned above, while it is good to be mission driven and not think about money, our competition does, which makes many of the functions at Mozilla challenging. In theory, we could "play by their rules", but then what would make Mozilla any different than any other company?

      > 3) How do you look at hiring? Can you talk about some of the mistakes you've made hiring (and also seen others make)? What have you learned about hiring A+ talents?

      On the first iteration of the growth team at Mozilla, I hired directly for the team and the people I hired were very different from everyone else. They were a bit of jack-of-all-trades or "T-shaped people" as HR folks would say. They could write some code, create a crazy spreadsheet, and the next day give a presentation to the executives on a win without breaking a sweat. These unicorns are hard to find and it is also difficult to train someone into the growth mindset. I have seen that with the right mindset, experience doesn't matter that much. Someone fresh out of school could have more impact than someone with a decade of experience in the right situation.

      In the second iteration of the growth team, we turned many teams in marketing and product into growth teams focused on one growth lever each. That means most of the hiring I am involved in now is finding people with a deep understanding of one growth lever, but also have a variety of other experiences and most important the growth mindset. I like to find people who big potential and get them placed on another team and it is amazing to watch them grow themselves and the metrics of the team they are on. As a company grows and matures, you can't expect all of the people doing growth being on one single teams.

      > 4)How do you look at balance in your professional and personal life? When it comes to work, how do you decide what you have to work on today(i'm sure you have alot of fires to put out everyday)?

      I write this answer with a little pain in my neck. ha. This is actually a struggle for me in balancing my professional and personal life. Why? Because I love my professional life so much. It doesn't feel like work to me and I'm excited nearly (well almost nearly) every day when I jump back into it. Though like any passion, you can take it too far and it then becomes your life. I have a lot of things I enjoy outside of work and that includes sports car racing and instructing at race tracks. I have found that when I make growth or my professional life, my entire life, it in the short-term has a positive impact on my work output, but I quickly burn out mentally and physically. There is only so many hours in a week and you need to give your mind and body a break. If not, you cannot really enjoy either the professional or personal side. Now, if I didn't love growth so much, I wouldn't have this problem, but I'll take loving what I do versus hating it, any day. :-)

  • SS

    Saulo Sobanski

    about 1 year ago #

    Be welcome, Chris!

    Actually, I have a question about how you manage your Growth Team on Mozilla, keeping track of all processes, experiments and metrics from different professionals who are working together. Which are your personal "must-have" tools, such as a workflows and canvas, to keep track and control everything without losing your mind?

    All the best!

  • MM

    martín medina

    about 1 year ago #

    Chris thanks for coming on here and doing this AMA!

    What are some important metrics that you’re tracking for Firefox? Has that changed over time? Do you guys have a “north star” metric?

    How do you guys view Tor use with regards to growth for Firefox?

    Lastly, what is your favorite type of car to race?

  • JM

    Jason Meresman

    about 1 year ago #

    Hi Chris - thanks for taking the time for today's AMA.

    What acquisition channels have you found that work for both desktop and the mobile app? Are they the same or different?

    Also, have you seen any emerging acquisition channels you find yourself excited to try?

  • VM

    vivek muralee

    about 1 year ago #

    Hi Chris
    Thanks a lot for doing this AMA. I am sure every GrowthHacker is eager to learn from you.

    1) What would you have liked to do differently if you had a chance to redo the Mozilla growth journey?
    2)What is the biggest hurdle to success for emerging startups?
    3)How can startups be more stable and have sustained growth in niches that have heavy competition from big players?
    4)What single belief/motto is your success mantra?
    5)What event has been the turning point of your life and provided you the most inspiration?

    Thanks again Chris

    Vivek

    • CM

      Chris More

      5 months ago #

      Hi Vivek!

      > 1) What would you have liked to do differently if you had a chance to redo the Mozilla growth journey?

      Wow, that's a good one. I've been doing growth pretty much my entire career. I just didn't call it growth, because I thought it was some weird mix of disciplines I invented for myself that focused on outcomes that were critical to the business. When I realized that what I have been doing forever is Growth, I just re-named what I was doing. I do wish I would have discovered it earlier for both myself and Mozilla. Before growth, it was hard to describe why I was pushing so much on both acquisition and retention. Once I called it growth internally and educated people on the process and methods, it gave everything I was doing and my teammates much more purpose. It kind of reminds me of a quote from Flight Club: "It was right on everyone's face. Tyler and I just made it visible. It was in the tip of everyone's tongue. Tyler and I just gave it a name. ". If I had to do it over again, I would have started with calling it Growth in 2011.

      > 2)What is the biggest hurdle to success for emerging startups?

      Like others in Growth, product/market fit is probably the most difficult followed up my getting the initial traction. Sure you have to have a product that people want, but you also have to get it in front of enough eyes for people to grow it organically. The app and web world is so saturated now with so many great ideas and niche solutions that it is easy to get lost in the mix. You really have to set yourself apart from the rest and make that very apparent from the start. If your app is similar to other app or website, it won't matter if your features are better, people will see it as basically the same and not even go deep enough to discover your awesome features. First make a product that people want, make that product so apparently different from other products, and then get users to see how it is different the moment they try it. If you call that the "aha!" or "wow!" moment, get that right in your onboarding. Also, remember that onboarding starts from the moment that a user has intent and not just after they installed it. It is the entire journey.

      > 3)How can startups be more stable and have sustained growth in niches that have heavy competition from big players?

      Yes, you can have sustained growth, but it may just be at a low magnitude or very slow growth rate. Maybe for some small startups, they don't need to be big, but the investors probably would argue otherwise. If you are the little product competing with the big players, try to show how you are different more than you are better. Find an audience that may care more about your mission or values regardless of just utility and features and growth that niche audience.

      > 4)What single belief/motto is your success mantra?

      Assume nothing. Even if you think you know everything.

      > 5)What event has been the turning point of your life and provided you the most inspiration?

      During my Computer Science University days, I took electives in Philosophy, Logic, and Sociology. Those classes gave me a deeper understanding of humans and even in the randomness of life, there are patterns and predictability. Those class and the professors I met, changed me forever and thus I kept my entire tech career focused as much as possible on impacting the end user experience.

  • AA

    Anuj Adhiya

    about 1 year ago #

    Hey Chris

    Great to have you on!

    I recently watched this presentation by @francescostl on some experiments run during his internship at Firefox that yielded a few big wins: https://growthhackers.com/videos/firefox-growth-mozilla-s-comeback-story-1f0e2172-30ca-4022-952a-3802911cba1f/ (for everybody else - there's some good testable ideas here)

    Could you talk about any other experiment(s) from your tenure that've yielded huge wins or critical insights?

    Cheers!

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
72
72