Leave a comment
Get the GH Bookmarklet

AMAs

Vivian is a software engineer at Pinterest where she focuses on growing users at scale. Pinterest is the world's first visual discovery engine that helps users discover and do what they love, with over 100 billion recipes, home hacks and style ideas.

In the past three years, she has worked on all parts of the user acquisition and retention funnel, from signup all the way through the resurrection of churned users. She has primarily focused on building growth features on iOS and recently has started working on improving the backend infrastructure for our onboarding experiences. Back when she joined, user growth was slowing and a Growth engineering team was formed with a mandate to accelerate growth, particularly internationally. Now more than half of our 175 million monthly active users live outside the US and over 75% of signups are from outside the US.

Vivian lives in San Francisco and has an adorable new puppy named Knuckles (you can follow her on Instagram @knucklesthepup). She's happy to discuss anything from user growth to engineering at scale to coffee and naming pets after classic Sonic characters.

You can follow her on Twitter: @vivqu

She will be live on Aug 15 starting at 930 AM PT for one and half hours during which she will answer as many questions as possible.

  • SP

    Shanik Patel

    5 days ago #

    Hi Vivian,

    I'm curious about the evolution of the onboarding flow for the Pinterest mobile app. Can you describe the experiments that you did in order to determine the order of the account creation, preferences, and product tutorial steps? Is there a specific metric that you optimize towards in onboarding experiments?

    • VQ

      Vivian Qu

      4 days ago #

      Hi Shanik, thanks for asking a question! We overall focus on activated users, which we consider as converting a new user into a weekly active user (WAU) after 28 days. Pinterest is not necessarily a service that drives users to visit every day but we want to make sure we are consistently providing value for people's lives, hence why we focus on WAUs.

      In general, we focus on the different stages of the user funnel which is how we think about specific features like account creation, tutorial steps, etc. Acquisition is how we actually get users to the website or the Pinterest mobile apps. Activation is getting a brand new user to turn into a WAU. We utilize notifications and emails to keep a user engaged. Finally, we have a team focused on resurrecting users who have signed up but didn't find the immediate value when they first signed up.

      Using those broad stages, we've tested different experiences that can move people from one stage of the user funnel farther along towards a retained, highly engaged user. For instance, in terms of account creation, we have tested not requiring creating an account to see content on Pinterest. Success is determined by if this user is more likely to understand the value and later come back for more content and eventually signup. Similarly for product tutorials, we have tested which actions or areas to emphasize actually cause users to become more engaged long term.

      • MB

        Mike Bossard

        1 day ago #

        Your description of the key stages of user engagement and how you measure is very helpful. Thanks!

    • TM

      Ty Magnin

      2 days ago #

      Hey Shanik - look forward to your AMA coming soon - saw this question and Vivian's response and thought this post I wrote after gathering info from an outsiders perspective might help give you some other ideas. It's experiment focused and includes screenshots: https://www.appcues.com/blog/casey-winters-pinterest-user-onboarding/

  • AL

    Arsene Lavaux

    6 days ago #

    Bonjour Vivian,

    Thank you for doing this AMA. Knuckles is so cute. Just started to follow his adventures on Instagram via @REveo.co

    I have a couple of questions for you:

    1) You mentioned you worked on onboarding at Pinterest. Is there one key learning that really jumped at you over that time?
    2) How important is it to localize international growth for mobile?

    Merci Vivian!

    • VQ

      Vivian Qu

      4 days ago #

      Hi Arsene, it's great to be here. Yes, Knuckles is super cute isn't she?

      1) One of the most important takeaways I've gained over the last three years is that growth problems aren't static. It's very natural that the types of issues change over time. Even just looking at international growth, when we first started thinking hard about how to accelerate growth we were limited by the content that was available in those countries. Pinterest overall was working towards localizing content in key markets like UK, France, and Germany. So initially we didn't have much local content and not only were we focusing on growing the number of users, we wanted to grow the number of users that would create or generate additional local content. Eventually we grew this specific user base enough that we could start focusing on users who were more interested in browsing or searching through existing content.

      5 Share
      • VQ

        Vivian Qu

        4 days ago #

        2) I think the answer to #1 also is relevant to your second question! Localization in i18n markets for Pinterest is extremely important, for both content and translation. We eventually built engineering solutions and recommendation engines for every major key market we were supporting. That being said, you do need to have quick, faster solutions while you are growing the right group of users and the right content. The key here is to have a playbook for what you need to do when entering new markets, and continue to adapt this over time as you learn along the way. That way every new market is not a completely new problem you have to solve.

        4 Share
  • DH

    David Hoos

    4 days ago #

    Hey Vivian. How much of a role has UX played in the growth of Pinterest?

    • VQ

      Vivian Qu

      4 days ago #

      Hi David! Great question: UX is a very important part of understanding growth at Pinterest. Even now we have a dedicated UX researcher who helps set up UX evaluation sessions with real users and synthesizes the information we gather. We're lucky to be able to conduct in-house interviews with locals, remote evaluation with services like UserOnboarding, as well as international research where we fly small groups of PMs, engineers, and designers out to evaluate a specific market. This has been critical in gathering general understanding of new users as well as helping us understand the experience of specific features and experiments. Oftentimes we'll discover that the quantitative data is mixed or unclear, which is why having qualitative understanding is so important. From my experience working on onboarding specifically, new users (especially those from international markets!) are so different from power users that you need all types of data to really understand what they're thinking. I have so much context about Pinterest that I often forget how hard of a time new users have understanding the basic value of our service, especially in countries where we have low market penetration.

      4 Share
  • GH

    Glen Harper

    4 days ago #

    Thank you for joining us today, Vivian.
    Can you share any details about a growth experiment that your team ran that was either a big win or led a big learning?

    • VQ

      Vivian Qu

      4 days ago #

      Hi Glen! Definitely I'd be happy to share.

      One of the biggest wins we saw in our onboarding experiments was radically simplifying the product for new users and then progressively re-introducing features. We hid a lot of actions that a user could take (ex. sharing, saving the image to their camera roll, etc) because we wanted to focus new users on the primary actions we wanted them to learn. Essentially, the key is to "show and not tell" the new users, a mantra we say often on the team. We previously tried a lot of ways to explain to the new user or point to the right place they should look at, and turns out no one really reads or follows directions. The downside of this approach is that it often leaves new users with a different initial experience than they would have as a core user of Pinterest. It also adds extra engineering work to maintain the difference between new/old users and you have to make sure that anything that is added for old users is tested and verified as important for new users as well.

  • TN

    Tri Nguyen

    4 days ago #

    Hi Vivian
    What does your typical week look like?
    Can you walk us through what you do and how?

    • VQ

      Vivian Qu

      4 days ago #

      Hi Tri! Yeah sure I'd be happy to talk about my week. Generally I split my time like so: 60% engineering, 20% experiments, and 20% process and organizational work.

      "Engineering" means actually implementing new features, refactoring code, and reviewing my teammates changes. This is the bread and butter of an individual contributor engineers work. Included in this time is interfacing with cross-functional team members that I sit with, such as product and design and other teams if collaboration is required.

      By "experiments" I am referring to configuring our A/B experiments, running pre- and post-analysis, and setting up qualitative research with our UX researcher. Other companies often have PMs take on most of these responsibilities but at Pinterest (and especially on Growth) we are very data-driven and run many experiments. Individual engineers are usually the ones that lead the experiments and synthesize the learnings to share with the rest of the team.

      Finally, the last bucket includes process and organizational work. One of the most interesting parts about being at a fast-growing company is that a lot of the engineering process is not solved. I have been able to have impact and influence how we improve our processes and systems to move faster, build more things, and share learnings better. I also actively mentor more junior engineers and help them gain technical expertise and confidence. This is an area that is not for all engineers, but I personally really enjoy contributing to the team and company in this way.

  • JP

    John Phamvan

    4 days ago #

    Hi Vivian

    What tools are the growth team using at Pinterest for experimentation & analytics right now?
    What has/have been the most recent one(s) you'll have added to your arsenal - and why?

    • VQ

      Vivian Qu

      4 days ago #

      Hi John! Great question. We luckily are able to take advantage of a completely in-house A/B experimentation framework with an entire data infrastructure engineering team supporting it. We process about 50TB of data every day. Note that Pinterest started building our in-house framework before commercial solutions such as MixPanel were available. We have a host of different services we use for analytics, like Graphite and Qubole.

      The biggest "tool" that we are currently exploring is potentially integrating React Native into our apps. RN offers potential benefits of increased developer productivity and iteration speed with the possibility of shared code. As a growth team, one of the biggest limiting factors is how quickly we are able to get new features out and start getting quantitative data as well as whether we have enough available engineering resourcing (there's never enough!). I'm involved in this exploration and very excited for the possibilities.

  • JF

    Javier Feldman

    4 days ago #

    Hi Vivian
    Now that you have so many users from around the world, what would you say are the top 3 lessons have you/your team has learned about international growth & localization?

    3 Share
    • VQ

      Vivian Qu

      4 days ago #

      Hi Javier! Great question, the three top lessons that come to mind are (1) continuously re-evaluating what "state" of growth you're in, (2) getting qualitative knowledge and (3) systematically considering the tradeoffs between hyper-localization and a general solution.

      1) I mentioned this in an earlier response but you never really "solve" growth because as you enter new markets or increase your penetration of a specific market, the problems keep changing. Two years ago most of our users were from the US, and now more than 60% of our users are international. Our strategy for growing overall users now is different than it was a few years ago. In international markets, we started off trying to acquire users who would repin and generate local content for us. Now that we have a robust set of local content in our major markets, we are trying to capture and engage people who are less interested in generating content. Experiments that succeeded a few years ago might not work now. So as much as possible, creating a systematic way of thinking about problems is the most useful rather than solving specific growth problems.

      • VQ

        Vivian Qu

        4 days ago #

        2) When you initially enter international markets, you don't have many users. This means running A/B experiments either take forever to get significant results or you have to make calls without hard evidence in the interest of moving fast. This is why qualitative data is so valuable and getting on the ground in those markets can help you make informed (although not guaranteed!) decisions. Some of the most illuminating experiences for me have been going to Germany and understanding how privacy-conscious the Germans are compared to US users. We would not have been able to understand that specific insight just from user behavioral data and from there build specific experiences for the German market.

    • VQ

      Vivian Qu

      4 days ago #

      3) Now I'm going to contradict myself a little bit :) In some cases you need to change the product to fit the needs of the specific country or market. For German users, we need to make it clear the extent of required data from the user and what we're doing with it since most users are very concerned with privacy. On the other hand, users in Brazil have lower quality internet connections and tend to sign up many different services. We need to adapt the product to be able to flexibly respond to all these different types of users. However there are important tradeoffs to consider when localizing too much: creating a specific onboarding experience for each market incurs a lot of technical maintenance burden. It's important to align the team on when the tradeoff to hyper-localize is worth the cost and when it's not.

  • RG

    Raj Goodman

    5 days ago #

    Hi Vivian,

    Firstly, thanks for agreeing to do the AMA.

    To give you context, I live in the UK and soon we will be hit by a major new regulation of General Data Protection Regulation (GDPR). It's certainly going to have in impact on Pinterest as well. The regulation says that European citizens' information held by any organisation needs to be 'double opted' in, stating the exact usage of the data. So data which was before only collected for signup can't be used for marketing, unless it was made clear to the user upfront.

    My question is, with this regulation in place, we are mainly restricted to social media and advertising to market our services. Social media's engagement levels are low and advertising can be expensive. Bearing this in mind how would you advice startups in the UK to market their products.

    FYI: We are a consultancy now developing SaaS based products.

    • VQ

      Vivian Qu

      4 days ago #

      Hi Raj! Thanks for asking a question :)

      I'm not familiar with the details of the specific regulation but I can certainly empathize with how much of a burden this adds to the daily operation of the team. We've had to build one-off disclosure and notice mechanisms in the past in order to comply with legal regulations. As an engineer, it's not a fun experience since you're not actively learning or growing the product, only doing something because the company has to.

      That being said, I would respond with two questions for you to think about regarding marketing/advertising: (1) do you need to have aggregate signup data in order to build an effecting marketing campaign? and (2) what barriers do users face in understanding the specific value provided by your SaaS products?

      1) Even though we're a hugely scaling company, a lot of our marketing operations are still grassroots or low budget. We recently launched a huge NYC ad campaign that was all videos and photos of employees. We also invest a lot to leverage our most important type of user, "pinfluencers." These are power users of Pinterest who love the service and happily advocate for our product. We have a community ops team that throws events for our pinfluencers and thinks of ways to showcase them and help them gain even more value from the service. So are there people who you can leverage for your own products that can be advocates? Additionally, can you identify specific users for whom it's more useful to be "double opted" than others?

      • VQ

        Vivian Qu

        4 days ago #

        2) Despite our successful growth over the last few years, we have only started considering paid acquisition in recent months. Most of our traffic comes from SEO and I would highly encourage you try to leverage SEO techniques, in addition to other ways you are growing your userbase. We have an entire engineering team within Growth building scalable engineering solutions for SEO. You can read more about it here: https://medium.com/@Pinterest_Engineering/demystifying-seo-with-experiments-a183b325cf4c

        But even once you get the user in the door through SEO or otherwise, do they immediately get the value you present? This is where having an effective onboarding process and demonstrating clear, immediate value to the user becomes important.

  • MD

    Mark Anthony de Jesus

    4 days ago #

    What is your process for discovering and communicating growth insights with the growth engineering team and across other teams?
    What does the typical flow look like from posing a question of your data to analyzing it to coming up with a hypothesis and proposed course of action or experiment?

  • AA

    Anuj Adhiya

    4 days ago #

    So cool to finally have you on, Vivian!
    What is your (as in personally and/or at Pinterest) biggest growth challenge right now?
    How are you tackling it?

  • DH

    Dani Hart

    4 days ago #

    How does the team decide which growth lever you should be focusing on at any given time? Are specific people/teams assigned to each lever perpetually? Also, within a given lever how does the team prioritize experiment ideas?

  • DO

    Danielle Olivas

    4 days ago #

    How do you'll manage growth sprints in conjunction with the normal development roadmap?
    How is the handoff done when learnings from experiments need to be incorporated into product?
    What are the biggest challenges with this interaction that you've faced?

  • JD

    James Dunn

    4 days ago #

    Hey Vivian
    What do you think it is about the growth engineering team that makes it as effective as it is?

  • GN

    gigipandora Ng

    4 days ago #

    Hi Vivian,

    Great to have chance to ask you some questions here.

    1) How to define the product strategy to reach critical mass?
    2) What's the relationship between growth and UX?
    3) How to get the first 1,000 users of Pinterest?

    Many Thanks!

  • VN

    vyshakh nair

    4 days ago #

    Hi vivian ,

    So glad to see you sharing so much knowledge here.

    Can you tell how did you align your team to growth experiments and take us through the process of sprint planning for the growth experiments especially regarding marketing.

    Thanks in advance . You are awesome .

  • BI

    Benjelloun Ibrahim

    4 days ago #

    Hi Vivian,

    Thanks for joining us today,

    If you want to acquire and retain users for pinterest, what will be the strategy to apply that come from you prior learning?

    Thanks in advance

  • LB

    Leonardo Bertolini

    4 days ago #

    Hey Vivian,
    How deeply are you guys watching Facebook and Instagram, and are you guys striving to maintain the difference that you have with them in terms of ease of use, reach-ability, and overall ROI for content creators? Reason being, is Facebook and Google make it increasingly difficult for brands and entrepreneurs to not only reach their audience at a reasonable price, but the platform as a whole is constantly changing and with every knew update, content creators and brands have to find a way to circumvent the "algorithm" and as a result, authenticity is lost and consumers become more and more skeptical.

    Thanks

  • VQ

    Vivian Qu

    4 days ago #

    That's all the time I have today, thanks everyone! See you on twitter- I'm @vivqu

  • VQ

    Vivian Qu

    4 days ago #

    Hey all! I won't be able to answer the remaining questions, but I'm working on some longer form blog posts that will answer some of the questions. Follow our engineering blog to get all kinds of cool growth and non-growth engineering info: https://medium.com/@Pinterest_Engineering

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
45
45