Leave a comment
Get the GH Bookmarklet

AMAs

Bilal Mahmood is the CEO of ClearBrain (clearbrain.com), and a firm believer that every human should be able to leverage AI in making better decisions.

While a PM of Data Science at Optimizely, he deployed a team of engineers and data scientists to help their marketing team predict conversion and churn. His cofounder Eric Pollmann was Google Ad’s first SRE, scaling their predictive recommendations 10X to millions of dollars per day.

They saw the opportunity to democratize what they had built internally to any company out-of-the-box and built ClearBrain to be a platform that can help any team predict any user action, without having to write a line of code.

Bilal also believes in the need for human context in data - tools without strategy are meaningless. He recently compiled The Growth Playbook, collaborating with growth experts at companies like Dropbox and Stitch Fix to illustrate step-by-step growth strategies in leveraging data into action.

Broadly, Bilal enjoys talking about why machine learning can (and should) be automated, how to ethically leverage first-party data for customer retargeting, and selecting a modern data and marketing stack to tackle these challenges.

Ask Bilal all your questions on growth, ML and AI! He'll be answering them live on Sep 18th.

  • CS

    Cecilia Schmitz

    1 day ago #

    Hey, Bilal!

    It's a pleasure to have you at this AMA.

    I went to a lecture about chat bots and AI a few months ago. I remember one of the participants asked about the risk of all human actions being replaced for "machines" in a near future.

    What do you typically answer when you face these sorts of questions?

    • BM

      Bilal Mahmood

      1 day ago #

      My personal perspective is that technology doesn't replace humans - it augments and repurposes it, enabling humans to focus on other problems. (case in point, agriculture was the dominant source of employment in the US a century ago, and despite being automated, overall unemployment is not any higher).

      Further, I think AI can help with automation of rote repetitive tasks (which a human doesn't need to do), or tasks requiring processing large amounts of data (which a human could never do). But in both contexts, I've found the need for human context in data to be essential.

      Take for instance K-Means Clustering techniques in ML. This is a common technique for grouping your users into categories of groups. The ML model can group the users into say n-groups based on similarity to one another, but the model doesn't know what makes each group similar. It's up to a human often to categorize the clusters of users based on further analysis.

      3 Share
  • DC

    Davi Candido

    1 day ago #

    Hi Bilal, thanks for doing this AMA. Loved the post about feature importances at https://blog.clearbrain.com/posts/explaining-your-machine-learning-model-or-5-ways-to-assess-feature-importance. I'm using random forrest from scikit learn python package to assess the feature importances of my data set. However I'm struggling to find the most important feature from a single data point... I don´t even know if that is possible or if I have to use other techniques. I know the AMA is about AI/ML in marketing but I'll use an ocean example to illustrate my question (I'm a oceanographer who works with marketing).

    Let's say I have a time series data from a sensor in the ocean which collects 5 variables (air temperature, water temp, etc) and we add other categorical data such as moon cycle, season, lat, long, etc. Now I want to predict the tidal range and try to identify anomalous events to help authorities in management plans. Using feature importances I can identify the variables that contribute more in predicting the extreme events. But I can't point with some degree of confidence what feature contribute more to a specific extreme event that happened.

    Is it possible to measure that? And if yes, what techniques do you recommend?

    Thank you

    • BM

      Bilal Mahmood

      1 day ago #

      Happy to try and help!

      Sounds like you're approaching this in the right way so you're totally on track.

      On the question of assessing feature importance to specific extreme events, maybe try the following:

      Let's say when you include all of the features in the model, a specific data point Y is given a score of X. For each feature, one at a time, you could exclude Y's value of that feature (presuming you already have a way with dealing with absent features from your model), and re-score Y.

      Calculate the % difference in output, and rank-order your features that way.

      Hopefully that helps!

      • DC

        Davi Candido

        1 day ago #

        Nice one! I've iterated the feature importance values over time to see the changes but I think your way will work for me.

        Thanks.

  • DC

    Davi Candido

    1 day ago #

    Hey, I have another question for you.

    How can data scientist and/or engineers (who knows a little bit more about ML, data science, read a plot, confidence interval, what is a boxplot, linear regression, etc), translate the data, the findings, the forecasts, the predictions, the plots in a more friendly language to the ones who are not comfortable with these (generally the ones who'll need this to make decisions and take actions)?
    I've seen a lot of people looking at reports and data, but still make decisions gut related because it's hard to interpret. And if you make easy ones with less information then what's the point?

    I think is not that easy to teach and engage colaborators in a data-driven culture but I do believe that is better than dummy plots and reports.

    Did you deal with this in your past experience?

    • BM

      Bilal Mahmood

      1 day ago #

      Making machine learning accessible to non-technical stakeholders is definitely a tough challenge.

      One thing I've found in the past that helped is to use models that make more intuitive sense to humans - i.e. decision trees and logistic regressions. The simpler the model, often has more accessibility.

      With respect to reporting / visualizations / graphs, its more of an art than a science I've found to finding the right presentation to convey the insights you're going for. So no easy solution I can provide, other than I think it is important to spend time on figuring out how to convey the story you want to in your data.

      I generally think the tradeoff between complexity or accuracy vs. interpretability - its better often to strive for interpretability. In the end, the purpose of an analysis is for humans to take action on it - so I'd trade 10% accuracy for actual deployment in practice.

  • AL

    Arsene Lavaux

    2 days ago #

    Bonjour Bilal,

    Thanks for doing this AMA.

    Convolutional neural networks and many more AI-driven techniques are making interesting inroads in computer vision at an increasingly faster pace.

    What could be the consequences for future growth marketing testing?

    Where do you see the line between human an AI brains in growth in the near future?

    Merci!

    • BM

      Bilal Mahmood

      1 day ago #

      Convolutional neural networks (CNNs) and Deep Learning techniques are definitely making inroads in the broader ML space, especially as the cost of compute drops. They're extremely powerful for processing large amounts of data or deriving signal from non-linear datasets, which is why they've lent themselves well to computer vision problems thus far.

      In the context of growth marketing though, I'd actually contend that CNNs are a bit of a sledgehammer approach when a regular old hammer will do. Most of the problems in growth marketing tend to be focused on classifying user behavior (predicting conversion or churn, or product recommendations), for which simpler classifiers like logistic regression or random forest are sufficient. Much cheaper to compute than a CNN and likely not too different in performance.

      That said, CNN is definitely still best in class for image classification. Potential applications I could see are in product catalog categorization for ecommerce companies. Otherwise, I feel you can get by with simpler models for most growth marketing use cases.

      3 Share
  • RT

    Rogier Trimpe

    1 day ago #

    Hi Bilal, big fan of the stuff you guys are doing at Clearbrain. Feel free not to answer any sensitive questions, but here are mine:

    - How do you feel dirty data and seasonality influences the ability to apply machine learning?

    - What do you believe is the minimum number of data points needed to be able to make reasonably accurate predictions or for it to be worthwhile using ML?

    - What do you believe is the best way to use MVT/Multi-armed testing in combination with ML Predictions? Is it even possible to use testing as a feedback loop, or should it just be used as an overall 'uplift' test?

    2 Share
    • BM

      Bilal Mahmood

      1 day ago #

      Thanks for the kind words Rogier! Happy to clarify on these facets, as they are questions we get often.

      Dirty data is certainly an issue - the statement "garbage in, garbage out" is pretty apt. A ML model is only as good as the data you're collecting about your users, so it is important to instrument your product correctly beforehand. There's a good post from our Growth Playbook which walks through how to do this correctly.

      That said, there are techniques to account for dirty data automatically - dimensionality reduction, ridge regression, etc. are automated processing techniques which can help distinguish the signal from the noise without too much manual hand-tuning.

      Seasonality is also important, but differs business to business. Seasonality in media vs. travel will differ, and affect user predictions. The primary way to account for this in your model is have longer training periods - use behaviorial data over longer cohorts of time rather than just recent time periods - evaluate m/m and y/y trends as inputs into your models.

      3 Share
      • BM

        Bilal Mahmood

        1 day ago #

        With respect to the minimum amount of data needed for ML to be useful or accurate, I'll answer in two parts.

        For a model to be accurate, its more of assessing the ratio of the # of users who perform an action vs your total user base. For instance if you have say a total of 1000 users, and you're trying to predict an action that only 10 users (1%) actually do in a given day, you're unlikely to get an accurate model. But if you were trying to predict an action that 100 users (10%) did its more likely to be accurate.

        A very cursory rule of thumb we've seen though is that you need about 25K - 50K records / samples / users to have an accurate model, but somewhere closer to 100K users to be able to run statistically significant experiments.

        3 Share
  • SE

    Sean Ellis

    1 day ago #

    Thanks for doing this AMA with us. Can you help us understand the main benefits of being able to predict user actions? Is it mostly about focasting or can you use the information to actually change likely user actions (like reducing churn)? Any specific examples would be awesome.

    • BM

      Bilal Mahmood

      1 day ago #

      Happy to help!

      Forecasting is definitely a common application for using ML in growth - linear regression techniques are the most commonplace in this regards, say for predicting sales or purchase forecasts.

      However, as you likely intuited, simply knowing someone is going to convert or churn isn't helpful unless you can actually change their behavior. ML can be useful here as well, and i see three broad categories where this comes into play:

      1. Retargeting: One way to influence the outcome of conversion or churn, is retargeting users at the right time and place for incremental lift. If you know each user's probability to convert or churn, you can also predict the incremental likelihood you can change that probability if they see an ad vs. email vs. push, etc. You can in turn retarget users in the appropriate channel in the day/week they have the highest likelihood to be influenced.

      2. Personalizing the Content: Once you know whom to target and when, you can also show them a personalized message. This is most commonly done in ecommerce with product recommendations, or in media with content recommendations. If you know someone is interested in a bagel vs. a doughnut for instance, you can personalize the ads they see to show a bagel if they have the highest probability to buy it in a given week - and in turn drive incremental conversions because you personalized the experience.

      3. Personalizing the Price: Once you know whom to target and what to target them with, you can also adjust the price or offer. Many companies offer discounts and coupons for their products. But giving a 20% discount to every user means you lose potential revenue from people who had a high probability to purchase (as they were going to purchase with or without the discount). Using predictive modeling, you can adjust discounts proportional to a user's probability to convert, and maximize revenue.

      2 Share
  • AA

    Anuj Adhiya

    1 day ago #

    Hey Bilal

    Great to have you on!
    1. Are there any actions or sets of actions (in general or within certain kinds of verticals/businesses) that are more difficult to make predictions off of?
    If yes, why?

    2. Is there a way for someone who has no technical background to be able to learn/teach themselves about machine learning?
    If yes, what resources would you recommend?

    • BM

      Bilal Mahmood

      1 day ago #

      There are several factors that affect the accuracy of a prediction: the # of samples, the # inputs, the % of users performing the predicted action, and the latency of the signal or predicted action.

      With this in mind, I've found that if the action you are trying to predict his very high latency, its often hard to predict. This is common for predicting churn or even upsell in enterprise SaaS businesses. These events happen quarterly (if not yearly) which makes it difficult to predict without just being a lagging indicator.

      Trying to predict actions or consumption of a physical good can be difficult - primarily because the data collection from a digital input perspective are very difficult. For instance if you have a company like BarkBox, which delivers a physical subscription box - it would be very hard to predict churn - because the only two digital touchpoints you'd see are when they purchase and when they cancel. You have no digital footprint of what they did inbetween that time to derive signal on intent.

      To get started in ML, I'd strong recommend Andrew Ng's course on Coursera: https://www.coursera.org/learn/machine-learning

      There's also a super helpful guide in our Growth Playbook which walks through how you can use RFM and Clustering techniques as well: https://playbook.clearbrain.com/blog/easy-breezy-behavioral-segmentation

      3 Share
  • MK

    Mariana Klober

    1 day ago #

    Hi Bilal,

    Thank you so much for doing this! I'd love to hear your opinion about the following:

    Most of us see technology as a way of being more productive and reaching goals faster and more effectively.

    I like to think that once we're getting there with less effort, professionals will be able to work fewer hours (and spend more time with family, friends and enjoying the nature). Do you see that happening at all or do you think we'll have more demands coming up and will be kept just as busy (or more)?

    Thanks again!!

    • BM

      Bilal Mahmood

      1 day ago #

      Looking at trends in technological changes over the last century, unfortunately the work week seems to have only increased.

      My personal perspective is this is due to reasons two-fold: a) humans fundamentally derive purpose from their work, b) as technology automates certain facets of work, humans retrain and repurpose their skillsets to more complex tasks.

      Case in point, in the early 20th century there were literal human calculators employed. With the advent of the actual calculator, these humans became some of the first computer programmers. So despite automation of their original task, they transitioned to working on more complex tasks.

      3 Share
  • DC

    Davi Candido

    1 day ago #

    We are conducting an R&D AI/ML project here in our company and until now I'm the only person in this project (data scientist and engineer with database, backend, frontend and devops experience). However the project will migrate from R&D to production soon and we will need to deploy an engineering team to scale and grow this project. What are your recommendations to start, which skills should we focus more at the beggining? Do you think it is better to have people who knows a little bit of everything or one person devoted to a specific part of the project? How do you scale a data science team?

    Thanks in advance

  • DH

    Dani Hart

    1 day ago #

    Hi Bilal,

    I know I'm a bit late here... I'm curious your take on the ethical side of AI and machine learning. How do you ensure you're building AI in a scientific way where unintended effects/consequences are discovered and improved upon?

    Looking forward to learning from you.

    Best,
    Dani

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
24
24