Bridging the Gap Between AI and Business Intelligence

An opinion piece by the CEO and co-founder of Pecan.ai, a predictive analytics company.

Dr. Zohar Bronfman, CEO and co-founder of Pecan.ai

August 22, 2022

7 Min Read

Data can be a company’s most valuable asset, providing the basis for predicting everything from future revenue to buying behavior and customer retention. Many companies have well-established business intelligence (BI) teams that review and analyze historical data for performance and management trends.

But when companies want to move beyond traditional historical analysis to incorporate predictive analytics and artificial intelligence (AI), they face challenges in finding the talent and tools they need. Data scientists are hard to hire, and they are trained to focus more on research and model accuracy than on achieving specific business results.

For businesses to make the most of their data, the key is to bridge the chasm between data science and BI. Both domains analyze data to propel the business forward, but each has strengths and limitations.

Classical BI

Classical BI is well understood: It is mainly focused on interpreting past events and trends and presenting them in easy-to-digest aggregated reports and dashboards. A limitation of BI is that the insights generated are usually hypothesis-driven, meant to explain why a particular trend or behavior happened in the past by looking at a large segment of people with similar characteristics.

But without machine learning (ML), BI cannot provide precise, hyper-granular insights down to the individual customer level. Most BI teams also do not have the deep statistical analysis training needed to implement predictive modeling.

This is where data science is supposed to help. So far, data science has often fallen short of that promise for many businesses. Predictive and prescriptive models are hard to deploy, and most projects never make it to production. At the same time, companies are challenged to quantify the business impact that their ML and AI investments have generated.

To address the shortage of talent and the disconnect between data science and business priorities, new advanced analytics solutions help companies leverage the business analytics talent they already have.

Business analysts typically work closely with specific departments or lines of business, so these professionals know how their organizations capture data and how they create and measure business value. In addition, many of today’s business analysts are eager to access automated statistical analysis, machine learning, and data cleansing so they can focus on interpreting and applying predictive models that provide more value to the company.

BI teams know the data and what is important to the business. They can work with business leaders to answer questions that include the following: What metrics are you looking to improve? Are you trying to grow revenue, reduce churn or increase customer lifetime value? These different goals will point to unique approaches to analyzing data.

Applying AI

Applying AI capabilities to BI data has moved analytics from looking at the past in aggregate to predicting the future of an individual customer and highlighting marketing opportunities. There are many questions this could help answer:

  • How often should a mobile game publisher offer a specific promotion to a player to bring them back to the game?

  • How much discount should an e-commerce company offer to win back a customer who has not made a purchase in the past two months but whose predictive lifetime value puts them in the VIP category?

  • If the customer is 90% likely to return on their own, should the marketing team spend their marketing dollars to retarget them or divert the funds to a different program or campaign?

BI can only show you that there is a connection between players and customers receiving special offers and returning to play or buy again — but that connection only reveals that people like free stuff and discounts. It does not tell us which customers will really like a particular offer at a specific moment in the future.

Instead of making the same offers to a large cohort of people, predictive analytics can identify which customers are most likely to return on their own and which need the nudge of a promotion. With this information, a company can target its marketing to the specific customers who will respond best to this nudge at the right time.

Business efficiency anchored in precision and automation is critical to gaining and maintaining scale, especially when resources are limited by challenging market conditions. Predictive models provide a glimpse of the customers’ future. When built and deployed in software platforms that bring together BI and data science, these predictive capabilities become accessible to many companies.

The chasm between data science and business analytics needs to close if we want to maximize the opportunities for very capable, data-rich BI teams to bring more value to enterprises.

Making the leap to AI predictions

To start leveraging their BI data and make the leap to AI predictions, companies should consider these steps toward maximizing their data and their team’s potential.

  1. Start with the question in mind. Focus on which business needles you want to move and have a concrete understanding of how you can use predictive analytics to make that happen. If a customer has not purchased something recently, it is important to incentivize them to do so, but finding the right combination of incentive and channel can be tricky. This is a problem that AI can help solve. For example, data analysts can employ a predictive model-based scoring system to automate the identification of customers who may respond to a bigger discount and be retained longer. That capability means you can predict and shape future customer outcomes with a deeper understanding of what makes each customer come back.

  2. Don’t stress about 'perfect' data. A new data project can require weeks of validation and data preprocessing. If you have business analysts on your team who employ tools like Looker and Tableau, you probably have plenty of data for them to analyze. You do not need to make sure every data point is accounted for. You can use the data you already have to create predictive analytics. How? Use your BI-ready data, which means the data is already in a state in which you can drive classic analytics, and select a predictive analytics solution that automates time-consuming data preparation to create an AI-ready dataset. It could save you months of data preprocessing — before any feature engineering can take place and a single model is created.

  3. Design A/B tests to validate the accuracy of predictions. An A/B test is one of the fastest ways to try out new changes and approaches like predictive modeling. Once a model has been developed, the effects of using it should be tested against a control group that is handled with your business-as-usual methods, such as business rules you may have been using to determine offers provided to customers. If you do not test how the model integrates into your business processes and compare its impact to a control group, then you do not know for sure if the model will produce your desired business outcomes.

  4. Enrich the data you have. Your internal transaction and customer data is an ideal starting point for predictive analytics. And, although it is not necessary, some businesses also benefit from enriching their data with external data sources, such as weather, holiday and public health data. Automating that enrichment is one of the fastest ways to ensure additional data streams continually add value to your models’ accuracy and usefulness.

  5. Plan for model monitoring and retraining. There is a common misconception that machine learning models become better over time completely on their own. The opposite is actually true — models typically have a short shelf life. They will work great for a while, but their performance will change over time as your business enacts different strategies based on the models and as customer behaviors shift. Many companies that hire data scientists find that they also need dedicated machine learning operations (MLOps) teams who handle model implementation and ongoing model management. However, to save time and resources, automated solutions can monitor and retrain models so they continue to deliver high performance and business impact.

About the Author

Dr. Zohar Bronfman

CEO and co-founder of Pecan.ai, Pecan.ai

Dr. Zohar Bronfman is the CEO and co-founder of Pecan.ai, a predictive analytics company. With deep expertise in computational psychology and data science, Zohar applied his inherent entrepreneurial spirit to co-found Pecan, right out of graduate school. Zohar holds two PhDs from Tel Aviv University – one in computational cognitive neuroscience and another in the history and philosophy of science and technology. He also holds a BA in economics from the Open University of Israel.

With deep expertise in computational psychology and data science, Zohar applied his inherent entrepreneurial spirit to co-found Pecan, right out of graduate school. Zohar holds two PhDs from Tel Aviv University – one in computational cognitive neuroscience and another in the history and philosophy of science and technology. He also holds a BA in economics from the Open University of Israel.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like