There are two visions for the future of data analysis and analytics. Tech and business magazines and IT evangelists see artificial intelligence (AI) and predictive analytics on the rise.
When I talk with business users and executives, especially ones already working with data warehouses, they see the future (and invest) in self-service business intelligence (SSBI).
One term – data analytics – having two meanings – AI and SSBI – this is the classic setup for misunderstandings, failing project pitches, and failed projects. But what exactly are the differences between AI and SSBI? And are they complementing or competing concepts?
The rise of self-service business intelligence
Business intelligence teams (and data warehouses) are established in many companies and organizations. They have been implemented years ago and are constantly in use. Data warehouses collect data from various (operational) databases, clean and transform it, and generate periodic reports and provide answers to ad-hoc queries. As a kind of grassroots movement, especially experienced users started to complain. They know the data in their data warehouse and its potential value for the business and their work. However, they cannot unlock all benefits. The limitations of their rigid business intelligence installation hinder them. Thus, the user feedback is often not flattering – too expensive, too inflexible, too slow.
This contrasts my personal observation that business intelligence and data warehouse engineers know the data models very well, often better than the business. Thus, if we want to understand why they are considered to be too expensive, too inflexible, and too slow, we should not look at the qualification of individual engineers and how they perform single tasks. The focus should be on the operational model. More precise criticism proves this:
Getting new data into a data warehouse is time-consuming. There is often not a lot of technical work to be done, but there are high lead times due to needed clarifications. What data is needed exactly and who can provide the data? Do you have to clean, filter, and transform the data? My personal experience with setting up new data feeds is that a few days of engineering can require weeks of clarification and specification. It takes time in larger organizations to find the experts. A data warehouse team has to deliver “good quality” data. This is part of their service offering. They cannot publish some data that might or might not be correct and which is regenerated differently every few days or outdated after a week.
Some data warehouse teams invest a lot of time in detailed specifications. It relates to the “new data” topic but has a different nuance. The root cause is a dysfunctional collaboration model. Detailed specifications are a consequence of being blamed and being made responsible for delivering something wrong – and not being able to prove the opposite or to fix it immediately. If a report takes four or five months for implementation, the business is furious if the result is different than expected. They are not furious about the misunderstanding. They are furious because they waited long. They still do not have the reports they need for their business – and fixing the misunderstanding can take some more weeks or months. To prevent – justified or unjustified – furious escalations, engineers invest more and more time in writing specifications and getting them approved.
Insufficient staffing: there is more demand than the BI team can handle. My experience with such services is that if the team provides superior service and/or there is no internal charging, there is always a lot of demand. The middle or senior management decides about the team capacity, the prioritization of tasks – and which requests are not considered.
Limited support for follow-up questions and investigations. This is the only more technical limitation. Often, answers to one question result in follow-up questions. If controllers realize that revenues drop in Western Europe, they want to know the exact countries and industry segments. The ability to drill-down or to submit slightly different queries is a natural need. Business users do not want to go through enough long circle of specification, development, and delivery, depending on the exact service model and the software product features.
Figure 1: Limitations of many business intelligence services from the users' perspective
Self-service business intelligence – the core concepts
The four points make it clear that ‘self-service business intelligence’ is more than enabling users to submit queries against an existing data warehouse with its current data set. Furthermore, the best indication of not matching the user expectation is when they ask the data warehouse team for reports containing all data. This is when they start querying the data themselves in Excel or Access and avoid the business intelligence service bottleneck as much as possible.
Many challenges are process-related, but the solution is software. SSBI software provides features that change how business users and business teams collaborate. I want to illustrate this by looking at some features of Microsoft Power BI. There are many more vendors, but a quick look at Google Trends proves that Power BI gets nearly as much attention as the Business Intelligence topic itself (Figure 2).
Figure 2: The rise of Power BI. A global Google Trends evaluation looking at this software solution compared with the general area of business intelligence
Users can create inspiring reports in Power BI with various diagrams including drill-down functionality or presenting numeric values on geographic maps. They can create complete cleansing and transformation pipelines using a GUI with features such as filtering or replacing values. This sounds like Excel on steroids, but it is a different software product class. Users define and execute reliable ETL processes on their local machine. This helps for periodic or recurring evaluations. In short, SSBI tools provide a comprehensive access layer for users for data access, for querying and transforming data, and for producing visually appealing reports (Figure 3, ①).
A trivial feature in Excel, but revolutionary in the world of databases, addresses the challenge of the long delays of getting new data into the data warehouse. Users can load data from their local laptops into Power BI and use them together with data stored in databases and data warehouses (Figure 3, ②). This offers users new opportunities for ad-hoc queries. They can run them without involving the business intelligence team. If they need a periodic report, they can implement a temporary solution to bridge the time till the business intelligence team implemented the final report – or they can hand over their reports to the business intelligence teams as a specification.
The ability to implement queries with a mixture of data warehouse and local data is also helpful from a governance perspective. It is always clear which data and reports are in the responsibility of a business intelligence team and which are in the user sphere.
Many companies benefit a fourth core concept of SSBI: enhanced collaboration features. They are rarely mentioned in articles, but Microsoft charges extra for them. This is a clear indication of the value of these features for larger companies. I also remember an event of another major player in the SSBI market, Qlik, some time ago in Zurich. One customer presentation was exactly about that: enabling controllers in an international organization to share and annotate reports instead of sending them around via Email and merging various feedback threads in yet other Excel sheet. SSBI can make copying, comparing, and consolidating Excel sheets obsolete (Figure 3, ③).
These three SSBI innovations do not relate to AI. This explains why pitching AI projects to the business does not work if they talk about analytics and have SSBI in mind. However, SSBI can serve as a kind of Trojan Horse. The tools usually allow integrating R or Python code (Figure 3, ④). As always, there are prerequisites and limitations – additional local installations might be needed, the modules work only on the presentation layer, etc. That is the “half-empty glass perspective”. The “half-full glass perspective” is that many users – all SSBI users – are closer using such tools in their company than ever before.
Figure 3: Self-service BI concepts
Comparing artificial intelligence use cases and self-service business intelligence use cases
AI derives insights automatically using algorithms, math, and statistics. In contrast, self-service business intelligence enables humans to analyze data more efficiently, to provide better reports, and share the reports. So, are AI and SSBI separate topics and not related to each other? Or, more provocatively, does AI make SSBI obsolete?
AI is great for finding patterns in large data sets. It all starts with a training set. The algorithms go through the set and derive and learn rules. These rules can be applied to new data the algorithm has not seen before. Looking at the behavior of today’s customers, companies can estimate how lucrative a new client is. Certainly, humans can do the same: they load data into Excel or Power BI and try to find correlations between gender, age, income, etc. to estimate how much a new customer can pay for the next holiday trip. However, humans – with or without SSBI tool – have no chance against AI. AI analyzes thousands of input variables and looks at relationships between dozens of variables. Humans cannot do the same. Thus, AI makes the use case “tool for users to manually analyze large data sets” for SSBI obsolete, because this stands this should not be done by humans anymore. However, it depends on the company culture how they move. Not everyone is ready to switch to AI-generated knowledge.
There is also some good news for the SSBI vendors. There is a second reason why employees have to analyze large data sets: understanding data singularities. In companies, large reports often contain some “surprising” data constellations. The management has to understand the reason. Is it a mistake in the data or is the reality different than the expectation? Why is the revenue dropping in the US whereas the rest of the world prospers? Why do we see a peak in profit in Antarctica where we do not have any branch? Controllers spend a lot of time with such tasks. The tasks are about cleansing data and understanding whether data discrepancies make sense or are caused by errors. There is nothing to learn and generalize using AI.
Figure 4: Comparing the use cases for AI and SSBI
So, the conclusion is: vendors sell both artificial intelligence and self-service business intelligence for “data analytics.” Customers have to understand what they exactly want to achieve, analyzing data singularities or learning rules. AI beats humans in finding rules and patterns, though some employees want to continue doing their Excel analysis (and demand SSBI to optimize doing their obsolete work). Here, the concept of artificial intelligence and self-service business intelligence compete. In contrast, AI is of no help when it comes to understanding data singularities. This is the true domain of self-service business intelligence. So, from a high-level perspective, self-service business intelligence complements artificial intelligence, because they have unique strengths best suited for different use cases.
Klaus Haller is a Senior IT Project Manager with in-depth business analysis, solution architecture, and consulting know-how. His experience covers Data Management, Analytics & AI, Information Security and Compliance, and Test Management. He enjoys applying his analytical skills and technical creativity to deliver solutions for complex projects with high levels of uncertainty. Typically, he manages projects consisting of 5-10 engineers.
Since 2005, Klaus works in IT consulting and for IT service providers, often (but not exclusively) in the financial industries in Switzerland.