Importance of Business Context in AI projects
Scenario 1: You have developed a model that predicts propensity of current customers to buy your other B2B SaaS products, deploying multiple data sources and using ensemble of models for the best predictive power, it took huge amounts of efforts and resource utilization. You provided a rank ordered list of prospects to the users of this output, the Sales Representatives, across geographies. They discard the list right away saying that they have taken months to develop relationship with their customers and now you say that they are unlikely to buy our product based on some machine output.
Scenario 2: You developed a model for predicting attrition probability for your retail banking customers. This list is provided to the marketing team which could then design retention campaigns for these customers to provide them incentives for not closing their accounts. After while, the marketing team gets back saying that the output is useless for them since they would need least 3 months’ time for preparing customized campaigns, and your model is predicting attrition within the same month or point in time. Those customers that the team is reaching out have already attrited or have processed the account closure formalities already.
Scenario 3: You have designed an Anomaly Detection system which could identify whether any specific input to a process flow is very far off from normal. It could detect anomalies with a certain confidence and prompt the user that anomaly has been detected in the user input. When the system is deployed, most of the end users provide their feedback demanding an explanation as to why their input is flagged as Anomaly. They discard this system and ignore the warning message.
All above scenarios are classic examples of designing best solutions for the wrong problem. Often when we think of Artificial Intelligence, the “something exciting” image always overpowers whether it makes any business sense. This happens when we do not visualize end to end process flows in AI projects, especially not ensuring all entities are involved and onboarded on the idea right from beginning of the project. In above scenarios, only after the entire system was developed, the team gathered inputs and feedback from end users of the model, which should have ideally been given primary importance in initial phase.
In Scenario 1, you need to understand the pain points of your sales teams first who are consumers of this model. Based on their inputs, the model output could be presented in a way that is easier and tempting for the Reps to consume. For eg. Instead of providing just a list of customers most likely to purchase the product along with a probability score, we can add elements like current product penetration, onboarding date, affluency, share of wallet, service requests, last transaction date etc. and bucketing the probability score into High/Medium/Low which will give Sales Reps a solid reason as to why it is important to contact High probability customers first. These features will anyways be inputs to the model, but it is important to understand that the Reps might have no idea about how a machine learning model works. Especially in B2B companies where sales cycles are extended over months and relationship building is a critical task, this 360-degree view empowered by propensity to purchase will aid the Reps in faster decision making.
Scenario 2 is a very common example of undermining the importance of Time in predictions. If I create my target variable for the model as “Account Closure = Yes”, and identify patterns that lead to customers taking this action, I might be ignoring the fact that customer’s “Mindset” to close the account might have infested 3–4 months before actual closure. Imagine that you are unsatisfied with your bank’s services and want to close your Savings account, what will you do? Will you suddenly wake up one fine morning and rush to the bank for processing closure formalities? Most likely you will be first taking out money and transferring to other accounts, not depositing anything in your account for weeks, and would be reaching out to contact centers for understanding formalities. All these are triggers or “Mindset” for closure and it is this phase which is most critical for reaching out to the customer rather than after they have already processed closure formalities.
Now coming to Scenario 3, the most common and challenging problem, how do you explain your model outputs to the consumers? Without keeping the end users in mind, it doesn’t matter how accurate you are in your predictions, if they do not understand the output, they will simply not consume it. To generate explanations although there are good resources available like Shapley values or LIME, it is still challenging to represent them in plain simple language without any tech heavy jargons, because we are not designing it for us to consume but for a person who might not even know what data science means or can do. Keeping things simple is an art in the era of advanced AI. Specific to this scenario, the problem could be pivoted to understanding groups of similar looking data points and generating manual explanations for what is represented in each bucket, and overlaying this information on the Anomaly model predictions.
These are just a few examples of the challenges in AI projects which require a good holistic understanding of the business problem and spending time in defining and structuring the problem. Many times, the teams focus on the HOW part more, with little attention to the WHAT and WHY, which creates issues further down the pipeline and can result in wasted time and effor
