Why are so many AI projects doomed for failure?
The elephant in the room is actually human
By Shahar Cohen, PhD, Head of Verix AI Lab.
One of my first data science projects sought to improve the targeting process of a sales organization. Before we came in, sales reps were expected to choose their own target lists, competing with each other on a single pool of leads.
Danny*, one of the dozen reps, was accountable for a staggering 40% of total sales, and of course was well compensated with extremely high commission on his sales. Clearly, Danny knew something that others simply did not.
Management hired us, hopping to scale up sales by revealing Danny’s secret sauce and automating it in a data science model, so other reps can perform as well as Danny. At the kickoff meeting the VP sales stated there were enough good leads for everyone to ensure that all the reps, including Danny, will end up getting better results. But after the meeting over a pint of beer, Danny advised us not to take the project, as it was bound to fail.
Three months later we’ve implemented a model for scoring leads according to their propensity to buy, which produced excellent lead scoring. However, to our disappointment, actual sales based on the automatically generated lists were a complete failure and soon enough, the organization went back to their old method. Apparently, Danny who had a strong incentive to preserve the existing situation, was very influential on many aspects of operations, and no matter what we did, he made sure it fails.
As frustrating as the failure was, it taught me an important lesson: it is the human intelligence that makes data science projects succeed or fail. Not the artificial one.
Data science projects, as any other organizational project, are intended to improve the business. We talk a lot about the scientific and technical challenges, yet even if we manage to develop perfect models to address the business objectives, we must bear in mind that AI solutions eventually affect the daily lives of humans. And humans tend to have different opinions on how they want things to be done. Apparently, organizational politics is the # 1 reason AI projects fail.
The term organizational politics brings up negative sentiments of self-serving behaviors. Since promoting meaningful endeavors within an organization requires the cooperation of numerous stakeholders, more often than not we’re up to a clash in perspectives and opinions about the situation.
Healthy organizations have a structured way to balance between the ability of stakeholders to express opinions and affect decisions and their responsibility to commit to resolutions, even if they disagree with them.
AI projects have an air of mystery about them, leading to emotions and misconceptions that magnify the impact of organizational politics. In most cases, only a few stakeholders really understand AI and how it can help the business, while others have opinions based on professional pride, fear, and sheer conservatism that often lead to an organizational conundrum.
Over the years, I developed a set of tips to help navigate the organizational politics rather than clashing with them:
- Management commitment: to promote change, management must align the organization with decisions that are made. Listen to objections and carefully resolve them to arrive at full management commitment.
- Clear vision: management needs a clear vision of the project to direct the organization in the right way. Have a clear vision of goals, scope, and expectations from the project.
- Communication: to align the organization behind the vision, it has to be clearly and consistently communicated to all stakeholders
- Mapping participants and listening to all voices: as part of the learning process it is important to map every stakeholder who will be affected by the new AI solution. Do not narrow your discussion to only decision makers. Listen to every stakeholder and be sensitive to different opinions.
Takeaway: although organizational politics is the # 1 reason for failure of AI projects, it is here to stay as an integral part of organizational life. Having a strong management commitment, a clear vision that is well communicated within the organization and listening to different opinions of stakeholders will dramatically increase your chances to succeed with your AI project.
* Some names and identifying details have been changed to protect the privacy of individuals
