Lessons Learned – CDO Survey Executive Summary
September 28, 2023Lessons from the Leading Edge:
Executive Summary
For enterprises who are in the next wave of putting ML into production, it is critical to understand what CDOs/CDAOs at leading-edge enterprises are doing to succeed and what is still an obstacle for them.
Discovering How to Get ROI from Machine Learning (ML)
With a $15.7 trillion (U.S.) potential contribution to the global economy by 2030 (PwC), demand for artificial intelligence (AI) has never been higher. But, artificial intelligence is still a project for early adopters in many cases, without the well-documented best practices and wealth of knowledge of a mature technology.
In fact, Gartner reports that 90% of AI initiatives fail to produce substantial return on investment (ROI), with about half never making past the prototype stage.
potential contribution to the global economy by 2030
But some AI initiatives are in production and generating value for the organizations conducting them. What are these leading-edge firms doing that allows them to succeed when others do not? More importantly, how can others learn from their best practices and the things that did not work for them?
Wallaroo.AI set out to answer these questions.
We commissioned NewtonX, the world’s leading B2B market research company with 100% verified research across 140 industries and 1.1 billion professionals, to reach out to ML leaders at leading-edge firms who have successfully put models in production to find out. This report summarized their findings.
ML in Production Today
There were several clear themes in the research, regarding the value of ML already in production:
The Leading Edge is Already Finding Value in Their Production ML Initiatives
Early adopters do not always realize the promise of new technologies. However, despite all the challenges of being first, 92% of AI team leaders at leading-edge organizations felt they are already finding that their AI initiatives are generating value.
The value leading-edge enterprises are gaining comes in the areas of personalizing customer experience, fraud detection, optimizing sales and marketing, and improving realtime decision making, among other use cases.
Current Investment and Future Plans Will Fuel Growth, Create Challenges for Others
Gains from current initiatives are at least partly the result of the strong investment made by leading-edge enterprises.
Partly due to the perceived value they feel their AI initiatives are already achieving, leading-edge organizations plan to invest significantly in their AI initiatives in the future (61% will more than double investment). They also intend to scale significantly within the next three years.
However, this investment (including cloud cycle costs, staffing costs, etc.) and scale will also impact other enterprises seeking the same human resources, vendors who support the ML ecosystem, and other elements of the AI ecosystem.
Key Challenges
Addressing the ML Skills Gap
Skilled ML experts are a critical factor to the success of leading-edge enterprises, according to CDO survey responses on several questions. In fact, 71% of respondents have more than 100 staffers working on ML. However, acquiring and retaining these ML experts represent some of the greatest challenges these organizations faced.
The Platform is the Challenge
Being at the leading edge means overcoming hurdles in whatever way you can. For enterprises trying to put ML into production and scale to reach ROI, that often means creating in-house ML frameworks that are later found to be suboptimal. In-house development teams then need to rework the software, which impacts staff learning curves, cost, and time-to-value But data shows that these are all challenges that CDO survey respondents are already facing, even before they make any platform revisions. Fifty-one percent cited the effort required to integrate tools as a major challenge in getting ML into production, while another 51% said that the number of discrete tools needed throughout the process was a major challenge.
ML Production Best Practices and Challenges by Stage
Where AI teams choose to deploy their models – most often in the cloud – has a major impact on the value they are able to generate for their organizations. However, leadingedge organizations encounter challenges across the entire M production process that stem from the ML production framework they use and the intricacies of working in the cloud.
Takeaways
- The research shows that leading-edge enterprises face many challenges that impact cost and productivity because of the dependency on hard to find and retain ML experts, the choice of cloud, and use of in-house developed ML production frameworks.
- All AI teams will be looking for and trying to retain ML experts but, fortunately, all leadingedge enterprises have strong ideas on how to do so that AI team leaders can use to acquire and retain ML experts that other organizations can use. There is also the option of working with a commercial vendor that has ML experts who can assist.
- To overcome the challenges faced by leading-edge enterprises, two viable options are available to the next wave of AI team leaders. They could adopt a purpose-built ML production platform or they could develop an in-house framework that includes the capabilities of commercial ML platforms. The challenge with an in-house framework that includes the latest innovations is that it would need to operate seamlessly across all phases of the ML production process without taking the significant time necessary to develop and test such a complex technology.
Next Steps
