3 Ways to Make Your Data Science Project a Success
Data science work in commercial companies sits at a nexus of IT, data, ML/statistics and business. There are a lot of moving parts, and to consistently deliver successful data science projects, you need to get multiple things just right. And it’s not that easy.
So, what does it take to make a data science project successful? I'll share three of my biggest learnings below.
Don't Fight Windmills
As with so many things in life, data science project success is not primarily what you do (even though that's very important), but rather what you avoid doing. And the first thing you should avoid doing is inventing projects yourself. It's the quickest way to shoot yourself in the foot and set the whole team up for failure.
No, what you've read in a very exciting blog post somewhere is not going to work at your company. No, you cannot use that idea from that great talk you heard at that conference. And no, the project you want to do because you've always wanted to try out that cool modelling technique is not going to be successful.
Believe me, I'm talking from painful experience.
Your inspiration about problems that are worth solving should come from one source and one source alone: your business stakeholders. They're the only people with knowledge of pain points that you should be working on.
Here's another tip: they don't talk data science jargon. So read the quarterly business review numbers carefully, familiarize yourself with their KPIs, catch your stakeholders in lunch lines in the cafeteria (preferably in the middle of the queue, so they can't run away) and just talk to them about the problems they are facing. You'll be amazed at the number of useful data science project ideas you can generate out of those conversations.
It's not Data Science, It's Business Transformation
At first glance, it might seem obvious that a data science project should be run by the data science team. However, this common assumption is actually misguided.
Data science projects are usually very intimately connected to business. They are typically about making a certain process in business smarter or automating some decisions. For example:
If you're modelling which clients will go to a competitor because of dissatisfaction (and what might persuade them not to), you're automating part of the work the customer experience department is doing.
If you're predicting how much product you have to buy to keep your FMCG company's stock balanced, you're affecting inventory managers' day-to-day work.
And guess what? These business departments own the processes you're trying to change. They know best how to change them and have the proper authority to do so. They are the ones who should use your shiny product in the end. It was their domain before you came to do the project, and it will be their domain when you leave for the next big thing.
So, take a deep breath and repeat after me: It's not a data science project with a business component. It's a business transformation project with a data science component.
Once you make this mental shift, it becomes obvious: business should own the project and you should run it with them. This will involve them deeply in the project, shape the project so that it actually solves a real pain point, will make the whole thing more fun for you and the rest of the data science team, and deliver results.
Oh, and you won't be stuck with the impossible task of making a whole business transformation happen (to just implement a data science product) with no competence of the business or the authority to make changes happen.
Have a Really Multi-disciplinary Team
We all know you need data scientists and data analysts on the data science team. If you're up against a tough enough technical challenge, a data engineer or even a software engineer is warranted. And to be honest, this composition looked multi-disciplinary enough for me for way too long.
But having a strong technical team is only one half of the equation. If you've read through to here, you can probably guess where I'm going with this: if you're dealing with a business transformation, you need to have very strong business competence on the team. Otherwise, your brilliant technical solution might never deliver actual business value.
The typical way I've seen of incorporating business competence in a data science team is through two positions I'll outline below.
1. Data Science Delivery Manager
This person is the lynchpin of your operation. They know what data science can be used for and - no less importantly - what it cannot. They know the stages of a data science project and how to drive each one with maximum efficiency (or minimal drama). They can sniff out a brewing problem or an obstacle a mile away and prepare mitigations. They know who to involve at what point and in what capacity. And they're laser-focused on delivering value.
They are your orchestrators.
In my experience, they're also the hardest competence to find on the market. We've come to a point where data scientists are not that hard to come by - they even give out very high-quality bachelor’s degrees for that stuff these days. But DS delivery managers are - again in my experience - still a rare breed.
The obvious way to get them is to take a good IT project DM and retrain them in data science project delivery. Smart business analysts can also pivot into the role relatively easily. And I've heard of several companies that have full-blown internal academies to train people in this competence en masse.
2. Analytics Translator
As mentioned above, data science teams do not usually know much about the business they're working with. It becomes very easy to build a great solution to the wrong problem or worse, build a great solution to the right problem that cannot be deployed because some nuance in business process was not taken into consideration. So it's very important to have a very significant business competence represented in the team.
This role has evolved in recent years and has been called everything from analytics translator (which I like the most, to be frank) to connector. But the mandate is broadly similar everywhere: to understand the business context and the place of analytical solution in it, and use this knowledge to help shape the business processes and the solution in a symbiotic way.
There are many paths to analytics translator role, but in my experience it works best if this person (the more senior and experienced, the better) comes directly from the business you're working with. They leave their permanent post only temporarily for the duration of the project and will spend the majority of their time - preferably even 100% - on the project. They:
bring the knowledge of the current business processes
advise on the constraints your analytical product will have to be built around
design the new business process that will incorporate your analytical product
own and run the new business process once you're done with the project.
For example, if you want to build and effectively deploy a model that predicts call centre loads (to plan the workforce needs adequately), you want to get someone senior from the planning department. They will:
Show you around and help you understand where they're having problems with planning and of what kind and where you could have the largest impact
Give you clear directions of what the solution should look like (e.g., tell you that you need to predict loads in 30 minute chunks over the next two weeks cause this is how the operators' schedule is built every day)
Define a "good enough" solution and validate the results of your model against it
Help integrate the predictions in the planning process and train the planning department staff in the new process
Own and monitor the performance of model-based planning and alert you if something goes wrong and needs attention
Conclusion
So there you have it: let business define projects, understand that you're doing business transformation, and bring a multi-disciplinary team with both technical and business competence to tackle the problem.
Have you bumped up against these challenges? How did you go about solving them? Share your experiences in the comments below, and let's learn from each other!