According to IDC, 45% of the time is wasted every week because data workers are unsuccessful in their activities. The wasted time is attributed to more than just a lack of skills in creative and analytical thinking, and a lack of data visualization skills.
It also has to do with a large amount of data sources, the different types of data sources, and the types of outputs being asked.
One delegate to a FutureCFO roundtable confessed that she eventually gave up on the notion of relying on spreadsheets to perform data crunches because of the number of rows her team had to deal with, the number of spreadsheets they process at any given time, and the sheer inefficiency and complexity the whole experience has become.
Sexiest Job of the 21st century
In an article on Harvard Business Review (HBR), authors Thomas H. Davenport and D.J. Patil observed demand for data scientists has been growing and that even at startups the compensation is competitive.
What is interesting is that data scientists are no longer the backroom geeks content at hiding behind their desks. They want to be in the thick of a developing situation, with real-time awareness of the evolving set of choices it presents, as noted in the HBR article.
The chief data scientist manages data-driven functions including overseeing data management, creating data strategy, and improving data quality. They also help their organisations extract the most valuable and relevant insights from their data, leveraging data analytics and business intelligence.
The data scientist role is critical for organizations looking to extract insight from information assets for “big data” initiatives and requires a broad combination of skills that may be fulfilled better as a team.
What’s expected of data scientists
Geoff Soon, managing director of South Asia, Snowflake, says there are three distinct skill sets that a data scientist possesses: domain knowledge of the business, strong background in mathematical statistics, and having a technical background.
The combination of the three is difficult to find in one person. “That's why we've seen some organisations take a hybrid approach by getting people with maybe one or two of those skill sets with an individual with complementary skill sets, so they can perform the same design,” said Soon.
He noted a trend to offset the limited pool of data scientists in the market. There is an effort to make the skills more accessible by bringing in more data science principles to what data workers have been doing for quite some time.
Top challenges to data workers
Soon acknowledge that data scientists still face challenges when it comes to access to fresh meaningful data. In many organisations, data is being tracked across silos. He noted that data scientists want standard transactional data, but they also want to understand how customers are interacting with the digital channels.
He also commented that the COVID-19 has changed markets more than the last decade. Data scientists, for instance, now need access to third-party data to be able to refine their models.
“The biggest challenge is getting access to this data which currently sits in silos, and being able to adequately leverage, third-party datasets, when data scientists create these machine learning and data science models,” he continued.
Critical skills lacking
“Fundamentally, the role of a data scientist is to be able to sell a particular outcome to a particular business stakeholder,” Soon opined. There is a need to have a strong business alignment and a clear understanding of the technical side.
He cautioned that data science for the sake of error, experimentation or innovation is not going to help. He reiterated the importance of identifying good problems in data science.
“Once we have a consensus on how these problems can be solved, next is look at how we can create a technical foundation that will enable the problem to be solved and maintain on an ongoing basis,” he added.
He stressed that a lot of effort will go into building the foundation for enabling this understanding, including identifying data sources – both internal and external, what data is needed and how this data can be accessed quickly on an ongoing basis.
He cautioned that access to data is only the beginning. ‘How to execute’ is the next step.
“A lot of innovation has been done to reduce the amount of programming required by data scientists. Many tools are taking some of that off the plate, making it easier for the data scientists to rely on their domain knowledge,” he continued.
Grow the data mindset
Soon conceded that data scientists are not a cheap resource. He hinted throwing money by bringing external skills into the role may create resentment within existing organizational structures.
With technology helping to simplify the work, a key aspect of being a data scientist is understanding what business problems can be solved. He suggested that leadership first look at what is already available internally.
“The people that often work or have been with an organisation for quite some time have that domain knowledge. It’s a matter of upskilling them to deliver value back to their organisation., he concluded.
Click on the PodChat player to listen to Soon elaborate on the challenges, opportunities and issues facing data science initiatives and teams in the quest to better understand and utilise data.
- What is a data scientist and where do they come from?
- What are the top challenges data scientists face in their day-to-day work?
- What are the key skill sets needed for today’s data scientist (technical and soft skills)?
- Who decides what the role of a data science team is, the qualifications, etc?
- How can a public/private partnership enable a strong pool of such talent and bridge the current gaps?
- Should data scientists care about data privacy?
- What is your advice to leadership about acquiring and nurturing a data science team?
