Despite all the advances in technology, including advanced analytics, data management, artificial intelligence and machine learning, the CEOs that participated in the EY study, acknowledged significant gaps remain in their organisations’ ability to generate value from their data.
Other interesting points, 41% report being able to combine machine and human data effectively to make informed decisions. But the biggest chasm is on the issue of data and trust – only 34% of CEOs say customers trust them with their data.
The consequence of a hybrid workforce
According to TJ Chandler, managing director, APAC at Fivetran, the shift to hybrid workforce started even before COVID-19 came into the picture.
“The whole flexible workplace, the ability of knowledge workers to be dispersed and distributed, is something that's been going on for over a decade. What we saw through the pandemic was an acceleration of that. We all had to work from home, and therefore, there were some different strains placed on the network infrastructure and on data,” he observed.
However, Chandler opined that with the change in the work environment came the recognition of the importance of data. Creating monthly reports was no longer sufficiently. There is now a greater reliance on looking at dashboards, evaluating statistics, and needing to have that data at the point of consumption.
Dual imperative for the CIO
“We need real-time information as circumstances around COVID, businesses, the economy, transportation, everything was shifting on a daily or weekly basis. Having that real-time decision-making information became super critical, and everyone realised that,” he opined.
According to Chandler, during COVID-19, the CIO has two imperatives thrust on him: leverage data in the cloud and make teams more efficient in doing so.
Data engineer in a data-driven organization
IDC says enterprises in the later stages of their digital transformation (DX) initiatives are looking to extract more value from their data in order to help employees increase efficiency, augment decision making, and generate data-driven revenue.
Chandler believed that this desire to be more data-driven has only come up because organisations now have access to more data. However, as the volume of data rises, organisations will eventually realise they need to work smarter, not harder, to access those many sources and find a way to bring them all into some form of repository or accessible place where then, reports can be written, dashboards can be accessed, transformations can happen and real intelligence can be extracted so that the business can make truly data-driven decisions.
“Being data-driven now really means we have to work smarter in collecting consolidating and managing an ever-increasing number of data sources and making it consumable ready for everyone across the organisation, including individual contributors, it's not just for executives anymore,” said Chandler.
Prevailing misconceptions about being data-driven
Will having access to data mean being smarter at making decisions? Chandler puts a caveat on this.
He believes that there needs to be an acknowledgement that reality is continuously changing and becoming ever more informed, which can change some fundamental assumptions.
This can be very unsettling for business managers and owners and executives who have based their strategies off data that they had at the beginning of the year, and as they see new data coming in from new sources, it challenges those assumptions,” he opined.
“They have to question, “did we have the right strategy, do we need to pivot? We saw numerous examples through 2020 of companies saying the world is changed and is literally changing again before our eyes. I need more information.”
“With data continuing to stream into the organisation, it is important that organisations keep their data-based assumptions flexible and malleable,” he added.
The future of data engineers
Caroline Evans, a recruitment manager specialising in data engineering roles, at Burtch Works in the US, see a blurring of lines between data engineers and data scientists. She observed that data engineers are being asked to do more, including DevOps and data modelling.
Maxine Beaychemin, a former data engineer at Airbnb and Facebook, describe the role of data engineers as building tools, infrastructure, frameworks, and services.
Chandler is optimistic when it comes to the career prospects (and demand) for data engineers. He predicts that the work that data engineers do in terms of building commodity-style, monotonous API connectors is about to change.
That change brings opportunity for them. He is quick to clarify that data engineers’ jobs are not going to become obsolete but what they have previously been doing will.
“There will always be a need and a demand to engineer something more complex. There is always going to be another source (of data). There's the Internet of Things, there's new data sources and new requirements that need great engineers to solve those problems.
“We also know that we'll need engineering expertise to train machine learning models and predictive analytics models to help develop AI into tools that are really consumer-ready. I don't think we're there yet, but data engineers, their expertise will be needed to help make those changes possible,” he concluded.
Click on the PodChat player above and listen as Chandler elaborates on the role of data engineers as organizations push the boundaries of their data-driven initiatives.
- In 2020 COVID-19 saw an increase in the use of the cloud. This as the workforce moved to a hybrid model way of work. The result is an increase in data traffic possibly putting a strain on the infrastructure supporting the business. Is this a fair assessment?
- We are into 2021, do you see the same thing occurring throughout this year?
- What is the role of a data engineer? What are the types of data engineers typically found in a business? What job titles usually connote data engineers?
- CIOs and CFOs we spoke to have voiced the importance of being more data-driven. At the operational level, what does this mean to the business?
- What sort of investments are required to keep data engineers productive?
- What are the key points outlined in the Dimensional Research and Fivetran study?
- What is your take data engineers? Evolving?
