May 62 min read

Exploring data challenges: Insights from our community survey

The image is a promotional graphic titled "Insights from our Data challenges survey," featuring a stylized illustration with a vibrant blue and purple background. It includes two cartoon-style figures: one sitting atop a large star, using a laptop, and another seated on the ground also using a laptop. The design is embellished with decorative stars and circles, conveying a lively and engaging theme to highlight the importance of the survey's findings.

In April 2024, Cloud Shuttle conducted a survey of data professionals in Australia and New Zealand. We were interested in exploring the common challenges that data professionals face in data platform engineering and management.

The job category breakdown of the respondents were as follows: Data managers (36.4%), Data engineers (31.8%), Data architects (9.1%), Data scientists/analysts (9.1%) and other (e.g. consultant).

The majority of respondents were located in Australia (84%) with a minority in New Zealand (16%). A wide range of company sizes were represented in the survey, from small startups to large enterprises, highlighting the widespread nature of many of these challenges.

The infographic titled "Data challenges 2024 community survey results" showcases the findings of a survey conducted by Cloud Shuttle among data professionals in Australia and New Zealand. It presents the distribution of respondents by location, job role, and company size, emphasizing that the majority are from Australia and are either data managers or data engineers. It highlights the most cited data challenges faced by these professionals, including data observability and tooling, data quality, data governance and compliance, and data integration. The visual aims to spotlight common hurdles in data management and platform engineering.

Notable trends

Survey respondents listed their biggest challenges (in order) as: data governance and compliance, data quality, data observability and tooling and data integration.

Data observability and tooling

The lack of centralised platforms for monitoring data events was raised as a common problem. Respondents wrestled with data silos and quality detection issues, resulting in reduced trust in the data outputs and an ultimate slowing in productivity due to the time lost navigating and managing complex, redundant data workflows.

Data governance and compliance

Data governance frameworks remained a stumbling block, with many respondents mentioning that their governance was not up to scratch. There were often inconsistent standards and unclear data ownership. Poor documentation and communication made the loss of institutional knowledge a much higher risk for businesses.

Data Integration

Many teams struggle with the ingestion and acquisition of data across disparate systems, hindered by the sheer number of tools available in the market, integration challenges, legacy systems and growing technical debt.

Data quality

These challenges have a common result: less-than-ideal data quality issues and data that is less useful as a strategic business asset. If the data quality is poor, it’s more difficult to drive actionable insights and establish a solid foundation to inform business decisions.

Conclusion

While the sample size was small (<100), it highlights shared obstacles faced by data professionals in the region. These challenges aren’t isolated ones and could be a good indicator of broader trends affecting data management across different industries and organisational sizes. Identifying and understanding these commonalities is the first step towards developing more effective strategies and solutions to uplift our data practices.

At Cloud Shuttle, we are familiar with the common data challenges organisations of various sizes faced. And we’re committed to partnering with our clients to overcome these issues. If you're experiencing any of these issues or looking for ways to leverage your data more effectively, get in touch today!

Get in touch