Insights & Articles
Getting started with Lakekeeper and Trino OPA (Data Governance)
Modern data platforms require strict governance to maintain compliance and security. This guide demonstrates how Trino query engine and Lakekeeper catalog can be integrated to achieve data governance in your data platform. Trino respects access policies defined at Lakeeper catalog level.
Peter Hanssens
Mar 03, 2025
Breaking into Data Engineering
The Cloud Shuttle Graduate Data Engineering Workshop is a 5-day paid training program designed to help recent graduates, final-year students, and career switchers break into data engineering. Created by an industry veteran with 15+ years of experience, the workshop addresses the growing demand for data engineers (with a 30% CAGR since 2019) by providing hands-on, practical experience that universities often lack. Participants learn essential skills like building ETL pipelines, working with cloud platforms, and using tools like Python, SQL, and Airflow. The program also includes career preparation support and aims to bridge the gap between theoretical knowledge and real-world job requirements, backed by success stories of graduates landing positions at companies like TikTok and major consultancies.
Peter Hanssens
Feb 21, 2025
Why Data Products are the Future of Data Engineering in Australia?
Data products are revolutionizing data engineering in Australia, marking a shift from traditional ETL pipelines to more sophisticated, product-centric approaches. As evidenced by success stories from Latitude Financial, Newcrest Mining, and Woolworths, Australian organizations leveraging modern tools like Backstage, Mage, and Apache Iceberg are seeing significant improvements in their data operations. McKinsey reports that companies adopting a data-as-a-product mindset are 2.5 times more likely to excel in data-driven decision-making, highlighting the transformative potential of this approach. This evolution is enabling faster deployment, enhanced team collaboration, improved data quality, and reduced technical debt across various sectors in the Australian market.
Peter Hanssens
Jan 25, 2025
Streamlining Data Migration for an Energy Giant
Cloud Shuttle successfully transformed a leading energy company's data infrastructure by migrating from Matillion to a modern stack using Snowflake, dbt, and Airflow. In just five months, our team of 2.5 FTEs migrated over 300 models while ensuring zero disruption to business operations. The project included implementing automated orchestration, establishing S3 as a handoff point, and conducting thorough regression testing throughout the migration. This strategic modernisation resulted in a 30% increase in operational efficiency, enabling real-time data accessibility and empowering teams with faster, more reliable data processing capabilities for improved decision-making.
Peter Hanssens
Oct 24, 2024
Driving Data Transformation for a Leading Insurance Provider
When a leading insurance provider found their legacy SSIS systems creating bottlenecks and driving up costs, Cloud Shuttle stepped in with a transformative solution. By migrating over 100 SQL models from SSIS to dbt on Snowflake, we helped modernize their entire data infrastructure with just one FTE. The migration eliminated data silos, simplified ETL processes, and reduced infrastructure costs while providing seamless cross-product insights. This strategic transformation not only solved immediate operational challenges but also positioned the client's data infrastructure for future growth and innovation in the competitive insurance landscape.
Peter Hanssens
Oct 02, 2024
GraphSummit Sydney 2024 recap: Innovations from the frontier of Data and AI
Neo4j's GraphSummit World Tour 2024 brought together 150 industry leaders in Sydney to showcase how Australian and New Zealand organizations are leveraging graph technologies with AI to solve real-world challenges. Highlights included Commonwealth Bank's unveiling of GraphIT, their network infrastructure digital twin, SparkNZ's innovative use of graph databases for RFP automation, and McKinsey's sobering perspective on GenAI implementation challenges. The summit featured practical workshops on architecting graph applications and enabling GenAI breakthroughs with knowledge graphs, while Cloud Shuttle demonstrated our DataEngBytes chatbot powered by Neo4j, LangChain, and Amazon Bedrock. As graph databases continue to evolve, the event highlighted their growing importance in everything from supply chain optimisation to drug discovery.
Peter Hanssens
May 13, 2024
Exploring data challenges: Insights from our community survey
Cloud Shuttle's 2024 survey of data professionals across Australia and New Zealand reveals the pressing challenges facing the industry today. Drawing insights from diverse roles including data managers, engineers, architects, and scientists, the survey highlighted four key pain points: data governance and compliance, data quality, observability and tooling, and data integration. Despite varying company sizes and locations, professionals consistently reported struggles with centralized monitoring, unclear governance frameworks, complex integration challenges, and maintaining data quality. While the sample size was modest, these findings underscore the shared obstacles in modern data management and the need for strategic solutions to enhance data practices across the ANZ region.
Peter Hanssens
May 06, 2024
Snowflake Data Cloud Summit 2024: Builders keynote
Day 3 of Snowflake Summit 2024 delivered an engaging Builders Keynote that showcased the platform's seamless AI and data capabilities through the return of "Tasty Bytes"—Snowflake's fictional food truck company. Through three practical demonstrations, the team illustrated how to implement customer sentiment analysis, create domain-specific chatbots, and perform network graph analysis, all within Snowflake's unified platform. Highlights included Vimeo's real-world implementation of LLMs for video analysis, Cash App's network science applications using Relational AI, and practical demonstrations of how Snowflake's tools can turn complex data challenges into actionable insights—all while maintaining data security and governance within the platform.
Peter Hanssens
May 05, 2024
Snowflake Data Cloud Summit 2024: opening keynote recap
Live from San Francisco's Moscone Center, Day 1 of Snowflake's Data Cloud Summit kicked off with CEO Sridhar Ramaswamy's bold vision for Enterprise AI democratization. The opening keynote featured major announcements including Polaris Catalog, an open-source catalog for Apache Iceberg, and two significant NVIDIA partnerships—Snowflake Arctic's support for TensorRT-LLM and NeMo Retriever integration with Cortex AI. A highlight was the virtual appearance of NVIDIA CEO Jensen Huang, who discussed AI's accelerating advancement rate of 2x every six months. The day concluded with an insightful panel of Chief Data Officers from JP Morgan Chase, Ericsson, NYC Health + Hospitals, and Booking.com, exploring how different industries are navigating the enterprise AI landscape while balancing innovation with governance and security.
Peter Hanssens
May 04, 2024
Snowflake summit: platform keynote recap 2024
Day 2 of Snowflake's Data Cloud Summit packed a punch with major platform announcements focused on Enterprise AI capabilities. Co-founder and President of Product Benoit Dageville kicked things off by positioning Snowflake as the ideal Enterprise AI platform, highlighting five critical components: data, elastic compute, world-class AI models, security and governance, and collaboration. The summit unveiled several exciting updates, including Iceberg Tables going GA across all clouds, enhanced data governance features through Snowflake Horizon, and significant AI/ML developments like Snowflake Copilot reaching GA. A standout announcement was Cortex AI's Fine-Tuning entering public preview, allowing users to customize LLMs at one-tenth the cost of training from scratch. With demonstrations from customer success stories like Siemens and Pizza Hut, Snowflake reinforced its commitment to simplicity, efficiency, and robust data foundations for Enterprise AI applications.
Peter Hanssens
May 04, 2024