The talk covers the challenges and strategies involved in handling tens of billions of records daily and during outages in a petabyte-scale cloud migration. It highlights the use of Databricks and Azure SQL Hyperscale, custom connectors for recovery, client-specific authentication, and performance optimization using clustered columnstore indexes. The presentation focuses on bold decisions and practical methods for building scalable, secure, and resilient data pipelines.
Balázs Szántó is a Technical Architect in Data Engineering field at Accenture and a recognizedDatabricks Champion. With over a decade of professional experience, he currently leads amultinational team of Data and DevOps engineers on a large-scale enterprise project. Hisexpertise lies in designing and implementing scalable data platforms, with a strong focus onDatabricks and Microsoft Azure technol...