Speaker
Abstract
Most organizations use databases to efficiently capture, store, organize and read day-to-day operational data such as customer and transaction details. However, databases are not the optimal solution for every data processing task. This session will highlight when data lakes can offer a strong alternative to traditional databases and data warehouses. Data lakes are now a standard part of successful organizations’ information systems, primarily because data lakes frequently offer greater flexibility than databases. Data has a life cycle. If data is first written into a file (such as computer log files), by using a data lake architecture, the data in the log files are immediately readable using SQL. Because many people first encounter SQL when using databases, they assume that SQL requires that the data be in a database. However, when using data lake architectures, SQL can be used to get immediate answers from data in collections of files. The data may be in its raw form, or transformed into file formats optimized for fastest data retrieval. Data lakes also simultaneously support analysis using artificial intelligence and machine learning too. For this session, we are going to use an open-source technology to read data from log files stored on Backblaze B2 Cloud Storage. The open-source tool we will highlight is Trino, which is a powerful SQL query engine. We will use the publicly available Backblaze Drive Stats data as our sample data set. Drive Stats is a public, open data set containing over 9 years of daily metrics, including drive failures, on all hard drives in Backblaze’s cloud storage infrastructure. Currently, Drive Stats comprises over 300 million records, consuming 90 GB of storage in CSV format, with over 200,000 records, or 75 MB of data, added every day. In the presentation, we will share the data engineering experience we gained in working with the Drive Stats data, as well as insights we were able to gain by being able to run analytical queries on the entire data set for the first time. Attendees will be provided access to the Backblaze Drive Stats data for their own use, publicly hosted on B2 Cloud Storage. Using endpoint connections that we will share with you, attendees will be able to get up and running their own test environments to get hands-on experience of the ease and power of data getting business insights from a data lake.
Topics
Sponsored session
QCon San Francisco 2022 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Sponsored Solutions Track IFrom the same track
Monday 24 October
11:50 Seacliff ABC Sponsored Development Is Web3 Here to Stay? Richard Bair VP of Software Engineering @Hedera You may be familiar with the current reputation of web3, but are you up to date on the advantages of distributed ledgers applied in the real world? Join us for a deeper dive into how companies are using this innovative technology today. 13:40 Pacific LM Sponsored DevOps Service Abstractions to Cloud Service Providers: A Tale of Trade-Offs Oscar Mullin VP of Technology @Mercado Libre 14:55 Pacific LM Sponsored Agile Building Agile Data Architectures in Support of Digital Twins and Data Products Stuart Sim Leader @Build by McKinsey Agile software development and elastic cloud foundations have enabled on-demand expansion of compute functions from real-time processing to Machine Learning at scale but Data has been left behind. 16:10 Pacific LM Sponsored Database Our Journey Into High Performance and Reliable Document Databases with RavenDB Trevor Hunter Chief Technology Officer @Kobo Inc. When I started at Kobo, we needed to look beyond the relational and into document databases. 17:25 Pacific DEKJ Sponsored Database Optimal Data Storage Choices - Data Lakes vs Databases Greg Hamer Data and Application Architecture Specialist @Backblaze Session Solutions Track Session 1 Details coming soon.