Python , pilvipalvelut ja lohkoketjut

When working with engineering customers, a typical ETL scenario is reading in some periodically incoming data files that are in a CSV or Excel format, and running them through an ETL pipeline. The data might be manually edited by a customer at some point, or it might come automatically from some other source. While having the data streamed as something else than static files would probably be better, you often don’t get to choose the method of delivery. So you need a way to efficiently load in a bunch of files, extract the data in them, do some transformations and finally store the end result to a database.

This week we participated in a one day AWS Immersion Day - Data Foundation with Apache Iceberg on AWS. Here’s a summary of our learnings from the event and some thoughts on how Apache Iceberg could be utilized in our data-oriented business cases.