Python , pilvipalvelut ja lohkoketjut

Hetzner Cloud is an EU-based relatively cheap cloud computing platform, at least when comparing to something like AWS. They offer a simple set of services required in everyday small-scale projects: servers, volumes, simple load balancing features and an S3-compatible object storage. But how does it fare from an automation point of view?

When working with engineering customers, a typical ETL scenario is reading in some periodically incoming data files that are in a CSV or Excel format, and running them through an ETL pipeline. The data might be manually edited by a customer at some point, or it might come automatically from some other source. While having the data streamed as something else than static files would probably be better, you often don’t get to choose the method of delivery. So you need a way to efficiently load in a bunch of files, extract the data in them, do some transformations and finally store the end result to a database.

This week we participated in a one day AWS Immersion Day - Data Foundation with Apache Iceberg on AWS. Here’s a summary of our learnings from the event and some thoughts on how Apache Iceberg could be utilized in our data-oriented business cases.