Articles
Thoughts, tutorials, and deep dives on software engineering and technology.
Demystifying Table Formats: Hive vs. Iceberg vs. Hudi
Understand the differences between file formats (Parquet, ORC) and table formats, how traditional Hive falls short on modern cloud storage, and how Apache Iceberg and Apache Hudi bring ACID transactions, partition evolution, and lakehouse capabilities to data lakes.
Building Scalable REST APIs with Go
A practical guide to designing and building production-ready REST APIs using Go, covering project structure, middleware patterns, and database integration.
Comparing various storage formats for object storage compatibility
We have been using parquet storage format for storing our petabyte scale tables in Google Cloud Storage using Hive as our meta store. We encountered s...
Spark Jobs and Table Storage
Performance of a spark/distributed application depends on how the input tables are stored. Let’s analyze how a spark application’s performance is depe...
Key Considerations While designing ETL Jobs
A typical data processing job in airflow has the following components...
Learnings from implementing Hive on Google Cloud Storage
Cloud Storage is the managed object store offering from google cloud. In our organization we are making use of cloud storage as the backend for storin...