Key Considerations While designing ETL Jobs
Originally published on Medium ↗
A typical data processing job in airflow has the following components
- A high level airflow DAG.
- Tasks inside each DAG.
- A processing engine to execute each task. eg: Spark.
How do we decide what tasks are involved a DAG ? Why cant an airflow DAG contain just one task which will execute all the needed steps in a processing engine like spark as shown below ?
How do we decide if we should create a new task in airflow DAG or if we should do the processing in the processing engine ( spark/SQL ) itself?
How many data assets is it creating. It is best if each airflow task is creating only one data asset. If the job is creating more than one data asset it is best to divide them into two separate airflow tasks.
Volume of data: If the processing job is handling too much data. Then it is best to handle it in separate jobs. That way load per job comes down and even if one job fails the retry is handling much less data and will take us considerably less time to re-process and process.
Idempotency : When the task is retriggered multiple times is the effect Idempotent ?