How to run Bash Script file in Airflow

In our previous article, we talked about what is Airflow and its installation on ubuntu.

We will learn about airflow BashOperator. And then we will understand airflow BashOperator with easy example.

So let’s get started:

The Airflow BashOperator is a basic operator in Apache Airflow that allows you to execute a Bash command or shell script within an Airflow DAG.

The BashOperator is very simple and can run various shell commands, scripts, and other commands.

The Bashoperator in airflow can be imported by typing the below command:

from airflow.operators.bash import BashOperator

The BashOperator class from the airflow.operators.bash_operator module, which is part of the airflow core package. You can create an instance of BashOperator and use it in your DAG once you have imported the operator.

Airflow bash operator run shell script:
Below is a simple example of airflow BashOperator.
A user can also run the shell script using the airflow bashOperator.

Let’s take the below example:

from airflow import DAG

from airflow.operators.bash import BashOperator

from datetime import datetime



default_args = {

    'owner': 'airflow',

    'depends_on_past': False,

    'start_date': datetime(2023, 7, 17),

    'retries': 0,

}



test_dag = DAG(

    'bashscript',

    default_args=default_args,

    schedule_interval="@weekly"

)



# Define the BashOperator task

bash_task = BashOperator(

    task_id='bash_task_execute_script',

    bash_command='./bash_script.sh',

    dag=test_dag

)



# Set task dependencies

bash_task

Create a file called bash_script.sh and paste the below line:

#! /bin/bash
# print hello BashOperator message
echo "Hello BashOperator"

We created the sh file in the dag directory.

Now let’s start writing our python code:

The above code is a simple DAG definition using Airflow’s BashOperator to execute a bash command.

The DAG is named “bashscript” and is scheduled to start on July 17th, 2023. schedule can be optionally made daily or weekly.
The DAG has only one task, which is the “bash_task_execute_scriptv”.
The task executes a bash command using the BashOperator. The bash command simply echoes the string “Hello BashOperator” to the console.

Now deploy the DAG and observe the output:
The task within the DAG is a BashOperator that executes a shell script named “bash_script.sh” using the bash_command parameter. The task is named “bash_task_execute_script”.

We trigger our dag and we see successful completion.

We go to the task log:

Let’s go to graph view in airflow and check the logs for output:
We can observe that it prints “Hello BashOperator” in the task log.

Overall, the BashOperator is a powerful operator in Airflow that allows you to execute bash commands or shell scripts within your DAGs.

The BashOperator is very easy to use, and you can pass any bash command or script using the bash_command parameter. You can also set the task dependencies in a DAG to determine the order of execution of the tasks.
The BashOperator is a valuable tool in Airflow for automating a wide range of tasks in a DAG, and it’s a great way to integrate shell scripts into your workflow.

What’s Next?

How to use the Postgresql in the Airflow