Skip to content

Airflow Deployment

AirflowRunner is a utility for managing and orchestrating Airflow DAGs. It is designed to provide a command-line interface (CLI) for creating, describing, showing, deleting, and getting the status of Airflow DAGs.

This class uses the Airflow models to interact with DAGs and DockerOperator to run tasks in Docker containers. It is aimed to simplify the deployment and management of Airflow tasks, providing a straightforward way to deploy DAGs with Docker tasks from the command line.

CLI Usage

genius airflow sub-command

Sub-commands
  • create: Create a new DAG with the given parameters and Docker task. genius airflow create [options]
  • describe: Describe a specific DAG by its ID. genius airflow describe --dag_id example_dag
  • show: Show all available DAGs in the Airflow environment. genius airflow show
  • delete: Delete a specific DAG by its ID. genius airflow delete --dag_id example_dag
  • status: Get the status of a specific DAG by its ID. genius airflow status --dag_id example_dag --airflow_api_base_url http://localhost:8080/api/v1

Each sub-command supports various options to specify the details of the DAG or the Docker task, such as the schedule interval, start date, owner, image, command, and more.

Example

Creating a new DAG: 

genius airflow create --dag_directory ~/airflow/dags \
                      --dag_id my_dag \
                      --image python:3.10-slim \
                      --command "echo Hello World"

Attributes:

Name Type Description
dag_directory str

Directory where DAGs are stored. This path should be known to Airflow.
Methods
  • create: Method to create a new DAG based on the provided parameters and template.
  • describe: Method to describe a specific DAG by its ID, showing details like tasks and schedule.
  • show: Method to list all available DAGs.
  • delete: Method to remove a specific DAG by its ID from the directory.
  • status: Method to fetch and display the status of a specific DAG using Airflow's REST API.
Note
  • Ensure that the Airflow environment is properly configured and the specified DAG directory is correct.
  • Make sure that the Airflow REST API base URL is accessible if using the status command.

__init__()

Initialize the AirflowRunner class for managing Airflow DAGs.

create(args)

Create a new DAG with a Docker task using the provided arguments.

Parameters:

Name Type Description Default
args Namespace

Namespace containing all the arguments needed for creating the DAG.

 required

delete(dag_id)

Delete a specific DAG by removing its file from the DAG directory.

Parameters:

Name Type Description Default
dag_id str

The ID of the DAG to delete.

 required

describe(dag_id)

Describe the details of a specific DAG.

Parameters:

Name Type Description Default
dag_id str

The ID of the DAG to describe.

 required

Returns:

Type Description
None

The DAG object if found, None otherwise.

run(args)

Execute the command based on the parsed arguments.

show()

Show all available DAGs by listing their IDs.

status(dag_id, airflow_api_base_url)

Get the status of a specific DAG using Airflow's REST API.

Parameters:

Name Type Description Default
dag_id str

The ID of the DAG to get the status of.

 required
airflow_api_base_url str

URL of airflow for calling its APIs.

 required