Skip to main content
This guide will show you how to securely connect SYNQ to your Airflow setup.We need this information so we can extract metadata about Airflow. By default, all tasks and all DAGs will be reported to us.To be able to finish this guide, you’ll need the following:
→ Access to modify your Airflow configuration code⏱️ Estimated time to finish: 10 minutes.

​
Setup with OpenLineage

  1. In the SYNQ app, create a new Airflow integration. Make sure that you select OpenLineage as the processing mode.
  2. Once the integration has been created, use the generated SYNQ token to configure Airflow to securely forward your events to our ingestion service.
Airflow uses a transport configuration option to define how to forward these events.
  • URL: https://developer.synq.io
  • Endpoint: /api/ingest/openlineage/v1
  • API Key: Generated by the Airflow integration.
With that information, you can follow the Airflow OpenLineage documentation to configure your Airflow instance.

​
Example OpenLineage Transport Value

{
  "type": "http",
  "url": "https://developer.synq.io",
  "endpoint": "/api/ingest/openlineage/v1",
  "auth": {
    "type": "api_key",
    "api_key": "st-***" // redacted
  }
}

​
Setup with DataHub

  1. Install the required dependencies in your Airflow
pip install 'acryl-datahub-airflow-plugin>=1.1.0.4' apache-airflow-providers-openlineage
For detailed installation instructions and compatibility information, see the DataHub Airflow documentation.
  1. Setup the REST hook
  • conn-host: https://datahubapi.synq.io/datahub/v1/
  • conn-password: Token from SYNQ you obtain when you click ‘Create’ on this page
airflow connections add  --conn-type 'datahub_rest' 'datahub_rest_default' --conn-host '<host from SYNQ>' --conn-password '<token from SYNQ>'

​
Airflow 2.7+ with OpenLineage Provider

If you’re using Airflow 2.7+, the native Airflow OpenLineage provider will improve the quality of lineage and metadata information obtained from your Airflow setup. The OpenLineage provider package is already included in the installation above since the DataHub plugin requires it. For AWS MWAA, add both acryl-datahub-airflow-plugin and apache-airflow-providers-openlineage to your requirements.txt file.

​
Log Forwarding

Log forwarding is required to include full task failure logs in SYNQ alerts. SYNQ supports multiple methods for forwarding Airflow logs:

​
AWS MWAA CloudWatch Logs

For AWS Managed Workflows for Apache Airflow (MWAA), you can forward logs from CloudWatch using the synq-aws-cloudwatch Lambda function:
  1. Deploy the Lambda function: Use the synq-aws-cloudwatch repository to deploy a Lambda function that forwards CloudWatch logs to SYNQ.
  2. Configure log forwarding: The Lambda automatically forwards Airflow logs from CloudWatch to SYNQ when properly configured with your SYNQ credentials.
  3. Set up log group subscription: Configure CloudWatch to trigger the Lambda when new Airflow logs are available.

​
API Endpoint

You can also send logs directly to SYNQ using our API endpoint. This endpoint accepts Airflow log data and forwards it to the SYNQ platform for processing and analysis.

​
Remote Logging (S3/GCS)

SYNQ supports uploading logs to SYNQ after files are created on S3 or GCS using Airflow’s Remote Logging feature. For assistance with this setup, please reach out to SYNQ support.

​
Read more

For more information about the plugins SYNQ uses to collect relevant task execution and lineage data, we provide the following links:
⌘I