How to create auto-trigger in Azure Data Factory

I am beginning a series of concise technical narratives on the straightforward utilization of Azure cloud services, accompanied by relevant use cases. Today, I will demonstrate how to configure an automatic trigger for a monitoring storage account that, upon detecting new arrivals, will initiate our pipeline in Azure Data Factory. We must start by setting up a few essential services.

• Azure Data Factory
• Azure Storage Account

The next step involves executing our pipeline only if something arrives in the designated blob container. Azure Data Factory supports three triggering methods:

1. By scheduler
2. By blob storage changes
3. By event grid using custom events

Let’s choose blob storage changes. This trigger relies on an event system grid topic. Additionally, Azure Data Factory supports only two types of events: blob created and blob deleted.

Additionally, we can include specific expressions to trigger the process only when a particular file arrives. In our case, we specify files with the extension ‘.tif’.

The next task involves reading metadata from this file and passing it to the subsequent stage. For this activity, we need to include the Get Metadata activity in our pipeline.

However, before inserting this activity, we need to create two input parameters to be used as input from the trigger. Our trigger should pass these parameters when it starts.

Then, we need to map it with a trigger output, like @trigger().outputs.body.fileName and @trigger().outputs.body.folderPath

Let’s create activity ‘Get Metadata’

This activity requires Dataset to read the data.

In a dataset, it is essential to map parameters to pass data values from the trigger and to select the field properties required from the metadata. By doing so, we ensure that the necessary information is accurately extracted and utilized. This meticulous mapping and selection process is crucial for maintaining the integrity and usability of the dataset in various applications.

That is it. We need to validate all changes and publish.

As a result, you can see after uploading files with .tif extensions.

As input, you can

{
    "dataset": {
        "referenceName": "FileDataSet",
        "type": "DatasetReference",
        "parameters": {
            "filename": "4.tif",
            "filepath": "provider1"
        }
    },
    "fieldList": [
        "itemName",
        "lastModified",
        "contentMD5"
    ],
    "storeSettings": {
        "type": "AzureBlobStorageReadSettings",
        "enablePartitionDiscovery": false
    },
    "formatSettings": {
        "type": "BinaryReadSettings"
    }
}

As output,

{
 "itemName": "4.tif",
 "lastModified": "2025-02-04T23:05:54Z",
 "contentMD5": "syu6QF5LAgiTSX9Y0tANUw==",
 "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Central US)",
 "executionDuration": 0,
 "durationInQueue": {
  "integrationRuntimeQueue": 0
 },
 "billingReference": {
  "activityType": "PipelineActivity",
  "billableDuration": [
   {
    "meterType": "AzureIR",
    "duration": 0.016666666666666666,
    "unit": "Hours"
   }
  ]
 }
}

In the upcoming story, I will incorporate table storage to transmit the status of the pipeline execution.

Happy coding.