How to update Azure Table Storage from Data Factory

Hey folks, here is a useful scenario for using Azure Data Factory. We need to update the table with the statuses of pipeline processing. The scenario is straightforward: when the pipeline starts, the table should be updated with a predefined status.

We encountered two challenges:

1. Identifying the right Azure Data Factory activity to use
2. Generating the correct request to update Azure Table

Regarding the Azure Data Factory activity, we have two options:

- Webhook
- Web

The main difference is that a webhook is needed when you have a callback URL that needs to be called when the webhook executes. In our case, we only need to call the REST API to update Azure Table, so the Web activity will be suitable.

For using the REST API, we must have an authorization method to execute our request. Based on official information we have different options:

We use SAS to pass our POST request https://myaccount.table.core.windows.net/mytable

    {   
    "PartitionKey": "{{$guid}}",  
    "RowKey":"myrowkey",
    "Status":"Processing"  
    }  

A partition key is required and we generate unique guid using Azure Data Factory function guid()

As a result, we have Web activity to create new entity inside Azure Table

{
    "name": "Update Azure Table",
    "type": "WebActivity",
    "dependsOn": [
        {
            "activity": "Get FileShare Data",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    "policy": {
        "timeout": "0.12:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": false,
        "secureInput": false
    },
    "userProperties": [],
    "typeProperties": {
        "method": "POST",
        "headers": {
            "Accept": "application/json;odata=nometadata",
            "Content-Type": "application/json"
        },
        "url": "https://datasourcestest.table.core.windows.net/{table}?sv={SAS token}",
        "body": {
            "value": "{   \n   \"PartitionKey\": \"@{guid()}\",  \n   \"RowKey\":\"myrowkey\",\n   \"Status\":\"Processing\"  \n}",
            "type": "Expression"
        }
    }
}

That is it, see you next time, happy coding.