Implementing Azure Data Factory Nodes

Recently I was working with a client to troubleshoot some Azure Data Factory connection issues with an on-premise runtime when connected to Dynamics365 Linked Service.

Part of that implementation required setting up the runtime as an on-premise implementation. The primary difference between an on-premise and azure runtime is that the on-premise is deployed as a VM in your Azure network which ensures data and signaling never leaves your network.

Setting this up as a single node implementation wasn’t too complicated and you can get away following the prescibed guidance, however things got a little dicey when we moved to pool our nodes together.

When deployed as nodes, you are configuring the system to work in a pool where each pool can be limited to concurrent nodes to maximize productivity

A few gotchas we ran into going through this process;

1. Your VNET matters – don’t separate your nodes across different VNETS, keep them on the same VNET, otherwise, your nodes won’t align.
2. Timezone – ensure your timezones are the same, otherwise you’ll have the same issues.
3. Ensure Auto-Update is turned on – we have yet to hit this issue but have heard from others where their nodes get out of sync and don’t function properly. We had no issue whether using the prescribed download on the configuration screen or the one from Microsoft.com

Once setup, you never have to check again (unless there is an issue).

If you don’t need a self-hosted (on-premise) runtime, go with the azure runtime the difference is minutes vs hours and some troubleshooting time mixed in for good luck.

If you have any questions on azure data factory, azure or dynamics 365, Ask us.