filter and lookup transformation
Hi, have 2 tables table1 .jobid 1 table2 tjob2.jobid1 and jobname='test' how to filter jobid1 where jobname<>'test'
Salesforce V2 Link Service doens't work. error message: a task was canceled Source=mscorlib
I am migrating the ADF salesforce connector to the new V2 connector and now testing it. When testing the connection for the link service, it is good. However, it failed in the copy activity. with message: Failure happened on 'Source' side.…
How do I call an API that redirects on the backend with a Copy Activity in Azure Data Factory without an authentication error?
I'm running into issues in Azure Data Factory (ADF) with an API I am working with. I am unfortunately not at liberty to share the API URL or much data on the call I am making, but I can say that I have been testing this API call in Postman and through…
ADF Pipeline runs successfully but data doesn't show up
Hi team Using Azure Data Factory pipeline using two web activity Source web is reading data from log analytics workspace w1 table t1 and I am passing the result of source web activity as request body of destination web activity then my destination web…
ADF unable to call internal API via web activity with self-host Integration Runtime
Hi, I am working on a project with Azure Data Factory. We have setup an internal application and provided a API for updating record, and we would like to trigger a POST call by web activity in ADF (with Self host IR), but failed with the follow…
Connect SharePoint Online List with Synapse Integrated Runtime Managed Virtual Network
Getting an error when attempting to create a Linked Service for SharePoint Online List in Synapse. I've followed the instructions and prerequisites on MS documentation, but getting the following error when testing the connection: An error occurred while…
Using cached data from one data flow activity as the source for another data flow activity
On this reference page... https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink ...MS says, "Write to activity output The cached sink can optionally write your output data to the input of the next pipeline activity. This will allow you…
How to store an array in a json file in ADF
I have a web activity with url drive/drive-id/items/folder-id/children then I have a filter activity that filters all folders and then I'm appending it in an array. Now what I want is to store all folder ids in a json file. and later I will load that…
Data factory Authentication With Azure SQL DB + Managed Identities + Federated Credential
I am facing ADF Auth issues with Azure SQL DB. ADF is in torus tenant and Azure SQL DB is in Microsoft tenant. Below are the steps i have followed to up FCI. Created User Assigned managed identity in Torus tenant. Created App registration in Torus…
Copy activity throwing an error, when a auto incremented column is there on the Sink side
Hello Team, I am trying to copy data from Amazon S3 to Azure SQL database. On Azure SQL side I have set an autoincrement column(Row_Index) in a table, however the copy activity throws an error that "Column 'Row_Index' does not allow…
How can I copy all files from sharepoint (nested folder structure) to ADLS container by using Azure data factory
I need help in building an iterative approach. Let's say we have used drives/drive-id/root/children and got initial level folders and file, files I will simply copy and if there is odata.nextLink that I can easily handle. Now for folders I will use for…
Azure Data Flow Sink - Connection string is not valid or missing Server/User/Database properties
Could you please assist here to resolve this issue below ? We are trying to read Delta table and store it as table in Azure SQL Database as using ADF Data Flow. Source Delta table via inline dataset from ADLS Gen2 its working fine and slink is getting…
CDC Start button disabled. Cannot Start CDC in Azure data factory
I created a CDC in adf and made a Git configuration to connect to Azure Devops Repos with a new branch created. Now, I checked out to a new branch. In the new branch, CDC is also there but i cannot start the CDC . It says Please publish to enable the…
how to retrieve the path to workspace_publish branch OR any branch in devops , use that path in azure powershell script to read file from that branch.
I want to retrieve path to the branch and folder inside it, basically i wan to read the jsons here and change the dev keyword with uat and save the file back to the sam folder in workspace_publish branch. I can change dev to uat, but am not able to…
ADF - CDC - enable and disable CDC tables during deployment
Hi, I have setup ADF pipeline for data sync of tables between Azure SQL Database to SQL Managed Instance databases. I am using Data Flow and CDC to track changes and sink activity to replicate it to target. However, when we deploy any table changes (add…
Error: 11402 -The value of the property '' is invalid: 'The remote name could not be resolved:
I am setting up a Linked service for ODATA in ADF, for IntegrationRuntime I am selecting "AutoResolveIntegrationRuntime" but connection to ODATA resource is failing: The value of the property '' is invalid: 'The remote name could not be…
REST API Connection through ADF with Auth Error
I am working on adding a data source through a rest api in a data factory data flow. The api requires an api key, which I have. I've configured the system with the base url, auth header, and with an anonymous auth type. When testing the connection in the…
Copy Data Activity failes on datatype uniqueidentifier when Staging is enabled and no column mapping configured.
Using Azure Data Factory Copy Data Activity from Azure SQL Database to Azure Synapse Analytics. The Copy Data Activity failed with InvalidCastException error 'The given value of type String from the data source cannot be converted to type…
How can I prevent Azure Data Factory (Web Activity) from handling Redirects?
We have a requirement to perform a series of callouts to get the report and we have built the pipeline and used the web activity to make the callouts. In that process, we’re getting the below error while making a get request using the web activity. …
In Azure Data Factory, is it possible to define link service and dataset with dynamic type?
In Data Factory, is it possible to define link service and dataset with dynamic type? Since the legacy Salesforce connector will soon be deprecated, I want to create a single link service and dataset to support both the legacy and new connector instead…