Azure Databricks Lakehouse Monitoring queries
Hi Team, I was exploring on Azure Databricks Lakehouse monitoring. I have few queries on this: When I am running a "refresh metrics" irrespective of an automated schedule or manual refresh, which compute does it run? There is no mechanism to…
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
CSV to XML conversion in databricks which have some blank values as well in csv
I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…
Reading data from Sharepoint Using Databricks
I am trying to read the SharePoint data from Databricks. But, not able to access it. Could you please let me know if it is possible to read the SharePoint data using Databricks. If possible then could you please share some code or Link where i can find…
Unable to downgrade Databricks workspace
When downgrading the Databricks Workspace, I receive the following message; However, none of the Enhanced Security options are currently enabled; Could you please help me identify the cause of the error?
Special character handling for file processing
Hello, I have some CSV files as feeds into datalake storage, this file will contain data /records with some special characters eg '/' We need to process the file from one container to another and we will need to remove some of the these special…
Is dynamic SQL Queries supported on Azur Databricks SQL Cluster?
Hello, I'm planning to implement Dynamic SQL function to query data on Databricks table. Tables and access for the users are governed by a custom access matrix using the Unity catalog. The problem is that in a custom matrix, there are two types of users:…
Error accessing Azure sql from Azure databricks using jdbc authentication=ActiveDirectoryInteractive
Getting below error while accessing Azure sql using jdbc from Azure databricks notebook, com.microsoft.sqlserver.jdbc.SQLServerException: Failed to authenticate the user p***** in Active Directory (Authentication=ActiveDirectoryInteractive). Unable to…
Azure Databricks test cases and Git commit
Hi friends, we have testing our test cases in a testing environment, and these are many tests, and want to test them as per test uses- cases and before committing into Git. Since there are two many of them I do not want to do it manually for each…
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
Clusters are failing to launch. Cluster launch will be retried.
Hi, I am a newbie. Can someone show me how I can fix the below please? Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores…
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
How to change databricks location
i want to change the existing location(region) for databricks and along with other apps as well
How do I add an inbound security rule if there is an default DenyAllInbound Rule that causes an error when attempting to create an inbound rule?
|Received an email with: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. To support infrastructure …
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Indexing a Pyspark dataframe
Hey guys, I am having a very large dataset as multiple parquets (like around 20,000 small files) which I am reading into a pyspark dataframe. I want to add an index column in this dataframe and then do some data profiling and data quality check…
How do I figure out what public IP ranges my Databricks workspace clusters are coming from?
Relatively new to Databricks. I have an existing workspace that was created years ago. It is vnet-injected but it has secured cluster connectivity (SCC) disabled. I need to know the outbound IP addresses/ranges the clusters would communicate on to…
[Databricks] Clusters are failing to launch. Cluster launch will be retried.
Hi all, I am a complete newbie on Databricks Azure. I have encounterd the below issue which I think is stopping me from running query. Any help will be much appreciated. Thanks. Billy Clusters are failing to launch. Cluster launch will be…