Microsoft Fabric is getting closer to general availability and while there are already many features available, not everything is supported yet. This blog post focuses on how to handle authentication within Fabric Notebooks using secrets without storing them in your code. Refresh your knowledge on why we should not store secrets in clear text and discover a secure way to use Key Vault in combination with Fabric Notebooks. By the end of this blog post, you know why and how to access secrets from Key Vault. If you are only interested in the code, jump to the solution here.
The Dangers of Clear Text Secrets in Your Code
In today’s software development landscape, security is paramount. Storing secrets like API keys, database credentials, or access tokens in clear text within your code is a risky practice that should be avoided at all costs. Consider these reasons why you should avoid hardcoding secrets in your code:
Security Risks
Perhaps the most compelling reason is the inherent security risks. When secrets are hardcoded into your source code, they become readily accessible to anyone with access to the codebase. This includes not only your development team but also potential malicious actors who may gain unauthorized access.
Clear text secrets in code are comparable to leaving your front door wide open and hoping that no one will walk in. If an attacker gains access to your source code, they can easily locate and misuse these secrets to compromise your systems and data. The possible consequences are data breaches, financial losses and damage to your organization’s reputation.
Lack of Control
Another significant drawback is the lack of control. Over time, as your application evolves and the team working on it expands, managing secrets becomes increasingly complex. When secrets are hardcoded, making changes or revoking access can become an impossible task.
Consider a scenario where you need to rotate an API key or update a database password due to a security incident or routine maintenance. If these secrets are scattered throughout your codebase, tracking them all down and ensuring that the changes are properly propagated can be error-prone and time-consuming. This lack of control can lead to potential security lapses and operational headaches.
What is Azure Key Vault
Azure Key Vault serves as a cloud-based solution to store and access secrets securely. In this context, secrets can be a wide array of data which you want to keep save. This can include API keys, passwords, certificates, or cryptographic keys. By integrating Azure Key Vault with your applications, you can store secrets in a centralized and highly secure repository, separated from your source code. This approach not only mitigates the security risks associated with clear text secrets but also streamlines the management of sensitive information.
Storing the secrets in Key Vault is a very good solution but you have to make sure your process is able to access the secrets. In existing Azure ELT Tools like Data Factory or Synapse Workbooks, there is the option to authenticate to Key Vault via linked services. Currently, this option is not available in Fabric and there needs to be another way.
The Issue with Fabric Notebooks
In Azure, we prefer to use managed identities for authentication. Managed identities provide a secure and streamlined way for applications and services to authenticate and access Azure resources without the need to store or manage credentials. But currently, there is no option to use a managed identity with the Fabric Notebooks. On the Fabric Roadmap, Data source connectivity using workspace identity is estimated to release in Q1 2024.
“Fabric will support workspace identity to help organizations avoid downtime due to credential expiration and to improve their security for authentication in connectors. You’ll also be able to configure workspace identity as the authentication method for data sources like Azure SQL DB, Azure Data Lake Storage Gen2 and others, which support AAD authentication. Power BI role-based access control will prevent unapproved use of the workspace identity.”
Until then, we need to have a solution to use app credentials but in a secure way. Hardcoding them is not an option and the way to go is to use Key Vault. But how to access Key Vault without a managed identity?
Solution: Implementation
To access Key Vault from a Fabric Notebook, use the PyTridentTokenLibrary to handle the call while using an access token. Specify the name of the secret in Key Vault and provide the URL of the Vault. If you don’t have a Key Vault, you need to create one in your Azure tenant. Double check that the current user has the permissions to read the secret in your Vault. Otherwise the retrieval will not work. Simply being the creator of the Key Vault won’t be enough.
Code
# Import wrapper
from trident_token_library_wrapper import PyTridentTokenLibrary
# Generate access token in current context
access_token = mssparkutils.credentials.getToken("keyvault")
# Set the variables for the Key Vault
secretName = "fabric-blog-dummy-password"
keyVaultURL = "https://dev-kv-blog.vault.azure.net/"
# Retrieve secret
pswrd = PyTridentTokenLibrary.get_secret_with_token(keyVaultURL,secretName,access_token)After retrieving the secret, printing out the variable does not yield the password itself but a string showing [REDACTED]. This is because Fabric protects the secret from leaking via your console output.
Code
# Try to print the retrieved string
print(f"The retrieved password is: {pswrd}")Despite the printout not showing the actual secret text, the value is stored internally. The secret can be used for example to call an API endpoint.
Code
# Demo API call with password
import json
import requests
api_url ="https://dummyjson.com/auth/login"
body = json.dumps({"username": "kminchelle", "password": pswrd }) # 0lelplR
headers = {"Content-Type":"application/json"}
response = requests.post(api_url, data=body, headers=headers)
print(response)
print(response.content)This solution is not perfect and we are eagerly waiting for the update to use workspace identities. But until then, at least there is an option to prevent cleartext passwords in the code.
More about Fabric:
2 responses to “How-To Access Azure Key Vault Secrets from Fabric Notebook”
Will the provided solution work when try to run notebook from pipeline? Under which user identity pipeline is run?
Hi Mykhailo
Yes you can run this notebook from a pipeline. I did some quick testing for you regarding the used identity (see below). What i can say is: its best to have one user owning the notebook, the pipeline and also triggering/scheduling the pipeline run with the same user. Also regularly login with the user to ensure the token can be generated.
Tests
User A: has Key Vault Access
User B: has no Key Vault Access
| Owner Notebook | Owner Pipeline | Pipeline Run by | Success |
| User A ————– | User A ———– | User A ———– | Yes |
| User A ————– | User B ———– | User A ———– | Yes |
| User A ————– | User B ———– | User B ———– | Yes |
| User B ————– | User A ———– | User A ———– | Yes |
| User B ————– | User B ———– | User A ———– | No |
| User B ————– | User B ———– | User B ———– | No |