Admittedly this blog is a little late to the party. A few months ago, everybody jumped on the hype train surrounding Microsoft’s latest innovation, Microsoft Fabric. But perhaps, in this case, tardiness is a virtue, since it has allowed me the time to look beyond the initial wave of excitement and truly understand what makes Fabric a game-changer for Power BI developers. In my opinion it is not just another toolkit or a marketing gag, it’s a transformative addition to the Power BI ecosystem. If you use Power BI to develop reports on an enterprise level, Microsoft Fabric is set to bring meaningful changes your way. While Microsoft Farbric is transformative in many ways around the whole ETL process, this blog will focus on its specific implications for Power BI developers.
1. Direct Lake query mode
The introduction of Direct Lake query mode within the Microsoft Fabric ecosystem brings substantial advantages for Power BI Developers. To fully grasp its significance, let’s consider the challenges associated with the traditional query modes, Direct Query and Import Mode.
Direct Query has its merits, particularly in terms of data security and Continuous Integration and Continuous Delivery (CI/CD). It enables the management of Row-Level Security (RLS) at the source, which is huge benefit for data governance and safes efforts because there is no necessity to implement RLS rules for each dataset. Plus, it doesn’t embed data within PBIX files, which aligns well with CI/CD workflows. This is especially important in regions like Switzerland, where strict data regulations require the permanent deletion of customer data on demand. A task made impossible if data remains recoverable within Git repositories due to its attachment to PBIX files. However, as datasets grow, Direct Query’s performance can decline significantly, impacting the user experience drastically.
In contrast, Import Mode distinguishes itself by its exceptional query performance, achieved through the use of a caching layer in Power BI Service that applies high levels of compression and optimization for querying data, resulting in efficient data retrieval and processing. However, one of its notable drawbacks is that it embeds the dataset within the PBIX file, which as already mentioned presents challenges for Git integration. Additionally, Import Mode requires the redundant definition of Row-Level Security policies for each individual dataset, adding complexity to data governance efforts.
Direct Lake Query represents a valuable compromise that bridges these gaps. It retains the advantages of Direct Query, allowing for RLS management at the source and keeping data out of the PBIX file, thus addressing the concerns related to CI/CD. Direct Lake provides these benefits while offering almost the same query performance as Import Mode. This performance enhancement is achieved by leveraging the optimized delta Parquet file format within the OneLake ecosystem. With Direct Lake, Power BI Developers can implement efficient data security and Git integration, all without compromising query performance.
2. OneSecurity
Expanding on the concept that Direct Lake enables you to handle Row-Level Security at the data source and not in each dataset, it’s important to note that your data source can be OneLake, the central Delta Lake of Microsoft Fabric. OneSecurity is expected to be a crucial component of it, allowing organizations to define access controls and RLS/OLS policies uniformly across their data assets. This improves consistency of security policies while reducing efforts for data governance. This uniform security policy goes beyond Power BI. In a data ecosystem where data products are not only consumed by Power BI but also by other applications like machine learning models, having a standardized security approach is a significant benefit. It streamlines data governance, enhances overall data security, and ensures consistent measures.
It’s important to mention that OneSecurity, has been announced but specific details are yet to be disclosed. For large enterprises, the functionality of OneSecurity will be pivotal, as its capabilities will play a central role in shaping the success and reliability of data governance solutions within Microsoft Fabric. So although there are no details yet it can be assumed that Microsoft is investing heavily in its functionality, given that a bad implementation will make Fabric as a whole unusable for a large share of their target market.
3. Deployment Pipelines
Discussing the practicality of utilizing Microsoft Fabric within large enterprise setting brings to light a pressing concern: the prospect of a functional deployment pipeline. Historically, the implementation of such pipelines has fallen short. Even for moderately complex project scenarios the functionality was not sufficient, necessitating the creation of custom deployment pipelines using the Power BI API.
The Microsoft Fabric web application builds upon the foundational structure of the existing Power BI service. This expansion introduces exciting possibilities, such as Dataflows Gen2 and Lakehouses, as novel types of artifacts that can seamlessly integrate with the Power BI Service. While Microsoft always found a way around admitting it, it remains evident that Power BI lacks fundamental support for essential CI/CD functionalities. Consequently, the viability of these new artifact types becomes questionable for numerous organizations unless a robust solution for deployment pipelines is established. Now that there is enough pressure for a suffisticated implementation I am certain we will see a functioning solution rather sooner than later.
4. Developer Mode & Git Integration
In my opinion Power BI Developer Mode is an essential feature that enhances the development and collaboration capabilities within the Power BI ecosystem in large enterprise scenarios. Although it may not have a direct connection to Microsoft Fabrics, it is worth noting that it coincides with the General Availability release of Microsoft Fabrics, further empowering developers. By enabling the saving of files as .PBIB, Power BI allows users to store dataset and report metadata as json files, thereby opening up a range of functionalities. Developers can leverage source control systems like Git to track version history, perform comparisons between different revisions, and easily revert to previous versions. This integration with source control facilitates co-development and collaboration, allowing teams to resolve conflicting changes and review alterations through pull requests. Furthermore, Power BI Developer Mode supports Continuous Integration and Continuous Delivery (CI/CD) workflows, where changes can be subjected to quality gates like code reviews and automated testing before reaching production. With the Git integration of Power BI in Azure DevOps all these features become even more accessible. Additionally, it enables programmatic generation and editing of artefact definitions, enabling developers to write scripts or applications based on public documentation or Microsoft-provided libraries to modify various artefacts within Power BI.
5. Data Activator
The introduction of “Data Activator” in Microsoft Fabrics represents a significant leap in transforming data insights into actionable outcomes. While Power BI excels at gaining and delivering insights, the ultimate goal is to create actionable insights, and Data Activator effectively bridges this gap. This feature operates through a streamlined 3-step process: first, it connects to a wide array of data sources within Microsoft Fabric, including Power BI datasets and eventstreams, continuously monitoring them for actionable patterns. Second, Data Activator empowers users to define these actionable patterns in a single, centralized location, offering flexibility ranging from simple threshold triggers to more intricate patterns. One can even use their own trained machine learning models to find triggerable patterns. Finally, when Data Activator detects these a suiting pattern, it triggers an action. These actions can take various forms, such as sending email notifications or Teams alerts to the relevant personnel within the organization. Furthermore, Data Activator can seamlessly integrate with workflow tools such as Power Automate to trigger automated processes or initiate actions within the organization’s line-of-business applications.
Conclusion
In summary, Microsoft Fabric presents a promising landscape for data management and analytics, with its Direct Lake Query mode striking a balance between performance and data governance. While awaiting specifics, OneSecurity holds the potential to standardize security policies across data assets, benefiting large enterprises. The need for robust deployment pipelines remains, but with increasing demand, a solution is likely on the horizon.
Developer Mode and Git integration empower collaboration, aligning with Microsoft’s commitment to developers. Finally, Data Activator introduces a powerful tool for turning data insights into actionable outcomes, enhancing the value of Microsoft Fabric. As this platform continues to evolve, it has the potential to transform data-driven decision-making, despite ongoing challenges.
More about Fabric: