Learn how OneLake can bring your data and your people together
If you want to write a book or make a movie, you must first define a world in which to tell your story. Does your story take place on this planet, in a “galaxy far, far away”, or in some parallel multiverse? Next, you’ll need a collection of flawed heroes and nearly impossible problems to solve. With the right combination of scenarios and characters in crisis, you can keep an audience on the edge of their seat for the entire journey to a gritty and inspiring finish.
For those of you involved in the corporate world of business intelligence, you’re probably thinking, “That’s not a science-fiction thriller, that’s my day job!” Your company probably has multiple data systems that don’t talk to each other, managed by diverse departments with different directives, and charismatic characters who are either entrenched in technological traditions or craving the coolest cure-all platform. That makes for a new adventure every day.
Though there are many tools and platforms that promise to give your company a competitive advantage, this article will focus on the opportunities Microsoft’s Fabric and Power Platform provide to bring together your data and your people to solve some of your company’s key problems.
Fabric Foundation
It doesn’t take long in a company’s growth for the various departments to establish multiple systems of record.
- Finance can use an ERP system to track sales, operational expenses, and other key data needed for financial statements and analysis.
- HR can have their own system to manage employee records and job placement.
- Manufacturing can have an MES system for tracking routings and BOM’s so they can manage shop floor processes and plan capacity.
- IT needs to oversee the security and integrity of these systems, track tickets for problems, and integrate all this data together.
- Business users meanwhile will constantly leverage the tools available to them to do their day job (think Excel, Access, or SharePoint) and build out many business-critical processes that IT doesn’t have the capacity to support.
Though these systems may start independently, it won’t take long before these different departments need to borrow data from each other and thereby create the multi-verse of “truth” used to manage the business. This has been the inevitable challenge of growing companies for decades. Now as machine learning and large language models are beginning to open new insights into a company’s data, it’s even more important to have a solid data foundation to build on.
This “problem” is also the “opportunity” Microsoft is hoping to help solve, with Fabric OneLake, which provides a foundation for wrangling a company’s data into submission. With this platform, data can be stored in one place and curated into a single universe of truth based on the assigned roles of each person in the process.
Bronze
For starters, the data engineers have tools like Data Factory embedded in this platform to grab and integrate data from all kinds of data sources. This data is stored in a Bronze layer (structured or unstructured, batched or streaming). This provides a baseline of all the raw ingredients (unchanged) for future business intelligence.
Silver
Next, the raw data is copied and enriched in the Silver layer with your developer’s favorite tools (Spark, Python, and/or SQL). Here field names and data types are standardized, duplicates removed, and data validated against basic business rules. Likewise table keys are developed to support data connections, and dimension tables established to track changing values over time.
Gold
Finally, the Gold layer provides data warehouse tables and certified semantic models to support your data scientists and data analysts across all departments. Not only does this support Power BI visual analysis, but this also supports the Spark/Python notebooks that data scientist would leverage for machine learning and AI initiatives in the business.
Explore end-to-end analytics with Microsoft Fabric - Training | Microsoft Learn
Power Platform for Business Users
Power Platform low-code tools include:
- Microsoft Power BI for ad hoc analysis of curated enterprise and local data.
- Power Apps to create forms to collect and manage structured data on internal processes . As the platform grows, pro-code tools are also becoming more integrated into this platform to help IT teams participate in speeding up development pipelines.
- Power Pages allow the creation of simple, external facing web pages to support processes beyond a company's boundaries.
- Power Automate handles background application processes including approvals and business process flows.
- Copilot Studio enables teams to create their own custom chatbots and AI tools to leverage across your business
- Dataverse undergirds this whole platform. It provides data storage (which can be connected to Fabric) for custom applications and Dynamics 365. It also supports all the meta-data needed to support the rest of Platform’s tools just mentioned.
Data Migration Reality Check
As you consider the opportunities to migrate to Microsoft Fabric and Power Platform, remember to think past only the “happy path” of your data processes. The truth is that many processes in business can get built as band-aids to problems and were never intended to be long term fixes, and they often prove far more complicated than expected.
Lean Handles
Lean Thinking people often create Spaghetti Diagrams to uncover the crazy reality of the current situation before they start process improvement projects. In a manufacturing scenario this may show how parts move around the shop floor. In an administrative process this may represent how people interact with each other and the multi-faceted ways they send, copy/paste, transform, VLOOKUP(), or manually retype data for a project or report. In data processes, this can be the various data pipelines and transformations required to create a report.
As a developer, you’re now at a critical fork in the road. What do you do with everything you’re discovering about this new process you’re trying to improve?
Do you dive in headfirst because a “digital process has got to be more efficient than the current process”?
A quick warning. Not every process is “ready for prime time.” If your discovery process continues to turn up surprises and the business team is using the data team to find out what the process actually is, then you’re probably setting yourself up a long, painful process of rework. It will probably help everyone if the business takes time to discover ALL the variations of what’s actually happening with the data before you start work. A good business analyst who asks good questions can help expedite this process.
Building the MVP
Once you’re ready to start building out your process, you should start with the simplest version you can imagine. This MVP (minimum viable product) will help make sure your process actually solves the problem it’s designed for. Now you can watch your customers begin to interact with the new platform and process and see where the pain points arise. A modern digital process can be VERY different than what your customer is used to.
Baby Steps with Small Sprints
With a very loose definition of scrums and sprints, now you gather your customer feedback and build the next version. The quicker you can get feedback, the quicker you can iterate your design. Also, as your customer tinkers with the design, you’ll be amazed at all the new ideas that come up in the new process that nobody could have even imagined at the beginning of the process.
Fabric and Power Platform for the Win
Do you often spend more time debating the accuracy of numbers in presentations rather than creating solutions to the problems?
Do you miss market opportunities because it took too long to gather data to respond in a timely manner?
Do your data scientists spend more time cleaning data instead of creating insights and fine-tuning their machine learning model?
Do your data scientists and data engineers trip over each other as they try to leverage different languages and tool sets to do their different jobs most efficiently?
If you can relate to any of these issues, take time to investigate how Microsoft Fabric and Power Platform can help you wrangle your data into submission and unify your people around your core business mission.
Rob Davies is a Senior Application Development Consultant at DesignMind.
Learn about DesignMind's Data and Analytics Solutions, including Fabric, OneLake, Data Lakehouse, Data Migrations, and Data Analytics.