Mirosoft Fabric: Infrastructure & Capacity
Infrastructure and capacity in Microsoft Fabric
As a SaaS solution, Microsoft Fabric enables users to efficiently manage all relevant data workloads without the need for complex infrastructure provisioning. Even though Microsoft Fabric offers numerous advantages, the right capacity planning in particular is crucial for successful use. In this article, we take a detailed look at the fabric infrastructure and show you how to avoid performance problems and unnecessary costs with the right capacity strategy.
How is the Microsoft Fabric infrastructure structured?
The Microsoft Fabric infrastructure is hierarchical and based on a clear structure of tenants, capacities and workspaces, which together ensure efficient management and processing of data.
Tenant: A tenant represents the top level of the hierarchy, usually corresponds to the entire company or organization and is linked to your Microsoft Entra directory. You can create multiple workspaces and capacities within a tenant.
Capacity: A capacity is a dedicated pool of resources within your tenant. The size of the capacity, measured in capacity units (CU), determines the available computing power and directly influences the performance of your workloads. It plays a central role in resource allocation and pricing. A capacity can hold several workspaces.
Workspace: These serve as containers for different projects, teams or departments and contain various elements such as lakehouses, data pipelines or reports. In this area, teams can work together on data, reports and pipelines.
Capacity models in Microsoft Fabric
A central element of Microsoft Fabric are the capacity models that control resource allocation and utilization.
F-SKUs
The capacity in Microsoft Fabric is defined by various F-SKUs, each of which provides a certain number of CUs (capacity units). These range from F2 (2 CUs) to F2048 (2048 CUs), allowing flexible scaling according to requirements. As the capacity is not defined in advance for specific workloads, companies can distribute their resources dynamically between different fabric services. Two billing models are available and can be purchased via the Azure portal:
- Pay-as-you-go: Billing according to actual consumption in CUs per second. Ideal for companies with variable usage.
- Reserved capacity: Advance booking for 1 or 3 years with up to 41% discount. Suitable for companies with regular usage and long-term planning.
Which capacity level is right for my company?
Choosing the right capacity depends on the specific requirements and workload of your company. Specific capacity prices also depend on the data center you choose.
For sound capacity planning, you should use the Microsoft Fabric Capacity Metrics app. The application helps you to monitor resource utilization and determine the optimal capacity size for your specific workloads.
The following can be used as a rough guide for capacity:
Small Power BI projects and data engineering pipelines: For smaller BI projects, an F4 or F8 license may already be sufficient. Depending on the use case, these may need to be scaled up at a later stage.
Real-time data processing: Applications with low latency and high parallelism, such as streaming data analysis or IoT data processing, benefit from an F256 or F512 capacity. If even faster processing is required, an F1024 SKU may be necessary.
Data science and machine learning: An F256 is recommended for AI models, neural networks and deep learning calculations. If computationally intensive training processes are carried out with large data sets, an F1024 capacity may be required.
As workload requirements vary, it is advisable to start with a smaller capacity and test the scalability flexibly in order to achieve optimum performance.
How do I create a Microsoft Fabric Capacity?
Scaling and capacity management
Microsoft Fabric uses automatic mechanisms to process workloads efficiently. Bursting, smoothing and throttling are three essential concepts that influence how computing capacities are used.
Bursting: Dynamic resource expansion
With bursting, the platform can use more resources than were originally allocated in the event of sudden load peaks in order to ensure consistent performance. This additional resource utilization is regarded as "borrowing" future capacities. This allows tasks that would normally take longer to complete to be completed more quickly. However, it is important to note that these borrowed resources must be "paid back" later.
Smoothing: Even distribution of resource utilization
To minimize the effects of bursting, Microsoft Fabric implements so-called smoothing. This distributes the consumption of additional resources over a certain period of time:
- Interactive operations (Interactive Compute): The resource usage is smoothed over at least 5 minutes.
- Background operations (background compute): Smoothing takes place over a period of 24 hours.
This procedure prevents short-term load peaks from overloading the capacity.
Throttling: protective mechanism in the event of overload
Even with bursting and smoothing, it can happen that the capacity is overloaded over a longer period of time. In such cases, throttling acts as a protective mechanism:
- Interactive Delay:
If the capacity is fully utilized for the next 10 minutes, new interactive requests are delayed by 20 seconds. This applies, for example, to direct user queries such as Power BI queries or SQL queries in lakehouses.
- Interactive rejection:
New interactive requests are rejected in the event of a sustained overload of 60 minutes. However, planned background operations, such as ETL pipelines or Spark jobs, will continue to be executed.
- Background rejection:
If the overload persists for more than 24 hours, all requests, both interactive and background operations, are rejected until the resource utilization falls below the specified limit again.
This tiered approach ensures that system stability is maintained and critical background processes can continue to run for as long as possible, while interactive requests are regulated early to protect the overall performance of the fabric capacity.
Best practices for capacity management in Microsoft Fabric
Optimal management of capacities in Microsoft Fabric requires strategic monitoring, performance optimization and scaling according to demand. The following examples help to use resources efficiently and avoid bottlenecks.
1. continuous monitoring and resource management
Structured monitoring of capacity utilization enables early identification of performance problems.
- Fabric Capacity Metrics App: This app provides a detailed insight into the use of available resources and helps to identify capacity limits in good time.
- Capacity notifications: It is advisable to set up warning messages that indicate critical utilization peaks in order to be able to react in good time.
- Workload management: The allocation of capacities to specific workloads ensures that business-critical applications always have sufficient resources.
2. optimization of workloads to increase performance
Efficient use of resources ensures improved query speeds and prevents unnecessary overloads. Here's how you can optimize your workloads, for example:
- Optimized data models: The use of compressed storage formats such as Delta Tables and Parquet reduces resource consumption and speeds up queries.
- Partitioning large tables: The targeted subdivision of data volumes improves processing efficiency and reduces latencies.
- Adaptation of the storage strategy: Depending on the use case, a mixture of DirectQuery for real-time analyses and import mode for high query speeds can be useful.
3. scaling of capacity in line with demand
If performance bottlenecks occur repeatedly despite optimization measures, a targeted expansion of capacities may be necessary. This could look like this, for example:
- Capacity increase: If continuously high CPU utilization or memory bottlenecks occur, upscaling to a higher capacity level may be necessary.
- Workload-specific adjustments: Limiting individual operations and avoiding resource-intensive processes during peak times contribute to load balancing.
- Monitoring for scaling decisions: A regular analysis of capacity utilization helps to determine the right time for scaling.
These measures enable the sustainable and economical use of Microsoft Fabric so that resources can be provided efficiently and bottlenecks avoided.
Success factors for optimal use of Microsoft Fabric
Microsoft Fabric is a powerful, scalable data platform that combines flexibility and high performance. To exploit its full potential, well thought-out capacity planning, efficient organization of your workspaces and optimization of workloads are crucial. Continuous monitoring helps you to identify bottlenecks at an early stage and make adjustments. It is best to start small and scale up gradually as required.
