In this article, I’d like to convey a simple use case for using Azure Resource Groups for Azure deployment scripts to keep BI environments synced for development and operational efficiency.
In my previous postings, I’ve talked about Azure Analysis Services and the importance of stretching out BI environments. In this post, I would like to put those two ideas together using Azure deployment techniques.
Positing a hypothetical business case:
How can the organization have a fluid transition between developing, testing, and using our AAS instance?
Considerations for Build
Our hypothetical business has given us some factors to consider:
- Have at least three environments: Develop. Test. Production.
- Keep the environments consistent.
- Keep costs down.
- Data and operations are to remain in Azure.
We have some questions to answer, then, when providing a solution:
- How do we make changes to all environments to keep them consistent?
- How can we optimize cost?
- Which Azure resources will we use?
- Which Azure deployment scripts will we use?
We will have the following Azure resources to facilitate data flowing into an AAS instance, aiding subsequent reporting mechanisms:
- Azure SQL Server
- Raw data database
- Data warehouse database
- ETL from the raw database to the data warehouse database, using SSIS in Azure Data Factory
- Azure Analysis Services, reading data from the data warehouse database
We now have the necessary resources to move data from Azure SQL Server, via Azure Data Factory, to Azure Analysis Services.
All of these resources are grouped under my resource group “wideworldimportersRG”. We will use this as our baseline to deploy our separate environments.
First, let’s create some Resource Groups that will act like environment containers. Let’s just call them “Testing” and “Production”.
My already-created resource group “wideworldimportersRG” can act as our development/prototype group.
We now have these three environment containers set up as Resource Groups:
Using the Resource Manager, we are able to deploy almost all resources from “wideworldimportersRG” to “Testing” and “Production.”
Since certain items would have to be changed, such as server names, the templated Azure deployment can be configured for each Resource Group. You could then save the script for later use.
Deploying an entire resource group is work up-front in terms of parameter settings and naming resources as to not conflict with existing ones. But it is worth going through the exercise to keep the environments in sync.
Each resource can be deployed on its own, however, if mass Azure deployment is not an option.
- Setting up an Azure SQL Server for the “Testing” environment was a manual task.
- Deploying the schemas and data to the server’s databases subsequently was via SMSS
- Since Data Factory is its own entity, I was able to just set up another Runtime Integration for the “Testing” environment in the existing Data Factory instance.
- I had to deploy the semantic models to the AAS instance in “Testing” once the instance was deployed, via Visual Studio.
- I had to add myself as an admin on the instance to deploy the models.
- There are limitations to shifting resources.
Post Azure Deployment
After working through naming conflicts and the above caveats, I have the objects needed to set up my “Testing” environment container and start working with it.
This sets the baseline to deploy further to other environment containers, such as QA, UAT, Staging, or a Sandbox. Deploying this to Production would finish out our hypothetical scenario.
However, in a real-world scenario, there are more nuances and moving pieces. So let’s go over some best practices to consider in a more general sense.
Resource Naming Conventions
A suggestion would be to tack on “_[resource group name]” to the resource. So we would have wwidemo_production and wwidemo_testing as examples of naming the SQL Server instances for each Resource Group.
There is also the option of Azure Tags, which categorize resources on a name/value pair system.
Each resource in Azure will cost based on its computing abstraction. The Azure pricing calculator can give you a good look at how much usage will cost for a Resource Group.
Keep in mind the following when keeping costs down, as Azure is all about charging per compute cycle:
- Non-production environments should only compute on-demand and should be in the lowest necessary pricing tiers.
- Environment resources should only be available during business/usage hours, or by special request otherwise.
- Production resources should be scalable to the business’ common denominator for demand.
- Only store data that is necessary for each environment. Have a data retention policy that fits business need as leanly as possible.
- Automate per resource availability and scalability as much as possible, based on business need, by using Azure Automation or Azure Function Apps.
- Each environment will have a different flavor of this. For example you will probably not need to process a full AAS semantic model in Development daily, but you will in Production for reports needed by the business in the mornings.
Have a release cadence once Azure deployment scripts are flushed out (as I worked through above).
Changes are to be aligned between environments to keep resource frameworks consistent. This will keep the organization from having disparate results when testing and using Azure resources.
Resource Group Management
It is recommended by Microsoft to keep resources that are related to each other within their own Resource Groups, separate from resources that are not related.
In our example above, we only have a few related resources, which makes our task simple. There might be many unrelated resources working to accomplish different goals in other scenarios.
One scenario would be an organization that has multiple departments whose resources will never interact with each other. In that case you could have Resource Groups such as:
- ACME_Sales_Development, ACME_Sales_Testing, etc.
- ACME_Procurement_Development, ACME_Procurement_Testing, etc.
In this business case, we had two databases, ETL operations moving data from one to the other, and an Analysis Server instance.
We deployed them to three different environment containers.
Development and Testing resources were scaled down in pricing tiers and usage to lessen cost.
Production pricing tiers and usage were leaned out to lessen cost, but still meet business needs.
- Consistent operations between environments
- Azure deployment plan when changes are made to the operational framework, starting with Development
- Cost mitigation for Azure resources between environment contexts
Thanks for reading!