Since 2016, AWS has included Quicksight as its Business Intelligence (BI) service. Quicksight facilitates business analytics thanks to its simplicity in generating visualizations and interactive dashboards, and its high capacity for real-time analysis of data. Despite this, major difficulties are still encountered when it comes to promoting the objects generated in Quicksight (analytics, dashboards and datasets) between environments or between accounts. There is no simple method where the data analyst who has generated Quicksight analytics and dashboards in a development environment can promote those analytics/dashboards to the production environment.
This article will try to explain a proposal developed to solve this problem, because it appears recurrently in those projects working with multi-account environments with Amazon Quicksight. This solution has been validated in productive environments by a data analyst, thus certifying its operation in most cases, and detecting those in which its operation has not been as expected.
Solution development
To implement this solution we have made use of boto3, the AWS SDK for Python. Boto3 is a Python library that facilitates communication with different AWS services, and in this case, it has been used to interact with the Amazon Quicksight service through its API. This API allows performing CRUD operations with the different Quicksight objects, thus facilitating the goal of migrating datasets, analytics and dashboard from an AWS source account to a target account.
When migrating Quicksight objects between environments, it is important to follow a specific order due to the presence of dependencies between them. For example, an analysis will depend on the datasets it uses.
During the migration process, objects will be taken from the source account and, using the Quicksight API, will be transformed into JSON objects, so that from these JSON objects Quicksight objects can be created in the target account.
The first of the objects to be migrated and created in the target account are the data sources1. Data sources are connections to the underlying data source, which can be a relational database, a NoSQL database or another type of data source. These need to exist in the target account before migrating the datasets, as they are referenced by the unique identifier of the data source.
To migrate these Quicksight objects, it is necessary to check which data sources exist in the source account, which can be done through the list_data_source method; then go through each of these data sources in the source account to check its configuration (with the describe_data_source method). And finally, create it in the destination account, through the create_data_source method.
It is important to use the same unique identifier (DataSourceId) as in the source account when creating the data source in the destination account, since this will facilitate the creation of the datasets later on, since the ID of the data source they reference does not have to be modified.
With the migrated data sources, the datasets can be migrated. This migration is performed in the same way as the previous one. First we check which data sets exist in the source account (list_data_sets); then we check their configuration (describe_data_set), and once their configuration is known, they can be created in the destination account using the create_data_set2 method. Once the datasets have been created, in order for the users of the target account to be able to view them from the console, their permissions must be updated using the update_data_set_permissions method.
Finally, to migrate the analyses and dashboards from the source account to the target account, it is necessary to follow a different procedure than the one previously used with the data sources and data sets. This is because the Quicksight API, when converting the analysis into a JSON type object with the describe_analysis method, results in a JSON object that is not compatible with the create_analysis method, since this method encounters format validity problems. To solve this problem, a Quicksight template must be used, which will act as a wildcard, facilitating the creation of the analysis and dashboard in the target account.
This template will be created from each of the analyses to be migrated, so there will be one template per analysis. The create_template method needs as parameters the arn of the analysis from which the template will be created and a list with the arn of the datasets used by this analysis. This data can be obtained through the describe_analysis method. With the template created, the root user of the target account must be granted the quicksight:DescribeTemplate and quicksight:UpdateTemplatePermissions permissions. And, in this way, the analysis and dashboard can be created in the target account from the template. To do so, the create_analysis and create_dashboard methods must be used, which, as shown in the diagram, make use of the SourceEntity parameter. This parameter is a dictionary with a field for the arn of the template and another for a list of arn of the datasets of the target account used by the analysis, or dashboard, that is being created.
Once both the analysis and the dashboard have been created, all that remains is to update their permissions so that those users, or roles, that need them can view them in the console and make modifications to them.
The proposed solution takes care of a first migration of those Quicksight objects that do not exist in the target account. But it is possible to add a conditional that checks if the Quicksight object exists before creating it. And, if it does, the object will be updated using the update_data_set, update_template, update_analysis and update_dashboard methods. In this way it will be possible to develop new versions of quicksight analysis and dashboards and migrate them to the productive environment when the version is stable.
What you should be aware of
During the validation phase of the dashboard migration process, there were some aspects in which the migration of the elements contained some difference with respect to the original report.
This is the case for the migration of documents containing tabs. One of the analyses contained three tabs, of which only two of them were decided to be published in the dashboard of the development environment. After performing the migration and checking the result in the production environment, the dashboard showed the three tabs of which the analysis consisted, so the selection of tabs to be displayed on the dashboard had not been transferred during the migration.
Also some visuals that were part of the analyses showed small differences in the colors used. This was especially true in those cases where the color to be used for a particular value had been selected manually, without the hexacolor being part of the main theme used for the rest of the analysis.
Finally, it should be noted that those options activated after the dashboard was created are not inherited after the migration. Options such as sharing the document with “All users of the account” or activating “Detectable in Quicksight” must be performed directly on the already migrated dashboard, even if they were already activated in the original report, since they are options that are not part of the analysis from which the dashboard is created.
Conclusions
Using the AWS SDK for Python, boto3, and the Quicksight API, it has been made easier for data analysts to migrate data sets, analytics, and dashboards effectively in multi-account AWS environments.
The migration of Quicksight objects between environments has been a complex task but thanks to the proposed solution validated in production environments, an automatable, reliable and efficient mechanism has been generated.
AWS is actively working on the optimization of this process, with the aim of facilitating each of the steps that must be followed to achieve this migration of Quicksight resources. In the meantime, this solution will help ease the waiting time and ensure that all the gear runs smoothly.
(1) Data sources can be configured independently in the target account without the need to perform the migration. But if this method is chosen, then there must be some kind of equivalence between the unique identifiers (ID) of the data sources in the source account and those created in the target account. This is an indispensable field for the migration of the data sets.
(2) During the development of the solution, a dependency between some data sets had to be resolved. This was due to the fact that the Quicksight dashboards used, made use of row level security (RLS), to filter the data according to the permissions of each user. And, for this reason, some datasets had a dependency on the one that marked the row-level restriction, so this dataset had to be migrated first.
Image: Unsplash | Shahadat Rahman
Data Analyst in Keepler: “Problem solver, especially ones related to data and technology. Always focused on improving, being updated and sharing what I've learned. If you love to analyze sports based on data, we will be great friends :)”
I am a technology enthusiast. I enjoy learning, which allows me to step out of my comfort zone to face new challenges. I am currently focused on learning about the cloud and AI, so Keepler is an ideal partner.
0 Comments