SUCCESS CASE #BigData

CECOC Platform: Unifying Data and Optimizing Processes

Acciona is a Spanish company with a global presence in 65 countries across five continents. They are a group that specializes in developing and managing infrastructure such as water, concessions, construction, and services, with a focus on renewable energies. Acciona invests in sustainable projects and is committed to leading the transition to a low-carbon economy by prioritizing technical excellence and innovation.

Their activities cover the entire value chain, including design, construction, operation and maintenance, providing sustainable solutions to the major challenges of our era, including the energy transition and environmental conservation.

Leading the Energy transition through digitization.

The energy transition has become a necessity, especially in recent years, due to the implementation of mandatory decarbonization and zero emissions policies. This highlights the importance of focusing on renewable energies and digitalization.

The goal is to lead the transition to a low-carbon economy, so it is necessary to invest in projects of technical excellence and technological innovation to design a better planet and contribute to economic and social development.

The challenge Acciona Construction had to face was to build, develop and deploy a data platform in Google’s public cloud  for the ingestion, consolidation and productivity of data generated from multiple sources of on-site production. 

In order to meet the challenge, Acciona Construction had to address several requirements, which included:

  • The construction of big data architectures in the public cloud
  • The ingestion of data from the identified sources
  • The processing and consolidation of data through the development of ETLs
  • The accompaniment of the Acciona Construcción IT team
Solution on Google Cloud Platform

Keepler teamed with Acciona IT to design CECOC data platform, a data lake based on three layers that are ingesting a series of sources following a mainly batch model, specifically databases and some APIs. Although, other types of ingest based on streaming were also started to be considered for other cases, such as heavy machinery data. In 2022 was made a proof of concept for tunnel boring machines, which will be taken up again during 2023.

  • The first layer is a Raw layer based on Cloud Storage.
  • The intermediate layer, which we call Common, is based on BigQuery.
  • The final layer is a consumption layer in which the technology is proposed specifically for each case. Mainly in BigQuery, although other storage systems such as relational databases or noSQL databases, such as DataStore, are being considered.

From this consumption layer there are a number of applications, especially reports in PowerBI and analytics applications that consume this data. Recently, CloudComposer, a managed AirFlow environment, is being introduced as an orchestrator of the platform’s ingest and transformation processes.

The Google Cloud Platform services used were as follows:

  • Cloud Storage: Raw Layer / Common Layer
  • Cloud IoT Core: Connection
  • API Gateaway: Connection / Serving
  • Cloud Pub/Sub: Ingestion
  • Cloud Functions: Ingestion / Transform and Load / Serving
  • Cloud Dataflow: Ingestion / Transform and Load
  • SFTP Server Marketplace: Ingestion / Serving
  • Cloud Monitoring and Cloud Logging: Monitoring
  • Cloud IAM: Security / Serving
  • Secret Manager: Security
  • Key Management Service: Security
  • Cloud Data Catalog: Validation and Metadata / Functional Validations
  • Cloud Dataproc: Validation and Metadata
  • Cloud Composer: Common Layer
  • Cloud Bigtable: Consumption Layer
  • Cloud Datastore: Consumption Layer
  • Cloud SQL: Consumption Layer
  • BigQuery: Common Layer / Consumption Layer
Benefits

  • Acciona Construction now has a platform in which to unify all data generated from multiple sources with the creation of the CECOC platform they achieved a centralized repository designed to store, process and protect large amounts of data.

  • Acciona Construction has adopted data governance, data quality and process monitoring.

  • Establishment of best practices and optimization of processes regarding data.

Keepler is a full-stack analytics services company specialized in the design, construction, deployment and operation of advanced public cloud analytics custom-made solutions. We bring to the market the Data Product concept, which is a fully automated, public cloud services-based, tailored software that adds advanced analytics, data engineering, massive data processing, and monitoring features. In addition, we help our customers transition to using public cloud services securely and improve data governance to make the organization more data-centric.

Let’s talk!

If you want to know more or if you want us to develop a proposal for your specific use, contact us and we’ll talk.