Use case #BIGDATA #IoT
Data Lake of manufacturing
in the cloud
Cepsa is a global energy company which operates in an integrated manner at all stages of the hydrocarbon value chain, in addition to manufacturing products from raw materials of plant origin and having a presence in the renewable energy sector.
It has more than 85 years of experience and a team of nearly 10,000 professionals, with technical excellence and adaptability. It is present in the five continents through its business areas of Exploration and Production, Refining, Chemicals, Marketing, Gas and Electricity, and Trading.
Energy and industrial sector is in the midst of change.
This change goes hand in hand with a new industrial revolution called Industry 4.0 where the use of data has a particular relevance.
Current systems of control and events historification have proved to be significantly limited when enabling to integrate and analyse information together with data external to their own plants. Besides, those systems have closed licencing models that penalizes the client when integrating external information, such as Lab Data, weather information, costs and prices information…
The solution and the main AWS services used
IoT standardization protocols enable to use current platforms of plants control but adding limitless historification functionality, cheap and with large capacities to integrate external data and to perform sophisticated analysis on them.
With this solution Cepsa is seeking to build a Data Lake in the cloud that centralizes the information coming from hundreds of thousands of sensors installed in their manufacture plants, that integrates additional sources to enrich this information and that enables to exploit the data using advanced analytics processes, visualization and Business Intelligence tools.
Data Lake is capable of intaking, processing and making available to platform users an average of two thousand signals per second rapidly in a Near-Real Time model, as well as persisting the information in a historic of several years with a projected growth at a Petabytes level.
The solution is completely based in the use of managed services, obtaining a serverless implementation easy to maintain, robust, secure and scalable. The main services used are:
- AWS IoT as MQTT messaging central broker.
- AWS Greengrass for the integration with on-premises sensors via MQTT and OPC-UA.
- Amazon Kinesis to process information in Near Real Time.
- Amazon S3 as storage main repository.
- AWS Athena to consult Data Lake using SQL.
- AWS Lambda and AWS Fargate to execute application logic.
- AWS Glue as ETL tool and Data Catalogue.
- AWS ElasticSearch as indexed data repository for time series.
- Amazon DynamoDB as metadata storage.
- AWS Database Migration Service for the migration and replication of on-premises databases.
- The pay per use model of the public cloud has enabled Cepsa to have a solution without major initial investments and a low experimentation cost.
- Given that it is a solution fully implemented by managed services, the operational cost is reduced.
- All the pieces of the solution scale horizontally, and so the integration of more sensors does not lead to a bottleneck in the platform.
- It is an open system that allows integrating any tool of information exploitation that may be deployed on AWS.
- The system works with services such as S3 or DynamoDB, which provide high availability and a great solidity by default.
- The cost of information storage in S3 en bruto is so low compared to traditional systems (changing from a scale of millions to one of thousands of euros) that Cepsa is able to storage all the values issued by all the sensors without having to apply mechanisms of values interpolation and approach.
Keepler is a boutique company of professional technology services specialized in design, construction, deployment and software solutions operations of Big Data and Machine Learning for big clients. They use Agile and Devops methodologies and native services of the public cloud to build sophisticated business applications focused in data and integrated with different sources in batch mode and real time. They have Advanced Consulting Partner level and have a technical workforce with 90% of their professionals certified in AWS. Keepler is currently working for big clients in different markets, such as financing services, industry, energy, telecommunications and media.
If you want to know more or if you want us to develop a proposal for your specific use, contact us and we’ll talk.