The Client:
Our client is a Middle Eastern company, a leader in producing devices and systems for quality control and electricity metering.
They have developed a SaaS platform for simple and convenient monitoring and display of consumption parameters of all types of energy (electricity, gas, water, fuel). This solution is implemented as a cloud service maintained 24/7 and provides the ability to obtain any data in real time using both local and remote access with modern gadgets.
Architecting a Energy Analytics Platform with IoT and Cloud Solutions
The Business and Technical Challenges:
Our client already has a data monitoring system, but many things need to be improved. They are currently working on a new cloud-based and scalable one. Our data engineers had several business objectives:
-
create data pipelines to collect data from a vast number of industrial customers' devices;
-
process this data for analytics;
-
develop BI dashboards displaying various aspects of energy consumption
The Solution:
1) We build data pipelines that gather data from various sources (APIs, customer files, metering devices, and previous monitoring systems).
For the primary storage of raw data, we used the Microsoft Azure infrastructure, Data Lake, enabling efficient and organised data migration.
2) The data was processed using Microsoft Azure Databricks. We also developed several Databricks notebooks to prepare the data for analytics. It included processing IoT data from various metering devices.
3) We created data pipelines in Microsoft Azure Data Factory to automate and schedule data collection and preparation tasks, ensuring seamless data migration and transformation.
4) We used the Power BI platform to create comprehensive BI reports and visualizations.
The Tech Stack, Used in the Project:
-
Data processing: Python
-
Access Rights Management: Microsoft Entra ID
-
Version Control and CI/CD: Microsoft Azure DevOps
-
Database Deployment: Microsoft Azure virtual machines, Microsoft SQL Server
-
Sensitive Data Storage: Azure Key Vault
-
Computing Resources for Power BI: Power BI Embedded Capacity
The Result:
DataEngi helped the client to create data pipelines to collect, process, and prepare data for analytics, which are displayed in the user's personal account on the website as information panels (dashboards).
Users have different requirements for resource consumption analytics. Very few comprehensive solutions simplify the provision of analytical information to customers in different sections. Our data engineers helped create simple and convenient access to analytical data for our client's customers.
The Data Security:
We employ best practices of cloud provider solutions to ensure compliance with national and regional security standards, such as the General Data Protection Regulation (GDPR).
