The Client:
Industry/Sector: E-commerce
Company size: Enterprise
Headquarters: USA
Our Customer is a consumer goods company specialising in brand development and operating third-party businesses on an e-commerce platform. It has gained significant
attention and success by identifying promising brands and developing them. Their goal is to help the operations of these brands, leveraging their expertise in marketing, supply chain management, and e-commerce to drive growth and profitability.
They help sellers scale their businesses by giving them access to capital, expertise, and resources in product development, marketing, inventory management, and distribution.
Empowering of Data Analytics: Data Engineering, Data Pipelines, and Data Quality in E-commerce
The Business and Technical Challenges:
-
Correct inconsistent reporting data for analytics
-
Сollect data from different sources (scraping data, e-commerce stores API data)
-
Data Pipelines creation for Business Intelligence
-
Data Quality checks for consistent data delivery for analytics (in BI and WEB-dashboards)
-
Discovery of the new stores (by product categories)
-
Prevent data loss: duplicate data import, processing considering duplicates
-
Provide data loss resilience
The Solution:
Our role: augmentation of our Customer`s team with Data Architect, Data Quality Engineer and Data Engineers.
The Tech Stack, Used in the Project:
-
Apache Airflow
-
Python
-
AWS Lambda, S3, Redshift, RDS, Kubernetes
-
Snowflake
-
Periscope (Sisense for Cloud Data Teams)
-
Terraform
The Result:
We helped to build a Pipeline system for collecting the data from dozens of MWS sources, processing them via different layers, and preparing data for reporting.
We provided monitoring, data quality checks of final data, sales and supply chain reports.
The Data Security:
Our Data Analytics Certified engineers follow the best practices of AWS security rules when building Data Systems. They used VPN, Okta and Tailscape services.
