The Client:
Treat Technology, Inc. is a customer acquisition channel that delivers leads to e-commerce brands. It is an operator of a customer database platform intended to help businesses to manage their customers. The company offers one unified view to individual
customer interactions and behaviours. It uses AI to generate customer-specific videos, pics, and content, enabling marketers and brands to experiment and identify segments faster using real-time customer behaviour. Treat enables marketers to harness the power of AI-driven creative insights to generate highly personalised content without compromising user privacy.
Headquarters: United States
Our customer clients are e-commerce brands with stores on platforms like Shopify and data in marketing platforms such as Klaviyo and other platforms like Facebook, Google Ads, and Google Analytics.
Our data engineers were involved in helping in building a platform that enables seamless data integration from different e-commerce and marketing platforms to create a single source of truth for making analytical reports.
Architecture Design and Data Integration from Different
e-commerce Platforms into Central Data Storage
The Business and Technical Challenges:
-
Build a platform that will make it easy for merchants (companies, online- stores) to connect various e-commerce platforms and other marketing platforms. Our Customer needed to collect data on users, products, campaigns, etc., in a central place to build analytics.
-
Create a Buyer Profile by Integration from different e-commerce platforms that is a single source of truth the analytics.
The Solution:
-
Help the client design the data system architecture
-
Implement data integration (ELT) from different e-commerce platforms into the central Data Warehouse
-
Build Data Pipelines
-
Build a central Data Warehouse
-
Write data connectors
We investigated the business and technical challenges to understand whether the Airbyte platform would work for our aims. Next, we designed the architecture and service to configure a merchant's ELT. After that, we used Python to develop custom Airbyte connectors for Shopify, Klaviyo e-commerce sources (for import into BigQuery) and developed CI/CD deployment scripts using Terraform for use with GitHub.
The Tech Stack, Used in the Project:
Technologies, we used:
-
Python, Airbyte, Airflow, Terraform, AWS (AWS Lambda, AWS SQS, Fargate, AWS Cloudwatch, AWS ECR, EKS), BigQuery (GCP), dbt, Kubernetes, PostgreSQL.
Platforms, we worked with:
-
Shopify, Klaviyo, Facebook, Google Ads, Google Analytics, Recharge.
The Result:
We successfully helped design and implement a comprehensive solution for our customer to make the E-commerce Data platform. Now it is easy to add e-commerce data sources to this platform and, as a result, to support updating a single source of truth to build analytical reports.
To do this, our engineers:
-
implemented a comprehensive solution to collect and aggregate data from diverse e-commerce platforms
-
designed a scalable Data Warehouse architecture
-
created custom data connectors using Airbyte, Python, and Airflow. Leveraged AWS services, including Lambda, SQS, and Fargate, along with BigQuery on GCP for efficient storage and analysis
-
enabled seamless data preparation for AI processing, ensuring high-quality input for advanced AI applications.
The Data Security:
We followed AWS and GCP security best practices and security guidelines. Our data engineers used:
-
The AWS Well-Architected Framework especially Security Pillar
-
The Google Cloud Architecture Framework: Security, privacy, and compliance
-
VPN, Private/Public subnets. Production and development environments separated in different AWS Accounts. Used AWS Secrets Manager for sensitive data.
