At Zola Electric, our mission is to provide energy access for billions. We aim to enable our customers, through a mission-specific technology platform, to grow, sustain, and supply 3 billion people and all the homes, schools, clinics, and businesses that support them with reliable and affordable energy. To achieve this, we’ve built a sophisticated data architecture that enables us to efficiently ingest, process, analyze, and act upon vast amounts of data from our smart devices and software applications.
An Architecture for Reliable Energy Delivery
Our existing architecture consists of integrated components that enable efficient data processing and analytics:
Storage Layer:
- Amazon Redshift: Primary data warehouse for hot storage and analytical processing
- Amazon S3: Cold storage for offloaded data, optimizing cost efficiency
Data Processing:
- ETL Pipeline: Orchestrates data movement from various sources
- Looker: Provides analytics and visualization capabilities
This architecture helps in efficiently storing and processing large data volumes from devices and applications, providing a foundation for deriving insights to support key business processes and decision making.
Scaling Our Impact With Data
In recent years, we’ve recognized the need to enhance our data architecture to keep pace with our growing scale, evolving business needs, and ambitious goals. As we expand our operations and customer base, we must ensure our data platform can handle the increasing volume, variety, and velocity of data while delivering insights faster and more cost-effectively.
Moreover, we see a significant opportunity to leverage advanced analytics and machine learning to drive operational improvements, enhance customer experiences, and unlock new revenue streams. To realize this potential, we need a data architecture that can support sophisticated ML workloads and enable rapid experimentation and deployment of new models. At the same time, we’re aware of the need to strengthen our data governance practices, security controls, and compliance posture.
Baking True Intelligence Into Energy Management
In recent months we’ve embarked on a strategic initiative to enhance our data architecture. For instance, we are enhancing our ML pipeline to generate personalized recommendations, making it more robust with multi-zone storage on S3, adding Feature Store in Redshift with real-time serving via Redis, leveraging SageMaker for ML training and inference, and strengthening data quality checks and error handling.
Our data architecture now supports holistic support for current and planned Gen AI use cases, including:
- Vector database
- Feature store
- Integration layer to leverage third party large language models (LLMs) such as OpenAI
This powerful combination enables us to support advanced LLM use cases, such as analyzing call transcripts to understand customer sentiment and intent, and building AI assistants that can engage in natural conversations to answer customer questions and troubleshoot issues.
Delivering Proactive Customer Experience
Currently, our users can access a UI that allows them to view the status of our devices. However, this process is largely manual, requiring users to individually check each device for alerts, system health, and other key metrics. Our vision is to automate and optimize this process through predictive analytics and proactive recommendations. We’re building a ML pipeline that incorporates historical user actions from the UI to understand typical user behaviors and preferences.
Using this rich dataset, our ML models generate personalized recommendations for each user, highlighting the top actions they should take based on the current state of their devices and their historical usage patterns. These recommendations are tailored to specific use cases. Our ultimate goal is to create an intelligent, intuitive, and highly automated UI that minimizes manual user interactions and proactively guides users to the most impactful actions. By transitioning from static, predefined screens to dynamic, data-driven interfaces, we can improve the efficiency and effectiveness of our maintenance and support operations.
Technical Implementation
To bring this vision to life, we’re investing heavily in our data foundation. We have implemented a robust data pipeline that gathers telemetry data from our device service at different intervals, depending on the device type. We are collecting invaluable historical user action data from our databases. All of this data flows into our custom ETL application and the backbone of our data processing. It loads data from sources, performs necessary transformations and calculations, and orchestrates data movement between our storage layers.
Our storage architecture on Amazon S3 is organized into distinct zones:
- Landing zone for raw data
- Standardized zone for transformed data
- Curated zone for ML ready datasets
- Archive zone for historical data storage
At the heart of our architecture is Amazon Redshift, our cloud data warehouse. Redshift stores cleaned, transformed data in a structured format. We’ve also implemented a Feature Store in Redshift, with real-time serving capabilities via Redis cache, to centrally manage our ML features.
Analytics and ML For Better Decision-Making
With data flowing smoothly and stored efficiently, we power analytics and ML workloads. Looker, connected to Redshift, enables self-serve reporting and visualization. The platforms directly access the data warehouse for embedded analytics and operational reporting. In our ML pipelines, we train both tenant-specific and cross-tenant models to capture unique patterns within each tenant’s data while learning from commonalities across tenants.
Our models cover using time-series, classification models and regression. For training Infrastructure, we have leveraged SageMaker for ML training and Inference. It’s a preferred choice as it helps with seamless integration with AWS data storage, has managed infrastructure reduces DevOps overhead and supports built-in model versioning and A/B testing.
High Confidence Through Monitoring
To ensure data quality, we have smoke tests in place that check for data inconsistencies, data freshness, duplicate records and consistency in data between S3 and Redshift. To strengthen our data quality and error handling techniques, we are working on additional methods like exponential backoff paired up with dead letter queue and anomaly detection.
To ensure fair comparisons and model generalization, we apply thoughtful feature normalization strategies. Tenant-specific features, like average task completion time and installations per month, are normalized within each tenant using z-scores. Cross-tenant features are normalized across all tenants to create relative metrics, enabling us to identify efficiency differences and best practices.
Performance at Scale
Cost optimization is crucial to us and we have achieved it through implementing multi-zone storage architecture on Amazon S3, centralized feature store in Redshift combined with Redis caching reducing redundant computation of features, adding more data validation checks that catch issues early, cross-tenant models that share learning and reduce redundant training and strategic use of SageMaker for scalable compute.
Our architecture ensures scalability across multiple dimensions. Data processing is scalable as we use S3 for storage with different zones, Redshift for analytical workloads with ability to scale compute independently and Redis Cache for real-time feature serving reduces database load. We have addressed model training scalability by using SageMaker that provides auto-scaling for training jobs, separate models per tenant allows parallel training and Redshift WLM queues help manage resource allocation.
Protecting Customer Trust
Throughout this journey, data governance remains top priorities. Our robust access controls, encryption practices, and audit mechanisms ensure that data is only used for its intended purposes and never exposed improperly. The strategy implements encryption at rest for RDS databases using AWS KMS and enables SSL/TLS for data in transit, requiring minimal code changes while maintaining existing data flows.
The implementation involves a staged migration process that includes creating encrypted snapshots of existing RDS instances, launching new encrypted instances, and updating connection strings in the ETL application layer. The benefits of this approach include AWS-managed encryption with automatic key rotation, minimal performance impact, and database-level security with built-in encryption for both data at rest and in transit.
Accelerating Towards a Sustainable Future
We envision a future where every aspect of our business, from demand forecasting to system maintenance to customer support, is powered by real-time insights and predictive intelligence. A future where our teams spend less time reactively addressing issues and more time proactively driving innovation and impact.
We believe this future is well within reach, and the benefits – for our business, our customers, and our planet – will be immeasurable.
