DSI Underground is a well-known global leader in underground mining and tunneling solutions. With operations in over 70 countries and more than 2,000 employees, the company has grown exponentially, primarily through organic expansion and acquisitions. However, this growth has presented challenges in managing data across multiple time zones and ERP systems.
Additionally, several critical data access processes were complicated and difficult, leading to inefficiencies and increased workloads. Our company provided an all-encompassing solution to these obstacles by implementing a robust, event-driven architecture that streamlined data management while automating essential processes. In this blog post, we will delve into the problems faced by DSI Underground, the solutions provided by our experts, and our plans for future architectural changes.
Through collaboration with ITMAGINATION, DSI Underground achieved remarkable optimization of its operations. The incorporation of automated systems resulted in reduced workload and enhanced efficiency. Our partnership exemplifies the significance of working together to effectively tackle intricate business challenges. We would like to take you through our past achievements and future aspirations.
As DSI Underground expanded, handling data from multiple entities located in different time zones became more demanding. The organization had to manage more than 20 entities distributed across four regions, including EMEA, APAC, LATAM, and North America. Furthermore, the data management process was complicated using various ERP systems like Navision and Optima in addition to local systems with distinct data structures.
Consequently, activities such as sales performance reporting, budgeting, and forecasting became challenging to handle efficiently and consumed a lot of time. This resulted in decreased productivity and possible inaccuracies in decision-making processes.
Furthermore, the rapid development and acquisitions of new entities meant that DSI Underground had to adapt its data management approach to accommodate new data sources and systems. The existing data management infrastructure struggled to keep pace with these changes, making it harder for the company to achieve a unified view of its operations and make informed business decisions.
DSI Underground aimed to enhance their collaboration with key customers by providing them with transparent and readily available access to critical data. To achieve this objective, they needed to enhance the accessibility of their existing data.
These were order previews and status updates, invoice details presentation, delivery notes management, client inventory tracking, stock levels monitoring, product listing, certificates and approvals display, dashboard creation, multilingual support, digital document repository management, contact with sales, and product catalog maintenance.
The absence of transparency in data proved to be expensive as it led to poor communication and limited access to crucial documents like certificates, delivery notes, and invoices. To enhance the response time of DSI towards their customers' requirements and streamline their data retrieval procedures, we aimed to make significant improvements.
Our Solution: Event-Driven Architecture and Consolidated Data Solutions
DSI Underground's data management challenges were addressed by implementing an event-driven architecture through Azure Event Grid. This innovative approach facilitated near real-time data processing and enhanced flexibility in managing data from diverse sources. The company was able to respond rapidly to changes in business conditions, guaranteeing that pertinent information was continuously available for analysis.
In addition, we devised separate integration instances for each entity employing Azure Data Factory (ADF) and Data Lake, ensuring smooth information flow between systems. This strategy helped us handle the intricacies of handling multiple ERP systems and time zones, simplifying the process of managing and analyzing data from various sources.
To consolidate data effectively, we developed a data warehouse with data marts, which enabled further analysis in Tabular Model. This new architecture not only improved data management across different time zones and ERP systems but also streamlined reporting processes, making it easier for DSI Underground to access and analyze crucial business information. The benefits of the new architecture include increased efficiency, reduced manual intervention, and improved data accuracy.
Additionally, our solution provided a scalable foundation for future growth, ensuring that DSI Underground's data management infrastructure could adapt to accommodate the company's ongoing expansion. This scalability allowed DSI Underground to continue focusing on its core business operations while relying on our expertise to maintain a robust, efficient data management system.
Furthermore, we utilized several Azure services to enhance automation across various business processes:
- Data Factory: The ETL (Extract, Transform, Load) process was automated using Data Factory, streamlining data integration and management.
- Data Lake v2: We leveraged hierarchical namespaces to improve data organization and storage.
- Event Grid: To support event-driven type of architecture
- Logic Apps: To automate using low-code approach data model refresh
- Azure Analysis Services: To give access for end-users to good quality data model with predefined measures and facts
- Power BI: The implementation of Power BI provides reporting and data insights, further enhancing the automation capabilities.
- SonarCloud (CI/CD): To improve code quality and set a baseline for future technical debt, we implemented SonarCloud in the continuous integration and continuous deployment process.
- Integration Tests (CI/CD): We incorporated integration tests into the software development process to improve overall software quality and reduce potential issues.
By incorporating these services and tools, DSI Underground has experienced a remarkable enhancement in the efficiency of its business processes. Automation of several tasks and minimization of manual intervention have contributed to this improvement.
Consequently, the company can concentrate on its primary business operations and provide superior service to clients. To maximize access to information, we developed a Progressive Web App (PWA) that is compatible with almost all devices with web browsers. Our app retrieves data from the data warehouse and presents it in a user-friendly format.
The app is an MVP of DSI’s Customer Portal. The portal would allow to:
As DSI Underground continues to grow and evolve, ITMAGINATION is committed to ensuring that their data management and automation infrastructure keeps pace with the changing needs of the business.
A Lakehouse is a unified data platform that combines the best features of data lakes and data warehouses. It provides ACID transactions and data governance typical of enterprise data warehouses while offering the flexibility and cost-efficiency of data lakes. This enables business intelligence (BI) and machine learning (ML) on all data, stored in massively scalable cloud object storage using open-source data standards.
In the context of the Databricks Lakehouse, the primary component includes Delta tables, which are tables that support ACID transactions, data versioning, ETL, and indexing, ensuring data reliability and consistency.
By implementing a Lakehouse, businesses can build a single source of truth for their data, making it easier to discover, collaborate, and gain insights from the data. This approach will enable DSI Underground to further streamline its data management processes and reduce the number of data factory instances required.
The implementation of the Lakehouse concept offers several advantages, including cost reduction, improved scalability, and enhanced data integration. By consolidating data management processes and reducing the number of data factory instances, DSI Underground can save costs on infrastructure and maintenance. With a more scalable solution for data management, the Lakehouse approach ensures that the company's infrastructure can handle future growth and expansion.
Additionally, having a single platform for data storage and processing allows for easier integration of data from multiple sources, resulting in better overall data quality and insights. We aim to work closely with DSI Underground to ensure a seamless transition to the Lakehouse concept in the upcoming months while minimizing any disruptions to their business operations.
DSI Underground was confronted with significant obstacles in managing data across multiple time zones and ERP systems, in addition to automating crucial business processes. We resolved these challenges by providing comprehensive solutions, implementing a sturdy, event-driven architecture to streamline data management, and automating processes through the use of Power BI and Azure services.
As the company expands, we remain dedicated to supporting their changing needs while ensuring that their data management and automation infrastructure remains efficient, scalable, and future proof.
Our Lakehouse approach is an excellent solution for businesses seeking cost savings, improved scalability, and enhanced data integration. Our team can work closely with you to ensure a smooth transition with minimal disruption to your operations. If you are encountering difficulties managing data across multiple systems and automating essential business processes, we encourage you to explore our customized data solutions.
.svg){kind=small}
