Data - ValueLabs

End-to-End Case Study

Geo location

Australia

Platform

Web

Engagement

ETM

Service

Data

Challenges

Based in Australia, the company partnered with several of the shipping industry’s largest companies to roll out a program to eradicate questionable practices that could endanger the safety of crews, ships, and the environment. This program proved so successful that the business opened a US office in 2006 and a European office in 2007.

Complex

Low reliability

Data lag

Risk assessment

Ensure easy assessment of the risk involved in shipping their cargo with specific vessels in the flagship vetting service.
Build a real-time system to predict risk rating in seconds, whereas the earlier system used to take hours to calculate.
Fast and more reliable results in a short time.
Provide auto-scalability to reduce batch processing time.

Our Solution

We entered the picture as a data analytics solution provider. Our team focused on the client’s data warehousing, migration, and reporting systems. New data management structures and practices were necessary to help the client further reduce the time it took to calculate risk ratings, relieving the strain on its digital infrastructure.

Solution Impact

Reduced risk calculation

time from minutes to fractions of a second

Real-time data available

for business reports and decisions

Built star schema architecture

for fact, dimension, and marts

Our Approach

We built a data warehouse for the client using Snowflake which provides instant computation and out-of-the-box integration. It captures every activity within the application and sends it out to the warehouse for risk rating calculation and real-time history updates for the shipping vessels.

Transforming the Medical Device Industry – Leverage Data for Better Business Insights

Read Blog

Migrated legacy warehouse to AWS-integrated modern data warehouse

We planned to include migration as part of our data engineering services and migrated the legacy warehouse to an AWS-integrated modern DWH. We also migrated the SSIS/warehouse to the Snowflake environment.

The client’s legacy warehouse came with landing, staging, enterprise, fact, and dimension structures. However, the warehouse took 24 hours to update data after a job had been executed. The time lag was inefficient and was not feasible in the long run. We targeted to make it as close to real-time as possible. To undertake this approach, we migrated from the SSIS legacy system. It is hosted on Soft Layer to a Snowflake environment integrated with AWS glue. Our action plan retained most of the existing SSIS components, such as schema structure, views, SSIS jobs, and table structures.

Tech Stack

Enabling a cloud-based data warehouse for swift and accurate decision making

The Australian maritime giant’s existing database was on SQL Server. Its poor performance was severely impacting our client’s operational efficiency. Based on our analysis of their multitude of problems, we migrated it to the cloud DWH for quicker availability and better performance. Our team built it on Snowflake and the data pipelines using AWS Glue and Python, as well as Lambda functions.

We achieved a sharp reduction in DWH processing time and thus performed a timely revision of the vessel ratings. This solution could ingest real-time data into the database for use in downstream applications and analytics. Therefore, stakeholders had access to near-real-time information on vessel incidents, events, inspections, and ratings for better business decisions and risk management.

Enabling a cloud-based data warehouse for swift and accurate decision making

Real-time analytical report generation as per user’s needs

The cloud-based warehouse is rebuilt on each event using data in the Event Store, MDM, and other data feeds. This data is then used to generate reports: Cognos Report and Tableau.

After selecting a maritime vessel, they are integrated into the UI and presented to a user. Most reports implemented are tables of text. So far, we have successfully implemented around 20 reports on Cognos. On the other hand, Tableau is used for analytical reports that are provided to Maritime Risk Rating business users. These reports are not integrated into the Qi user interface.

Real-time analytical report generation as per user’s needs

Business Impact

Throughout our six-year relationship with the client, we’ve faced several challenges. Despite the roadblocks, our team developed a data analytics solution that offered breakneck computation speeds and allowed instant integration with the client’s vetting app. It offered auto-scalability and reduced batch-processing time. Secondly, it ensured that the leading risk rating company and its customers could use real-time data to inform their decision-making and draft business reports. Finally, it reduced the cost of infrastructure maintenance.