indigoChart Menu

Highlights

A Consumer Defensive company that is a pioneer in the vaping industry faced challenges in their data processing and reporting timelines due to file arrival delays. They needed a way to streamline their operations and raise the general effectiveness of their data handling procedures. By utilizing indigoChart's services, an automation solution was successfully developed to handle these pain points. The following are some noteworthy aspects and outcomes of their journey:

Problem Statement: Current state and pain areas

Facing several challenges with file arrival delays and their data processing workflow:

• File Processing Timeline:This typically required several hours to load and process retail sales data files that were expected to be received from vendors via SFTP early on Monday morning. After that, it took a few extra hours for the data to be made available in Looker for reporting. Any deviation from the predetermined schedules remained undetected until users reported data unavailability.

• Unknown Delays: The planning and implementation of the data processing workflow were impacted by the file arrival delays, which were not discovered until late in the day.

• Inefficiencies: An entire day had to be spent interacting with vendors and loading the data on Monday evening due to file arrival delays, which meant that reporting data was only accessible much later in the week.

Areas of Value Add: Design and implementation of a bespoke File Monitoring System (FMS).

indigoChart’s services brought significant value through the design and implementation of a File Monitoring System (FMS) to address the file arrival delay challenges:

• Continuous Integration Continuous Deployment (CI/CD): The FMS solution was deployed and integrated seamlessly by implementing a CI/CD strategy made possible by Github Actions.

• Configurable and Scalable Architecture: The FMS, deployed within AWS, was designed to be configurable and scalable, allowing for easy additions, updates, or deletions of file types. To ensure flexibility for future loads, it could manage a variety of file scenarios, such as hourly, daily, weekly, or monthly arrivals.

• Leveraging AWS and Snowflake: AWS cloud functions, along with Prefect 2.0 Cloud, were employed to design an architecture that could store and retrieve files efficiently.  Apart from processing data within Snowflake, it was also employed for storing and retrieving files and processes specific metadata along with logging information, used for analysis and process adjustments.

• Workflow Automation and Notifications: The FMS utilized Prefect Automation integrated with email blocks for workflow notifications by invoking JIRA APIs. If files were not received in a timely manner for processing, this allowed for quick notifications to the relevant team and enabled prompt action.

Solution Design for FMS

Impact: Improved processing times

Partnering with indigoChart resulted in several positive impacts on their file arrival and data processing practices:

• Enhanced Configurability: The IT data team of the customer was able to become fully independent by configuring the FMS solution to handle any file scenarios. This workflow automation greatly increased overall productivity and flexibility in managing various file formats.

• Timely Actions and Improved Reporting: Vendors were instantly notified if files were not received on time, allowing speedier processing of the retailer's sales data thanks to the FMS's capacity to detect delays in file deliveries. Decisions were made more quickly, and reports were completed on time as a result.