Highlights

Cloud data platforms like Snowflake are essential due to their ability to offer scalable, flexible, and efficient solutions for managing vast amounts of data. Snowflake’s elastic scalability allows businesses to adjust compute and storage resources dynamically, ensuring cost-efficiency and performance even as data volumes grow. It supports multi-cloud environments, enabling organizations to avoid vendor lock-in and easily share data across platforms. With built-in support for both structured and semi-structured data, Snowflake simplifies handling diverse data types and consolidates data storage, analytics, and machine learning workloads on a unified platform. Its ability to auto-scale resources and optimize query performance makes it a powerful tool for real-time analytics and collaboration, while also offering secure data sharing capabilities and a marketplace for data monetization. This flexibility and ease of use is why Snowflake and similar other cloud data platforms occupy a position within an organization’s data ecosystem.

Key Points:

• Elastic scalability with separated compute and storage.

• Multi-cloud support and secure cross-cloud data sharing.

• Handles diverse data types and workloads (structured, semi-structured, and un-structured).

• High performance with auto-scaling, efficient query optimization.

• Real-time collaboration, data sharing, and monetization via a Data Marketplace.

Problem Statement:

While Snowflake offers flexibility and cost-efficiency through its cloud-based, pay-as-you-go model, costs can quickly escalate if not managed efficiently.

Since compute and storage are billed separately, excessive or inefficient use of resources, such as running unnecessary compute clusters or failing to suspend idle warehouses may lead to higher costs.

Moreover, over-provisioning compute power for simple tasks or running large queries without optimization can result in significant expenses.

Storing excessive data without regular clean-up or archiving can also contribute to increased storage costs.

Additionally, egress and ingress costs (moving data between cloud regions or sharing large datasets) can add up.

To control costs, businesses need to monitor usage carefully, optimize queries, and utilize Snowflake's auto-suspend and auto-resume features to avoid wasting resources.

How indigoChart helped?

Managing Cloud or Cloud Platform costs is as important as scaling and optimizing performance. Recently, we worked with a client facing unexpected Snowflake costs due to rapid data growth and inefficient resource utilization on both front of Compute and Storage. Through a detailed analysis of their usage patterns, we identified several areas of improvement, including optimizing query performance, optimizing Storage, adjusting warehouse sizing, and leveraging Snowflake's auto-suspend feature more effectively. By implementing best practices and providing tailored recommendations, we helped the client significantly reduce their Snowflake spend while maintaining high performance and scalability. In this writeup, we’ll walk through the key strategies we used to achieve these savings and ensure cost-efficiency without compromising on data capabilities.

• Snowflake Telemetry Collection: indigoChart’s automated solution enables maintaining history of Snowflake usage data from Snowflake shared databases for accounts for long term storage as Snowflake provides telemetry for only up to 365 days. Our automated solution leverage Snowflake tasks, Stored Procedures, and Configuration Tables loading data at a set schedule.

• Storage Optimization: Based on loaded telemetry, we focused on storage pattern, growth pattern, type of database, schemas and tables created. We then focused on Time Travel and failsafe storage, analyzed the pattern with auto ML functions indicating measures to be taken immediately for optimizing storage costs. We helped reduce storage costs up to 60%.

• Compute Optimization: For Compute optimization, based on telemetry data across the accounts, we realized the spend was more towards CTAS and INSERT OVERWRITES, and transformation SQL which also use to run for more than 2 hours getting queued most of the time on virtual warehouse.

• Bespoke Reports: A few value added reports were developed as below:

  
  

  • Current Usage in Credit and Dollars

  • Top n users

  • Top n queries

  • Usage by Department

  • Usage by Warehouse (Ingestion/Transformation/Consumption)

  • Warehouses need more clusters

  • Warehouses need scale up

  • Warehouses eligible for Acceleration Services

• Storage and Compute Anomaly detection: We have prebuilt solutions developed to forecast Snowflake usage that provides alerts when there is anomaly in usage pattern based on highest threshold set.

Options to implement the solutions and its Key Features

The solution can be implemented in two ways:

a) indigoChart Managed Services option:

• Using an indigoChart managed services option (as indicated by the image below), health reports, action plans, and cost-control measures can be generated for you as long you are able to provide Snowflake "telemetry" data using the Database Sharing feature within Snowflake. This does not require any other organizational data and metadata to be shared.

b) Client deployment option:

• Initial set up requires only a couple of hours.

• With a client deployed end to end solution approach, all reports created within client Snowflake account.

Regardless of the option chosen:

• Costs reduction can be observed from day one post solution deployment.

• Effective implementation of cost show back right down to the query level.

• Snowflake object’s cost is tracked as and when used.

• Clean Snowflake environment with automatic suggestions on dormant objects not required anymore.

Overall, as a result of these efforts, we helped our client realize on an average 60% savings in Storage and around 40% in Compute costs, month over month.