yayabobiEvery penny counts, right? When optimizing your tech stack, you'll likely look first at the bigger picture to identify software vendors that don't serve their purpose or generate ROI. After that, you can dig down to a granular level and audit platforms like your data warehouse to look for ways to reduce data costs and maximize efficiencies.
This statement is particularly true when it comes to storage and computing costs, as 82% of companies cite cloud computing spending as a major concern. Cloud data warehouse platforms such as Snowflake allow users to scale storage independently, making it ideal for organizations that want to save time and resources on-premise with server setup, maintenance, and support.
In this article, we'll delve deeper into Snowflake and explore 8 surprising ways to reduce costs, to help you gain the most out of this service.
What is Snowflake?
Snowflake is a cloud-based data warehousing platform designed to store and analyze large amounts of data. It uses a unique architecture that separates storage and computing, allowing users to scale their storage and processing independently. Snowflake also offers advanced security features, including end-to-end encryption, multi-factor authentication, and granular access controls, making it a trusted solution for storing and processing sensitive data.
Snowflake uses a multi-cluster, shared data architecture that decouples storage, compute resources, and system services. Snowflake's architecture allows the platform to automatically deliver the optimal set of IO, memory, and CPU resources for each workload and usage scenario. Snowflake offers native support for all semi-structured data formats without compromising completeness, performance, and flexibility. Its VARIANT data type allows users to store semi-structured data records in a native form inside relational tables, allowing schema-less storage options for all JSON, Avro, XML, and Parquet data records.
Key Use Cases for the Snowflake Platform
[
Snowflake's benefits, like performance, speed, storage, accessibility, concurrency, data sharing, availability, and security, make it an ideal choice for modern businesses that need to manage and analyze large volumes of data. Here are three key use cases for Snowflake:
1. Data Warehousing and Analytics
Snowflake is designed to handle large volumes of structured and semi-structured data. Hence, it is an ideal platform for data warehousing and analytics. With its scalable and elastic architecture, users can easily store, query and analyze large data sets, regardless of the complexity or the size. The platform's automatic optimization and partitioning of data enable users to run fast queries even on large and complex datasets.
2. Real-time Data Processing
Snowflake's unique multi-cluster architecture enables users to process data in real-time. With its built-in support for streaming data, you can ingest and process real-time data from various sources, including social media feeds, website clickstream data, and IoT devices. This feature allows businesses to react quickly to changing market conditions or customer needs and gain a competitive advantage.
3. Data Sharing and Collaboration
Snowflake's Secure Data Sharing feature allows you to share database objects like tables, secure views, and external tables with other Snowflake accounts. It's important to note that shared data won't contribute to the other person's monthly data storage costs because the sharing service doesn't copy or transfer actual data between accounts. Instead, you'll only pay for the compute resources used to query the shared data. Snowflake does not require complex data integration or ETL processes for secure communication. This behavior allows businesses to collaborate effectively and share data-driven insights to improve decision-making.
How Much Does Snowflake Cost?
We can divide Snowflake pricing into 4 main categories:
● Compute costs: These costs depend on the time duration your warehouses are running. For example, if you schedule your warehouse to run 5 minutes each hour, you will be only charged for the 5 minutes.
● Storage costs: This is calculated based on the daily average of data stored across database tables, clones, and failover regions.
● Data transfer costs: In Snowflake, you must pay a cost to transfer data from one Snowflake region to another. You will be charged per byte.
● Serverless features costs: If you use Snowflake's serverless ELT add-on features like Snowpipe, you will be charged for actual compute usage since Snowflake manages the warehouses.
However, these costs may change based on the size of the business. For example, SMBs may use fewer computing and storage resources and do not require additional features. On the other hand, large enterprise companies might need more computing, storage, data transfer resources, and advanced features to handle larger volumes of data.
8 Surprising Ways to Reduce Snowflake Costs
1. Change your warehouse size
[
](https://plazagonzalo.medium.com/snowflake-snowpro-certification-data-warehouses-76956b82ab)
Reducing the size of a Snowflake virtual warehouse is an easy solution to reduce Snowflake costs. Different warehouse sizes have different numbers of available nodes, and a query will utilize all the nodes available within the warehouse to execute itself. If a query is small and doesn't use all the available nodes, you might end up paying for unused resources.
It's best to start with a small or medium-sized warehouse and experiment to find the optimal size for your use case. Increasing the warehouse size doesn't always result in better performance, and creating separate warehouses for different functions of the data stack can help optimize costs.
2. Suspend warehouses that are sitting idle
If you are not using your virtual warehouses, you can prevent unnecessary charges by suspending them. You can use Snowflake's automatic feature for suspending inactive warehouses or manually adjust the time it takes for a warehouse to shut down after executing a query. Similarly, you can enable the auto-resumption feature to resume warehouse activity upon receiving queries.
3. Decrease the warehouse's auto-suspend period
One of the most effective ways to reduce Snowflake costs is by decreasing the auto-suspension period for your virtual warehouses. By default, the UI provides a minimum auto-suspend time of 5 minutes, which can be too long if you have a lot of processes running simultaneously.
To optimize cost efficiency, you can manually create your warehouse and set a custom auto-suspend time using the following command:
alter warehouse DBT_WH set auto_suspend=30;
The optimal auto-suspend time is 30 seconds. This duration allows consecutive queries to execute without interruption while minimizing unnecessary downtime and cost.
Once you click the warehouse settings on the UI, you should see an auto-suspend time of 30 seconds instead of 5 minutes. Since every second counts in cost optimization, small steps like adjusting auto-suspend times can make a big difference in the long run.
4. Update the query timeout default value
By default, Snowflake allows a statement to run for 48 hours (or 172,800 seconds) before automatically aborting it. So, even if a query was initiated by mistake and takes an extended period to compute, Snowflake will still charge you for the processing time.
However, you can adjust the query timeout value by updating the STATEMENT_TIMEOUT_IN_SECONDS parameter.
By setting an appropriate statement timeout, you can prevent unnecessary charges and optimize cost-efficiency for your Snowflake usage. You can check out the Snowflake documentation for more info.
5. Using the resource monitor tool
It is essential to establish resource monitors to ensure your Snowflake usage does not exceed your budget. You can configure these monitors to issue notifications and/or automatically cease warehouse operations when consumption levels exceed predetermined thresholds. To avoid reaching your hard limit, you should set a notification threshold. You can set these alerts at an account level within Snowflake platform, which includes all warehouses or individual warehouses. Alternatively, with Aggua's Cost Management & Optimization solution, you can even set notifications and alerts for every single job running in your entire data pipeline. You can better monitor your Snowflake data storage and computing costs in real-time, and ultimately reduce your Snowflake expenditure.
6. Use materialized view
Another effective method to decrease your Snowflake expenses is optimizing the runtime of your Snowflake queries. In Snowflake, materialized views are executed once against all the data in a table. Upon the initial run, it processes all available data but only processes new data added to the table. This is accomplished using INSERT and UPDATE commands.
This approach can be incredibly beneficial since it has the potential to save you a lot of computing costs. Rather than executing the query on all the data, it only runs on new data, which requires less time. It is important to note that materialized views are only suitable for data models that employ simple SELECT statements rather than those that involve joins or aggregation.
7. Avoid loading large files using the copy command
[
](https://docs.snowflake.com/user-guide/data-load-local-file-system)
When you have a large amount of data to migrate or move to the Snowflake data warehouse, you might use massive CSV or semi-structured data files to copy data to the relevant tables on Snowflake.
However, loading heavy files on Snowflake in one go is not advisable. Instead, it is better to break a single heavy file into multiple smaller chunks and load them using the COPY command. Snowflake divides the workload of the COPY command into parallel threads to load multiple files simultaneously. This function reduces the time required to compute the virtual warehouse, resulting in lower Snowflake credit consumption than loading a large file.
8. Use Zero-Copy cloning effectively
One unique feature of Snowflake is the* ability to create clones of databases, tables, and schemas that utilize pointers to live data without requiring additional storage. This feature can significantly reduce storage costs and time to configure the cloned environment.\
However, it's important to note that the storage fees will transfer to the cloned table if you delete the original table. Therefore, deleting the original and cloned tables is the best thing to do if you no longer use them.*
How to Save Up to 30% on Your Snowflake Expenses?
As a bonus tip, you can use an exclusive Data Management solution like Aggua as a layer on top of your Snowflake platform. Their Cost Management & Optimization tool is built from the ground-up to further reduce your Snowflake costs. Aggua allows your organization to track and report in real-time on granular-level data storage and computing cost changes across your entire data pipeline, upstream and downstream, through your preferred tech stack. You can also monitor the price of every query over time and gain visibility over your spending.
