Infrastructure: Azure Databricks
With business growth often comes increasing, hard-to-manage costs. For Insight, its Databricks consumption reached 10% growth a month, with its largest all-purpose cluster comprising approximately 220 average cores or 2,640 DBUs per day, and scaling up to 17 nodes.
The IT consulting firm was already leveraging several optimization solutions and best practices, including cluster rightsizing, spot instances, and configuration tuning, but the Databricks costs continued to grow, sparking the need to look for a way to contain cloud costs without disrupting the exciting, consistent expansion of business.
At an annual, internal event, a group of Insight teammates learned about Intel Granulate as a solution to optimize their cloud spend through application performance optimization. Soon after, Insight and Intel began an engagement to deploy a pilot of Intel Granulate on a Databricks cluster to see if it could help reduce spend.
Insight required the solution to be simple to deploy and necessitate minimal engineering efforts to maintain. They also needed the optimization to be effective and complementary to existing optimization efforts.
Insight decided to try out the Intel Granulate Databricks optimizations by applying them on one cluster to start, with plans to expand across other Azure workloads if successful.
Upon activation, Intel Granulate identified patterns of consumption and determined areas of inefficiency in cloud workloads. Enterprise-ready, a POC was developed in only one week, allowing the project to move forward at an accelerated pace, with no code changes needed.
The pilot proved to be successful, with several inefficiencies autonomously identified and optimized. This allowed Insight to reduce the average number of Databricks cores per hour required for its workloads by 21%, from 177.94 to 140.24.
Most importantly, the more efficient cloud distribution led to a 28% reduction in costs from cloud use and downstream costs like storage, networking, and adjacent 3rd party SaaS applications. With these savings, Insight is positioned to accelerate the remaining cloud usage while still lowering overall spend.
21% Reduction in Cores in Primary Databricks Cluster
HQ: Arizona, USA
Daily DBUs: 2,640
Insight Enterprises, Inc. is a Fortune 500 solutions integrator with more than 11,500 teammates worldwide helping organizations accelerate their digital journey to modernize their business and maximize the value of technology.
Insight enables secure, end-to-end transformation and meets the needs of their clients through a comprehensive portfolio of solutions and 33+ years of broad IT expertise.