Infrastructure: Apache Spark on Cloudera
Intel IT’s Enterprise Data Platform supports real-time and batch ingestion of structured, semi-structured, and unstructured data. Their data warehouses, together with their data lake, provide data to their visualization and data science platforms. These platforms are used by a variety of dashboards, application frameworks, and custom applications.
As a large IT shop, they continuously look for ways to reduce both operational expenses and infrastructure purchases, while simultaneously improving the user experience. Unfortunately, workload optimization can be disruptive and time consuming. Most optimization efforts require developers to take time away from business applications and product development to rewrite code – and sometimes, the tuning can lead to suboptimal performance.
Given Intel Granulate’s capability to optimize big data platforms with no hands-on development, Intel IT decided to test it in their Cloudera platform. First, an assessor from their Information Security team performed a risk assessment of Granulate. He reviewed several assessments by independent third parties, including Intel Granulate’s SOC 2 Type II Report, the results of penetration testing and remediation actions, and SecurityScorecard’s report. In that report, Intel Granulate earned an “A” rating of 91. The Intel Information Security team assessed that Intel Granulate met or exceeded the pertinent standards.
Intel IT initially deployed Intel Granulate in their Quality Assurance environment to validate no disruption to business operations. The optimization solution ran for about two to three weeks, learning about the platform’s Spark jobs.
Then they deployed the solution to all the nodes in their production environment. Intel Granulate identified optimization opportunities at the job level behind the scenes, with no human involvement. Intel IT then activated the optimization solution and moved to the benchmark phase.
Intel IT gathered metrics for seven days. During this time, Intel Granulate dynamically optimized more than 1000 jobs – equivalent to approximately 42,000 executed Spark jobs/applications. The results were compelling: an average 25% reduction in memory utilization and 44% reduction in CPU utilization. Now Intel IT can run more jobs concurrently without adding memory or compute infrastructure, which will help them avoid capital and operational expenses in the future.
Intel Granulate shows the average memory and CPU overcommitment from April 21-27, 2023
Given the outstanding results Intel IT saw with Intel Granulate, they are planning to continue to run it in their Cloudera platform. In addition, they started a new PoC running Intel Granulate in one of their public cloud environments. They are also exploring the use of the optimization solution in streaming platforms such as Apache Kafka, and log analytics platforms like Elasticsearch.
*Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy
Technology
HQ: Santa Clara, California
Employees: 125,000
Intel IT plays a central role in increasing the value of Intel’s business. The division works at the boundaries of innovation every day, developing data-driven solutions to improve the operations and processes of a global technology leader.
Their deep knowledge and experience as IT professionals enables them to develop and deploy cutting-edge technologies that deliver strong business value to our own and others’ enterprise IT organizations.
