Optimizing Spark and PySpark for Improved Big Data Performance

Continuous and autonomous optimization for Spark and PySpark workloads empowering more efficient data engineering, data science and machine learning

One optimization solution for all Spark use cases

Achieve faster, more efficient Spark applications with Intel Tiber App-Level Optimization

Batch/streaming data

Data science at scale

SQL analytics

Machine learning

Integrates with all major data storage and infrastructure

Complete more Spark jobs in less time

Intel Tiber App-Level Optimization allows data science, data engineering and data analysis teams to improve Spark and PySpark performance

Spark dynamic allocation

Optimized dynamic allocation and removal of executors based on the job patterns and predictive idle heuristics

JVM execution for Spark

JNI overhead reduction, execution control flow and reflection overhead optimization

Memory arenas optimization

Release of memory space and object sizes to reduce allocation overhead

Crypto & compression acceleration

Leveraging Crypto architecture, accelerators, and instruction sets for operations

Python optimization for PySpark

Automatic profile guided inlining of hot-path functions and optimized code based on each node processor architecture and generation in the cluster

“We saw the effect on the costs right away. After implementing Intel Tiber App-Level Optimization we saw a 50% memory reduction and a 20% CPU reduction, which eventually translated to an 18% cost reduction… Looking forward, we’re going to install Intel Tiber App-Level Optimization on all the other apps that we have”