Dynamic Allocation in Spark
February 18, 2017 Leave a comment
Dynamic Allocation is a spark feature that allows addition or removal of executors launched by the application dynamically to match the workload.
Unlike static allocation of resources (prior to 1.6.0) where spark used to reserve fixed amount of CPU and Memory resources, in Dynamic Allocation its purely based on the workload.
Note: This is the main difference between Spark Standalone Architecture (static allocation) and Spark YARN/Mesos Architecture.
// flag to enable/diable DA feature spark.dynamicAllocation.enabled: true/false // Application starts with this many executors spark.dynamicAllocation.minExecutors: m // Application can increase to this many executors at max spark.dynamicAllocation.maxExecutors: n // For the FIRST Time when this time is hit, number of executors // will be increased spark.dynamicAllocation.sustainedSchedulerBacklogTimeout: x secs // Next Time onwards, whenever this time is hit, it increases // number of executors till maxExecutors is hit spark.dynamicAllocation.schedulerBacklogTimeout: y secs // It releases the executor when it sees no Task is scheduled // on Executor for this time spark.dynamicAllocation.executorIdleTimeout: z secs
There were few issues with dynamic allocation in Streaming Applications because
- Executors may never be idle as they run for every N secs
- Receiver will be running on a Slot/Core inside Executor which is never finished and hence idleTimeout will never be hit
- …
https://issues.apache.org/jira/browse/SPARK-12133
// For streaming applications, disable above switch and enable below one spark.streaming.dynamicAllocation.enabled
Recent Comments