easy_sql.spark_optimizer

Module Contents

Classes

SparkDynamicConfig

Functions

clear_temp_views(spark)
get_spark(app_name, conf = None)

class easy_sql.spark_optimizer.SparkDynamicConfig(max_shuffle_partitions=None, min_shuffle_partitions=None)

Parameters

• max_shuffle_partitions (Optional[int]) –

• min_shuffle_partitions (Optional[int]) –

use_adaptive_shuffle_partitions(self, spark)

Parameters

spark (pyspark.sql.SparkSession) –

Return type

SparkDynamicConfig

use_max_shuffle_partitions(self, spark)

Parameters

spark (pyspark.sql.SparkSession) –

Return type

SparkDynamicConfig

use_min_shuffle_partitions(self, spark)

Parameters

spark (pyspark.sql.SparkSession) –

Return type

SparkDynamicConfig

easy_sql.spark_optimizer.clear_temp_views(spark)

Parameters

spark (pyspark.sql.SparkSession) –

easy_sql.spark_optimizer.get_spark(app_name, conf=None)

Parameters

• app_name (str) –

• conf (Optional[Dict]) –