Welcome to Easy SQL’s documentation!

Easy SQL is built to ease the data ETL development process. With Easy SQL, you can develop your ETL in SQL in an imperative way.

It defines a few simple syntax on top of standard SQL, with which SQL could be executed one by one. Easy SQL also provides a processor to handle all the new syntax.

Since this is SQL agnostic, any SQL engine could be plugged-in as a backend. There are built-in supported for several popular SQL engines, including SparkSQL, PostgreSQL, Clickhouse, Aliyun Maxcompute, Google BigQuery. More will be added in the near future.

Contents

• Easy SQL
  • Background
    • Why imperative SQL
  • Design principal
  • Language features in Easy SQL
• Build and install Easy SQL
  • Install Easy SQL
  • Building Easy SQL
• Quick start
  • First ETL with Easy SQL
    • For spark backend
    • For postgres backend:
    • For clickhouse backend:
    • For flink backend:
    • For other backends:
  • Run ETL in your code
  • ETL Language support
• Syntax
  • Language features in Easy SQL
  • Syntax in Easy SQL
    • The imperative structure
    • Variables
    • Temporary tables
    • Function calls
    • Control execution flow
    • Write data
    • Templates used to reuse code
    • Include other ETL code snippets
    • Debugging support
    • Debugger interface
    • Write data
    • List variables
    • SQL actions
• Debug ETL
  • Start to debug
  • Debuger API
• Test ETL
  • ETL testing challenges
  • Testing ETL in Easy SQL
    • Test suit
    • Test case
  • Run test
  • Run test programmatically
• Linter
  • Command Line Interface
  • Programmatical usage
• Functions
  • Use functions in ETL
    • Implement your customized function
    • Use function in ETL
    • Register functions programmatically
  • Builtin and operator functions
  • Functions implemented in Easy SQL
    • Functions for spark backend
      • Partition functions
      • Column functions
      • Cache functions
      • Parallelism functions
      • IO functions
      • Alert functions
      • Table functions
      • Model functions
      • Analytics functions
      • Lang functions
      • Test functions
    • Functions for rdb backend (PostgreSQL Clickhouse BigQuery)
      • Partition functions
      • Column functions
      • Alert functions
      • Table functions
      • Model functions
      • IO functions
      • Analytics functions
      • Test functions
• UDFs
  • For spark backend
    • Register and use scala UDF/UDAF
    • Register and use python UDF
  • For flink backend
    • Register and use scala UDF
    • Register and use python UDF
  • For other backends
  • Register and use UDF programmatically
  • UDF reference
    • Spark UDFs
    • PostgreSQL UDFs
    • Clickhouse UDFs
• Variables
  • Variables to control data saving
  • Variables to control execution behaviour
  • Variables for function calling
• Flink backend
  • ETL mode
  • Connector configuration
  • Register catalogs and tables
  • Job configuration
  • prepare-sql directive
  • Cannot create output table automatically
• API Reference
  • ingest_funcs
    • Module Contents
      • Functions
        • ingest_cdc_pg
  • ingest_hudi_funcs
    • Module Contents
      • Functions
        • read_hudi
        • read_kafka
        • write_hudi
  • easy_sql
    • Subpackages
      • easy_sql.cli
        • Submodules
          • easy_sql.cli.backend_processor
      • easy_sql.sql_linter
        • Subpackages
          • easy_sql.sql_linter.rules
        • Submodules
          • easy_sql.sql_linter.sql_linter
          • easy_sql.sql_linter.sql_linter_cli
          • easy_sql.sql_linter.sql_linter_reportor
      • easy_sql.sql_processor
        • Subpackages
          • easy_sql.sql_processor.backend
        • Submodules
          • easy_sql.sql_processor.common
          • easy_sql.sql_processor.context
          • easy_sql.sql_processor.funcs
          • easy_sql.sql_processor.funcs_common
          • easy_sql.sql_processor.funcs_flink
          • easy_sql.sql_processor.funcs_rdb
          • easy_sql.sql_processor.funcs_spark
          • easy_sql.sql_processor.report
          • easy_sql.sql_processor.sql_processor
          • easy_sql.sql_processor.step
        • Package Contents
          • Classes
      • easy_sql.udf
        • Submodules
          • easy_sql.udf.check
          • easy_sql.udf.udfs
      • easy_sql.utils
        • Submodules
          • easy_sql.utils.db_connection_utils
          • easy_sql.utils.flink_test_cluster
          • easy_sql.utils.io_utils
          • easy_sql.utils.kv
          • easy_sql.utils.object_utils
          • easy_sql.utils.sql_expr
    • Submodules
      • easy_sql.data_process
        • Module Contents
          • Classes
          • Functions
      • easy_sql.local_spark
        • Module Contents
          • Classes
      • easy_sql.logger
        • Module Contents
          • Functions
          • Attributes
      • easy_sql.report
        • Module Contents
          • Classes
      • easy_sql.spark_optimizer
        • Module Contents
          • Classes
          • Functions
      • easy_sql.sql_processor_debugger
        • Module Contents
          • Classes
      • easy_sql.sql_tester
        • Module Contents
          • Classes
          • Attributes
  • sql_config
    • Module Contents
      • Classes
      • Functions
        • EasySqlConfig
          • abs_sql_file_path
          • from_sql
          • get
          • is_batch
          • is_streaming
          • prepare_sql_list
          • tables
          • task_name
        • FlinkBackendConfig
          • flink
          • flink_conf_command_args
          • flink_configurations
          • flink_tables_file_path
        • SparkBackendConfig
          • spark_conf_command_args
          • spark_submit
        • parse_backend
        • parse_vars

Indices and tables

• Index

• Module Index

• Search Page