当前位置: 首页 > article >正文

Spark eventlog

Eventlog 示例
{
    "Event": "org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart",
    "executionId": 0,
    "rootExecutionId": 0,
    "desc ription": "select round(a, 2), a from double_table",
    "details": "org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMetho    dAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain    (SparkSubmit.scala:1029)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)",
    "physicalPlanDescription": "==   Physical Plan ==\n* Project (3)\n+- * ColumnarToRow (2)\n   +- Scan parquet spark_catalog.default.double_table (1)\n    \n\n(1) Scan parquet spark_catalog.default.double_table\nOutput [1]: [a#0]\nBatched: true\nLocation: InMemoryFileInde    x [file:/home/hadoop/files/double_table]\nReadSchema: struct<a:double>\n\n(2) ColumnarToRow [codegen id : 1]\nInput [1]: [a#0]\n\n(3) Project [codegen id : 1]\nOutput [2]: [round(a#0, 2) AS round(a, 2)#1, a#0]\nInput [1]: [a#0]\n\n",
    " sparkPlanInfo": {
        "nodeName": "WholeStageCodegen (1)",
        "simpleString": "WholeStageCodegen (1)",
        "children": [
            {
                "nodeName": "Project",
                "simpleString": "Project [round(a#0, 2) AS round(a, 2)#1, a#0]",
                "children": [
                    {
                        "nodeName": "ColumnarToRow",
                        "simple String": "ColumnarToRow",
                        "children": [
                            {
                                "nodeName": "InputAdapter",
                                "simpleString": "InputAdapter",
                                "children": [
                                    {
                                        "nodeName": "Scan parquet spark_catalog.default.double_table",
                                        "simpleString": "FileScan parquet spark_catalog.default.double_table    [a#0] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/hadoop/files/d    ouble_table], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:double>",
                                        "children": [

                                        ],
                                        "metadata": {
                                            "Locat ion": "InMemoryFileIndex(1 paths)[file:/home/hadoop/files/double_table]",
                                            "ReadSchema": "struct<a:double>",
                                            "Format": "Par    quet",
                                            "Batched": "true",
                                            "PartitionFilters": "[]",
                                            "PushedFilters": "[]",
                                            "DataFilters": "[]"
                                        },
                                        "metrics": [
                                            {
                                                "name": "number of     files read",
                                                "accumulatorId": 5,
                                                "metricType": "sum"
                                            },
                                            {
                                                "name": "scan time",
                                                "accumulatorId": 4,
                                                "metricType": "timing"
                                            },
                                            {
                                                "nam e": "metadata time",
                                                "accumulatorId": 6,
                                                "metricType": "timing"
                                            },
                                            {
                                                "name": "size of files read",
                                                "accumulatorId": 7,
                                                "metricTyp e": "size"
                                            },
                                            {
                                                "name": "number of output rows",
                                                "accumulatorId": 3,
                                                "metricType": "sum"
                                            }
                                        ]
                                    }
                                ],
                                "metadata": {

                                },
                                "metrics": [

                                ]
                            }
                        ],
                        "met adata": {

                        },
                        "metrics": [
                            {
                                "name": "number of output rows",
                                "accumulatorId": 1,
                                "metricType": "sum"
                            },
                            {
                                "name": "number of input b    atches",
                                "accumulatorId": 2,
                                "metricType": "sum"
                            }
                        ]
                    }
                ],
                "metadata": {

                },
                "metrics": [

                ]
            }
        ],
        "metadata": {

        },
        "metrics": [
            {
                "name": "durat    ion",
                "accumulatorId": 0,
                "metricType": "timing"
            }
        ]
    },
    "time": 1741661558528,
    "modifiedConfigs": {

    },
    "jobTags": [

    ]
}

== Physical Plan ==
* Project (3)
+- * ColumnarToRow (2)
   +- Scan parquet spark_catalog.default.double_table (1)

对应于

==     Physical Plan ==\n* Project (3)\n+- * ColumnarToRow (2)\n   +- Scan parquet spark_catalog.default.double_table (1)\n    \n\n
(1) Scan parquet spark_catalog.default.double_table
Output [1]: [a#0]
Batched: true
Location: InMemoryFileIndex [file:/home/hadoop/files/double_table]
ReadSchema: struct<a:double>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [a#0]

(3) Project [codegen id : 1]
Output [2]: [round(a#0, 2) AS round(a, 2)#1, a#0]
Input [1]: [a#0]

对应于

 "physicalPlanDescription": "==     Physical Plan ==\n* Project (3)\n+- * ColumnarToRow (2)\n   +- Scan parquet spark_catalog.default.double_table (1)\n    \n\n(1) Scan parquet spark_catalog.default.double_table\nOutput [1]: [a#0]\nBatched: true\nLocation: InMemoryFileIndex [file:/home/hadoop/files/double_table]\nReadSchema: struct<a:double>\n\n(2) ColumnarToRow [codegen id : 1]\nInput [1]: [a#0]\n\n(3) Project [codegen id : 1]\nOutput [2]: [round(a#0, 2) AS round(a, 2)#1, a#0]\nInput [1]: [a#0]\n\n",

http://www.kler.cn/a/587587.html

相关文章:

  • [文献阅读] 可变形卷积DCN - Deformable Convolutional Networks
  • 服务性能防腐体系:基于自动化压测的熔断机制
  • 【软考-架构】3.4、数据库新技术-SQL语言
  • 基于牛优化( OX Optimizer,OX)算法的多个无人机协同路径规划(可以自定义无人机数量及起始点),MATLAB代码
  • 【eNSP实战】将路由器配置为DHCP服务器
  • 数据库原理10
  • 锂电池保护板测试仪:守护能源安全的科技卫士
  • 【STM32】USART串口收发HEX数据包收发文本数据包
  • 力扣 11.盛水最多的容器(双指针)
  • QT核心类:基础类、GUI类、多媒体与图表、网络与数据库
  • 游戏引擎学习第160天
  • 【GPT入门】第21课 langchain核心组件
  • vscode--工作区和相对路径
  • 【Linux内核系列】:文件系统收尾以及软硬链接详解
  • 在windows上通过idea搭建doris fe的开发环境(快速成功版)
  • Redis 数据结构及使用场景介绍
  • Linux练级宝典->动态库和静态库
  • Vue 3 vs Vue 2:深入解析从性能优化到源码层面的进化
  • 深入React Redux:原理剖析与高效实践指南
  • Sequelize:Node.js 项目中数据库管理的 “秘密武器”