概念
构建完整的销售数据流水线
学习如何通过加载原始数据、清洗和转换数据、将其存储在 Delta 表中，以及使用 SQL 进行分析，构建端到端的 Lakehouse 流水线。
1name,region,sales 2Alice,East,100 3Bob,West,200 4Alice,East,150 5Charlie,North,300 6David,South,250 7Emma,West,300 8Bob,West,100 9Charlie,North,200
...
Notebook showing uploading CSV file from your machine
清洗并转换原始数据，生成 Silver 层
1from pyspark.sql.functions import col 2 3# Remove duplicates based on key columns 4silver_df = raw_df.dropDuplicates(["name", "region"]) 5 6# Cast sales column to integer 7silver_df = silver_df.withColumn("sales", col("sales").cast("integer")) 8 9silver_df.show()
...
Notebook output displaying the cleaned Silver layer DataFrame
将清洗后的数据集保存为 Gold Delta 表
1silver_df.write.format("delta") \ 2.mode("overwrite") \ 3.saveAsTable("gold_sales")
...
Notebook output confirming Delta table creation
用 SQL 查询计算各区域的销售总额
1%sql 2SELECT region, SUM(sales) AS total_sales 3FROM gold_sales 4GROUP BY region
...
Notebook output showing total sales grouped by region
使用 display 函数可视化聚合后的销售数据
1sales_summary = spark.sql(""" 2SELECT region, SUM(sales) AS total_sales 3FROM gold_sales 4GROUP BY region 5""") 6 7display(sales_summary)
...
Notebook chart displaying total sales by region