Dateframe vs RDD

Dataframe advantage

• schema
• built-in optimization: 包括predictive pushdowns/constant folding/loop unrolling等solution进行自动优化，相较RDD执行效率更高
• compact binary representation: 使用列式存储，optimization across operators(跨算子优化)
• ease of use
• integrated with the rest of spark:

Dataframe operations

• common relational operators
  • projection(select)
  • filter(where)
  • join
  • groupBy
• 通过AST(sbstract syntax tree)组合算子，而不是立即触发计算

Dataframe creation

Filter

Sort

Distinct

With column

Alias

Map

map()

flagmap()

mergge columns

Spark SQL

User-definede functions

---

链接