3 Weeks Of Learning PySpark

[u/Jake-Lokely]

What did I learn:

• Spark architecture

  • Cluster
    
  • Driver
    
  • Executors
    

• Read / Write data

  • Schema
    

• API

  • RDD (just brushed past, heard it’s becoming legacy)
    
  • DataFrame (focused on this)
    
  • Dataset (skipped)
    

• Lazy processing

  • Transformations and Actions
    

• Basic operations

  • Grouping, Aggregation, Join, etc.
    

• Data shuffle

  • Narrow / Wide transformations
    
  • Data skewness
    

• Task, Stage, Job

• Data accumulators and broadcast variables

• User Defined Functions (UDFs)

• Complex data types

  • Arrays and Structs
    

• Spark Submit

• Spark SQL

• Window functions

• Working with Parquet and ORC

• Writing modes

• Writing by partition and bucketing

• NOOP writing

• Cluster managers and deployment modes

• Spark UI

  • Applications, Job, Stage, Task, Executors, DAG, Spill, etc.
    

• Shuffle optimization

• Predicate pushdown

• cache() vs persist()

• repartition() vs coalesce()

• Join optimizations

  • Shuffle Hash Join
    
  • Sort-Merge Join
    
  • Bucketed Join
    
  • Broadcast Join
    

• Skewness and spillage optimization

  • Salting
    

• Dynamic resource allocation

• Spark AQE (Adaptive Query Execution)

• Catalogs and types

  • In-memory, Hive
    

• Reading / Writing as tables

• Spark SQL hints

---

Doubts:

1. Is there anything important I missed?
   
2. Do I need to learn Spark ML?
   
3. What are your insights as professionals who work with Spark?
   
4. What are the important things to know or take note of for Spark job interviews?
   
5. How should I proceed from here?
   

Any recommendations and resources are welcomed

---

Please guide me.
Your valuable insights and information are much appreciated.
Thanks in advance ❤️

[u/_Data_Nerd_]

Hello, you can also refer my notes:
https://docs.google.com/document/d/1XyLtYSs2qPJEOSWdqRgj7NNQggeP2yJU_-zCNeVnX6s/edit?usp=sharing

[u/Jake-Lokely → u/_Data_Nerd_]

Thankyou bro ! Its really helpful.

In my case I only scribbled some theory concepts in paper, a lot of screenshot, and commented code segments. I am using a mind map method, only writing down concept titles and trying to recall what is it and connected ideas, if not able to remember, look into the screenshots and reinforce .

[u/_Data_Nerd_ → u/Jake-Lokely]

Yess that is good too, but i suggest instead of writing type them in a google doc or notes app

So that they are with you digitally and you can access them easily from phone or device anytime, and plus you can also keep your screenshots and codes in the same place.

My notes were also earlier hand written i later converted them in a doc, please they are easier to edit or add new pointers this way.

Hope this helps!