Processing nested json dynamically in PySpark (databricks)

 Below youtube video provides excellent way to process complex json dynamically in PySpark (databricks)

Corresponding code is in github

https://github.com/raveendratal/ravi_azureadbadf/blob/main/azure_realtime_scenarios/dynamic_json_process.ipynb

How to remove Duplicates in DataFrame using PySpark

 Below ways can be used to remove duplicates from a dataframe in PySpark:

1. distinct
1. df.distinct()
2. df.distinct(["Column1","Column2"])
2. dropDuplicates
1. df.dropDuplicates()
2. df.dropDuplicates(["Column1","Column2"])

How to Sort a Dataframe in PySpark

 Below are different ways to sort a dataframe:

1. sort
1. df.sort(df.ColumnName)
2. df.sort(df.ColumnName.desc())
3. df.sort(df.ColumnName.desc(), df.ColumnName2)
4. df.sort(col("ColumnName"))
2. orderBy
1. df.orderBy(df.ColumnName)
2. df.orderBy(df.ColumnName.desc())
3. df.orderBy(df.ColumnName1.desc(), df.ColumnName2)
4. df.orderBy(col("ColumnName"))

How to filter a DataFrame using PySpark

 Multiple ways to filter dataframe data:

1. filter
1. df.filter(df.ColumnName ==VALUE)
2. df.filter(col("ColumnName") == VALUE)
3. df.filter((col("ColumnName1") == VALUE) | (col("ColumnName2") == VALUE))
4. df.filter((col("ColumnName1") == VALUE) & (col("ColumnName2") == VALUE))
5. df.filter(col("ColumnName") != VALUE)

How to ADD New Columns in DataFrame using PySpark

 Below are different ways to add new columns to dataframe in PySpark:

1. withColumn and lit
1. df.withColumn("NewColumnName", lit("default value for new column"))
2. withColumn and col (Derived column)
1. df.withColumn("NewColumnName", col("Column1") * col("Column2"))
3. select
1. df.select(lit("default column value").alias("NewColumnName"), col("Column1"), col("Column2"))