Challenge: Customer Segmentation

You are given a flights dataset as a list of rows. Load it into a DataFrame using createDataFrame and segment airlines by their operational profile using K-Means clustering. Complete all steps and store results in the specified variables:

1. Fill nulls in Delay and Length with 0;
2. Aggregate by Airline to compute:
   • AVG_DELAY – average Delay;
   • AVG_LENGTH – average Length;
   • TOTAL_FLIGHTS – count of flights. Store the result in airline_df;
3. Build a Pipeline with VectorAssembler on ["AVG_DELAY", "AVG_LENGTH", "TOTAL_FLIGHTS"] and KMeans with k=3, seed=42, maxIter=5 – no scaling needed;
4. Fit the pipeline and transform airline_df – store the result in clustered_df;
5. Store the number of rows per cluster as a list of tuples [(cluster_id, count), ...] sorted by cluster_id in cluster_counts.

Print cluster_counts and show clustered_df sorted by prediction.