Lernen Selecting, Filtering, and Renaming Columns

Swipe um das Menü anzuzeigen

The three most common DataFrame operations are selecting the columns you need, filtering rows by condition, and renaming columns for clarity. These map directly to SQL SELECT, WHERE, and AS.

Selecting Columns


              1234567891011121314151617
            
import urllib.request
from pyspark.sql import SparkSession

urllib.request.urlretrieve(
    "https://staging-content-media-cdn.codefinity.com/courses/aa80ac56-0d50-49e8-9231-2c2374cd3e9d/flights.csv",
    "flights.csv"
)

spark = SparkSession.builder \
    .appName("SelectFilterRename") \
    .master("local[*]") \
    .getOrCreate()

flights_df = spark.read.csv("flights.csv", header=True, inferSchema=True)

# Selecting specific columns
flights_df.select("AIRLINE", "ORIGIN_AIRPORT", "DESTINATION_AIRPORT", "DEPARTURE_DELAY").show(5)

Filtering Rows


              1234567891011
            
from pyspark.sql.functions import col

# Filtering flights with arrival delay over 60 minutes
delayed_df = flights_df.filter(col("ARRIVAL_DELAY") > 60)

# Combining conditions
long_delayed_df = flights_df.filter(
    (col("ARRIVAL_DELAY") > 60) & (col("DISTANCE") > 1000)
)

long_delayed_df.select("AIRLINE", "ORIGIN_AIRPORT", "DESTINATION_AIRPORT", "ARRIVAL_DELAY", "DISTANCE").show(5)

Use col() from pyspark.sql.functions for conditions – it is more explicit and avoids ambiguity when working with multiple DataFrames.

Renaming Columns


              1234567
            
# Renaming columns for readability
renamed_df = flights_df \
    .withColumnRenamed("ORIGIN_AIRPORT", "FROM") \
    .withColumnRenamed("DESTINATION_AIRPORT", "TO") \
    .withColumnRenamed("ARRIVAL_DELAY", "DELAY_MINUTES")

renamed_df.select("AIRLINE", "FROM", "TO", "DELAY_MINUTES").show(5)

Adding a Derived Column


              123456789
            
from pyspark.sql.functions import col

# Adding a boolean column indicating whether the flight was significantly delayed
flights_df = flights_df.withColumn(
    "IS_DELAYED",
    col("ARRIVAL_DELAY") > 15
)

flights_df.select("AIRLINE", "ARRIVAL_DELAY", "IS_DELAYED").show(5)

War alles klar?

Danke für Ihr Feedback!

Abschnitt 1. Kapitel 9

Fragen Sie AI

Fragen Sie alles oder probieren Sie eine der vorgeschlagenen Fragen, um unser Gespräch zu beginnen

Abschnitt 1. Kapitel 9