In Apache Spark, DataFrames and Columns are central concepts for handling and processing structured and semi-structured data.

Here’s a detailed overview.

Spark DataFrame

It is conceptually equivalent to the table in a relational database or a DataFrame in Pandas, but additionally - it is distributed across a cluster.

Creation

DataFrames can be created in several ways, including:

• From existing RDDs;
• From structured data files;
• From a table in a database.

For example, here we are creating DataFrame using existing RDD:

python
Copy code

Here - from a JSON file:

python
Copy code

Finally - from a table into database:

python
Copy code

Operations with DataFrame

Here are the most important operations with DataFrame in Spark.

Transformations

• select - used to select specific columns:

python
Copy code

• filter - applies a filter to rows and columns:

python
Copy code

• groupBy - used to aggregate data based on a column value:

python
Copy code

• join - combines two DataFrames based on a common column:

python
Copy code

Actions

• show - displays several first rows of the DataFrame:

python
Copy code

• collect - retrieves entire dataset as a list of rows:

python
Copy code

• count - counts the number of rows into DataFrame:

python
Copy code

Spark Column

Columns are used to refer to the data contained in specific fields of the DataFrame, manipulate and transform it.

As well as in case of DataFrame, conceptually Columns similar to Series object in Pandas library.

Working with Columns

We can select column from DataFrame, using the next expression:

python
Copy code

We can create new columns, for example, using expressions:

python
Copy code

Or using user-defined functions:

python
Copy code

Also we can rename columns:

python
Copy code