Mixing Languages with Magic Commands

One of the most powerful features of the Databricks notebook is its polyglot nature. You are not forced to choose between being a "Python person" or a "SQL person." Instead, you use the best tool for the specific step of your project. This is made possible through Magic Commands.

The Default Language vs. Magic Commands

Every notebook has a "Default Language" which you chose when you created the file. You can see this language (e.g., Python) next to the notebook name at the top of the screen. In a Python notebook, every cell assumes it is Python unless you tell it otherwise. To override the default for a single cell, you start the cell with a Magic Command.

Switching to SQL with %sql

We have already touched upon %sql. When you are in a Python-heavy notebook, you often want to quickly check the contents of a table using a standard SQL query.

%sql
SELECT count(*) FROM main.default.sample_sales_records

By placing %sql at the very top, the Databricks cluster ignores the Python interpreter for that cell and uses the SQL engine instead. This allows you to perform data exploration with the speed of SQL and then move to the next cell to continue your Python logic.

Documentation with %md (Markdown)

Data work is collaborative, which means your code needs to be readable by others. The %md magic command allows you to write Markdown, which is a simple way to create formatted text, headers, and bullet points.

When you run a cell starting with %md, it doesn't execute code; it renders the text as a professional-looking document.

%md
# Project: Diamonds
This notebook explores the **Diamonds Sales Data** from two years ago.
* Step 1: Data Cleaning
* Step 2: Aggregation

Running Python in a SQL Notebook with %python

The reverse is also true. If you are working in a SQL-default notebook, you can use %python to perform a task that SQL isn't designed for, such as a complex loop or a machine learning prediction.

Other Useful Magics

Beyond switching languages, there are magics for interacting with the environment:

• %run: Used to run another notebook from within your current notebook. This is helpful for importing helper functions or shared variables:

• %pip: Used to install specific Python libraries directly into your cluster's session without leaving the notebook:

By mastering these commands, your notebooks transition from being just "code files" into becoming comprehensive, well-documented data stories.