Parsing and Storing Blockchain Data

When working with blockchain networks, you often encounter raw data in formats like JSON or CSV. JSON (JavaScript Object Notation) is the most common format for blockchain transaction data, as it is lightweight, human-readable, and easily parsed by Python. CSV (Comma-Separated Values) is also widely used when exporting or sharing structured tabular data, such as transaction logs or address balances.

Parsing blockchain data involves reading the raw data, extracting relevant fields, and converting it into a structured format suitable for analysis. Strategies for parsing include using Python's built-in json library to load JSON data and tools like csv or pandas for handling CSV files. You need to identify which fields are important, such as transaction hashes, sender and receiver addresses, block numbers, and values transferred. Often, blockchain data contains nested structures, so you must carefully navigate the hierarchy to extract the necessary information.

import json

# Example raw blockchain transaction data (as JSON string)
raw_json = '''
[
    {
        "hash": "0xabc123",
        "from": "0x111...",
        "to": "0x222...",
        "value": "1000000000000000000",
        "blockNumber": 1234567,
        "timestamp": 1717000000
    },
    {
        "hash": "0xdef456",
        "from": "0x333...",
        "to": "0x444...",
        "value": "500000000000000000",
        "blockNumber": 1234568,
        "timestamp": 1717000600
    }
]
'''

# Parse the JSON string
transactions = json.loads(raw_json)

# Convert to a structured format (list of dicts with selected fields)
parsed_data = []
for tx in transactions:
    parsed_data.append({
        "hash": tx["hash"],
        "from": tx["from"],
        "to": tx["to"],
        "value_eth": int(tx["value"]) / 10**18,  # Convert from Wei to Ether
        "block": tx["blockNumber"],
        "timestamp": tx["timestamp"]
    })

# Display structured data
for entry in parsed_data:
    print(entry)

Once you have parsed the raw blockchain data into a structured format, you may need to clean and transform it before analysis. Data cleaning involves removing duplicates, handling missing or malformed values, and standardizing formats (such as converting timestamps or normalizing address case). Transformation steps might include converting values from Wei to Ether, extracting date and time from timestamps, or creating new fields that summarize or categorize transactions.

By ensuring your data is clean and well-structured, you make downstream analysis more reliable and efficient. The parsed data structure—such as a list of dictionaries with keys like hash, from, to, value_eth, block, and timestamp—is ideal for further processing or exporting to a format like CSV. This allows you to leverage Python's data analysis tools and easily share results with others.

import csv

# Assume parsed_data is the cleaned and transformed list of transaction dicts
parsed_data = [
    {
        "hash": "0xabc123",
        "from": "0x111...",
        "to": "0x222...",
        "value_eth": 1.0,
        "block": 1234567,
        "timestamp": 1717000000
    },
    {
        "hash": "0xdef456",
        "from": "0x333...",
        "to": "0x444...",
        "value_eth": 0.5,
        "block": 1234568,
        "timestamp": 1717000600
    }
]

# Save to CSV file
with open("transactions.csv", "w", newline="") as csvfile:
    fieldnames = ["hash", "from", "to", "value_eth", "block", "timestamp"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for tx in parsed_data:
        writer.writerow(tx)