Continuum Analytics: Documentation

# Pandas to Blaze

This page maps pandas constructs to blaze constructs.

## Imports and Construction

    import numpy as np
    import pandas as pd
    from blaze import data, by, join, merge, concat

    # construct a DataFrame
    df = pd.DataFrame({
       'name': ['Alice', 'Bob', 'Joe', 'Bob'],
       'amount': [100, 200, 300, 400],
       'id': [1, 2, 3, 4],
    })

    # put the `df` DataFrame into a Blaze Data object
    df = data(df)

Computation | Pandas | Blaze
--- | --- | ---

Column Arithmetic

    df.amount * 2

    df.amount * 2

Multiple Columns

    df[['id', 'amount']]

    df[['id', 'amount']]

Selection

    df[df.amount > 300]

    df[df.amount > 300]

Group By

    df.groupby('name').amount.mean()
    df.groupby(['name', 'id']).amount.mean()

    by(df.name, amount=df.amount.mean())
    by(merge(df.name, df.id),
       amount=df.amount.mean())

Join

    pd.merge(df, df2, on='name')

    join(df, df2, 'name')

Map

    df.amount.map(lambda x: x + 1)

    df.amount.map(lambda x: x + 1,
                  'int64')

Relabel Columns

    df.rename(columns={'name': 'alias',
                       'amount': 'dollars'})

    df.relabel(name='alias',
               amount='dollars')

Drop duplicates

    df.drop_duplicates()
    df.name.drop_duplicates()

    df.distinct()
    df.name.distinct()

Reductions

    df.amount.mean()
    df.amount.value_counts()

    df.amount.mean()
    df.amount.count_values()

Concatenate

    pd.concat((df, df))

    concat(df, df)

Column Type Information

    df.dtypes
    df.amount.dtype

    df.dshape
    df.amount.dshape

Blaze can simplify and make more readable some common IO tasks that one would want to do with pandas. These examples make use of the odo library. In many cases, blaze will able to handle datasets that can’t fit into main memory, which is something that can’t be easily done with pandas.

    from odo import odo

Operation | Pandas | Blaze
--- | --- | ---

Load directory of CSV files

    df = pd.concat([pd.read_csv(filename)
                    for filename in
                    glob.glob('path/to/*.csv')])

    df = data('path/to/*.csv')

Save result to CSV file

    df[df.amount < 0].to_csv('output.csv')

    odo(df[df.amount < 0],
        'output.csv')

Read from SQL database

    df = pd.read_sql('select * from t', con='sqlite:///db.db')

    df = pd.read_sql('select * from t',
                     con=sa.create_engine('sqlite:///db.db'))

    df = data('sqlite://db.db::t')