Continuum Analytics: Documentation

# Split-Apply-Combine – Grouping

Grouping operations break a table into pieces and perform some reduction on each piece. Consider the `iris` dataset:

    >>> from blaze import data, by
    >>> from blaze.utils import example
    >>> d = data('sqlite:///%s::iris' % example('iris.db'))
    >>> d
        sepal_length  sepal_width  petal_length  petal_width      species
    0            5.1          3.5           1.4          0.2  Iris-setosa
    1            4.9          3.0           1.4          0.2  Iris-setosa
    2            4.7          3.2           1.3          0.2  Iris-setosa
    3            4.6          3.1           1.5          0.2  Iris-setosa
    4            5.0          3.6           1.4          0.2  Iris-setosa

We find the average petal length, grouped by species:

    >>> by(d.species, avg=d.petal_length.mean())
               species    avg
    0      Iris-setosa  1.462
    1  Iris-versicolor  4.260
    2   Iris-virginica  5.552

Split-apply-combine operations are a concise but powerful way to describe many useful transformations. They are well supported in all backends and are generally efficient.

## Arguments

The `by` function takes one positional argument, the expression on which we group the table, in this case `d.species`, and any number of keyword arguments which define reductions to perform on each group. These must be named and they must be reductions.

    >>> by(grouper, name=reduction, name=reduction, ...)

    >>> by(d.species, minimum=d.petal_length.min(),
    ...               maximum=d.petal_length.max(),
    ...               ratio=d.petal_length.max() - d.petal_length.min())
               species  maximum  minimum  ratio
    0      Iris-setosa      1.9      1.0    0.9
    1  Iris-versicolor      5.1      3.0    2.1
    2   Iris-virginica      6.9      4.5    2.4

## Limitations

This interface is restrictive in two ways when compared to in-memory dataframes like `pandas` or `dplyr`.

1. You must specify both the grouper and the reduction at the same time
2. The “apply” step must be a reduction

These restrictions make it much easier to translate your intent to databases and to efficiently distribute and parallelize your computation.

## Things that you can’t do

So, as an example, you can’t “just group” a table separately from a reduction

    >>> groups = by(mytable.mycolumn)  # Can't do this

You also can’t do non-reducing apply operations (although this could change for some backends with work)

    >>> groups = by(d.A, result=d.B / d.B.max())  # Can't do this