Automatic re-sizing with ByRow in combine

[pdeffebach]

perhaps for combine and ByRow we should re-size tables

julia> df = DataFrame(a = [1, 2]);

julia> combine(groupby(df, :a), :a => (t -> [100, 200]) => :b)
4×2 DataFrame
 Row │ a      b     
     │ Int64  Int64 
─────┼──────────────
   1 │     1    100
   2 │     1    200
   3 │     2    100
   4 │     2    200

julia> combine(groupby(df, :a), :a => ByRow(t -> [100, 200]) => :b)
2×2 DataFrame
 Row │ a      b          
     │ Int64  Array…     
─────┼───────────────────
   1 │     1  [100, 200]
   2 │     2  [100, 200]

No big deal if not. I actually think this change might not be that hard.

This would be a breaking change so it's fine to wait until much further down the line.

[bkamins]

:a => ByRow(t -> [100, 200]) is an equivalent of (t -> [100, 200]).(df.a) and what you report is exactly expected. Otherwise there would be no way to obtain the result you present in your question if it were asked for.

Actually the hard case is when you have groups having more than one row in them, and then you need something like (I am writing it verbosely for clarity):

julia> df = DataFrame(a = [1, 1]);

julia> combine(groupby(df, :a), :a => (x -> reduce(vcat, (t -> [100, 200]).(x))) => :b)
4×2 DataFrame
 Row │ a      b
     │ Int64  Int64
─────┼──────────────
   1 │     1    100
   2 │     1    200
   3 │     1    100
   4 │     1    200

or use flatten for post processing

[jkrumbiegel]

But as I wrote on Slack, could an alternative be passing something akin to => AsTable in this case, which would do the flattening automatically? Flattening is not hard per se, but it does require to spell out all the new column names again, which could get tedious quickly.

Maybe => Flattened or something like that?

[bkamins]

@jkrumbiegel - we have more space here to show the examples. Can you please give the exact use case you have? I am hesitant to add Flattened option as I feel it would not be needed frequently. But maybe I am wrong - so it would be best to work on a concrete example to work out the best approach.

[jkrumbiegel]

Sure, it is exactly the situation that one has a function returning multiple rows, which is applied ByRow. I think this is quite common, at least I encounter it frequently when cleaning or preparing dataframes for analysis.

You just need a dataframe in which a column contains some kind of complex object, which actually encodes data that belongs to separate rows.

Let me just repeat the example from Slack here so it doesn't get lost:

df = DataFrame(:participant => [1, 2], :blockdescription => ["a,false,b,true", "a,true,b,true"])

function extract_trials(blockdescription)
    temp = map(Iterators.partition(split(blockdescription, ","), 2)) do (condition, success)
        condition, parse(Bool, success)
    end
    (condition = first.(temp), success = last.(temp))
end

@pipe df |>
    groupby(_, :participant) |>
    combine(_, :blockdescription => ByRow(extract_trials) => AsTable)

This gives

2×3 DataFrame
 Row │ participant   condition                    success    
     │ Int64         Array…                       BitVector  
─────┼──────────────────────────────────────────────────────
   1 │           1   SubString{String}["a", "b"]  Bool[0, 1]
   2 │           2   SubString{String}["a", "b"]  Bool[1, 1]

Where it is an extra step of work to repeat the column names and flatten the dataframe via flatten(df, [:condition, :success]) to:

4×3 DataFrame
 Row │ participant   condition    success 
     │ Int64         String       Bool    
─────┼────────────────────────────────────
   1 │           1   a            false
   2 │           1   b            true
   3 │           2   a            true
   4 │           2   b            true

So I think it's not uncommon to have a row-wise computation resulting in several output rows, and there should be a standard idiom for that, instead of having to go in manually and repeating the newly created column names in a flatten command.

A sink descriptor that says "the output of this call consists of multiple rows each time, and all new columns should be flattened".

[bkamins]

Just to clarify - I have a problem with your example that each group contains only one row. Is this on purpose or not? Do you want in general to allow for multiple rows per group?

The question is that ByRow is designed to process things rowwise, so in your case a more natural thing to write for me would be:

julia> df = DataFrame(:participant => [1, 2], :blockdescription => ["a,false,b,true", "a,true,b,true"])
2×2 DataFrame
 Row │ participant  blockdescription
     │ Int64        String
─────┼───────────────────────────────
   1 │           1  a,false,b,true
   2 │           2  a,true,b,true

julia> df.id = axes(df, 1)
Base.OneTo(2)

julia> function extract_trials(blockdescription)
           temp = map(Iterators.partition(split(blockdescription, ","), 2)) do (condition, success)
               condition, parse(Bool, success)
           end
           (condition = first.(temp), success = last.(temp))
       end
extract_trials (generic function with 1 method)

julia> @pipe df |>
           groupby(_, :id) |>
           combine(_, :blockdescription => (x -> extract_trials(only(x))) => AsTable)
4×3 DataFrame
 Row │ id     condition  success
     │ Int64  SubStrin…  Bool
─────┼───────────────────────────
   1 │     1  a            false
   2 │     1  b             true
   3 │     2  a             true
   4 │     2  b             true

the point is that in your pipeline you do not take into account the grouping variable in any way (there is no logical link between rows in a single group), so the question is why is it (i.e. the group-by step) present there at all?

[jkrumbiegel]

It should also work for multiple rows, in this example there is just one. You are right that the example is not very good in that way, that the grouping is equivalent to doing a row-wise transformation. I have to think about it more, whether my problem stems from a missing feature in DataFrames, or my own misconception of the problem here.

So far, I still believe that it's useful to have ByRow return multiple rows, no matter if that's happening with or without groupby.
Of course one can always write the concatenation logic and pass that function without ByRow, but it saves a lot of work not having to write the vector form, and not having to do the concatenation.

[bkamins]

I have to think about it more, whether my problem stems from a missing feature in DataFrames, or my own misconception of the problem here.

To be clear - if we decide this is something that is needed I am OK to add it, just it cannot be added to ByRow, but it must be a separate mechanism (and then the question is if it is common enough to justify its existence).

and not having to do the concatenation.

Indeed this is what combine does by default on GroupedDataFrame by design if it gets a multiple-row output (and I take advantage of this fact in my solution).

However, the question is:

1. do we need a special syntax in DataFrames.jl mini-language (this should not be done lightly)
2. or we can have a lightweight function that does the concatenation and then you can just compose this in your function call (this is what I would prefer)

So to be clear instead of

:col => ByRow(some_fun) => Flatten(:outcol_name)

I would prefer to have

:col => (x -> generic_flatten_function(some_fun.(x))) => :outcol_name

as this is more flexible. The reason is that Flatten() behavior has to be hardcoded in DataFrames.jl, while generic_flatten_fucntion can live on its own independent on DataFrames.jl. This is much better as DataFrames.jl is a core package and generic_flatten_fucntion is a function that different users might want to work differently, so it is better not to hard-code it into the package. The rules in https://dataframes.juliadata.org/latest/man/split_apply_combine/#The-Split-Apply-Combine-Strategy are alredy very complex.

For vectors generic_flatten_function is just reduce(vcat, ...). For NamedTuples there is no such function AFAICT but it can be easily added.

[bkamins]

@quinnj - actually I think TableOperations.joinpartitions could be used for this. Could you please comment what would be the preferred way to use it to convert:

[((a=[1], b=[2]),(a=[3], b=[4])]

into a table with columns :a and :b containing [1,2] and [3,4] respectively?

Thank you!

[pdeffebach]

I still think this is a good idea.

1. a => ByRow(t -> [100, 200]) is an equivalent of (t -> [100, 200]).(df.a) and what you report is exactly expected. Otherwise there would be no way to obtain the result you present in your question if it were asked for.

Without ByRow, you are required to use combine(df, :a => (t -> Ref([100, 200])) to get a vector of vectors. We can make the same requirement with ByRow, making things more consistent.

2. I don't think adding Flatten is a good idea. It's too niche a use to have a special syntax for. However I think flattening the column is the most consistent default

3. The key here is not that each group has one row. The key is that combine re-sizes the data frame while other functions do not. You could have combine(df, :a => ByRow(t -> [100, 200])) and it would produce a data frame with 2x the number of rows as the original.

4. To be clear - if we decide this is something that is needed I am OK to add it, just it cannot be added to ByRow, but it must be a separate mechanism (and then the question is if it is common enough to justify its existence).

I don't fully understand this logic. In my mind, ByRow only guarantees the input is a NamedTuple, but doesn't have to enforce anything about the output.

Nonetheless, I am fine with the current behavior until after 1.0.

[bkamins]

Just to be clear:

combine(_, :blockdescription => ByRow(extract_trials) => AsTable)

is exactly the same as

combine(_, :blockdescription => (x -> extract_trials.(x)) => AsTable)

and this will not change. (DataFrames.jl is not at the stage of design any more 😄)

So the discussion is how should:

combine(_, :blockdescription => (x -> extract_trials.(x)) => AsTable)

be treated. And again - this is already settled - we have a set of rules how "a vector of something" is processed and this will not change.

What we could do is either:

1. add new "verbs" or "nouns" in our mini-language to change how "a vector of something" is processed
2. add utility functions that process "a vector of something" into "a vector of something else" (without affecting the mini-language)

And I am just saying that I prefer option 2. over option 1., as the mini-language should be as minimal as possible (there are already requests to add AsVector, RowNumber, proprow). The difference between these three things and the thing that is requested here is that "flattening" can be done without using the mini-language, whereas AsVector, RowNumber, proprow have to be built-in, as otherwise the things that they are intended to produce are very difficult to do without them.

The key is that combine re-sizes the data frame while other functions do not.

This is an orthogonal issue. The thing that is different is that select and transform are target shape aware (and enforce this shape) while combine accepts any output shape.
In particular this means that ByRow must work the same way both for combine and select because ByRow definition is independent from which function calls it. ByRow can be even used without select or combine present at all, e.g.:

julia> ByRow(sin)(1:10)
10-element Array{Float64,1}:
  0.8414709848078965
  0.9092974268256817
  0.1411200080598672
 -0.7568024953079282
 -0.9589242746631385
 -0.27941549819892586
  0.6569865987187891
  0.9893582466233818
  0.4121184852417566
 -0.5440211108893698

and it is essentially a broadcasting operation:

julia> @code_warntype ByRow(sin)(1:10)
Variables
  f::Core.Compiler.Const(ByRow{typeof(sin)}(sin), false)
  cols::Tuple{UnitRange{Int64}}

Body::Array{Float64,1}
1 ─ %1 = Base.getproperty(f, :fun)::Core.Compiler.Const(sin, false)
│   %2 = Core.tuple(%1)::Core.Compiler.Const((sin,), false)
│   %3 = Core._apply_iterate(Base.iterate, Base.broadcasted, %2, cols)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(sin),Tuple{UnitRange{Int64}}}
│   %4 = Base.materialize(%3)::Array{Float64,1}
└──      return %4

julia> @code_warntype sin.(1:10)
Variables
  #self#::Core.Compiler.Const(var"##dotfunction#253#1"(), false)
  x1::UnitRange{Int64}

Body::Array{Float64,1}
1 ─ %1 = Base.broadcasted(Main.sin, x1)::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(sin),Tuple{UnitRange{Int64}}}
│   %2 = Base.materialize(%1)::Array{Float64,1}
└──      return %2

Is this now clearer what I want to say?

[bkamins]

Just to be clearer what I want to say is that one can write:

julia> df = DataFrame(a = [1, 2]);

julia> flattenme(x) = reduce(vcat, x)
flattenme (generic function with 1 method)

julia> combine(groupby(df, :a), :a => flattenme∘ByRow(t -> [100, 200]) => :b)
4×2 DataFrame
 Row │ a      b
     │ Int64  Int64
─────┼──────────────
   1 │     1    100
   2 │     1    200
   3 │     2    100
   4 │     2    200

and flattenme is not a part of the mini language, but gets the job done. Exactly the same way how skipmissing works for skipping missing values. We do not have to have skipmissing as a part of the mini-language to use it.