Column safety

[maxigit]

Is your feature request related to a problem? Please describe.
At the moment the type of a DataFrame doesn't provide any information regarding the columns it holds (names and/or type) and there is no way to verify as compile time that a column exists and has the right type.

I've been using Haskell and R do forecasting and I am planning to rewrite some of the R bit either to python (using panda dataframe) or Haskell. My experience so far has been than R was brilliant for prototyping, the lack of type safety wasn't a problem at first as I could get instant feedback on what was wrong. However finding problem later on ( I mean years later, I still use R script that I wrote in 2012) was more tricky mainly to the fact that I couldn't know what columns was in a dataframe.

A typical example would be reading 2 or 3 csv, joins them and the script crashing because for some reasons, one of the csv was empty, no column so later on in the pipeline one of the column has been lost and there is no easy way to find where the column should come from.

Unless I am missing something, the DataFrame in this package will have the same issue. We have indeed some type safety but not the one needed to solve this issue.

And the other and , plain Haskell with vector or list of records provides this safety.
Join between two vector of records can result for example as a These a b.
It becomes however quickly cumbersome and it is easy to have performance issue.

(I even created a package metamorphosis allowing using to Template Haskell, to merge records to a new one, but that also is a bit heavy).

Describe the solution you'd like
I don't have a fully fledged solution but I've thought of this problem quite a bit and I have a few idea.
Most of them would be being able to convert a typeless DataFrame to something with more information.

For example the type of the column can be added as a (type level) list of types as in DataFrame '[Int, Double, Double].
In a similar way, the name could be added using symbol as in DataFram '[("quantity", Int), ("price", Double)].

Another idea would be to add HasField constraints. It should then look like (HasField "quantity" d Int , HasField "price" d Double, d ~ DataFrame) => d

The most promising would to be to have something similar to justified-container where you can be given "proof" that the DataFrame as the columns you require.

Describe alternatives you've considered
Being able to parametrized (or convert) a DataFrame to something indexed by a record.
(Maybe like the Frame package). One of the problem here is that when doing join between to DataFrame, we need a 3 records (the union of both records joined). This could be made easier if Haskell supported row polymorphism. Maybe an alternative would be to add something in GHC to support it.

[mchav]

This library intentionally doesn’t fully encode the entire dataframe schema (names + types) in the DataFrame type. Libraries like Frames and javelin can give you stronger compile-time guarantees, but the tradeoff is that a lot of operations that are common in exploratory/interactive dataframe work become heavier: schema evolution is more cumbersome, and operations like pivot/melt/transpose (or anything that changes columns based on data) become awkward or require more type-level machinery. One of the goals here is to keep “time to first result/plot” low and keep schema-changing operations ergonomic.

That said, I agree with the underlying need: you want earlier failure + better localization when the schema isn’t what you think. The library’s approach is to provide some safety via a typed expression/transform DSL over an untyped dataframe (that approach is described here), but it won’t catch every “column exists and has type X” issue at compile time and still requires the programmer to keep some of the mental state of what goes where in their head.

Practically, the best pattern today is:

• do exploratory transforms in the dataframe DSL,
• assert schema at boundaries, then either do some computation or marshal into typed Haskell records for the parts that benefit most from full type safety.

For example the following code in Python gets all terminals' distance from a customer then return available terminals as a list:

def get_list_terminals_within_radius(customer_profile, x_y_terminals, r):
    x_y_customer = customer_profile[['x_customer_id','y_customer_id']].values.astype(float)
    squared_diff_x_y = np.square(x_y_customer - x_y_terminals)
    dist_x_y = np.sqrt(np.sum(squared_diff_x_y, axis=1))
    available_terminals = list(np.where(dist_x_y<r)[0])
    return available_terminals

The equivalent in Haskell would be:

import DataFrame.Monad

getListTerminalsWithinRadius :: (Double, Double) -> DataFrame -> Double -> [Int]
getListTerminalsWithinRadius (customerX, customerY) terminals radius =
    let
        dist terminalX terminalY = sqrt ((customerX - terminalX) ^ 2 + (customerY - terminalY) ^ 2)
        availableTerminals = evalFrameM terminals $ do
                                 distanceToTerminal <- deriveM "dist" (F.lift2 dist x_terminal_id y_terminal_id)
                                 filterWhereM (distanceToTerminal .< F.lit radius)
                                 inspectM (D.columnAsList terminal_id)
    in availableTerminals

You save yourself from a class of Pandas bugs but you don't really get proofs about your data + transformations. In general that's a hard problem to solve cause data is messy.

[maxigit]

The library’s approach is to provide some safety via a typed expression/transform DSL over an untyped dataframe (that approach is described here)

I discovered the FrameM monad just after posting this issue and it addresses indeed most of my concerns. I've rbeen ereading the dataframe doc (much have been added since I first saw it) and I must admit I am actually quite impressed.

Libraries like Frames and javelin can give you stronger compile-time guarantees, but the tradeoff is that a lot of operations that are common in exploratory/interactive dataframe work become heavier

I agree that at the moment the tradeoff of such library is too heavy (whic is why I am not using them) and some vector of typable (as I understand DataFrame roughly is) makes perfect sense. I am not talking about redesigning DataFrame but more add somehow a layer to add schema safery (a bit like FrameM does but at the type level).

For example, I do forecast, my dataframes deals with x, y y_hat (the forecast) and I want to compute some error errors. I want to try different models and compare them.

Ideally I would have some functions (let's use a fancy record syntax to express the schema

   readForecast :: FilePath -> DF {x :: Date, y :: Double}
   readActual :: FilePath -> DF { x :: Date, y_hat :: Double }
   foreceCast :: DF {x :: Date, y :: Double } -> { x :: Date, y_hat :: Double}

   computeError :: DF { x:: Date, y :: Double } -> DF { x :: Date, y_hat :: Double} -> DF {x :: Date, y :: Double, y_hat :: Double, error :: Double}

Without schema information it becomes

   readForecast :: FilePath -> DataFrame
   readActual :: FilePath -> DataFrame
   foreceCast :: DataFrame-> DataFrame
   computeError :: DataFrame-> DataFrame-> DataFrame

All the schema information is gone. At that point, there is no much difference between using panda or DataFrame.

You save yourself from a class of Pandas bugs
Yes and no. You seem to focus on EDA, using ghci or jupyter-book.
My experience with EDA is that when I do something wrong (typo, type mismatch etc ...) whether it is a bug (runtime error) or a compile error (statically deduced at compile time) doesn't really make a difference : I do some modification, see what happen, it either work or not, I fix and carry on. The type of error message is irrelevant (even though usually runtime errors are actually easier to understand that type error).

If it is to do pure EDA (without planning to "freeze" it later into something more robust) I'm not sure there is any benefit using Haskell, vs something like pandas , which works out of the box and have thousands of maintainers.

In general that's a hard problem to solve cause data is messy.

I think something can be done about it, something built on top of DataFrame, maybe involving TH or even LiquidHaskell, but it is not your priority for now.

I'll give a try to DataFrame, see how it compares with pandas and its ecosystem and see if the extra type safety is at the same time enough and worth it.
I'll give you my feedback and my though about how to improve things if you are interested.