Statistical Distances#

FairLens allows users to make use of a wide range of statistical distance metrics to measure the difference between the distributions of a variable in two potentially sensitive sub-groups of data. These metrics are available for individual use in the package fairlens.metrics, or they can be called using the stat_distance wrapper function.

Let’s import this method and load in the compas dataset.

In [1]: import pandas as pd

In [2]: import fairlens as fl

In [3]: df = pd.read_csv("../datasets/compas.csv")

In [4]:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20281 entries, 0 to 20280
Data columns (total 22 columns):
 #   Column                   Non-Null Count  Dtype  
---  ------                   --------------  -----  
 0   PersonID                 20281 non-null  int64  
 1   AssessmentID             20281 non-null  int64  
 2   CaseID                   20281 non-null  int64  
 3   Agency                   20281 non-null  object 
 4   LastName                 20281 non-null  object 
 5   FirstName                20281 non-null  object 
 6   MiddleName               5216 non-null   object 
 7   Sex                      20281 non-null  object 
 8   Ethnicity                20281 non-null  object 
 9   DateOfBirth              20281 non-null  object 
 10  ScaleSet                 20281 non-null  object 
 11  AssessmentReason         20281 non-null  object 
 12  Language                 20281 non-null  object 
 13  LegalStatus              20281 non-null  object 
 14  CustodyStatus            20281 non-null  object 
 15  MaritalStatus            20281 non-null  object 
 16  ScreeningDate            20281 non-null  object 
 17  RecSupervisionLevelText  20281 non-null  object 
 18  RawScore                 20281 non-null  float64
 19  DecileScore              20281 non-null  int64  
 20  ScoreText                20245 non-null  object 
 21  AssessmentType           20281 non-null  object 
dtypes: float64(1), int64(4), object(17)
memory usage: 3.4+ MB

Distance metrics from fairlens.metrics can be used by passing two columns of data to their __call__ method, which will return the respective metric or None if it cannot be computed. Different metrics take in different keyword arguments which can be passed to their constructor.

In [5]: target_attr = "RawScore"

In [6]: x = df[df["Ethnicity"] == "African-American"][target_attr]

In [7]: y = df[df["Ethnicity"] == "Caucasian"][target_attr]

In [8]: fl.metrics.KolmogorovSmirnovDistance()(x, y)
Out[8]: 0.26075238442125354

In [9]: _, bin_edges = np.histogram(df[target_attr], bins="auto")

In [10]: fl.metrics.EarthMoversDistance(bin_edges)(x, y)
Out[10]: 0.526609942555413

In [11]: ord = 1

In [12]: fl.metrics.Norm(ord=ord)(x, y)
Out[12]: 0.5231267237105415

The method stat_distance provides a simplified wrapper for distance metrics and allows us to define which sub-groups of data we want to measure the distance between using a simplified dictionary notation or a predicate itself.

In [13]: group1 = {"Ethnicity": ["African-American"]}

In [14]: group2 = df["Ethnicity"] == "Caucasian"

We can now make a call to stat_distance. The parameter mode is indicative of the statistical distance metric, and corresponds to the id function in the classes for the distance metrics. If it is set to “auto”, a suitable distance metric will be chosen depending on the distribution of the target variable.

In [15]: fl.metrics.stat_distance(df, target_attr, group1, group2, mode="auto")
Out[15]: (0.26075238442125354,)

We can see that the distance between the groups is the same as above. stat_distance has chosen KolmogorovSmirnovDistance as the best metric since the target column is continous.

It is possible to get a p-value back with the distance by using the p_value flag.

In [16]: fl.metrics.stat_distance(df, target_attr, group1, group2, mode="auto", p_value=True)
Out[16]: (0.26075238442125354, 6.76556519475455e-241)

Additional parameters are passed to the __init__ function of the distance metric. This can be used to pass keyword arguments such as bin_edges to categorical distance metrics.

In [17]: _, bin_edges = np.histogram(df[target_attr], bins="auto")

In [18]: fl.metrics.stat_distance(df, target_attr, group1, group2, mode="emd", bin_edges=bin_edges)
Out[18]: (0.526609942555413,)