Hierarchical imbalanced data generation
The purpose of this page is to continue the exploration of More precise imbalanced data generation. That post explained how to divide a binary classification data set into two subsets:
- one balanced subset with 50% positive, 50% negative labels.
- one imbalanced subset with an arbitrary proportion of negative labels,
p_neg. - such that the two subsets have the same number of samples.
That setup can be used for comparing imbalanced versus balanced training, in the context of our recently proposed SOAK algorithm, Hocking et al, Comp Stat Data Anal 2026. However that method does not allow for controlled comparison of training on different imbalance ratios, which would be interesting to explore how much class imbalance a learning algorithm can handle. That would require a hierarchical definition of imbalanced subsets, which was discussed in another previous blog (but without control for sample size). In this blog we explain a method which has both properties:
- subsets have equal sample sizes,
- and different imbalance proportions are hierarchical.
Both properties are desirable, to ensure controlled comparison. If we see a difference in test error rates, we want to be sure that the effect is due to the class imbalance, and not some other difference between train sets (like number of samples or which samples were used).
Problem with previous method
We are creating machine learning algorithms for imbalanced classification problems.
We would like to know if these algorithms work well for training on imbalanced data, and predicting on balanced data, and vice versa.
We therefore would like to create different subsets for testing.
Assume we start with the higgs data, binary classification with this many data in each class:
Tpos=5829123L
Tneg=5170877L
Below are the target proportions of the negative class:
(Target_prop <- 10^seq(-1, -5))
## [1] 1e-01 1e-02 1e-03 1e-04 1e-05
Below we repeat the calculations in the previous post:
library(data.table)
compute_target_counts <- function(p_neg, Tpos, Tneg){
n_pos_max <- 2*Tpos*(1-p_neg)/(3-2*p_neg)
n_pos_max_neg <- n_pos_max*p_neg/(1-p_neg)
n_pos_max_N <- (n_pos_max_neg+n_pos_max)/2
n_pos_max_neg_extra <- Tneg-n_pos_max_neg-n_pos_max_N
n_neg_max <- 2*Tneg*p_neg/(1+2*p_neg)
n_neg_max_pos <- n_neg_max*(1-p_neg)/p_neg
n_neg_max_N <- (n_neg_max_pos+n_neg_max)/2
##n_neg_max_pos_extra <- Tpos-n_neg_max_pos-n_neg_max_N
N_pos_neg <- as.integer(floor(
ifelse(n_pos_max_neg_extra<0, n_neg_max_N, n_pos_max_N)))
n_small <- as.integer(round(ifelse(
p_neg<0.5,
ifelse(n_pos_max_neg_extra<0, n_neg_max, n_pos_max_neg),
ifelse(n_pos_max_neg_extra<0, n_neg_max_pos, n_pos_max))))
n_large <- 2L*N_pos_neg-n_small
data.table(
p_neg,
N_pos_neg,
n_pos=ifelse(p_neg<0.5, n_large, n_small),
n_neg=ifelse(p_neg<0.5, n_small, n_large)
)[, let(
pos = n_pos + N_pos_neg,
neg = n_neg + N_pos_neg,
check_prop = n_neg/(n_neg+n_pos)
)][, let(
prop_diff=check_prop-p_neg,
extra_imb=n_pos+n_neg - 2L*N_pos_neg,
unused_neg=Tneg-neg,
unused_pos=Tpos-pos
)][]
}
p_neg <- sort(c(Target_prop, 1-Target_prop), decreasing = TRUE)
(prev_dt <- compute_target_counts(p_neg, Tpos, Tneg))
## p_neg N_pos_neg n_pos n_neg pos neg check_prop prop_diff extra_imb unused_neg unused_pos
## <num> <int> <int> <int> <int> <int> <num> <num> <int> <int> <int>
## 1: 0.99999 1723637 34 3447240 1723671 5170877 9.999901e-01 1.371344e-07 0 0 4105452
## 2: 0.99990 1723740 345 3447135 1724085 5170875 9.998999e-01 -7.309687e-08 0 2 4105038
## 3: 0.99900 1724775 3450 3446100 1728225 5170875 9.989999e-01 -1.304518e-07 0 2 4100898
## 4: 0.99000 1735193 34704 3435682 1769897 5170875 9.900000e-01 -4.034133e-08 0 2 4059226
## 5: 0.90000 1846741 369348 3324134 2216089 5170875 9.000001e-01 5.414944e-08 0 2 3613034
## 6: 0.10000 2081829 3747292 416366 5829121 2498195 1.000000e-01 4.803468e-08 0 2672682 2
## 7: 0.01000 1956081 3873040 39122 5829121 1995203 1.000010e-02 9.713299e-08 0 3175674 2
## 8: 0.00100 1944337 3884785 3889 5829122 1948226 1.000084e-03 8.383320e-08 0 3222651 1
## 9: 0.00010 1943170 3885951 389 5829121 1943559 1.000942e-04 9.417601e-08 0 3227318 2
## 10: 0.00001 1943053 3886067 39 5829120 1943092 1.003575e-05 3.575301e-08 0 3227785 3
Above we see the results from the previous post.
- it works for negative proportions greater than 0.5.
- There are always the same number of samples in the imbalanced and balanced subsets (
extra_imb=0in every row). - Every row has a valid
neg <= Tneg = 5170877(nonegvalues that are too large to create), even for large values of the target proportion for the negative class,p_neg >= 0.45. This is verified above: all rows have non-negative numbers in theunusedcolumns.
So each row defines a valid pair of subsets (one balanced, one imbalanced), but the rows are not consistent with each other.
Look at the N_pos_neg column. Values are not quite the same, so we can’t use this method to create hierarchical subsets.
We would like to have, for example:
p_neg=0.01andp_neg=0.1having the same sized subsets, andp_neg=0.01having a subset of the negative examples ofp_neg=0.1, andp_neg=0.1having a subset of the positive examples ofp_neg=0.01.
New hierarchical method
We assume there is a list of desired proportions of the negative class in the imbalanced subset:
Target_prop
## [1] 1e-01 1e-02 1e-03 1e-04 1e-05
The most extreme is:
(xp_neg <- min(Target_prop))
## [1] 1e-05
The main idea of the new method is to divide the whole data into three sets:
- two sets X and Y with equal numbers of positive and negative samples,
- one set E with some extra samples of either the positive or negative class.
For each target proportion of negative samples in the imbalanced subset, these sets will be used to create subsets:
- Xi versus Y: an imbalanced version of X, created by removing the first few rows of the minority class, and by adding the first few rows of E.
- X versus Yi: an imbalanced version of Y, created by removing the first few rows of the minority class, and by adding the first few rows of E.
- X versus Y: both subsets balanced, for comparing prediction error rates with a baseline training method.
Identifying active constraint
There are two inequality constraints to consider in the computation of N, the number of positive samples (and negative samples) in A and B (there are 2N samples in A, 2N samples in B, 4N samples total across A and B).
First constraint is the same as the previous method, N + n_pos <= Tpos.
If this is active, then the number of positive samples in the most imbalanced subset is the limiting factor.
(n_pos_max <- 2*Tpos*(1-xp_neg)/(3-2*xp_neg))
## [1] 3886069
(n_pos_max_neg <- n_pos_max*xp_neg/(1-xp_neg))
## [1] 38.86108
(n_pos_max_N <- as.integer(floor((n_pos_max_neg+n_pos_max)/2)))
## [1] 1943053
(n_pos_max_bal_neg <- n_pos_max_N*2)
## [1] 3886106
Tneg-n_pos_max_bal_neg
## [1] 1284771
We see above that there are many negative samples left over, so this works. Below we consider the other option, could N be limited by the number of negative samples in the balanced subset?
(Tneg_N <- as.integer(floor(Tneg/2)))
## [1] 2585438
(Tneg_n_pos <- Tneg*(1-xp_neg))
## [1] 5170825
(Tneg_imb_pos <- Tneg_n_pos+Tneg_N)
## [1] 7756263
Tpos-Tneg_imb_pos
## [1] -1927140
We see above a negative value, indicating this is not feasible. Maxing out the negative examples does not leave enough extra positive examples. The other constraint must be active.
Computing subset sizes
extab <- function(Tprop, N, Tpos, Tneg){
dt <- data.table(
p_neg=sort(c(0.5, Tprop)),
n_bal=2L*N
)[, let(
n_neg=as.integer(round(2*N*p_neg))
)][, let(
n_pos=n_bal-n_neg
)][, let(
n_imb=n_pos+n_neg,
unused_pos=Tpos-n_pos-N,
unused_neg=Tneg-n_neg-N
)][]
list(props=dt, params=dt[, data.table(
extra_pos=max(n_pos-N),
extra_neg=max(n_neg-N),
N)])
}
extab(Target_prop, n_pos_max_N, Tpos, Tneg)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 1e-05 3886106 39 3886067 3886106 3 3227785
## 2: 1e-04 3886106 389 3885717 3886106 353 3227435
## 3: 1e-03 3886106 3886 3882220 3886106 3850 3223938
## 4: 1e-02 3886106 38861 3847245 3886106 38825 3188963
## 5: 1e-01 3886106 388611 3497495 3886106 388575 2839213
## 6: 5e-01 3886106 1943053 1943053 3886106 1943017 1284771
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 1943014 0 1943053
We see above that n_bal == n_imb for each row, which means the two subsets have the same number of samples, and we can create hierarchical subsets.
Other constraint active
When would the other constraint become active? If there are not enough negative samples, for example ten times fewer:
Tpos=5829123L
Tneg=517087L
(n_pos_max <- 2*Tpos*(1-xp_neg)/(3-2*xp_neg))
## [1] 3886069
(n_pos_max_neg <- n_pos_max*xp_neg/(1-xp_neg))
## [1] 38.86108
(n_pos_max_N <- as.integer(floor((n_pos_max_neg+n_pos_max)/2)))
## [1] 1943053
(n_pos_max_bal_neg <- n_pos_max_N*2)
## [1] 3886106
Tneg-n_pos_max_bal_neg
## [1] -3369019
Above the negative number indicates that there are not enough negative samples for this constraint to be active. Below we compute the feasibility for the other constraint.
(Tneg_N <- as.integer(floor(Tneg/2)))
## [1] 258543
(Tneg_n_pos <- Tneg*(1-xp_neg))
## [1] 517081.8
(Tneg_imb_pos <- Tneg_n_pos+Tneg_N)
## [1] 775624.8
Tpos-Tneg_imb_pos
## [1] 5053498
Above we see a positive number, which indicates that the number of positive samples is feasible.
extab(Target_prop, Tneg_N, Tpos, Tneg)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 1e-05 517086 5 517081 517086 5053499 258539
## 2: 1e-04 517086 52 517034 517086 5053546 258492
## 3: 1e-03 517086 517 516569 517086 5054011 258027
## 4: 1e-02 517086 5171 511915 517086 5058665 253373
## 5: 1e-01 517086 51709 465377 517086 5105203 206835
## 6: 5e-01 517086 258543 258543 517086 5312037 1
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 258538 0 258543
Above we see extra_neg close to zero, indicating the number of negative samples was the limiting factor.
Large target proportion of negative samples in imbalanced subset
How does the algorithm work for target proportion greater than one half? In that case there are two inequalities to check:
N + n_neg <= Tneg2*N <= Tpos
Tpos=5829123L
Tneg=5170877L
(Target_prop <- 1-10^seq(-1, -5))
## [1] 0.90000 0.99000 0.99900 0.99990 0.99999
(xp_neg <- max(Target_prop))
## [1] 0.99999
(n_neg_max <- 2*Tneg*xp_neg/(1+2*xp_neg))
## [1] 3447240
(n_neg_max_pos <- n_neg_max*(1-xp_neg)/xp_neg)
## [1] 34.47274
(n_neg_max_N <- (n_neg_max_pos+n_neg_max)/2)
## [1] 1723637
(n_neg_max_bal_pos <- n_neg_max_N*2)
## [1] 3447274
Tpos-n_neg_max_bal_pos
## [1] 2381849
Above the positive number indicates that are enough positive samples for this constraint to be active. Below we compute the feasibility for the other constraint.
(Tpos_N <- as.integer(floor(Tpos/2)))
## [1] 2914561
(Tpos_n_pos <- Tpos*(1-xp_neg))
## [1] 58.29123
(Tpos_n_neg <- Tpos_N*2-Tpos_n_pos)
## [1] 5829064
(Tpos_imb_neg <- Tpos_n_neg+Tpos_N)
## [1] 8743625
Tneg-Tpos_imb_neg
## [1] -3572748
The negative number indicates this second constraint is not active.
General code for both cases
The function below works for both cases.
ntab <- function(Target_prop, Tpos, Tneg){
if(all(Target_prop<0.5)){
Tminor <- Tneg
Tmajor <- Tpos
p_minor <- Target_prop
}else if(all(Target_prop>0.5)){
Tminor <- Tpos
Tmajor <- Tneg
p_minor <- 1-Target_prop
}else stop("Target_prop should be numeric, with each entry greater than 0.5, or each entry less than 0.5")
xp_minor <- min(p_minor)
(n_pos_max <- 2*Tmajor*(1-xp_minor)/(3-2*xp_minor))
(n_pos_max_neg <- n_pos_max*xp_minor/(1-xp_minor))
(n_pos_max_N <- as.integer(floor((n_pos_max_neg+n_pos_max)/2)))
(n_pos_max_bal_neg <- n_pos_max_N*2)
(Tminor_N <- as.integer(floor(Tminor/2)))
N <- if(Tminor<n_pos_max_bal_neg)Tminor_N else n_pos_max_N
extab(Target_prop, N, Tpos, Tneg)
}
ntab(1-10^seq(-1, -5), 5829123L, 5170877L)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 0.50000 3447274 1723637 1723637 3447274 2381849 1723603
## 2: 0.90000 3447274 3102547 344727 3447274 3760759 344693
## 3: 0.99000 3447274 3412801 34473 3447274 4071013 34439
## 4: 0.99900 3447274 3443827 3447 3447274 4102039 3413
## 5: 0.99990 3447274 3446929 345 3447274 4105141 311
## 6: 0.99999 3447274 3447240 34 3447274 4105452 0
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 0 1723603 1723637
ntab(1-10^seq(-1, -5), 582912L, 5170877L)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 0.50000 582912 291456 291456 582912 0 4587965
## 2: 0.90000 582912 524621 58291 582912 233165 4354800
## 3: 0.99000 582912 577083 5829 582912 285627 4302338
## 4: 0.99900 582912 582329 583 582912 290873 4297092
## 5: 0.99990 582912 582854 58 582912 291398 4296567
## 6: 0.99999 582912 582906 6 582912 291450 4296515
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 0 291450 291456
ntab(10^seq(-1, -5), 5829123L, 5170877L)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 1e-05 3886106 39 3886067 3886106 3 3227785
## 2: 1e-04 3886106 389 3885717 3886106 353 3227435
## 3: 1e-03 3886106 3886 3882220 3886106 3850 3223938
## 4: 1e-02 3886106 38861 3847245 3886106 38825 3188963
## 5: 1e-01 3886106 388611 3497495 3886106 388575 2839213
## 6: 5e-01 3886106 1943053 1943053 3886106 1943017 1284771
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 1943014 0 1943053
ntab(10^seq(-1, -5), 5829123L, 517087L)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 1e-05 517086 5 517081 517086 5053499 258539
## 2: 1e-04 517086 52 517034 517086 5053546 258492
## 3: 1e-03 517086 517 516569 517086 5054011 258027
## 4: 1e-02 517086 5171 511915 517086 5058665 253373
## 5: 1e-01 517086 51709 465377 517086 5105203 206835
## 6: 5e-01 517086 258543 258543 517086 5312037 1
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 258538 0 258543
ntab(10^seq(-1, -2), 3000L, 2000L)
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 0.01 2000 20 1980 2000 20 980
## 2: 0.10 2000 200 1800 2000 200 800
## 3: 0.50 2000 1000 1000 2000 1000 0
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 980 0 1000
We see in all the results above that n_bal == n_imb.
Either positive or negative numbers are the limiting factor.
Creating CSV data based on counts
After having computed the target number of samples in each subset, we need to assign rows to A/B/E sets. Here is a dummy data table,
dummy.dt <- data.table(
y=rep(0:1, c(2000,3000)),
x1=NA, x2=NA) #example feature columns.
Tlist <- setNames(as.list(table(dummy.dt$y)), c("Tneg", "Tpos"))
Tlist$Target_prop <- 10^seq(-1, -2)
Tlist
## $Tneg
## [1] 2000
##
## $Tpos
## [1] 3000
##
## $Target_prop
## [1] 0.10 0.01
(count.list <- do.call(ntab, Tlist))
## $props
## p_neg n_bal n_neg n_pos n_imb unused_pos unused_neg
## <num> <int> <int> <int> <int> <int> <int>
## 1: 0.01 2000 20 1980 2000 20 980
## 2: 0.10 2000 200 1800 2000 200 800
## 3: 0.50 2000 1000 1000 2000 1000 0
##
## $params
## extra_pos extra_neg N
## <int> <int> <int>
## 1: 980 0 1000
Above we see the expected counts for a small problem with 2000 samples in A and B, with 980 extra positives.
set.seed(1)
(ind.dt <- data.table(dummy.dt)[
, row := .I
][sample(.N)][
, set := NA_character_
][])
## y x1 x2 row set
## <int> <lgcl> <lgcl> <int> <char>
## 1: 0 NA NA 1017 <NA>
## 2: 1 NA NA 4775 <NA>
## 3: 1 NA NA 2177 <NA>
## 4: 0 NA NA 1533 <NA>
## 5: 1 NA NA 4567 <NA>
## ---
## 4996: 0 NA NA 1397 <NA>
## 4997: 1 NA NA 2779 <NA>
## 4998: 0 NA NA 1255 <NA>
## 4999: 1 NA NA 2262 <NA>
## 5000: 1 NA NA 2606 <NA>
Above we see a random ordering of the data rows, set not yet assigned.
label.list <- list(pos=1,neg=0)
N <- count.list$params$N
for(label.name in names(label.list)){
label.value <- label.list[[label.name]]
label.extra <- count.list$params[[paste0("extra_", label.name)]]
set.values <- rep(c("X","Y","E"), c(N,N,label.extra))
label.i <- which(ind.dt$y==label.value)[seq_along(set.values)]
ind.dt[label.i, set := set.values]
}
ind.dt[, table(set, y, useNA="always")]
## y
## set 0 1 <NA>
## E 0 980 0
## X 1000 1000 0
## Y 1000 1000 0
## <NA> 0 20 0
Above we see set has been assigned,
- 1000 positive samples in each of X and Y,
- 1000 negative samples in each of X and Y,
- 980 positive samples in E,
- 20 unused positive samples (missing set).
We assign fold below too.
n.folds <- 5L
ind.dt[, fold := rep(1:n.folds, length.out=.N), by=.(set, y)][]
## y x1 x2 row set fold
## <int> <lgcl> <lgcl> <int> <char> <int>
## 1: 0 NA NA 1017 X 1
## 2: 1 NA NA 4775 X 1
## 3: 1 NA NA 2177 X 2
## 4: 0 NA NA 1533 X 2
## 5: 1 NA NA 4567 X 3
## ---
## 4996: 0 NA NA 1397 Y 4
## 4997: 1 NA NA 2779 <NA> 3
## 4998: 0 NA NA 1255 Y 5
## 4999: 1 NA NA 2262 <NA> 4
## 5000: 1 NA NA 2606 <NA> 5
Now we create the output columns representing the different subsets.
(out.unsort <- ind.dt[, data.table(
fold,
Xb_Yb=ifelse(set %in% c("X","Y"), set, NA))])
## fold Xb_Yb
## <int> <char>
## 1: 1 X
## 2: 1 X
## 3: 2 X
## 4: 2 X
## 5: 3 X
## ---
## 4996: 4 Y
## 4997: 3 <NA>
## 4998: 5 Y
## 4999: 4 <NA>
## 5000: 5 <NA>
The Xb_Yb column represents the random assignment of two balanced subsets (a baseline).
out.unsort[, table(fold, Xb_Yb)]
## Xb_Yb
## fold X Y
## 1 400 400
## 2 400 400
## 3 400 400
## 4 400 400
## 5 400 400
Above we can see the distribution of folds across subsets is uniform. Below we create the other output columns, representing mapping of data rows to splits with an imbalanced subset.
imb.counts <- count.list$props[p_neg != 0.5]
pos.part <- function(x)ifelse(x<0, 0, x)
for(cformat in c("Xineg%s_Yb", "Xb_Yineg%s")){
for(imb.i in nrow(imb.counts):1){
imb.row <- imb.counts[imb.i]
j.name <- sprintf(cformat, imb.row$p_neg)
set(
out.unsort,
j=j.name,
value=ind.dt$set)
imb.set <- ifelse(grepl("Xi", cformat), "X", "Y")
for(label.name in names(label.list)){
label.value <- label.list[[label.name]]
label.n <- imb.row[[paste0("n_", label.name)]]
find.rep.dt <- rowwiseDT(
find.set=, sign=, rep.set=,
imb.set, 1, NA, #rm
"E", -1, imb.set)#add
for(find.rep.i in 1:nrow(find.rep.dt)){
find.rep.row <- find.rep.dt[find.rep.i]
possible.indices <- ind.dt[, which(y==label.value & set==find.rep.row$find.set)]
change.n <- pos.part((N-label.n)*find.rep.row$sign)
change.indices <- possible.indices[seq_len(change.n)]
set(
out.unsort,
i=change.indices,
j=j.name,
value=find.rep.row$rep.set)
}
is.E <- which(out.unsort[[j.name]]=="E")
set(
out.unsort,
i=is.E,
j=j.name,
value=NA)
}
}
}
Finally we sort back to the original row order:
orig.ord <- order(ind.dt$row)
(with.y.set <- data.table(ind.dt[, .(y, set)], out.unsort)[orig.ord])
## y set fold Xb_Yb Xineg0.1_Yb Xineg0.01_Yb Xb_Yineg0.1 Xb_Yineg0.01
## <int> <char> <int> <char> <char> <char> <char> <char>
## 1: 0 Y 1 Y Y Y <NA> <NA>
## 2: 0 Y 2 Y Y Y Y Y
## 3: 0 Y 2 Y Y Y <NA> <NA>
## 4: 0 X 1 X <NA> <NA> X X
## 5: 0 Y 4 Y Y Y <NA> <NA>
## ---
## 4996: 1 Y 5 Y Y Y Y Y
## 4997: 1 Y 5 Y Y Y Y Y
## 4998: 1 E 4 <NA> X X Y Y
## 4999: 1 X 2 X X X X X
## 5000: 1 X 1 X X X X X
The column names can be interpreted as follows:
bsuffix for balanced:XbandYb.isuffix for imbalanced, with a specified proportion of negative labels:ineg0.1,ineg0.01.
Below we count each unique row pattern.
with.y.set[, .(rows=.N), keyby=names(with.y.set)]
## Key: <y, set, fold, Xb_Yb, Xineg0.1_Yb, Xineg0.01_Yb, Xb_Yineg0.1, Xb_Yineg0.01>
## y set fold Xb_Yb Xineg0.1_Yb Xineg0.01_Yb Xb_Yineg0.1 Xb_Yineg0.01 rows
## <int> <char> <int> <char> <char> <char> <char> <char> <int>
## 1: 0 X 1 X <NA> <NA> X X 160
## 2: 0 X 1 X X <NA> X X 36
## 3: 0 X 1 X X X X X 4
## 4: 0 X 2 X <NA> <NA> X X 160
## 5: 0 X 2 X X <NA> X X 36
## 6: 0 X 2 X X X X X 4
## 7: 0 X 3 X <NA> <NA> X X 160
## 8: 0 X 3 X X <NA> X X 36
## 9: 0 X 3 X X X X X 4
## 10: 0 X 4 X <NA> <NA> X X 160
## 11: 0 X 4 X X <NA> X X 36
## 12: 0 X 4 X X X X X 4
## 13: 0 X 5 X <NA> <NA> X X 160
## 14: 0 X 5 X X <NA> X X 36
## 15: 0 X 5 X X X X X 4
## 16: 0 Y 1 Y Y Y <NA> <NA> 160
## 17: 0 Y 1 Y Y Y Y <NA> 36
## 18: 0 Y 1 Y Y Y Y Y 4
## 19: 0 Y 2 Y Y Y <NA> <NA> 160
## 20: 0 Y 2 Y Y Y Y <NA> 36
## 21: 0 Y 2 Y Y Y Y Y 4
## 22: 0 Y 3 Y Y Y <NA> <NA> 160
## 23: 0 Y 3 Y Y Y Y <NA> 36
## 24: 0 Y 3 Y Y Y Y Y 4
## 25: 0 Y 4 Y Y Y <NA> <NA> 160
## 26: 0 Y 4 Y Y Y Y <NA> 36
## 27: 0 Y 4 Y Y Y Y Y 4
## 28: 0 Y 5 Y Y Y <NA> <NA> 160
## 29: 0 Y 5 Y Y Y Y <NA> 36
## 30: 0 Y 5 Y Y Y Y Y 4
## 31: 1 <NA> 1 <NA> <NA> <NA> <NA> <NA> 4
## 32: 1 <NA> 2 <NA> <NA> <NA> <NA> <NA> 4
## 33: 1 <NA> 3 <NA> <NA> <NA> <NA> <NA> 4
## 34: 1 <NA> 4 <NA> <NA> <NA> <NA> <NA> 4
## 35: 1 <NA> 5 <NA> <NA> <NA> <NA> <NA> 4
## 36: 1 E 1 <NA> <NA> X <NA> Y 36
## 37: 1 E 1 <NA> X X Y Y 160
## 38: 1 E 2 <NA> <NA> X <NA> Y 36
## 39: 1 E 2 <NA> X X Y Y 160
## 40: 1 E 3 <NA> <NA> X <NA> Y 36
## 41: 1 E 3 <NA> X X Y Y 160
## 42: 1 E 4 <NA> <NA> X <NA> Y 36
## 43: 1 E 4 <NA> X X Y Y 160
## 44: 1 E 5 <NA> <NA> X <NA> Y 36
## 45: 1 E 5 <NA> X X Y Y 160
## 46: 1 X 1 X X X X X 200
## 47: 1 X 2 X X X X X 200
## 48: 1 X 3 X X X X X 200
## 49: 1 X 4 X X X X X 200
## 50: 1 X 5 X X X X X 200
## 51: 1 Y 1 Y Y Y Y Y 200
## 52: 1 Y 2 Y Y Y Y Y 200
## 53: 1 Y 3 Y Y Y Y Y 200
## 54: 1 Y 4 Y Y Y Y Y 200
## 55: 1 Y 5 Y Y Y Y Y 200
## y set fold Xb_Yb Xineg0.1_Yb Xineg0.01_Yb Xb_Yineg0.1 Xb_Yineg0.01 rows
## <int> <char> <int> <char> <char> <char> <char> <char> <int>
Above we can see the desired properties.
- In the first half of the table, with
y=0, when a row has been removed inineg0.1, then it is also removed inineg0.01(hierarchical removal of negative samples as negative proportion is decreased). - In the second half of the table, with
y=1, when a row has been added inineg0.1, then it is also added inineg0.01(hierarchical addition of positive samples as negative proportion is decreased). - The same E rows are used for addition in Xi and Yi.
- The distribution of folds is uniform: theres are equal numbers of rows with each pattern in each fold (stratification).
Below we create a table for output, with rows in the same order as the original table.
(out.sort <- out.unsort[orig.ord])
## fold Xb_Yb Xineg0.1_Yb Xineg0.01_Yb Xb_Yineg0.1 Xb_Yineg0.01
## <int> <char> <char> <char> <char> <char>
## 1: 1 Y Y Y <NA> <NA>
## 2: 2 Y Y Y Y Y
## 3: 2 Y Y Y <NA> <NA>
## 4: 1 X <NA> <NA> X X
## 5: 4 Y Y Y <NA> <NA>
## ---
## 4996: 5 Y Y Y Y Y
## 4997: 5 Y Y Y Y Y
## 4998: 4 <NA> X X Y Y
## 4999: 2 X X X X X
## 5000: 1 X X X X X
The table above is a split info table that we can save alongside the original table.
fwrite_head <- function(dt){
fwrite(dt, data.file <- tempfile())
cat(head(readLines(data.file)), sep="\n")
}
fwrite_head(dummy.dt)
## y,x1,x2
## 0,,
## 0,,
## 0,,
## 0,,
## 0,,
fwrite_head(out.sort)
## fold,Xb_Yb,Xineg0.1_Yb,Xineg0.01_Yb,Xb_Yineg0.1,Xb_Yineg0.01
## 1,Y,Y,Y,,
## 2,Y,Y,Y,Y,Y
## 2,Y,Y,Y,,
## 1,X,,,X,X
## 4,Y,Y,Y,,
The output above shows the first few lines of the CSV file that describes the cross-validation experiment we have created.
Verification
Now we verify that the counts are reasonable.
check.dt.list <- list()
for(sub.col.i in 2:ncol(out.sort)){
sub.col.name <- names(out.sort)[sub.col.i]
two.cols <- out.sort[, c(1, sub.col.i), with=FALSE]
setnames(two.cols, c("fold", "subset"))
count.dt <- data.table(dummy.dt, two.cols)[, .(
rows=.N
), keyby=.(y, fold, subset)]
stats.dt <- dcast(
count.dt,
subset + y ~ .,
list(sum, var, length),
value.var="rows")
check.dt.list[[sub.col.name]] <- data.table(sub.col.name, stats.dt)
}
(check.dt <- rbindlist(check.dt.list))
## sub.col.name subset y rows_sum rows_var rows_length
## <char> <char> <int> <int> <num> <int>
## 1: Xb_Yb <NA> 1 1000 0 5
## 2: Xb_Yb X 0 1000 0 5
## 3: Xb_Yb X 1 1000 0 5
## 4: Xb_Yb Y 0 1000 0 5
## 5: Xb_Yb Y 1 1000 0 5
## 6: Xineg0.1_Yb <NA> 0 800 0 5
## 7: Xineg0.1_Yb <NA> 1 200 0 5
## 8: Xineg0.1_Yb X 0 200 0 5
## 9: Xineg0.1_Yb X 1 1800 0 5
## 10: Xineg0.1_Yb Y 0 1000 0 5
## 11: Xineg0.1_Yb Y 1 1000 0 5
## 12: Xineg0.01_Yb <NA> 0 980 0 5
## 13: Xineg0.01_Yb <NA> 1 20 0 5
## 14: Xineg0.01_Yb X 0 20 0 5
## 15: Xineg0.01_Yb X 1 1980 0 5
## 16: Xineg0.01_Yb Y 0 1000 0 5
## 17: Xineg0.01_Yb Y 1 1000 0 5
## 18: Xb_Yineg0.1 <NA> 0 800 0 5
## 19: Xb_Yineg0.1 <NA> 1 200 0 5
## 20: Xb_Yineg0.1 X 0 1000 0 5
## 21: Xb_Yineg0.1 X 1 1000 0 5
## 22: Xb_Yineg0.1 Y 0 200 0 5
## 23: Xb_Yineg0.1 Y 1 1800 0 5
## 24: Xb_Yineg0.01 <NA> 0 980 0 5
## 25: Xb_Yineg0.01 <NA> 1 20 0 5
## 26: Xb_Yineg0.01 X 0 1000 0 5
## 27: Xb_Yineg0.01 X 1 1000 0 5
## 28: Xb_Yineg0.01 Y 0 20 0 5
## 29: Xb_Yineg0.01 Y 1 1980 0 5
## sub.col.name subset y rows_sum rows_var rows_length
## <char> <char> <int> <int> <num> <int>
The table above has one row per combination of CSV column, subset, and label. We see that the results are reasonable.
- subset is either X or Y.
- number of rows per subset is always 2000.
- no variance between number of rows across folds, which means fold assignment respects stratification and subsets.
Higgs data
Below we compute CSV subset files for two random seeds, and for downsampling both classes.
(higgs.dt <- fread("higgs.csv", select="target"))
## target
## <num>
## 1: 1
## 2: 1
## 3: 1
## 4: 0
## 5: 1
## ---
## 10999996: 1
## 10999997: 1
## 10999998: 1
## 10999999: 0
## 11000000: 0
get_subsets <- function(y.vec, p_neg, n.folds=5L){
Tlist <- setNames(as.list(table(y.vec)), c("Tneg", "Tpos"))
Tlist$Target_prop <- p_neg
Tlist
(count.list <- do.call(ntab, Tlist))
(ind.dt <- data.table(y=y.vec)[
, row := .I
][sample(.N)][
, set := NA_character_
][])
label.list <- list(pos=1,neg=0)
N <- count.list$params$N
for(label.name in names(label.list)){
label.value <- label.list[[label.name]]
label.extra <- count.list$params[[paste0("extra_", label.name)]]
set.values <- rep(c("X","Y","E"), c(N,N,label.extra))
label.i <- which(ind.dt$y==label.value)[seq_along(set.values)]
ind.dt[label.i, set := set.values]
}
ind.dt[, fold := rep(1:n.folds, length.out=.N), by=.(set, y)][]
(out.unsort <- ind.dt[, data.table(
fold,
Xb_Yb=ifelse(set %in% c("X","Y"), set, NA))])
imb.counts <- count.list$props[p_neg != 0.5][order(abs(p_neg-0.5))]
pos.part <- function(x)ifelse(x<0, 0, x)
for(cformat in c("Xineg%s_Yb", "Xb_Yineg%s")){
for(imb.i in 1:nrow(imb.counts)){
imb.row <- imb.counts[imb.i]
j.name <- sprintf(cformat, imb.row$p_neg)
set(
out.unsort,
j=j.name,
value=ind.dt$set)
imb.set <- ifelse(grepl("Xi", cformat), "X", "Y")
for(label.name in names(label.list)){
label.value <- label.list[[label.name]]
label.n <- imb.row[[paste0("n_", label.name)]]
find.rep.dt <- rowwiseDT(
find.set=, sign=, rep.set=,
imb.set, 1, NA, #rm
"E", -1, imb.set)#add
for(find.rep.i in 1:nrow(find.rep.dt)){
find.rep.row <- find.rep.dt[find.rep.i]
possible.indices <- ind.dt[, which(y==label.value & set==find.rep.row$find.set)]
change.n <- pos.part((N-label.n)*find.rep.row$sign)
change.indices <- possible.indices[seq_len(change.n)]
set(
out.unsort,
i=change.indices,
j=j.name,
value=find.rep.row$rep.set)
}
is.E <- which(out.unsort[[j.name]]=="E")
set(
out.unsort,
i=is.E,
j=j.name,
value=NA)
}
}
}
orig.ord <- order(ind.dt$row)
out.unsort[orig.ord]
}
(higgs.p <- (10^seq(-3, -1))*0.5)
## [1] 5e-04 5e-03 5e-02
p.list <- list(neg=higgs.p, pos=1-higgs.p)
for(minor.class in names(p.list)){
p.vec <- p.list[[minor.class]]
for(seed in 1:2){
set.seed(seed)
(higgs.sub.dt <- get_subsets(higgs.dt$target, p.vec))
print(out.csv <- sprintf(
"higgs_subsets_seed=%d_pneg[%s,%s].csv",
seed, min(p.vec), max(p.vec)))
fwrite(higgs.sub.dt, out.csv)
}
}
## [1] "higgs_subsets_seed=1_pneg[5e-04,0.05].csv"
## [1] "higgs_subsets_seed=2_pneg[5e-04,0.05].csv"
## [1] "higgs_subsets_seed=1_pneg[0.95,0.9995].csv"
## [1] "higgs_subsets_seed=2_pneg[0.95,0.9995].csv"
higgs.sub.dt
## fold Xb_Yb Xineg0.95_Yb Xineg0.995_Yb Xineg0.9995_Yb Xb_Yineg0.95 Xb_Yineg0.995 Xb_Yineg0.9995
## <int> <char> <char> <char> <char> <char> <char> <char>
## 1: 3 Y Y Y Y Y <NA> <NA>
## 2: 1 X X <NA> <NA> X X X
## 3: 3 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 4: 1 X X X X X X X
## 5: 1 <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## ---
## 10999996: 1 X <NA> <NA> <NA> X X X
## 10999997: 4 X <NA> <NA> <NA> X X X
## 10999998: 3 Y Y Y Y <NA> <NA> <NA>
## 10999999: 2 X X X X X X X
## 11000000: 3 X X X X X X X
Above we see the last table created and saved to CSV.
table(higgs.sub.dt$fold, paste(higgs.sub.dt[["Xb_Yineg0.9995"]], higgs.dt$target))
##
## NA 0 NA 1 X 0 X 1 Y 0 Y 1
## 1 1 820641 344840 344840 689336 344
## 2 0 820640 344840 344840 689335 345
## 3 0 820640 344840 344840 689335 345
## 4 0 820639 344840 344840 689335 345
## 5 0 820639 344840 344840 689335 345
Above we see that the data counts across subsets and folds are as expected (X balanced, Y imbalanced mostly negative).
cat(system("du -ms higgs*csv", intern=TRUE), sep="\n")
## 5310 higgs.csv
## 141 higgs_subsets_seed=1_pneg[0.95,0.9995].csv
## 147 higgs_subsets_seed=1_pneg[5e-04,0.05].csv
## 141 higgs_subsets_seed=2_pneg[0.95,0.9995].csv
## 147 higgs_subsets_seed=2_pneg[5e-04,0.05].csv
Above we see that the CSV files we created are 141 MB (for large pneg) or 147 MB (for small pneg).
This makes sense because there are fewer negative samples, as shown below.
table(higgs.dt$target)
##
## 0 1
## 5170877 5829123
Conclusions
We have shown how to split a binary classification data set into two subsets.
- Either the two subsets are both balanced (baseline),
- or one or the other subset is imbalanced to given proportions.
- each subset always has the same sample size,
- and more imbalanced subsets are hierarchical: minor class samples are removed, major class samples are added (so there is a certain kind of continuity between imbalance proportions).
This code will be useful for creating imbalanced classification benchmarks, for comparing various machine learning algorithms.
Session info
sessionInfo()
## R Under development (unstable) (2026-07-28 r90311)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0 LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=fr_FR.UTF-8 LC_COLLATE=en_US.UTF-8
## [5] LC_MONETARY=fr_FR.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=fr_FR.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C
##
## time zone: America/Toronto
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] data.table_1.18.6.1
##
## loaded via a namespace (and not attached):
## [1] compiler_4.7.0 cli_3.6.6 tools_4.7.0 otel_0.2.0 knitr_1.51 xfun_0.60 rlang_1.3.0
## [8] evaluate_1.0.5