8  Why one subtraction is not enough

An unbalanced panel, and the loop that fixes it

8.1 Why you cannot just subtract twice

Chapter 3 subtracted one mean and got the right answer. The obvious extension is to subtract the month mean as well. It does not work:

one_pass <- lm(
  I(dpd30  - ave(dpd30,  SK_ID_PREV) - ave(dpd30,  MONTHS_BALANCE) + mean(dpd30)) ~
  I(raised - ave(raised, SK_ID_PREV) - ave(raised, MONTHS_BALANCE) + mean(raised)) - 1,
  data = d)
Table 8.1: One pass of double demeaning against the real thing 
                                                Method Estimate (pp)
                                                <char>        <char>
1:      subtract account means, then month means, once        -0.676
2: feols(dpd30 ~ raised | SK_ID_PREV + MONTHS_BALANCE)        +0.086

One pass gives -0.676 pp — the wrong sign again, on the very last step.

The reason is that the panel is unbalanced: accounts appear for different numbers of months and in different ones. Subtracting the month means puts some of the account means back, because the months an account appears in are not a random sample of all months. Subtract the account means again and you disturb the month means. fixest alternates the two projections until neither moves — the same idea, run to convergence instead of once.

The general point

Two-way demeaning is only exact on a balanced panel. On anything real it is an iteration, and software that does it properly is doing more than the formula in the textbook suggests. Worth knowing before you hand-roll it.

8.2 The panel is lopsided, and that is the whole reason

“Unbalanced” is a dry word for something quite dramatic here. Accounts enter and leave the sample at different times, so the number of them observed in a given calendar month, and the share of those that have had an increase, both move a great deal.

Figure 8.1: Each bubble is one calendar month. Size is how many accounts were observed in it.

On a balanced panel — every account present in every month — subtracting account means would leave the month means alone, and one pass would be exact. Here the months an account appears in are not a random sample of all months, so the two subtractions keep undoing each other.

8.3 Watching it converge

The fix is to alternate: take out account means, take out month means, take out account means again, and keep going until neither move. It takes fewer rounds than you might expect.

demean <- function(v, g) v - ave(v, g)

y <- d$dpd30; x <- d$raised
path <- rbindlist(lapply(1:8, function(pass) {
  y <<- demean(y, d$SK_ID_PREV); x <<- demean(x, d$SK_ID_PREV)   # accounts
  y <<- demean(y, d$MONTHS_BALANCE); x <<- demean(x, d$MONTHS_BALANCE)   # months
  data.table(pass = pass, est = 100 * sum(x * y) / sum(x * x))
}))
Figure 8.2: The estimate after each round of alternating subtraction.

One round leaves you at +0.120 pp — already the right sign, but 39% too big. By round 5 it agrees with fixest to four decimal places.

Note that this is a sequential round: accounts first, then months. The -0.676 pp above came from the simultaneous shortcut — subtract both means and add the grand mean back — which on an unbalanced panel is worse still. Neither is the estimator. The estimator is the loop.

That is the whole difference between the formula and the estimator. The textbook writes the within-transformation as one subtraction because on a balanced panel it is one. Software that handles real data runs it as a loop, and this is the loop.

That is the estimate. Now, why believe it?

Everything above runs from 08-convergence.qmd. Shared setup <80><94> the data loader, the plot theme and the inline formatters <80><94> is in R/_common.R. The panel itself is built by R/01-build.R.