To start, I apologize for this blog’s title but I couldn’t resist referencing to the Owen Wilson classic You, Me, and Dupree – wow! The other gold-plated candidate was U-statistics and You. Please, please, hold your applause.
My previous blog post defined statistical functionals as any real-valued function of an unknown CDF, , and explained how plug-in estimators could be constructed by substituting the empirical cumulative distribution function (ECDF) for the unknown CDF . Plug-in estimators of the mean and variance were provided and used to demonstrate plug-in estimators’ potential to be biased.
Statistical functionals that meet the following two criteria represent a special family of functionals known as expectation functionals:
1) is the expectation of a function with respect to the distribution function ; and
2) the function takes the form of a symmetric kernel.
Expectation functionals encompass many common parameters and are well-behaved. Plug-in estimators of expectation functionals, named V-statistics after von Mises, can be obtained but may be biased. It is, however, always possible to construct an unbiased estimator of expectation functionals regardless of the underlying distribution function . These estimators are named U-statistics, with the “U” standing for unbiased.
This blog post provides 1) the definitions of symmetric kernels and expectation functionals; 2) an overview of plug-in estimators of expectation functionals or V-statistics; 3) an overview of unbiased estimators for expectation functionals or U-statistics.
Kernels, degree, and symmetry
Consider a real-valued function such that for random variables ,
is referred to as a kernel of degree and is by definition, an unbiased estimator of . is said to be symmetric in its arguments, or symmetric, if it is invariant to permutation of its arguments . For example, a kernel of degree 2 is symmetric if .
If is not symmetric, a function can always be found that is symmetric. As a result of the ‘s being identically distributed, they may be considered “exchangeable” such that for any permutation of the random variables,
There are possible permutations of ‘s arguments. Then, since is an unbiased estimator of , the average of across all permutations of its arguments,
is both an unbiased estimator for and symmetric in its arguments such that
Thus, without loss of generality, the kernel may always be assumed to be symmetric.
As an example, is a symmetric kernel of degree 2 since and .
On the other hand, is an unbiased estimator of such that
but it is not symmetric as . The corresponding symmetric kernel of degree 2 is then
Any statistical functionals that can be expressed as the expectation of a symmetric kernel of degree with respect to
represent a special family known as expectation functionals or regular functionals. For a refresher on what means, refer to my previous blog post on plug-in estimators!
Common examples of expectation functionals include moments,
When , expectation functionals may also be referred to as linear functionals since for some mixture ,
V-statistics are plug-in estimators of expectation functionals such that for ,
As noted previously, the empirical cumulative distribution function defined as
is a valid distribution function which assigns mass to each . That is, for a random variable ,
Now, consider such random variables . Then, there are a total of equally-likely realizations of . For example, all could equal , all could equal , or anything in between such as .
The are independent and thus the plug-in estimator of an expectation functional is equal to
Since is discrete and the support of each is the random sample , the plug-in estimator can be explicitly expressed as,
so that is the sample average of evaluated at all possible realizations of , or equivalently, the sample average of evaluated at all permutations of size from .
Bias of V-statistics
When , the V-statistic takes the form of a traditional sample mean,
which is unbiased and its asymptotic distribution is provided by the central limit theorem,
Now, consider the form of when ,
Notice that the sum contains terms for which . We can expand the sum to make these terms explicit,
There are terms for which and terms for which . Taking the expectation of with respect to yields,
Since and are independent when , by definition such that,
Clearly and thus is a biased estimator of . is generally biased for as subscript duplication () results in dependence within its summands. Note however, the bias of approaches 0 as .
As an example, consider the plug-in estimator for the variance using the symmetric kernel we defined above,
Expanding the square and splitting the sum into and ,
The second sum is clearly zero and we are left with,
Now, taking the expectation of with respect to ,
Since the are identically distributed, let’s drop the subscripts and aggregate terms recalling that there are terms for which ,
Recalling that , simplifying leaves us with
which takes the general form of we derived previously.
Since is biased as a result of subscript duplication, a possible solution would be to restrict the sums to distinct combinations of subscripts . We can require that as a result of the kernel ‘s symmetry. Thus, there are such subscript combinations to consider. Let represent the set of all subscript combinations. Then, the resulting estimator of is the U-statistic
Now that all subscripts within the summands are distinct, we have
so that is unbiased for , hence the name!
Returning to the variance example, the corresponding U-statistic is
Taking the expectation of with respect to yields,
Since the are identically distributed, dropping the subscripts and aggregating terms gives,
so that the U-statistic is unbiased for the population variance as desired.
It can be shown that U-statistics are asymptotically normal. However, the central limit cannot be used to prove this result. Each appears in more than one summand of , making the sum of dependent terms. As a result, a clever technique known as H-projection, named after Wassily Hoeffding, is required.