Estimation Archives • Page 2 of 3 • Statisticelle

Nonparametric neighbours: U-statistic structural components and jackknife pseudo-observations for the AUC

Two of my recent blog posts focused on two different, but as we will see related, methods which essentially transform observed responses into a summary of their contribution to an estimate: structural components resulting from Sen’s (1960) decomposition of U-statistics and pseudo-observations resulting from application of the leave-one-out jackknife. As I note in this comment, I think the real value of deconstructing estimators in this way results from the use of these quantities, which in special (but common) cases are asymptotically uncorrelated and identically distributed, to: (1) simplify otherwise complex variance estimates and construct interval estimates, and (2) apply regression methods to estimators without an existing regression framework.

As discussed by Miller (1974), pseudo-observations may be treated as approximately independent and identically distributed random variables when the quantity of interest is a function of the mean or variance, and more generally, any function of a U-statistic. Several other scenarios where these methods are applicable are also outlined. Many estimators of popular “parameters” can actually be expressed as U-statistics. Thus, these methods are quite broadly applicable. A review of basic U-statistic theory and some common examples, notably the difference in means or the Wilcoxon Mann-Whitney test statistic, can be found within my blog post: One, Two, U: Examples of common one- and two-sample U-statistics.

As an example of use case (1), Delong et al. (1988) used structural components to estimate the variances and covariances of the areas under multiple, correlated receiver operator curves or multiple AUCs. Hanley and Hajian-Tilaki (1997) later referred to the methods of Delong et al. (1988) as “the cleanest and most elegant approach to variances and covariances of AUCs.” As an example of use case (2), Andersen & Pohar Perme (2010) provide a thorough summary of how pseudo-observations can be used to construct regression models for important survival parameters like survival at a single time point and the restricted mean survival time.

Now, structural components are restricted to U-statistics while pseudo-observations may be used more generally, as discussed. But, if we construct pseudo-observations for U-statistics, one of several “valid” scenarios, what is the relationship between these two quantities? Hanley and Hajian-Tilaki (1997) provide a lovely discussion of the equivalence of these two methods when applied to the area under the receiver operating characteristic curve or simply the AUC. This blog post follows their discussion, providing concrete examples of computing structural components and pseudo-observations using R, and demonstrating their equivalence in this special case.

Continue reading Nonparametric neighbours: U-statistic structural components and jackknife pseudo-observations for the AUC

Resampling, the jackknife, and pseudo-observations

Resampling methods approximate the sampling distribution of a statistic or estimator. In essence, a sample taken from the population is treated as a population itself. A large number of new samples, or resamples, are taken from this “new population”, commonly with replacement, and within each of these resamples, the estimate of interest is re-obtained. A large number of these estimate replicates can then be used to construct the empirical sampling distribution from which confidence intervals, bias, and variance may be estimated. These methods are particularly advantageous for statistics or estimators for which no standard methods apply or are difficult to derive.

The jackknife is a popular resampling method, first introduced by Quenouille in 1949 as a method of bias estimation. In 1958, jackknifing was both named by Tukey and expanded to include variance estimation. A jackknife is a multipurpose tool, similar to a swiss army knife, that can get its user out of tricky situations. Efron later developed the arguably most popular resampling method, the bootstrap, in 1979 after being inspired by the jackknife.

In Efron’s (1982) book The jackknife, the bootstrap, and other resampling plans, he states,

Good simple ideas, of which the jackknife is a prime example, are our most precious intellectual commodity, so there is no need to apologize for the easy mathematical level.

Despite existing since the 1940’s, resampling methods were infeasible due to the computational power required to perform resampling and recalculate estimates many times. With today’s computing power, the uncomplicated yet powerful jackknife, and resampling methods more generally, should be a tool in every analyst’s toolbox.

Continue reading Resampling, the jackknife, and pseudo-observations

Parametric vs. Nonparametric Approach to Estimations

Parametric statistics assume that the unknown CDF $F$ belongs to a family of CDFs characterized by a parameter (vector) $\theta$ . As the form of $F$ is assumed, the target of estimation is its parameters $\theta$ . Thus, all uncertainty about $F$ is comprised of uncertainty about its parameters. Parameters are estimated by $\hat{\theta}$ , and estimates are be substituted into the assumed distribution to conduct inference for the quantities of interest. If the assumed distribution $F$ is incorrect, inference may also be inaccurate, or trends in the data may be missed.

To demonstrate the parametric approach, consider $n = 100$ independent and identically distributed random variables $X_1, …, X_n$ generated from an exponential distribution with rate $\lambda = 2$ . Investigators wish to estimate the 75 $^{th}$ percentile and erroneously assume that their data is normally distributed. Thus, $F$ is assumed to be the Normal CDF but $\mu$ and $\sigma^2$ are unknown. The parameters $\mu$ and $\sigma$ are estimated in their typical way by $\bar{x}$ and $\sigma^2$ , respectively. Since the normal distribution belongs to the location-scale family, an estimate of the $p^{th}$ percentile is provided by,

$x_p = \bar{x} + s\Phi^{-1}(p)$

where $\Phi^{-1}$ is the standard normal quantile function, also known as the probit.

set.seed(12345)
library(tidyverse, quietly = T)

# Generate data from Exp(2)
x <- rexp(n = 100, rate = 2)

# True value of 75th percentile with rate = 2
true <- qexp(p = 0.75, rate = 2) 
true

## [1] 0.6931472

# Estimate mu and sigma
xbar <- mean(x)
s    <- sd(x)

# Estimate 75th percentile assuming mu = xbar and sigma = s
param_est <- xbar + s * qnorm(p = 0.75)
param_est

## [1] 0.8792925

The true value of the 75 $^{th}$ percentile of $\text{Exp}(2)$ is 0.69 while the parametric estimate is 0.88.

Nonparametric statistics make fewer distributions about the unknown distribution $F$ , requiring only mild assumptions such as continuity or the existence of specific moments. Instead of estimating parameters of $F$ , $F$ itself is the target of estimation. $F$ is commonly estimated by the empirical cumulative distribution function (ECDF) $\hat{F}$ ,

$\hat{F}(x) = \frac{1}{n} \sum_{i=1}^{n} \mathbb{I}(X_i \leq x).$

Any statistic that can be expressed as a function of the CDF, known as a statistical functional and denoted $\theta = T(F)$ , can be estimated by substituting $\hat{F}$ for $F$ . That is, plug-in estimators can be obtained as $\hat{\theta} = T(\hat{F})$ .

Continue reading Parametric vs. Nonparametric Approach to Estimations

Kernel Regression

Motivation

For observed pairs $(x_i, y_i)$ , $i = 1, …, n$ , the relationship between $x$ and $y$ can be defined generally as

$y_i = m(x_i) + \varepsilon_i$

where $f(x_i) = E[y_i | x = x_i]$ and $\varepsilon_i \stackrel{iid}{\sim} (0, \sigma^2)$ . If we are unsure about the form of $m(\cdot)$ , our objective may be to estimate $m(\cdot)$ without making too many assumptions about its shape. In other words, we aim to “let the data speak for itself”.

Simulated scatterplot of $y = f(x) + \epsilon$ . Here, $x \sim Uniform(0, 10)$ and $\epsilon \sim N(0, 1)$ . The true function $f(x) = sin(x)$ is displayed in green.

Non-parametric approaches require only that $m(\cdot)$ be smooth and continuous. These assumptions are far less restrictive than alternative parametric approaches, thereby increasing the number of potential fits and providing additional flexibility. This makes non-parametric models particularly appealing when prior knowledge about $m(\cdot)$ ‘s functional form is limited.

Estimating the Regression Function

If multiple values of $y$ were observed at each $x$ , $f(x)$ could be estimated by averaging the value of the response at each $x$ . However, since $x$ is often continuous, it can take on a wide range of values making this quite rare. Instead, a neighbourhood of $x$ is considered.

Result of averaging $y_i$ at each $x_i$ . The fit is extremely rough due to gaps in $x$ and low $y$ frequency at each $x$ .

Define the neighbourhood around $x$ as $x \pm \lambda$ for some bandwidth $\lambda > 0$ . Then, a simple non-parametric estimate of $m(x)$ can be constructed as average of the $y_i$ ‘s corresponding to the $x_i$ within this neighbourhood. That is,

(1) $\begin{equation*} \hat{f}_{\lambda}(x) = \frac{\sum_{n} \mathbb{I}(|x - x_i| \leq \lambda)~ y_i}{\sum_{n} \mathbb{I}(|x - x_i| \leq \lambda)} = \frac{\sum_n K\left( \frac{x - x_i}{\lambda} \right) y_i}{\sum_n K\left( \frac{x - x_i}{\lambda} \right) } \end{equation*}$

where

$K(u) = \begin{cases} \frac{1}{2} & |u| \leq 1 \\ 0 & \text{o.w.} \end{cases}$

is the uniform kernel. This estimator, referred to as the Nadaraya-Watson estimator, can be generalized to any kernel function $K(u)$ (see my previous blog bost). It is, however, convention to use kernel functions of degree $\nu = 2$ (e.g. the Gaussian and Epanechnikov kernels).

The red line is the result of estimating $f(x)$ with a Gaussian kernel and arbitrarily selected bandwidth of $\lambda = 1.25$ . The green line represents the true function $sin(x)$ .

Continue reading Kernel Regression