All About t-Tests

Author

Dr. Anisha Singh

Dataset

This script uses a dataset of reading scores from 72 participants assigned to one of two intervention types. Download it and save it in the same folder as your R script before running any code.

↓ scores.csv

Variables: scores (reading score), intervention (1 or 2) · 72 observations, 36 per group

Exploring the Data

Always start by loading and inspecting the dataset before running any tests.

library(readr)
scores <- read_csv("scores.csv")

head(scores)
summary(scores)

# A tibble: 6 × 2
  scores intervention
   <dbl>        <dbl>
1    170            1
2    175            1
3    165            1
4    180            1
5    160            1
6    158            1

     scores       intervention
 Min.   :152.0   Min.   :1.0
 1st Qu.:170.8   1st Qu.:1.0
 Median :182.5   Median :1.5
 Mean   :184.5   Mean   :1.5
 3rd Qu.:198.2   3rd Qu.:2.0
 Max.   :228.0   Max.   :2.0

describeBy() from the psych package gives a richer breakdown by group in one step:

library(psych)
describeBy(scores$scores, group = scores$intervention)

 Descriptive statistics by group
group: 1
   vars  n mean    sd median trimmed   mad min max range  skew kurtosis   se
X1    1 36  193 17.44    194  193.13 16.31 158 228    70 -0.11    -0.63 2.91
------------------------------------------------------------
group: 2
   vars  n mean    sd median trimmed   mad min max range skew kurtosis   se
X1    1 36  176 13.88  174.5  175.47 14.83 152 205    53 0.35    -0.76 2.31

Intervention 1 has a higher mean (193 vs 176) and slightly more spread (SD 17.44 vs 13.88). Both groups look roughly symmetric (skew near zero), which is a good sign for normality.

Checking Assumptions

Before running a t-test, check two key assumptions: normality of the data within each group, and homogeneity of variance (equal spreads) between groups.

Shapiro-Wilk Test of Normality

The Shapiro-Wilk test assesses whether a variable is plausibly drawn from a normal distribution.

H₀: The data are normally distributed
H₁: The data are not normally distributed

A non-significant result (p > .05) means we fail to reject normality — the data are consistent with a normal distribution. Subset by group before testing:

## Test normality within each group
shapiro.test(scores$scores[scores$intervention == 1])
shapiro.test(scores$scores[scores$intervention == 2])

    Shapiro-Wilk normality test

data:  scores$scores[scores$intervention == 1]
W = 0.98198, p-value = 0.8101


    Shapiro-Wilk normality test

data:  scores$scores[scores$intervention == 2]
W = 0.9688, p-value = 0.3937

Both groups pass the normality test: W values close to 1 and p-values well above .05 indicate no evidence of departure from normality. We can proceed with confidence.

When normality is violated

If the Shapiro-Wilk test is significant (p < .05), the t-test is still reasonably robust with large samples (n ≥ 30 per group). For small samples with clear non-normality, consider a non-parametric alternative such as the Mann-Whitney U test (wilcox.test()).

One-Sample t-Test

A one-sample t-test asks whether a sample mean is significantly different from a known or hypothesized population mean (\(\mu_0\)). The test statistic is:

\[t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}\]

The key difference from the z-test is that we use the sample standard deviation \(s\) instead of a known population \(\sigma\). This introduces more uncertainty, which is accounted for by the t-distribution (discussed further at the end of this script).

One-Tailed Test

A one-tailed test has a directional alternative hypothesis: we predict the mean is either greater than or less than \(\mu_0\).

Is the sample mean significantly higher than a national average reading score of 150?

## H0: mean = 150
## H1: mean > 150  (one-tailed, upper)
t.test(scores$scores, mu = 150, alternative = "greater")

    One Sample t-test

data:  scores$scores
t = 16.41, df = 71, p-value < 2.2e-16
alternative hypothesis: true mean is greater than 150
95 percent confidence interval:
 180.9963      Inf
sample estimates:
mean of x
    184.5

The t-statistic of 16.41 is far into the upper tail, and the p-value is essentially zero. We reject H₀ and conclude that the sample mean (184.5) is significantly higher than 150.

Two-Tailed Test

A two-tailed test has a non-directional alternative: we only predict that the mean is different from \(\mu_0\).

## H0: mean = 150
## H1: mean ≠ 150  (two-tailed)
t.test(scores$scores, mu = 150, alternative = "two.sided")

    One Sample t-test

data:  scores$scores
t = 16.41, df = 71, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 150
95 percent confidence interval:
 180.3081 188.6919
sample estimates:
mean of x
    184.5

The t-statistic is the same (16.41) — only the CI and p-value calculation differ. The 95% CI [180.31, 188.69] does not include 150, confirming the result.

One-tailed vs. two-tailed: which to use?

Use a two-tailed test unless you have a strong, pre-registered directional prediction. Two-tailed tests are more conservative and are the default in most psychological research. Switching to one-tailed after seeing your data inflates Type I error.

Two-Sample t-Test (Independent Samples)

A two-sample independent t-test compares the means of two separate, unrelated groups. We look at the sampling distribution of the difference between means.

There are two versions depending on whether we assume the two groups have equal population variances.

Equal Variances Assumed

## H0: mean(group 1) = mean(group 2)
## H1: means are not equal
t.test(scores$scores ~ scores$intervention, var.equal = TRUE)

    Two Sample t-test

data:  scores$scores by scores$intervention
t = 4.5757, df = 70, p-value = 2e-05
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
  9.590143 24.409857
sample estimates:
mean in group 1 mean in group 2
            193             176

Unequal Variances (Welch’s t-Test)

## Welch correction: does not assume equal variances
t.test(scores$scores ~ scores$intervention, var.equal = FALSE)

    Welch Two Sample t-test

data:  scores$scores by scores$intervention
t = 4.5757, df = 66.636, p-value = 2.125e-05
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
  9.58356 24.41644
sample estimates:
mean in group 1 mean in group 2
            193             176

Equal vs. unequal variances: Welch’s correction

When variances are unequal, the Welch version adjusts the degrees of freedom downward (here, from 70 to 66.636) to compensate for the additional uncertainty. The t-statistic stays the same, but the p-value changes slightly.

In practice: Welch’s t-test (var.equal = FALSE) is recommended as the default. It performs just as well as the equal-variance test when variances are equal, and it is more accurate when they are not. Always run Levene’s test first (see below) to check the assumption.

Paired Samples t-Test

A paired t-test is used when the same participants (or matched pairs) provide scores under two conditions. Instead of comparing two independent groups, we compute the difference score for each participant and test whether the mean difference is zero.

The formula is the same as the one-sample t-test, applied to the difference scores:

\[t = \frac{\bar{d}}{s_d / \sqrt{n}}\]

Here we use data from the mindfulness stress study on the slides. Stress was measured before and after one month of mindfulness practice for 10 participants.

before <- c(195, 198, 201, 187, 205, 155, 187, 190, 234, 187)
after  <- c(194, 190, 200, 200, 152, 185, 190, 220, 180, 180)

## H0: mean difference = 0
## H1: stress is higher before than after (one-tailed: before > after)
t.test(before, after, alternative = "greater", paired = TRUE)

    Paired t-test

data:  before and after
t = 0.52277, df = 9, p-value = 0.3069
alternative hypothesis: true mean difference is greater than 0
95 percent confidence interval:
 -12.0314      Inf
sample estimates:
mean difference
            4.8

The mean stress score dropped by 4.8 points on average, but this difference is not statistically significant (t = 0.52, p = .31). With only 10 participants and substantial variability in stress change across individuals, we do not have enough evidence to conclude that mindfulness reduced stress.

Effect Sizes

Statistical significance tells you whether an effect is real — but not how big it is. Effect sizes are standardized measures that express the magnitude of a difference in units that can be compared across studies.

Cohen’s d (Independent Samples)

Cohen’s d divides the mean difference by the pooled standard deviation:

\[d = \frac{\bar{x}_1 - \bar{x}_2}{s_\text{pooled}}\]

Conventional benchmarks: small = 0.2, medium = 0.5, large = 0.8.

## install.packages("effectsize")
library(effectsize)

cohens_d(scores$scores ~ scores$intervention)

Cohen's d |       95% CI
------------------------
1.08      | [0.58, 1.57]

- Estimated using pooled SD.

d = 1.08 is a large effect — the two interventions differ by more than one full standard deviation. Despite the non-significant Levene test (variances are fairly similar), the practical difference between groups is substantial.

Cohen’s d (Paired Samples)

For paired data, Cohen’s d uses the standard deviation of the difference scores:

\[d = \frac{\bar{d}}{s_d}\]

You can also compute it manually before passing it to cohens_d():

before <- c(195, 198, 201, 187, 205, 155, 187, 190, 234, 187)
after  <- c(194, 190, 200, 200, 152, 185, 190, 220, 180, 180)

## Manual calculation
m_D  <- mean(before - after)   # mean difference
sd_D <- sd(before - after)     # SD of differences

m_D                   # 4.8
sd_D                  # 29.04
m_D / sd_D            # Cohen's d = 0.165

## Or let the package do it
cohens_d(before, after, paired = TRUE)

[1] 4.8
[1] 29.03561
[1] 0.1653142

Cohen's d |        95% CI
-------------------------
0.17      | [-0.46, 0.79]

d = 0.17 is a small effect and the CI crosses zero, consistent with the non-significant t-test result.

Hedges’ g (Corrected for Small Samples)

For small samples (n < 20), Hedges’ g applies a correction factor that shrinks Cohen’s d slightly downward to reduce bias:

hedges_g(before, after, paired = TRUE)

Hedges' g |        95% CI
-------------------------
0.15      | [-0.42, 0.72]

Cohen’s d vs. Hedges’ g

Both measure effect size, but Hedges’ g is preferred when your sample is small (roughly n < 20). The correction is small but matters: here d = 0.17 shrinks to g = 0.15. With larger samples the two converge. When in doubt, report Hedges’ g — it is the more conservative and unbiased estimate.

Levene’s Test: Can We Assume Equal Variances?

Before choosing between the equal-variance and Welch versions of the independent t-test, check whether the two groups have similar variance using Levene’s test.

H₀: The variances are equal (homogeneity of variance holds)
H₁: The variances are not equal

A significant result means the equal-variance assumption is violated — use Welch’s t-test.

## Install lawstat once if you haven't already:
## install.packages("lawstat")
library(lawstat)

levene.test(scores$scores, group = scores$intervention)

    Modified robust Brown-Forsythe Levene-type test based on the absolute
    deviations from the median

data:  scores$scores
Test Statistic = 1.1056, p-value = 0.2966

Reading the Levene output

Look at the p-value. If p > .05, you fail to reject H₀ and may proceed with the equal-variance t-test. If p < .05, variances differ significantly — use Welch’s (var.equal = FALSE).

Here p = .297 — well above .05. The variances are not significantly different, so the equal-variance assumption holds. That said, Welch’s t-test (var.equal = FALSE) is always a safe default because it performs equally well when variances are equal and is more accurate when they are not.

Visualizing the Results

Box Plot

A box plot shows the median, interquartile range, and any outliers for each group.

with(scores, boxplot(scores ~ intervention,
                     xlab = "Intervention Type",
                     ylab = "Reading Scores",
                     main = "Reading Scores by Intervention"))

Bar Graph with Error Bars

Bar graphs with error bars (showing ± 1 standard error) are standard for reporting group means in psychology.

## First, compute the descriptive statistics needed for the plot
aggregate(scores$scores, list(scores$intervention), FUN = mean)
aggregate(scores$scores, list(scores$intervention), FUN = sd)
table(scores$intervention)

  Group.1   x
1       1 193
2       2 176

  Group.1        x
1       1 17.44379
2       2 13.87907

 1  2
36 36

## SE = SD / sqrt(n)
17.44379 / sqrt(36)   # SE for Intervention 1
13.87907 / sqrt(36)   # SE for Intervention 2

[1] 2.907298
[1] 2.313178

## Build the bar graph
library(ggplot2)

intervention.names <- c("Intervention 1", "Intervention 2")
mean.scores        <- c(193, 176)
se.scores          <- c(2.907298, 2.313178)
scores.dataframe   <- data.frame(intervention.names, mean.scores, se.scores)

ggplot(scores.dataframe, aes(x = intervention.names, y = mean.scores)) +
  geom_bar(stat = "identity", width = 0.4, color = "black", fill = "white") +
  geom_errorbar(aes(ymin = mean.scores - se.scores,
                    ymax = mean.scores + se.scores),
                width = 0.1) +
  labs(x = "Intervention Type", y = "Reading Scores")

Reading error bars

Each error bar extends ±1 standard error above and below the bar. When error bars from two groups do not overlap, this is a visual signal (though not a formal test) that the group means may differ significantly.

t-Critical Values and the Link to z

Use qt() to find the t-critical value for any combination of alpha and degrees of freedom — the value your t-statistic must exceed to reach significance.

## qt(p, df, lower.tail = FALSE)
## p = alpha/2 for a two-tailed test (0.025 at alpha = .05)
## df = degrees of freedom

qt(0.025,    5, lower.tail = FALSE)
qt(0.025,   10, lower.tail = FALSE)
qt(0.025,   25, lower.tail = FALSE)
qt(0.025,   30, lower.tail = FALSE)
qt(0.025,  100, lower.tail = FALSE)
qt(0.025, 1000, lower.tail = FALSE)

[1] 2.570582
[1] 2.228139
[1] 2.059539
[1] 2.042272
[1] 1.983972
[1] 1.962339

As df increases, t approaches z

Notice the pattern: as degrees of freedom grow, the t-critical value falls toward 1.96 — the familiar z-critical value for a two-tailed test at α = .05.

df	t-critical (α = .05, two-tailed)
5	2.571
10	2.228
25	2.060
30	2.042
100	1.984
1000	1.962
∞ (z)	1.960

This is not a coincidence. The t-distribution has heavier tails than the normal distribution to account for the extra uncertainty introduced by estimating the population SD from a small sample. As n (and therefore df) increases, that estimate becomes more precise and the t-distribution gradually narrows until it is indistinguishable from z.

You can verify this with qnorm():

## The z-critical value (equivalent to t with infinite df)
qnorm(0.025, lower.tail = FALSE)

[1] 1.959964

With 1000 degrees of freedom the t-critical value (1.962) is already within 0.002 of the z-critical value (1.960). By the time you have a few hundred participants, the practical difference between a t-test and a z-test is negligible — the reason we use t is that for small samples, those extra decimal points matter.