Luna Frauhammer
This is the power simulation for the study “Explaining the Link between Attitudinal Congruence in Social Media and Political Participation: A Metacognitive Approach”. It estimates power for three hypotheses referring to effects of attitudinal congruence on subjective knowledge, effects of attitudinal congruence on political participation, and the relationship between these two variables. The remaining hypotheses regarding processing fluency and perceived controversiality are not addressed separately since past literature does not allow adequate estimation of effects. However, we believe the calculated sample size to be large enough to detect any effects of practical relevance for these hypotheses as well.
Based on the below analyses, we want to collect data of 1,200 participants. Regarding the group differences, this allows the detection of small standardized effect sizes of 0.15 with power of 1 - \(\beta\) > 0.80. Regarding the regression hypotheses, this sample size allows the detection of a small standardized effect of b = 0.08 with power of 1 - \(\beta\) \(\approx\) 0.85. For a more detailed explanation of why we chose these values, which effects we expect, as well as exact power estimates, please consult the remaining document.
Our estimates of the effect sizes are based on a paper of Wojcieszak et al. (2016), which we believe to be the published study most similar to our aim. See section 3 for a summary of their effects. However, we assume that our results will be somewhat larger than the results displayed in their paper for the following reasons:
Our study implements a different media context and therefore a different operationalization of the independent variable attitudinal congruence. The Wojcieszak paper presented the participants news articles that were written in a balanced, pro- or counter-attitudinal way. We present the participants social media news posts with opinionated user generated comments. We believe this manipulation to evoke stronger perceptions of attitudinal (in)congruence. Also, they do not only entail the opinion of one author but of several commenters. Finally, the Wojcieszak paper contrasted the pro-and counter-attitudinal conditions with a balanced condition, while our main analyses directly compare those two conditions.
Since we cannot predict exact effect sizes, we chose to determine the smallest effect size of interest (SESOI) while still considering the effects of the Wojcieszak paper to get a realistic estimate. We set the SESOI at d = 0.15 for the group differences indicating a small effect (standardized Wojcieszak effects were around d \(\approx\) 0.10 for SK and d \(\approx\) 0.15 for PP). For the relationship between Subjective Knowledge and Political Participation, we expect to find the same relationship as Wojcieszak et al. (2016), since this effect does not depend on the experimental manipulation.
To ensure full transparency and interpretability, we nonetheless report simulated power for the effects that Wojcieszak et al. (2016) observed in section 2.2 of this document.
The data is simulated so that there is a group difference in Subjective Knowledge of 0.21 units of measurement (1 - 7 scale, \(sd_{SK}\) = 1.4, standardized difference = 0.15). Political Participation (PP) is simulated to partially depend on Subjective Knowledge (b = 0.09, standardized b = 0.084) but to also have a group difference of 0.21 units of measurement (1 - 7 scale, \(sd_{PP}\) = 1.55) independent of Subjective Knowledge, resulting in an average group difference of 0.23 units of measurement (standardized = 0.15).
For the relationship between SK and PP we use the same effect size observed by Wojcieszak et al. (2016) since this relationship does not depend on the experimental manipulation. For the group differences, we use slightly higher values (again, see section 2.2 for the simulations using the exact Wojcieszak results).
n <- m <- 500 # set group sizes, sample size = n + m
nsim <- 2000 # number of simulations
p_SK <- rep(NA, nsim)
p_PP <- rep(NA, nsim)
p_Reg <- rep(NA, nsim)
beta <- rep(NA, nsim)
Effect_PP <- rep(NA, nsim)
sd_PP <- rep(NA, nsim)
beta_SK <- rep(NA, nsim)
power_SK <- power_PP <- power_Reg <- 0
Results <- data.frame(GroupSize = NA, SK = NA, PP = NA, Regression = NA)
j <- 1
while(power_SK < 0.8 | power_PP < 0.8 | power_Reg < 0.85){
for(i in 1:nsim){
# Simulate data for subjective knowledge:
SK1 <- rnorm(n, mean = 5.31, sd = 1.4) # data group 1 (congruent)
SK2 <- rnorm(m, mean = 5.1, sd = 1.4) # data group 2 (incongruent)
SK <- c(SK1, SK2)
group <- rep(c(0, 1), c(n, m))
# Simulate data for political participation:
# 1) simulate relationship between SK and PP
PP <- 2.8 + 0.09 * SK + rnorm(n = n + m, mean = 0, sd = 1.55)
# PP = intercept + b1 * SK + e
# intercept and b1 are adopted from Wojcieszak et al. (2016)
# residual variance is set so that the PP variance is around 1.55
data <- data.frame(group, SK, PP)
# 2) simulate experimental effect independent from SK:
data[data$group == 0, ]$PP <- data[data$group == 0, ]$PP + rnorm(n, 0.105, 0.1)
data[data$group == 1, ]$PP <- data[data$group == 1, ]$PP - rnorm(n, 0.105, 0.1)
# perform hypothesis tests and extract p-value:
p_PP[i] <- t.test(PP ~ group, data, var.equal = FALSE, alternative = "g")$p.value
p_SK[i] <- t.test(SK ~ group, data, var.equal = FALSE, alternative = "g")$p.value
p_Reg[i] <- coef(summary(lm(PP ~ SK, data = data)))["SK", "Pr(>|t|)"]
}
power_SK <- mean(p_SK < 0.05) # Power: group difference Subjective Knowledge
power_PP <- mean(p_PP < 0.05) # Power: group difference Political Participation
power_Reg <- mean(p_Reg < 0.05)
Results[j, ] <- c(n, power_SK, power_PP, power_Reg)
j <- j + 1
n <- n + 50 # increase group size
m <- m + 50
}| GroupSize | SK | PP | Regression |
|---|---|---|---|
| 500 | 0.766 | 0.7540 | 0.7695 |
| 550 | 0.791 | 0.7695 | 0.8145 |
| 600 | 0.831 | 0.8215 | 0.8570 |
Note. The table shows the power for the group differeces in Subjective Knowledge (SK) and Political Participation (PP) as well as the regression of Subjective Knowledge on Political Participation (Regression) based on the group size (sample size = 2 * group size). The highlighted row represents the finally selected sample size.
n <- m <- 500 # set group sizes, sample size = n + m
nsim <- 2000 # number of simulations
p_SK <- rep(NA, nsim)
p_PP <- rep(NA, nsim)
p_Reg <- rep(NA, nsim)
beta <- rep(NA, nsim)
Effect_PP <- rep(NA, nsim)
sd_PP <- rep(NA, nsim)
beta_SK <- rep(NA, nsim)
power_SK <- power_PP <- power_Reg <- 0
Results <- data.frame(GroupSize = NA, SK = NA, PP = NA, Regression = NA)
j <- 1
while(power_SK < 0.8 | power_PP < 0.8 | power_Reg < 0.8){
for(i in 1:nsim){
# Simulate data for subjective knowledge:
SK1 <- rnorm(n, mean = 5.25, sd = 1.42) # data group 1 (congruent)
SK2 <- rnorm(m, mean = 5.1, sd = 1.42) # data group 2 (incongruent)
SK <- c(SK1, SK2)
group <- rep(c(0, 1), c(n, m))
# Simulate data for political participation:
# 1) simulate relationship between SK and PP
PP <- 2.8 + 0.09 * SK + rnorm(n = n + m, mean = 0, sd = 1.55)
# PP = intercept + b1 * SK + e
# intercept and b1 are taken from Wojcieszak et al. (2016)
# residual variance is set so that the PP variance is around 1.55
data <- data.frame(group, SK, PP)
# 2) simulate experimental effect independent from SK:
data[data$group == 0, ]$PP <- data[data$group == 0, ]$PP + rnorm(n, 0.1, 0.1)
data[data$group == 1, ]$PP <- data[data$group == 1, ]$PP - rnorm(n, 0.1, 0.1)
# perform hypothesis tests and extract p-value:
p_PP[i] <- t.test(PP ~ group, data, var.equal = FALSE, alternative = "g")$p.value
p_SK[i] <- t.test(SK ~ group, data, var.equal = FALSE, alternative = "g")$p.value
p_Reg[i] <- coef(summary(lm(PP ~ SK, data = data)))["SK", "Pr(>|t|)"]
}
power_SK <- mean(p_SK < 0.05) # Power: group difference Subjective Knowledge
power_PP <- mean(p_PP < 0.05) # Power: group difference Political Participation
power_Reg <- mean(p_Reg < 0.05)
Results[j, ] <- c(n, power_SK, power_PP, power_Reg)
j <- j + 1
n <- n + 50 # increase group size
m <- m + 50
}| GroupSize | SK | PP | Regression |
|---|---|---|---|
| 500 | 0.5035 | 0.6900 | 0.7650 |
| 550 | 0.5390 | 0.7240 | 0.8185 |
| 600 | 0.5790 | 0.7680 | 0.8415 |
| 650 | 0.6190 | 0.8085 | 0.8740 |
| 700 | 0.6385 | 0.8200 | 0.8815 |
| 750 | 0.6490 | 0.8215 | 0.8955 |
| 800 | 0.6705 | 0.8635 | 0.9265 |
| 850 | 0.7125 | 0.8850 | 0.9425 |
| 900 | 0.7295 | 0.8920 | 0.9585 |
| 950 | 0.7345 | 0.9085 | 0.9600 |
| 1000 | 0.7470 | 0.9160 | 0.9750 |
| 1050 | 0.7950 | 0.9440 | 0.9675 |
| 1100 | 0.7985 | 0.9380 | 0.9770 |
| 1150 | 0.7990 | 0.9440 | 0.9845 |
| 1200 | 0.8265 | 0.9620 | 0.9895 |
Note. The table shows the power for the group differeces in Subjective Knowledge (SK) and Political Participation (PP) as well as the regression of Subjective Knowledge on Political Participation (Regression) depending on the group size (sample size = 2 * group size). The highlighted row represents the finally selected sample size.
Subjective Knowledge:
M = 5.2, SD = 1.42, relationship SK ~ pro attitudinal exposure: a = 0.15 (0.05)
Political Participation:
M = 3.36, SD = 1.55,
Group Means (circa): M_balanced = 3.32, M_pro = 3.56, M_counter = 3.34
Relationship PP ~ pro attitudinal exposure: b = .20 (.06)
Relationship PP ~ SK: b = 0.09 (0.02)
Wojcieszak, M., Bimber, B., Feldman, L., & Stroud, N. J. (2016). Partisan news and political participation: Exploring mediated relationships. Political Communication, 33(2), 241-260.