Accepted Manuscript (AM) / Post-print (final draft post-refereeing)
This article has been accepted for publication and will appear in a revised form, subsequent to peer review and editorial input by Cambridge University Press, in British Journal of Nutrition, published by Cambridge University Press.
The validity of a web-based FFQ assessed by doubly labelled water and multiple 24-h recalls.
Medin AC, Carlsen MH, Hambly C, Speakman JR, Strohmaier S, Andersen LF.
Br J Nutr. 2017 Dec;118(12):1106-1117. doi: 10.1017/S0007114517003178. Epub 2017 Dec 5. PMID: 29202890
https://www.cambridge.org/core/journals/british-journal-of-nutrition/article/validity-of-a- webbased-ffq-assessed-by-doubly-labelled-water-and-multiple-24h-
recalls/68B7B7A770C87C7BCF98B408D40B2DA0
© 2017 Anine Christine Medin All rights reserved.
Title page
Title of the article:
The validity of a web-based food frequency questionnaire assessed by doubly labelled water and multiple 24-hour recalls
Authors’ names:
A.C. Medin1, M.H. Carlsen1, C. Hambly2, J.R. Speakman2, 3, S. Strohmaier4, 5, L.F.
Andersen1.
Authors’ affiliations:
1 Department of Nutrition, Institute of Basic Medical Sciences, University of Oslo, Oslo, Norway.
2 Institute of Biological and Environmental Sciences, University of Aberdeen, Aberdeen, Scotland, UK.
3 State key laboratory of Molecular Developmental Biology, Institute of Genetics and Developmental Biology, Chinese Academy of Sciences, Beijing, China.
4 Department of Biostatistics, Institute of Basic Medical Sciences, University of Oslo, Oslo, Norway.
5 Channing Division of Network Medicine, Department of Medicine, Brigham and Women's Hospital, and Harvard Medical School, Boston, USA.
Corresponding author:
A.C. Medin
Department of Nutrition, Institute of Basic Medical Sciences, University of Oslo, Oslo, Norway. Address: P.O. Box 1046, Blindern, N-0317 Oslo, Norway.
Phone: +47- 22851349 Cellphone: +47-47463893 Fax: +47-22851249 E-mail: [email protected]
Short title:
The validity of a web-based FFQ
Keywords:
dietary assessment; food frequency questionnaire; web-based; validation; doubly labelled water
Abstract 1
The aim of this study was to validate the estimated habitual dietary intake from a newly 2
developed web-based food frequency questionnaire (WebFFQ), for use in an adult population 3
in Norway. In total 92 individuals were recruited. Total energy expenditure (TEE) measured 4
by doubly labelled water was used as the reference method for energy intake in a subsample 5
of 29 women, and multiple 24-hour recalls (24HRs) were used as the reference method for the 6
relative validation of macronutrients and food groups in the entire sample. Absolute 7
differences, ratios, crude and deattenuated correlations, cross-classifications, Bland-Altman 8
plot, and plots between misreporting of energy intake (EI-TEE) and the relative misreporting 9
of food groups (WebFFQ-24HRs) were used to assess the validity. Results showed that 10
energy intake on group level was not significantly different from total energy expenditure 11
measured by doubly labelled water (0.7 MJ/day), but ranking abilities were poor (r= -0.18).
12
The relative validation showed an overestimation for the majority of the variables using 13
absolute intakes, especially for the food groups ‘vegetables’ and ‘fish and shellfish’, but an 14
improved agreement between the test and reference tool was observed for energy adjusted 15
intakes. Deattenuated correlation coefficients were between 0.22-0.89, and low levels of 16
grossly misclassified individuals (0-3%) were observed for the majority of the energy 17
adjusted variables for macronutrients and food groups. In conclusion, energy estimates from 18
the WebFFQ should be used with caution, but the estimated absolute intakes on group level 19
and ranking abilities seem acceptable for macronutrients and most food groups.
20
Introduction 21
An unhealthy diet is recognized as being among the main modifiable risk factors for the major 22
non-communicable diseases globally (1,2), thus measuring and targeting diet, is important.
23
However, as no objective biomarkers of total diet yet exist (3), dietary assessments cannot 24
avoid using some form of self-reported data. The limitations of self-reported data should not 25
be downplayed, and well-conducted validation studies are therefore extremely important, to 26
quantify how much the estimated dietary intake deviates from the unknown true intake.
27
Among the existing dietary self-report assessment methods, the food frequency questionnaire 28
(FFQ) and the 24-hour recall (24HR) are much used and validated tools; however, the FFQ is 29
especially found to have considerable limitations (4,5). The FFQ is nonetheless popular, 30
particularly in large epidemiological studies, because it is designed to capture the habitual 31
dietary intake, and it can be applied in large numbers of individuals, at a relatively low cost 32
(6,7). In comparison, the 24HR has proven superior to the FFQ in terms of accuracy (8), but 33
repeated recalls are needed when assessing the distribution of intakes in a group, or individual 34
intakes (6,7). 35
New technology has been proposed as a way to reduce the challenges associated with the self- 36
report dietary assessment methods; shifting from paper-based FFQs with limiting printed 37
formats, to web-based FFQs with possible skip algorithms and images for improved portion 38
size estimates (9). Web –and computer formats permit inherent error checks, avoiding 39
incomplete recordings and inconsistency, and add additional value in reducing the burden of 40
data handling (10,11). 41
A web- and image-based, self-administered food frequency questionnaire, the WebFFQ, has 42
been recently developed at the University of Oslo (UiO), to replace the much used paper- 43
based FFQ (12). As any new tool, the WebFFQ needs to be validated to reveal how it performs, 44
and to clarify how data from the WebFFQ can be used and interpreted in future studies.
45
The main aim of this study was to assess the validity of estimated intakes from the WebFFQ, 46
using two different reference methods; an absolute validation of energy intakes using doubly 47
labelled water (DLW), and a relative validation of macronutrients and food groups using 48
repeated non-consecutive 24HRs. A supplementary aim was to assess the validity of energy 49
intake (EI) estimated from the second reference method (24HRs) using DLW.
50
Methods 51
Design 52
A total of 92 participants were recruited over two rounds. Group 1, consisting of women only, 53
was recruited in November 2015, and the data collection was conducted from January to June 54
2016. Group 2, consisting of both women and men, was recruited and data collected, in the 55
period from March to December 2016.
56
Both written and verbal information regarding the study was provided to all participants. All 57
participants were instructed to fill out the WebFFQ, covering their habitual dietary intake, 58
over the last 12 months. Subsequently, four non-consecutive 24HRs were collected for all 59
participants by trained nutritionists, using telephone interviews. In addition, the participants in 60
group 1 had their total energy expenditure assessed by the doubly labelled water (DLW) 61
method.
62
Ethical statement 63
This study was conducted according to the guidelines laid down in the Declaration of Helsinki 64
and all procedures involving human subjects were approved by the Data Protection Official 65
for Research in Norway (NSD), project numbers: 44876 and 45712. Written informed consent 66
was obtained from all participants. No economical compensation or incentives were given to 67
the participants.
68
Recruitment 69
An overview of the recruitment process is shown in Figure 1. Group 1 was recruited using 70
Facebook, posters and word of mouth. During a period of two weeks, 58 women volunteered 71
to participate, of which 42 fulfilled the inclusion criteria. Out of these women, 32 with the 72
least similar traits, defined by age, self-reported body weight and height, self-reported 73
physical activity level, and area where they lived, were included in the study. This was done 74
to increase variability in the sample, and to include only the number of individuals needed, 75
based on sample size calculations. Before the commencement of the study, one participant 76
withdrew and was replaced by one of the 10 formerly omitted individuals, who fulfilled 77
inclusion criteria. All 32 completed all parts of the study.
78
Group 2 was recruited from a random selection of the Norwegian population aged between 79
18-70 years. The sample was drawn by the Norwegian Tax Administration. A total of 300 80
received invitations, out of which 200 were a random mix of both sexes and 100 were a 81
random selection of men. More men than women were invited in group 2, to equalize the sex 82
ratio in the entire sample. Potential participants were sent a written invite, followed up by a 83
phone call within one to two weeks. Text messages or voice-mail were used if no contact was 84
established, and if needed a new phone call was made again after a few days.
85
Inclusion and exclusion criteria 86
Stricter criteria were used for group 1 than for group 2, as the DLW method was used only in 87
group 1. However, all had to be between the age 18-70 years, born in Scandinavia, and have 88
access to a computer and internet. Any present or former students in nutrition or sports 89
nutrition were excluded.
90
In addition, those included in group 1 had to be healthy, female, have a BMI 18.5-35 kg/m2 91
and a domestic freezer in their home (for sample storage), and live within Oslo or surrounding 92
areas to fulfil the inclusion criteria. Women who were pregnant, breastfeeding or had given 93
birth during the last 10 months were excluded. Furthermore, women with self-reported weight 94
fluctuations >2.5 kg over the last three month period, women planning to increase or lose 95
weight, and professional athletes were also excluded.
96
The web-based food frequency questionnaire (WebFFQ) 97
The WebFFQ was developed by researchers from the Department of Nutrition and staff at the 98
University Center for Information Technology, both at the University of Oslo, based on the 99
experience from former paper based FFQs (13,14). 100
The WebFFQ is designed as a web-based, self-administered food frequency questionnaire, 101
assessing the habitual intake for an individual, asking about their diet over the past 12 months.
102
Access is provided by a direct link sent to each participant’s email. It contains 279 foods or 103
beverages, with images illustrating different portions sizes to help the portion size estimation.
104
Skip-algorithms are used to reduce the burden on the participants; that is, entire food main 105
categories (i.e. cereals) are bypassed if the participant indicates that such foods are never 106
consumed. Inherent error checks are used to minimize unintentional oversights: the 107
participant cannot proceed without ticking off the boxes for each question on each page.
108
Questions on background variables (i.e. age and educational level) are at the very end of the 109
FFQ. The data collected in the WebFFQ on frequency of consumption and portion sizes were 110
converted into grams per day, using standard procedures (15), before it was imported into the 111
food and nutrient composition database and calculation system KBS (KBS, version 7.3, 112
database AE14, University of Oslo, Oslo, Norway), to allow calculations of energy, nutrients 113
and food groups. Calculations of energy intake were done using standard procedures (SI 114
units) for the energy providing nutrients (16). 115
Doubly labelled water 116
Total energy expenditure (TEE) was measured using the doubly labelled water (DLW) 117
technique (17), in all participants in group 1, for comparison with estimates of EI from the 118
WebFFQ. This method has been previously validated on multiple occasions by comparison to 119
simultaneous indirect calorimetry in humans (18). 120
After completing the WebFFQ, participants were individually paid a total of three home 121
visits. During the first visit, they were provided with equipment for sampling and storage of 122
urine samples. Visit two included collection of a baseline (pre-dose) urine sample, to estimate 123
background isotope enrichment, and assessment of height and weight, before dosing with 124
DLW. A multi-sample protocol over a period of two weeks was used. The DLW doses with 125
mixed isotopes were prepared individually, based on participants self-reported body weight, 126
by technical staff from the Energetics group, University of Aberdeen, Scotland, UK. The 127
isotopes, 18O and deuterium, were purchased from Sercon (Crewe, UK). The calculated 128
enrichment of the mixed DLW was 109203.1 ppm 18O and 47193.7 ppm deuterium and the 129
dose was 1.2 ml per kg body mass. Dosing was done in the mornings, from a sealed cup, in 130
the fasting state. Two post-dose urine samples were collected by the participants the same day 131
to obtain the initial isotope enrichments: one approximately three-four hours after dosing, and 132
subsequently another in the evening. Further urine samples (evening void) were collected 133
every other day until day 14. Precise times of all samples were recorded. All urine samples 134
were kept frozen in the participants’ domestic freezers until the third home visit, during which 135
samples were collected and subsequently brought to the laboratory at the Department of 136
Nutrition, University of Oslo. Weight of the participants was also measured at the third home 137
visit, to assess weight stability during the sampling period.
138
Urine samples were thawed, well mixed and pipetted from the urine specimen containers into 139
cryotubes, which were kept at -80 degrees Celsius, until shipped on dry ice from Oslo, 140
Norway to, Aberdeen, Scotland, UK, where they were kept frozen until analysis. Blinded 141
analysis of the isotopic enrichment of urine was performed, using a Liquid Isotope Water 142
Analyser (Los Gatos Research, USA) (19). First, the urine was vacuum distilled (20), and the 143
produced distillate was used for analysis. Samples were run alongside five lab standards for 144
each isotope and International standards (GISP, SMOW and SLAP) to correct for day-to-day 145
variation, and the data was converted from delta values to ppm. For each sample, 15 replicates 146
were analysed. The average within day error in deuterium replicates after stability had been 147
reached was 0.05 ppm and for 18O was 0.12 ppm. The average between day error in deuterium 148
was 0.08 ppm and for 18O was 0.87 ppm. The mean isotope enrichments in each sample, after 149
accounting for background levels, were loge transformed and the elimination constants (ko and 150
kd) were calculated by fitting a least squares regression model to the loge transformed data. To 151
calculate the isotope dilution spaces (No and Nd), the back extrapolated intercept was used. A 152
two-pool model, using Schoeller et al.’s equation A6 (21), in its modified form (22) was used to 153
calculate rates of CO2 production as recommended for humans by Speakman (23) using an 154
assumed food quotient of 0.85 (24). 155
The interviewer-assisted computer-based 24-hour multi-pass recall module 156
Intake data from 24HRs were used as a relative reference method to the WebFFQ. An 157
interviewer-assisted and computer-based 24-hour multi-pass recall module, integrated and 158
directly connected to the nutrition composition database KBS (KBS, version 7.3, database 159
AE14, University of Oslo, Oslo, Norway) was used, as described elsewhere (25). In short, the 160
24HR-module is used in a three-step sequence; first, the interviewee freely describes what 161
was consumed the previous day; secondly the interviewer repeats all items that are reported, 162
chronologically, and adds questions about portion sizes, plausible overlooked extra items (i.e.
163
milk, if cereals are reported without milk), and possibly omitted eating occasions; finally, the 164
interviewer prompts for commonly forgotten items, including supplements. All participants in 165
the current study had access to a booklet with images of different portion sizes, in paper 166
format or electronically as a PDF file.
167
Three trained interviewers, all with five years of formal nutrition educational background, 168
conducted the interviews by telephone. Four non-consecutive 24HRs were completed for each 169
participant. One out of the four days had to be a Friday, Saturday or Sunday, as people tend to 170
eat differently on these days compared to the rest of the week (26). To avoid reactivity, 171
interviews were predominantly not pre scheduled (93%); that is, the participants did not know 172
in advance which days they were to be interviewed.
173
Anthropometrics 174
All participants self-reported weight and height in the WebFFQ.
175
Additionally, participants in group 1 had their weight and height measured in their home 176
during home-visits. Height was measured once using a portable stadiometer (Seca 213, Seca 177
GmbH & Co. KG., Hamburg, Germany) to the nearest mm. Weight was measured twice on a 178
digital scale (TANITA TBF-300, Tanita Corporation, Tokyo, Japan) to the nearest 0.1 kg;
179
first at the day of dosing, and secondly, the day after the last urine sample was sampled. Both 180
weight measurements were done in the morning, in the fasting state, after emptying the 181
bladder. Only underwear or very light clothing was allowed during weighing.
182
Other information 183
Questions regarding educational level, smoking habits and birth date were included in the 184
WebFFQ. Also, information regarding physical activity level was provided by group 1 185
participants over the phone, at the time of evaluation of possible inclusion in the study.
186
Statistical analyses 187
Descriptive statistics were computed for the total study sample, and by participant group and 188
sex, given as mean and SD or as percentage. Chi-square and Mann-Whitney tests were used to 189
compare groups. Paired sample t-test was used to compare measured weight at baseline and 190
the second weighing, and measured weight at baseline to self-reported weight, in group 1.
191
The absolute validity of estimated EI from the WebFFQ (EIFFQ), and for the mean of four 192
24HRs (EI24HR), was assessed for group 1 (n=29), using TEE from DLW (TEEDLW) as the 193
reference method. Mean and SD of EIFFQ, EI24HR and TEEDLW were computed, in addition to 194
ratios between their means. Further comparisons of means were done using paired sample t- 195
tests, after loge transformations, due to skewed data.
196
Crude Pearson’s correlations were calculated between EIFFQ and TEEDLW, and between EI24HR
197
and TEEDLW, using loge transformed data, to deal with the non-normally distributed data. To 198
take into account the within-person variation in EI in the 24HR-data, we calculated the 199
deattenuated Pearson’s correlation coefficient rd using the formula from Beaton et al (27), using 200
data on EI for each recording day, for each individual. Scatterplots were also created for EIFFQ
201
and TEEDLW, and EI24HR and TEEDLW, respectively.
202
A Bland-Altman plot was created for the difference between EIFFQ and the TEEDLW, and the 203
mean of the two.
204
To identify acceptable reporters of energy intake (AR), we calculated the ratio of EIFFQ to 205
TEEDLW. A perfect agreement between the methods would give EIFFQ: TEEDLW = 1. Due to 206
the skewness in EI data, the ratio was subsequently loge transformed. ARs were defined as 207
subjects within the range of the 95% confidence limits of agreement (95% CI) for EIFFQ: 208
TEEDLW, calculated in accordance with Black et al (28), on the loge ratio scale. Because the 209
WebFFQ refers to habitual intake, the number of assessment days can be taken as infinite; the 210
coefficient of variation (CV) for EIFFQ was therefore set to 0, whereas the CV for TEEDLW was 211
set to 8.2% (29), giving a 95% CI ±16% for the loge transformed EIFFQ: TEEDLW. Individuals 212
who were defined to be within these CL were defined as ARs.
213
Quartiles for EIFFQ, EI24HR and TEEDLW were created, and the WebFFQ’s and 24HRs’ ability 214
to correctly classify their respectively estimated EIs compared to TEEDLW were assessed.
215
A relative validation was conducted for the entire sample (n=92), assessing macronutrients 216
and food groups. Median intakes and 25 and 75 percentiles were calculated. Absolute intakes 217
are presented in g/day. Simple energy adjustments were done by calculating energy 218
percentage (E%) for macronutrients, and intakes per 10 MJ for fibre and all food groups.
219
Wilcoxon signed rank test for related samples, was used to test for differences in median 220
intakes between the WebFFQ and the 24HRs. The ratio of the WebFFQ to the 24HRs, using 221
median intakes, was also calculated. Crude Pearson’s correlations were calculated for 222
nutrients and food groups between the WebFFQ and the mean of four 24HRs using loge
223
transformed data. The formula from Beaton et al (27) was used to calculate deattenuated 224
Pearson’s correlation coefficient rd. The WebFFQ’s ability to correctly classify nutrient or 225
food intake of individuals compared to dietary intake data from the 24HRs was assessed.
226
Quartiles were created using estimated intakes from the WebFFQ and 24HR data for nutrients 227
and food groups using both absolute intakes and energy adjusted intakes. Proportions of 228
individuals classified into the same, adjacent and extreme opposite quartile were calculated.
229
Finally, the absolute difference between EIFFQ and TEEDLW was plotted against the difference 230
in grams between the WebFFQ and 24HRs, for the food groups having a significantly 231
different absolute estimated intake between the two methods. Pearson’s correlation 232
coefficients were subsequently calculated for the respective variables in these plots, except for 233
skewed variables in which Spearman’s nonparametric alternative was used.
234
All data analyses were conducted using IBM SPSS (version 22.0, 2013, IBM Corp, Armonk, 235
NY, USA) and MS Excel (version 2010, Microsoft, Redmond, WA, USA).
236
Power calculations 237
For the doubly labelled water analyses, in which only the participants in group 1 were 238
included, sample size was calculated based on the ability to identify acceptable reporters (AR) 239
of energy. ARs were defined as individuals within the 95% CI for EIFFQ: TEEDLW, described 240
previously. Thus, a difference of 16% between reported EI and TEEDLW needed to be 241
detectable. Using the equation from Cole (30), based on an expected mean EI of 8.0 MJ and SD 242
of 2.4 MJ sourced from the latest nationwide Norwegian dietary survey (31), a power of 80%
243
and a 5% significance level, a total of 27 participants were needed. To account for expected 244
dropouts and invalid samples, 32 participants were recruited.
245
For the relative validation analyses, all participants from both group 1 and group 2 were 246
included. Data from 92 participants was available. For a sample this size, a significance level 247
of 5% and 80% power, it would be possible to detect a correlation of minimum 0.26 (32). 248
Results 249
Characteristics of participants 250
Characteristics of the study sample are presented in Table 1. Out of the 92 participants, 37.0%
251
were male, 68.5% had higher education, and 10.9% were smokers. Mean age was 44.4 years, 252
and mean BMI was 24.5 kg/m2. Participants, in group 1 (all women), were different than 253
group 2, having a 1.0 kg/m2 lower average BMI (p=0.04), a higher educational level (p=0.02), 254
in addition to being 9 years younger on average (p<0.001). During the sampling period, we 255
observed a non-significant mean weight change of 0.1 kg between baseline and the second 256
weighing (p=0.72), implying that group 1 was weight stable. Additionally, no significant 257
difference was observed between the mean self-reported and measured weight in group 1 258
(p=0.98).
259
Absolute validity of estimated energy intake 260
Out of the 32 participants in group 1, three had non-valid samples and were consequently 261
excluded, leaving 29 to be included in the statistical analyses. The ratio of the elimination 262
constants ko/kd was 1.25 ± 0.001 and the dilution space ratio Nd/No was 1.05 ± 0.004. On average 263
across all individuals, the EIFFQ was 0.7 MJ (6%) lower, but not significantly different, than 264
the TEEDLW (p=0.22), on group level (Table 2). In comparison, on average the EI24HR was 265
underestimated significantly with 1.9 MJ (17%) compared to the TEEDLW (p<0.001).
266
Pearson’s correlation between EIFFQ and TEEDLW showed no significant linear relationship (r=
267
-0.18), see Figure 2 (A). The deattuenuated Pearson’s correlation observed between TEEDLW
268
and the EI24HR was stronger (r= 0.34), see Figure 2 (B).
269
The Bland-Altman plot in Figure 3 displays difference between energy estimates from the 270
WebFFQ and the DLW method, against the average of the measurements of each individual 271
in group1. Over-reporting and under-reporting of EI is spread widely but evenly out, 272
resulting in the small mean difference between the methods. The plot reveals that the 273
individual EIFFQ deviate largely from the individual TEEDLW and only 14 out of 29 individuals 274
were identified as acceptable reporters of EI (Figure 3).
275
Cross-classification between quartiles of EIFFQ and TEEDLW showed that 52% of the 276
participants were classified in the same or adjacent quartile, and 21% were grossly 277
misclassified (opposite quartiles). In comparison, for EI24HR and TEEDLW, the proportion of 278
individuals classified in the same or adjacent quartiles, versus the grossly misclassified were 279
66% and 7%, respectively.
280
Relative validity of macronutrients and food groups 281
The relative validity for the energy providing nutrients, including alcohol and fibre, and 282
several food groups, is presented as absolute intakes (Table 3) and energy adjusted intakes 283
(Table 4). The absolute estimated intakes (g/day) from the WebFFQ, were significantly 284
overestimated compared to the 24HRs, for 68% of the variables. ‘Cheese’ was the only 285
significantly underestimated variable. ‘Alcohol’ had the least discrepancy between the two 286
methods, and the largest overestimations by the WebFFQ were observed for ‘vegetables’ and 287
‘fish and shellfish’, followed by ‘cereals’, ‘fibre’ and ‘butter, margarine, oil’. Less 288
overestimation was observed for energy adjusted intakes, for which 32% of the variables were 289
significantly overestimated, 53% were not significantly different, and ‘cheese’ and ‘cakes’
290
were the only underestimated variables, by the WebFFQ relative to the 24HRs. The under- 291
and over-reporting of absolute estimated intakes of food groups by the WebFFQ relative to 292
the 24HRs, were mostly spread out between the over- or under-reporters of energy: No 293
significant correlations between energy deviations and these food deviations were observed 294
except for ‘fish and shellfish’, in which a significant positive correlation (r=0.48) was found.
295
See Figure 4 (A-D) for selected plots showing: ‘cheese’, ‘vegetables’, ‘fish and shellfish’ and 296
‘cereals’. Similar patterns were observed for the other food groups.
297
Crude and deattenuated Pearson’s correlations for absolute intakes varied from 0.19-0.69 and 298
0.22-0.89, respectively (Table 3). The strongest correlations were observed for ‘milk, cream, 299
ice cream and yoghurt’, ‘juice’ and ‘fruits and berries’, all at 0.80 or more after adjusting for 300
within-person variation. The weakest correlations were observed for ‘fibre’, ‘eggs’, ‘potatoes’
301
and ‘cakes’, all below 0.40, even for the deattenuated correlations. An improvement in the 302
linear relationship adjusted for within-person variation was observed for 68% of the variables 303
when shifting from absolute intakes to energy adjusted intakes (Table 3 and 4); the largest 304
improvements were observed for ‘vegetables’, ‘protein’ and ‘fibre’.
305
In Table 3, cross-classifications between quartiles of absolute intakes from the WebFFQ and 306
quartiles of absolute intakes from the 24HRs are shown. For the majority of the variables no 307
more than 5% of participants were grossly misclassified. The most correctly classified 308
variables were ‘milk, cream, ice cream and yoghurt’ and ‘juice’, whereas the least correctly 309
classified variables were ‘carbohydrates’, ‘fibre’, ‘vegetables’ ‘cakes’ and ‘fish and shellfish’.
310
The cross-classifications were improved when using energy adjusted intakes (Table 4) instead 311
of absolute intakes (Table 3). The variables ‘vegetables’ and ‘fish and shellfish’ had the 312
largest improvement; the percentage of grossly misclassified was reduced from 8% and 7% to 313
3% and 2%, respectively. Consequently, low levels of grossly misclassified participants (0- 314
3%) were observed for more than 63% of the energy adjusted variables.
315
Discussion 316
Results showed no significant difference between estimated EI from the WebFFQ and the 317
TEE from DLW on group level. However, the WebFFQ’s ranking abilities for energy intake 318
were unsatisfactory. By contrast, the 24HRs showed a significant underestimation of EI at 319
group level, but better ranking abilities for energy intake. When comparing absolute intakes of 320
macronutrients and food groups from the WebFFQ to the 24HRs, we observed a general 321
overestimation of estimated intakes by the WebFFQ on the group level, and Pearson’s 322
correlations in the range of 0.19-0.69. Adjusting for within-person variation improved 323
correlation coefficients, and the use of energy adjusted intakes compared to absolute intakes 324
improved both correlations and cross-classifications for most macronutrients and foods 325
groups.
326
Absolute validity of estimated EI from the WebFFQ 327
In a Norwegian validation study of a paper-based FFQ, on which the WebFFQ in our study 328
builds upon, DLW was used in a group of women; EI was underreported modestly by a mean 329
of 0.96 MJ/day (compared to 0.70 MJ/day reported here), but the Bland-Altman plot showed 330
large differences between the methods at the individual level (33). These results conform to the 331
observations in the present study. Based on this, it looks like the WebFFQ tool is neither 332
superior nor worse in estimating EI than the paper-based FFQ.
333
Underreporting of energy in dietary self-reported methods has been reported previously, 334
amongst others in the study of Freedman et al., who pooled results from five large validation 335
studies using recovery biomarkers, including TEE measured by DLW (8). Specifically, for 336
women, Freedman et al., report an average rate of under-reporting of EI of 28% with FFQs (8). 337
In comparison, the mean EI was only underreported by 6% in our study. This shows that on 338
group level, the WebFFQ seems to perform more superiorly than several other FFQs.
339
However, the group mean is a result of large over- and under-reporting of energy on the 340
individual level that cancelled each other out. The evenly spreading out of over- and under- 341
reporting of energy in the present study may have been influenced by the sampling, as we 342
attempted to increase the variability in age, BMI and physical activity. Moreover, Freedman 343
et al. reported deattenuated correlations for women in the range of 0.11-0.34 between the 344
estimated EI from the FFQ and TEE measured from DLW. Our observations from group 1 are 345
quite similar to these results, showing that our WebFFQ, like several other FFQs, is unsuited 346
for ranking individuals correctly according to reported EI.
347
Absolute validity of estimated EI from the 24HRs 348
For the 24HRs, we observed an underestimation of EI of 17%, compared to the TEE from 349
DLW, which is in line with the underreporting found for 24HRs in other studies among adults 350
in western countries (34). Despite a thorough multi-pass approach and the use of images for 351
portion size estimation, some foods or beverages were probably omitted or forgotten, and/or 352
portion sizes were underestimated, which previously have been identified as a source of error 353
(35). However, Pearson’s deattenuated correlation and cross-classification showed reasonable 354
ranking abilities. This is similar to observations from Freedman et al. who reported 355
deattenuated correlations for women in the range of 0.27-0.42 between the estimated EI from 356
the mean of three 24HRs and TEE measured from DLW (8). In our study we do not know what 357
foods or beverages contributed the most to the observed underreporting of energy in the 24HR 358
estimates, yet it is of importance to take the underreporting into account when interpreting the 359
results from the relative validation of the WebFFQ, in which the mean of four 24HRs was 360
used as the reference.
361
Relative validity of macronutrients and food groups estimated by the WebFFQ 362
A satisfying agreement on group level between the WebFFQ and mean of the four 24HRs 363
were observed for the macronutrients for energy adjusted intakes. However, for absolute 364
intakes, the WebFFQ overestimated the intake of all macronutrients significantly, relative to 365
the 24HRs, except for alcohol. This trend of overestimation by FFQs compared to multiple 366
24HRs or food records is also observed in a number of other studies (36-39), although reports on 367
underestimation are also found (40,41). We speculate that the observed overestimation of 368
absolute intakes of macronutrients by the WebFFQ may partly be artificially overestimated, as 369
a result of the underestimation of energy observed for the 24HRs, compared to the DLW data.
370
The observed ranking abilities of the WebFFQ, relative to the 24HRs for macronutrients, are 371
comparable to what have been found in other studies; the observed proportions of grossly 372
misclassified individuals for the E% of protein, fat and alcohol, except for carbohydrates, 373
were slightly lower in our study, compared to a Swedish relative validation study between two 374
web-based FFQs and a 7-days weighed food record (41). Moreover, the deattenuated energy 375
adjusted correlations for macronutrients found in the present study are also conforming to the 376
Swedish study (41), a study of an Ecuadorian FFQ compared to 3×24HRs (36), and a study of a 377
Chinese web-based FFQ compared to a 3-day record (37). 378
Food groups were also assessed in this validation study, because food groups and food 379
patterns are growingly used as a measure of dietary exposure (42). The WebFFQ overestimated 380
the absolute intake significantly for all food groups, in the range of 3-120%, except for 381
‘juice’, ‘cakes’, ‘eggs’, ‘cheese’ and ‘sweets, desserts, sugar’, demonstrating that the 382
agreement on the group level varied substantially. As speculated for the macronutrients, the 383
overestimation observed for food groups may partly reflect a true underreporting by the 384
reference instrument, rather than, or in addition to, an overestimation by the WebFFQ. Yet, 385
especially for ‘vegetables’ and ‘fish and shellfish’ the reported intakes from the WebFFQ are 386
remarkably large, relative to the 24HRs, even for the energy adjusted intakes. Due to the 387
extent of overestimation, we argue that this most likely reflects a true overestimating of these 388
variables, perhaps caused by a social desirability bias.
389
By combining data from the validation of estimated EI from the WebFFQ using DLW, and 390
the relative validation of the WebFFQ compared to the 24HRs, it was possible to demonstrate 391
how misreporting of different food groups was distributed in relation to misreporting of 392
energy. The plots showed that the direction and magnitude of misreporting of food groups 393
were mainly evenly distributed between acceptable reporters of energy and those who under- 394
reported or over-reported their EI by the WebFFQ, indicating that misreporting of energy is 395
associated with misreporting of many foods.
396
Comparing food groups across different studies can be challenging, because of discrepancies 397
in how foods are grouped, and due to cultural differences in what is eaten. Nevertheless, some 398
of our observations for Pearson’s correlations between estimated intakes of food groups (i.e.
399
vegetable, milk and milk products), are comparable and in line with results of ranking abilities 400
from other studies: including a paper-based Dutch FFQ (43), a Danish web-based FFQ (40) and 401
a Finnish paper-based FFQ study (39). This indicates that the observed acceptable ranking 402
abilities of the WebFFQ, for most energy adjusted food groups, relative to the 24HRs seems 403
to be in line with what is reported elsewhere.
404
Implications of energy misreporting on the relative validation between WebFFQ and the 405
24HRs 406
Because the intake of many nutrients, and especially the intake of energy providing nutrients 407
are correlated with total energy intake (44), one would expect the ranking abilities of a tool to 408
be fairly similar for energy and energy providing nutrients. Yet, we observed poor ranking 409
abilities for energy for the WebFFQ as compared to the objective DLW method, but 410
acceptable ranking abilities for the macronutrients, in the relative comparison between the 411
WebFFQ and 24HRs. Without nutritional biomarkers (3) for more nutrients or food groups, or 412
other objective reference methods, it is not possible to disentangle what this truly implies.
413
Nevertheless, we speculate if this could indicate that there are correlated errors between the 414
WebFFQ and 24HRs, which may falsely improve the agreement between methods (34). 415
However, ranking abilities for energy intake of the 24HRs assessed by the objective DLW 416
were moderately satisfactory. We argue that because the EI ranking ability of the 24HRs is 417
superior to that of the WebFFQ, the 24HRs seems an appropriate reference tool for 418
comparison with the WebFFQ.
419
Referring to previous arguments in this paper, the 24HRs proved to underestimate EI on 420
group level to a larger extent than the WebFFQ, and the general overestimation observed for 421
most macronutrients and food groups by the WebFFQ is probably partly reflecting the true 422
underestimation by the 24HRs. Thus, mean intakes on group level from the WebFFQ, seem to 423
be acceptable, with some exceptions.
424
Methodological considerations 425
The strength of the present study was the use of two different reference methods. The DLW 426
biomarker allowed an objective assessment of the energy estimates from the WebFFQ.
427
Moreover, the four repeated non-consecutive 24HRs used in the relative comparison between 428
methods enabled evaluation of estimates of the usual dietary intake. However, the number of 429
recalls needed to estimate usual dietary intake varies for different components of the diet (45): 430
Although as few as three to four repeats can be sufficient for the macronutrients validated in 431
the current study, this is in all probability not the case for episodically consumed foods. Still, 432
the number of recalls was restricted to four in this study, due to feasibility and limited 433
resources.
434
For the WebFFQ to be filled in by the participants under as unflawed conditions as possible, it 435
was administered as the first thing in the study, before the 24HRs for all participants, and 436
before the dosing of DLW and urine sampling in group 1. Therefore, the WebFFQ and 24HRs 437
diverge timeline wise: the WebFFQ covers the period before the 24HRs. A recent systematic 438
review and meta-analysis have demonstrated that there is seasonal variation in energy intake 439
and the intake of several foods or food groups (46); this may have attenuated the agreement 440
between the WebFFQ and the 24HRs. Group 1, in which the validity of EI was assessed using 441
the DLW method, consisted of women only; this constrains the generalizability of the results 442
to the general adult population, and is also a limitation of this study.
443
The web-format of our WebFFQ offer inherent error checks, skip-algorithms and images of 444
foods to improve portion size estimates. However, as discussed previously, we did not 445
observe noticeably different results compared to other studies, not even for a paper-based 446
Norwegian FFQ (33). No improvement in accuracy was observed for the web-format compared 447
to the paper-format in a study by Beasely et al. (47) either, and Ilner et al. (10) argue that the 448
fundamental issues with dietary self-reports are not bypassed by new technology. Thus, a 449
web-based FFQ is still an FFQ, and will still call for the ability to perform cognitively 450
complex tasks, including estimating the intake of episodically consumed foods.
451
Conclusion 452
The performance of the WebFFQ conformed to both similar paper-based FFQs and web- 453
based FFQs. For energy, the WebFFQ showed only an insignificant mean underestimation of 454
EI compared to measured TEE from DLW, but is not suitable to rank individuals correctly 455
according to their EI. The relative comparison between the WebFFQ and the mean of four 456
24HRs demonstrated that the estimated intakes on group level for most macronutrients and 457
food groups appear to be acceptable, except for ‘vegetables’ and ‘fish and shellfish’ which are 458
significantly and largely overestimated by the WebFFQ. The WebFFQ’s ranking ability for 459
macronutrients and most food groups appears to be satisfactory relative to the 24HRs. The 460
agreement between methods improved after energy adjustments. In conclusion, energy 461
estimates must be used with caution, but the WebFFQ’s ranking abilities and estimated group 462
intakes are mostly acceptable relative to the 24HRs, and may, therefore, be used in both future 463
nutrition epidemiology studies and dietary surveys, respectively. Further studies using 464
nutritional biomarkers or other objective reference methods are warranted to confirm these 465
results.
466
Acknowledgements 467
We thank Peter Thomson for conducting the laboratory analysis on the DLW, and Helene 468
Astrup and Ida Sofie Kaasa for conducting telephone 24HRs.
469
Financial Support 470
This study was funded by the Institute of Basic Medical Sciences, University of Oslo, with 471
supplementary funds from the Throne Holst Nutrition Research Foundation. The funders had 472
no role in the design, analysis or writing of this article.
473
Conflict of Interest 474
None.
475
Authorship 476
The authors’ roles in the study were as follows:
477
ACM, CH, JRS, LFA: conception and design; ACM: acquisition of data; ACM, MHC, CH, 478
JRS, SS, LFA: analysis and interpretation of data; ACM: drafted the manuscript; ACM, 479
MHC, CH, JRS, SS, LFA: critically revised the manuscript; LFA: supervision and obtained 480
funding.
481 482 483 484 485 486 487 488 489
490
Figure legends 491
Figure 1. Flow chart showing the recruitment process in a Norwegian validation study of a 492
web-based food frequency questionnaire (WebFFQ).
493
Figure 2. Plots showing A) the EI from a web-based food frequency questionnaire (WebFFQ) 494
plotted against the TEE from DLW and B) the mean EI from multiple 24HRs plotted against 495
the TEE from DLW (n=29).
496
Figure 3. Bland – Altman plot showing the difference between EI from a web-based food 497
frequency questionnaire (WebFFQ) and TEE from DLW plotted against the average of the 498
two methods. The black dots are individuals identified as acceptable reporters of EI. The grey 499
disrupted line displays the 95% confidence interval for the mean difference.
500
Figure 4. Plots showing the difference between EI from a web-based food frequency 501
questionnaire (WebFFQ) and TEE from DLW, plotted against the difference of estimated 502
intakes of foods between the WebFFQ and multiple 24HRs. The black dots are individuals 503
identified as acceptable reporters of EI. The horizontal line displays the point of 0 difference 504
between EI from the WebFFQ and TEE from DLW. The vertical, disrupted line displays the 505
point of 0 difference between the WebFFQ and 24HRs in the estimated food groups. A) 506
Cheese B) Vegetables C) Fish and shellfish D) Cereals.
507 508 509 510 511 512 513 514 515 516 517 518
519
References 520
1. Ezzati M, Riboli E (2012) Can noncommunicable diseases be prevented? Lessons from studies of 521
populations and individuals. Science 337, 1482-1487.
522 523
2. WHO (2015) Global status report on noncommunicable diseases 2014. Geneva: WHO.
524 525
3. Kuhnle GG (2012) Nutritional biomarkers for objective dietary assessment. J Sci Food Agric 92, 526
1145-1149.
527 528
4. Dhurandhar NV, Schoeller D, Brown AW et al. (2015) Energy balance measurement: when 529
something is not better than nothing. Int J Obes (Lond) 39, 1109-1113.
530 531
5. Subar AF, Freedman LS, Tooze JA et al. (2015) Addressing Current Criticism Regarding the Value 532
of Self-Report Dietary Data. J Nutr 145, 2639-2645.
533 534
6. Medical Research Council (2014) Dietary assessment - Food frequency questionnaire. The MRC 535
Diet and Physical Activity (DAPA) Measurement Toolkit http://www.dapa-toolkit.mrc.ac.uk/dietary- 536
assessment/methods/food-frequency-questionnaire/index.php (accessed March 2017).
537
7. National Institutes of Health, National Cancer Institute (2017) Dietary Assessment Primer, Food 538
Frequency Questionnaire at a Glance.
539
https://dietassessmentprimer.cancer.gov/profiles/questionnaire/index.html (accessed March 2017).
540 541
8. Freedman LS, Commins JM, Moler JE et al. (2014) Pooled results from 5 validation studies of 542
dietary self-report instruments using recovery biomarkers for energy and protein intake. Am J 543
Epidemiol 180, 172-188.
544 545
9. Kristal AR, Peters U, Potter JD (2005) Is it time to abandon the food frequency questionnaire?
546
Cancer Epidemiol Biomarkers Prev 14, 2826-2828.
547 548
10. Illner AK, Freisling H, Boeing H et al. (2012) Review and evaluation of innovative technologies 549
for measuring diet in nutritional epidemiology. Int J Epidemiol 41, 1187-1203.
550 551
11. Falomir Z, Arregui M, Madueno F et al. (2012) Automation of Food Questionnaires in Medical 552
Studies: a state-of-the-art review and future prospects. Comput Biol Med 42, 964-974.
553 554
12. Andersen LF, Solvoll K, Johansson LR et al. (1999) Evaluation of a food frequency questionnaire 555
with weighed records, fatty acids, and alpha-tocopherol in adipose tissue and serum. Am J Epidemiol 556
150, 75-87.
557 558
13. Nes M, Frost Andersen L, Solvoll K et al. (1992) Accuracy of a quantitative food frequency 559
questionnaire applied in elderly Norwegian women. Eur J Clin Nutr 46, 809-821.
560 561
14. Carlsen MH, Karlsen A, Lillegaard IT et al. (2011) Relative validity of fruit and vegetable intake 562
estimated from an FFQ, using carotenoid and flavonoid biomarkers and the method of triads. Br J Nutr 563
105, 1530-1538.
564 565
15. National Institutes of Health, National Cancer Institute (2017) Dietary assessment primer, 566
Software for Dietary Analysis - FFQs and Screeners 567
https://dietassessmentprimer.cancer.gov/learn/software.html (accessed May 2017).
568
569
16. Food and Agriculture Organization of the United Nations (2003) Food energy – methods of 570
analysis and conversion factors. Food and nutrition paper 77. Rome: FAO 571
17. Lifson N, McClintock R (1966) Theory of use of the turnover rates of body water for measuring 572
energy and material balance. J Theor Biol 12, 46-74.
573 574
18. Speakman JR (1997) Doubly labelled water: theory and practice. London: Chapman & Hall.
575 576
19. Berman ES, Fortson SL, Snaith SP et al. (2012) Direct analysis of delta2H and delta18O in natural 577
and enriched human urine using laser-based, off-axis integrated cavity output spectroscopy. Anal 578
Chem 84, 9768-9773.
579 580
20. Nagy KA (1983) The Doubly Labeled Water (3HH18O) Method: a guide to its use. Los Angeles, 581
CA: UCLA.
582 583
21. Schoeller DA, Ravussin E, Schutz Y et al. (1986) Energy expenditure by doubly labeled water:
584
validation in humans and proposed calculation. Am J Physiol 250, R823-830.
585 586
22. Schoeller DA (1988) Measurement of energy expenditure in free-living humans by using doubly 587
labeled water. J Nutr 118, 1278-1289.
588 589
23. Speakman JR, Nair KS, Goran MI (1993) Revised equations for calculating CO2 production from 590
doubly labeled water in humans. Am J Physiol 264, E912-917.
591 592
24. Black AE, Prentice AM, Coward WA (1986) Use of food quotients to predict respiratory quotients 593
for the doubly-labelled water method of measuring energy expenditure. Hum Nutr Clin Nutr 40, 381- 594
391.
595 596
25. Myhre JB, Loken EB, Wandel M et al. (2014) Eating location is associated with the nutritional 597
quality of the diet in Norwegian adults. Public Health Nutr 17, 915-923.
598 599
26. McCarthy S (2014) Weekly patterns, diet quality and energy balance. Physiol Behav 134, 55-59.
600 601
27. Beaton GH, Milner J, Corey P et al. (1979) Sources of variance in 24-hour dietary recall data:
602
implications for nutrition study design and interpretation. Am J Clin Nutr 32, 2546-2559.
603 604
28. Black AE (2000) The sensitivity and specificity of the Goldberg cut-off for EI:BMR for 605
identifying diet reports of poor validity. Eur J Clin Nutr 54, 395-404.
606 607
29. Black AE, Cole TJ (2000) Within- and between-subject variation in energy expenditure measured 608
by the doubly-labelled water technique: implications for validating reported dietary energy intake. Eur 609
J Clin Nutr 54, 386-394.
610 611
30. Cole TJ (1997) Sampling, study size, and power. In Design Concepts in Nutritional Epidemiology 612
[BMM Nelson, editor]. Oxford: Oxford University Press.
613 614
31. The Norwegian Directorate of Health (2012) Norkost 3 - En landsomfattende 615
kostholdsundersøkelse blant menn og kvinner i Norge i alderen 18-17 år, 2010-11 (Norkost 3 - A 616
nationwide dietary survey among 18-70 year old men and women in Norway, 2010-11). Oslo, 617
Norway: The Norwegian Directorate of Health.
618 619
32. StatsToDo (2014) Sample Size for Pearson's Correlation Coefficient Explained and Table.
620
https://www.statstodo.com/SSizCorr_Exp.php (accessed May 2017).
621
622
33. Andersen LF, Tomten H, Haggarty P et al. (2003) Validation of energy intake estimated from a 623
food frequency questionnaire: a doubly labelled water study. Eur J Clin Nutr 57, 279-284.
624 625
34. Thompson FE, Subar AF (2013) Dietary assessment methodology. In Nutrition in the Prevention 626
and Treatment of Disease, Third edition ed. [BC Coulston AM, Ferruzzi MG, editor]. San Diego:
627
Academic Press.
628 629
35. Jonnalagadda SS, Mitchell DC, Smiciklas-Wright H et al. (2000) Accuracy of energy intake data 630
estimated by a multiple-pass, 24-hour dietary recall technique. J Am Diet Assoc 100, 303-308; quiz 631
309-311.
632 633
36. Silva-Jaramillo KM, Neutzling MB, Drehmer M (2015) FFQ for the adult population of the capital 634
of Ecuador (FFQ-Quito): development, reliability and validity. Public Health Nutr 18, 2540-2549.
635 636
37. Feng RN, Du SS, Chen Y et al. (2016) An internet-based food frequency questionnaire for a large 637
Chinese population. Asia Pac J Clin Nutr 25, 841-848.
638 639
38. Hollis JL, Craig LC, Whybrow S et al. (2017) Assessing the relative validity of the Scottish 640
Collaborative Group FFQ for measuring dietary intake in adults. Public Health Nutr 20, 449-455.
641 642
39. Paalanen L, Mannisto S, Virtanen MJ et al. (2006) Validity of a food frequency questionnaire 643
varied by age and body mass index. J Clin Epidemiol 59, 994-1001.
644 645
40. Knudsen VK, Hatch EE, Cueto H et al. (2016) Relative validity of a semi-quantitative, web-based 646
FFQ used in the 'Snart Foraeldre' cohort - a Danish study of diet and fertility. Public Health Nutr 19, 647
1027-1034.
648 649
41. Christensen SE, Moller E, Bonn SE et al. (2013) Two new meal- and web-based interactive food 650
frequency questionnaires: validation of energy and macronutrient intake. J Med Internet Res 15, e109.
651 652
42. Nordic Council of Ministers (2014) Nordic Nutrition Recommendations 2012. Copenhagen, 653
Denmark: Nordic Council of Ministers.
654 655
43. Streppel MT, de Vries JH, Meijboom S et al. (2013) Relative validity of the food frequency 656
questionnaire used to assess dietary intake in the Leiden Longevity Study. Nutr J 12, 75.
657 658
44. Willett WC, Howe GR, Kushi LH (1997) Adjustment for total energy intake in epidemiologic 659
studies. Am J Clin Nutr 65, 1220S-1228S; discussion 1229S-1231S.
660 661 662
45. Pereira RA, Araujo MC, Lopes TS et al. (2010) How many 24-hour recalls or food records are 663
required to estimate usual energy and nutrient intake? Cad Saude Publica 26, 2101-2111.
664 665
46. Stelmach-Mardas M, Kleiser C, Uzhova I et al. (2016) Seasonality of food groups and total energy 666
intake: a systematic review and meta-analysis. Eur J Clin Nutr 70, 700-708.
667 668
47. Beasley JM, Davis A, Riley WT (2009) Evaluation of a web-based, pictorial diet history 669
questionnaire. Public Health Nutr 12, 651-659.
670 671
672
