Reproducibility of the printed version of the Webcas Questionnaire

Santos Legnani, Rosimeide; Legnani, Elto; Quentino, Rafael; da Silva, Michael; Araujo Bacil, Eliane; de Campos, Wagner

Original Article

Vol. 16 nº 2 - Abr/Jun - 2019
Pages:7-
15

Reproducibility of the printed version of the Webcas Questionnaire

Authors: ^¹, ^², ^³, ^⁴, ^⁵, ^⁶

¹PhD in Physical Education from the Federal University of Paraná (UFPR). Professor at the State University of Ponta Grossa (UEPG). Ponta Grossa, PR, Brazil

²PhD in Physical Education from the Federal University of Paraná (UFPR). Professor at the Federal University of Technology of Paraná (UTFPR). Curitiba, PR, Brazil

³Master’s student in Physical Education from the Federal University of Technology of Paraná (UTFPR). Curitiba, PR, Brazil

⁴PhD in Physical Education from the Federal University of Paraná (UFPR). Professor at the State University of Central-West (UNICENTRO). Guarapuava, PR, Brazil

⁵PhD in Physical Education from the Federal University of Paraná (UFPR). Professor at the Positivo University (UP). Curitiba, PR, Brazil

⁶Post-doctorate in Motor Development and Sports Studies from the University of Pittsburgh (PITT – USA). Professor at the Federal University of Paraná (UFPR). Curitiba, PR, Brazil

Correspondence:

Rosimeide Francisco Santos Legnani
Ponta Grossa State University – Uvaranas Campus – Department of Physical Education
Av. General Carlos Cavalcanti, 4748
Ponta Grossa, PR, Brazil. Zip code 84030-900
( legnanirosi@gmail.com )

Submitted on 08/05/2018
Approved on 28/01/2025

Keywords: Reproducibility of results; Questionnaires; Students.

Abstract

OBJECTIVE: To analyze the reproducibility of the printed version of the WebCas VI questionnaire.
METHODS: For the test and retest of the study, 255 students between 9 and 15 years old, intentionally selected from a public educational institution in Curitiba-PR, participated. The WebCas VI is composed of seven sections: sociodemographic variables, sleep time, physical activity, commuting to school, frequency of food consumption, consumption of alcoholic beverages and cigarettes, and socioeconomic issues.
DATA SYNTHESIS: Data analysis was performed using the percentage of agreement, Kappa PABAK (KP) and correlation coefficient of agreement.
RESULTS: The students had a mean age of 13.2 (±1.1) years and an estimated caloric expenditure of 2261.6 (±591.5) METs. The agreement percentages ranged from 40.8% to 99.3%. The lowest KP value was observed for the variable sweets (0.59; 95% CI: 0.51 – 0.67) and the highest for the variable cigarette consumption (1.00; 95% CI: 0.93 – 1.00). No variable presented weak or poor KP values. For continuous variables, the highest agreement correlation coefficient value was 0.83 (95% CI: 0.42 – 0.65) and the lowest was 0.11 (95% CI: 0.02 – 0.13). Only two variables analyzed presented weak agreement correlation coefficients (r < 0.40).
CONCLUSION: The reproducibility indicators of the WebCas VI questionnaire items were considered adequate for both categorical and continuous variables.

INTRODUCTION Health-related behaviors (HRBs) in children and adolescents can be completely different, and although many researchers study this variable, there is still much to be researched about it. HRBs tend to change according to several factors: culture, ethnicity, and customs can be influenced by external factors from childhood onwards and throughout life. Therefore, HRBs, such as physical activity level (PAL), sufficient hours of sleep, healthy diet, and abstinence from alcohol and cigarette consumption, can be related to numerous health benefits in children and adolescents ^1,2 . However, measuring these variables in this age group still presents a series of limitations for researchers in the health field, mainly due to the weaknesses in the process of constructing, testing, and validating questionnaires ³ . Considering these limitations, printed questionnaires are widely used to collect data and monitor these behaviors in different geographic regions and contexts, mainly due to the possibility of gathering a large amount of information at a low cost and easy to apply ^2,4 . Even if the instrument already exists and presents adequate psychometric characteristics, there is a need to test these qualities for use in a context other than that for which it was developed, which implies carrying out validity, reliability and reproducibility procedures ^5,6 . In Brazil, few studies have proposed developing instruments to assess CRS in adolescents ^3,7,8 . Even so, there does not appear to be a questionnaire to assess CRS in adolescents that includes the variable “sleep habits”, as well as one that proposes to assess Habitual Physical Practice (PHAF) based on the Compendium of Physical Activities ^{9 . Therefore, the objective of this study was to carry out the validity and reproducibility procedures of the Printed Version of the}Webcas Questionnaire ( Webcas VI) . METHODS Data collection was carried out in August and September 2014 (cross-sectional survey) by a trained team from the Center for Physical Activity and Health Studies of the Federal University of Paraná (UFPR), supervised by the main researcher. The study was approved by the Human Research Ethics Committee of UFPR, under opinion number: 684.147/2014 of June 11, 2014. The sample consisted of 255 intentionally selected students aged 9 to 15 years, of both sexes, enrolled in the daytime period of a public school in the urban area of Curitiba, Paraná. The instrument was developed to evaluate the CRS, its elaboration process and reproducibility in students in two stages: elaboration of the printed version and reproducibility procedures of WebCas VI. The structure of WebCas VI was elaborated by adapting sections extracted from instruments used in international studies. The questionnaire was divided into seven sections: sociodemographic variables: student code, school, city, shift, date of birth, school system (private, municipal, state, federal), body mass, height, sex and day to be recalled; time of sleep on the previous day; recall of activities performed on the previous day; type of transportation used to travel to school; eating habits (frequency of food consumption); sleeping habits; consumption of alcoholic beverages and cigarettes; socioeconomic questionnaire. The objective of the first section was to investigate the anthropometric and sociodemographic variables of the participants, and the second section assesses the duration of sleep on the previous night. The third section lists 244 physical activities (PA) with 35% of the metabolic equivalent (MET) values coming from research with children and adolescents, the remainder coming from PA listed in the adult compendium and corrected for application in adolescents ⁹ . Each listed activity ⁹ is equivalent to a MET value, which represents its relative intensity in multiples of the Resting Metabolic Rate (RMR) defined as 1.0 Kcal/kg (weight/hour). This was represented by a six-digit code, in which the first digit refers to the type of PA, the second digit to the body position during the activity, the third digit represents the context for each PA – being specific to each PA category, the fourth and fifth digits describe the specificity of each activity and the sixth digit describes the intensity reported during the PA ^10,11 . To facilitate the interpretation of these codes by WebCas VI respondents, eight PA domains were created: Arts; Domestic activities; Personal care; Dance; Gymnastics; Student and work activities; Sports; Leisure and recreation. In addition, three intensity categories were highlighted: low, moderate and high, as well as possibilities of performing these according to the body positions (lying down, sitting and standing), making an analogy with the list of PA presented to the students, and their respective values in METs. All PA listed in the compendium were included in one of the domains according to the intensity category and body position. This information allowed the calculation of daily energy expenditure (DEE), divided into fifteen-minute intervals. Energy expenditure (EE) is represented by a continuous variable in MET, measured in kilocalories per day (Kcal/day), or categorical, in the case of PA level (PAL) classified as sedentary, slightly active, active and very active. To calculate the EE of a young person in relation to a given PA, the MET values must be multiplied by the young person’s RMR, proceeding as follows: Kcal = MET value X RMR X body mass, duration of PA. Where: RMR = Kcal.kg ^-1 min ^-1 ; body mass (BM) = Kg, time = minutes ¹¹ . RMRs were calculated using the equations: RMR = 0.084 X BM + 2.122 (for boys) and RMR 0.047 X BM + 2.951 (for girls) ¹² . These results are expressed in MJ/day, and to determine the value in kilocalories per day (Kcal/day), the RMR result must be multiplied by 239 (constant). To identify the GED, all activities performed during a day were recorded, and then calculations were made for each activity performed during the recorded day. After calculating the RMR and GED, the students’ PAL was calculated according to the following equation: GED in kilocalories divided by basal caloric expenditure, resulting in an estimated proportion up to two and a half times above the RMR, according to the classification: sedentary [1-1.39], slightly active [1.4-1.59], active [1.6-1.89] and very active [1.9-2.5] ¹³ . Sections four and five refer to transportation habits (home/school/home), eating and sleeping habits, where the last week was used as a reference. Five groups were considered for the frequency of food consumption: fruits; vegetables; savory snacks, French fries, chips or hot dogs; sweets, stuffed cookies or chocolates; soft drinks or juices with added sugar. The option to study these types of foods is due to the greater availability of information from other surveys involving the same age group and the majority of these items being included in surveys conducted by the WHO ^4,14 , in addition to having a direct relationship with health levels and significantly representing the eating habits of adolescents. Sleep habits were investigated with questions related to: daytime sleepiness, naps or siestas, and the times the adolescent usually wakes up and goes to sleep. Section six gathered information regarding the frequency of alcohol consumption in the last 30 days, number of doses per occasion and cigarette consumption, and was based on the seven days prior to the survey ^15,16 . The last section of WebCasVI refers to the classification of the socioeconomic stratum of the participants. The socioeconomic level was assessed following the recommendations of the Brazilian Association of Research Companies ¹⁷ , using the following classification criteria: possession of movable property and the level of education of the head of the family, which classified the students into classes A, B1, B2, C1, C2, D and E. For this study, the samples were stratified into three classes: high (A + B1); medium (B2 + C1 + C2); and low (D + E). During the WebCas VI test and retest procedures, the researchers made four visits to the school. In the first visit, they presented the research project and its objectives to the students, distributing the Free and Informed Consent Form (FICF) and the Free and Informed Assent Form (FICF). On the second visit, students who agreed to participate presented the signed TALE and the TCLE, duly signed by their guardians, and then participated in an anthropometric assessment, using a scale (Plena brand) with a capacity of 150 kilograms and accuracy of 100 grams and a tape measure fixed to the wall and a wooden barrier, placed on the vertex of the skull. On the third visit to the school, the first application of WebCas VI (test) was carried out. Seven days later, the retest procedures were carried out. There was a sample loss of 111 (44.7%) students. Before the students began the test, the questionnaire was presented in detail in multimedia ( Datashow ), together with the PA compendium, printed on A3 paper, in color and delivered separately from the questionnaire. At this time, the methodological procedures pertinent to the PA recall were explained, which should be noted in the questionnaire. A 24-hour chart was presented (divided into hours and these into four 15-minute parts), and students were asked to remember the activities carried out the previous day. To do this, they were instructed to first consult the domain in which the activity was allocated (in the PA list), then the type of activity and its value in METs, for later note of each activity carried out in the 15-minute intervals. After the presentation and explanation of the other sections of WebCas VI, students were allowed to fill out one question at a time until they had completed all the items. As the students finished filling out the questionnaire, they were instructed to move on to the other sections: type of transportation to school, frequency of food consumption, alcohol and cigarette consumption, sleeping habits and the socioeconomic questionnaire. WebCas VI was administered by the main researcher and two assistants in approximately 35 – 40 minutes. Regarding the analysis of quantitative data, descriptive statistics procedures were performed (mean, standard deviation and frequency distribution). Reproducibility was assessed using the Kappa percentage of agreement adjusted for variables on an ordinal scale (PABAK-OS), or simply PABAK (K), which can be used to calculate the reliability between two applications of the same instrument, when analyzing a variable on an ordinal scale of three, four, five, six or seven categories ¹⁸ . The analyses related to K were performed directly on the page http://www.singlecaseresearch.org . Continuous variables were analyzed using the concordance correlation coefficient technique (CCC ρ _c ) ¹⁹ . The CCC ρ _c presents a measure of precision (ρ) and another of accuracy ( C _b ). Where: ρ is Pearson ‘s correlation coefficient and Cb is a bias correction factor that measures how much the best-fit line deviates from the 45º line through the origin, thus being a measure of accuracy. The suggested values for the interpretation of the Concordance Coefficients ( _ρc) _indicate the strength of the agreement are: <0.90 = poor; 0.90 – 0.95; moderate; 0.95 – 0.99, substantial and >0.99 = Almost Perfect20 ^. For the variables that did not present a normal distribution, the data normalization was performed in the statistical program Medcalc 15.2 for Windows, where the pre-established significance level of 5% (p<0.05) was adopted. RESULTS In the reproducibility process between the WebCas VI application replicates, 141 students were evaluated, with a mean age of 13.2 years (±1.1), body mass of 50.3 kg (±9.9), height of 1.56 cm (0.07), body mass index of 20.4 cm/m2 ⁽ 3.34) and estimated caloric expenditure of 2261.6 METs (591.5). Table 1 shows the sociodemographic characteristics of the study participants.

The results related to the questionnaire reproducibility procedures are presented in Table 2. The agreement percentages ranged from 40.8% to 99.3%, with higher values observed among the variables related to cigarette consumption 99.3% (141) and alcohol (doses) with 90.1% (128), and lower values among the variables related to the frequency of food consumption (fruits and vegetables), soft drink consumption 40.8% (58) and sweets 43.0% (61).

When considering the entire sample, it was found that the highest K values were observed in: commuting to school (K=0.84; CI:0.77 – 0.90), alcohol and cigarette consumption (alcohol consumption >5 doses; drunkenness); and daytime sleepiness, napping or siesta. No variable presented weak or poor Kappa values. After categorizing the analyses by sex, it was found that the highest K values were among boys in: cigarette consumption (K=0.98; CI:0.90 – 1.07) and napping or siesta (K=0.96; CI:0.87 – 1.04), with the lowest values among girls in the variables fruit consumption (K=0.56; CI:0.43 – 0.69) and sweets consumption (K=0.58; CI:0.45 – 0.70). When considering the sample as a whole (continuous variables), the highest CCC P _c values were observed in: time waking up on the weekend (r=0.70; CI: 0.42 – 0.65) and time waking up today (r=0.70; 95% CI = 0.60 – 0.77). The highest agreement value was observed in the variable time waking up from Monday to Friday, (r=0.87) for boys and (r=0.75) for girls. In the analyses related to the C _b indicators , the variables that presented the highest indicators were: time waking up today; time waking up from Monday to Friday; time sleeping from Monday to Friday; time waking up on the weekend and GEDA (C _b ≥0.99), demonstrating an accuracy classified as almost perfect. These indicators remained among boys in the same variables and suffered small reductions among girls. In the analyses related to P , the highest correlation value observed was in the variable, time of waking up from Monday to Friday, both among boys (r=0.87) and among girls (r=0.76). Table 3 presents more details about these indicators.

DISCUSSION The assessment of CRS in children and adolescents is of great importance to public health, given its strong relationship with implications for the health of young people in the short and long term. Among the main results of the WebCas VI reproducibility procedures, it can be highlighted that of the 12 categorical variables analyzed, nine presented agreement percentages higher than 50%; of those analyzed using Weighted Kappa ( Pabak ), seven (58.33%) presented very good agreement values (K≥0.84), namely: travel to school; alcohol consumption in 30 days; alcohol consumption >5 doses; drunkenness; cigarette consumption; daytime sleepiness and naps. Moderate Pabak values were observed in the variables that analyzed the frequency of consumption of sweets, fruits and vegetables (K=0.61 to K=0.65). The reproducibility indicators observed in this study were higher than those found in the literature ²¹ in students from Santa Catarina (K=0.23 to K=0.58) and similar to those found by Farias Júnior ³ , who identified Kappa values ranging from moderate to strong in most variables related to eating habits (K=0.44 to K=0.69). On the other hand, when analyzing the reproducibility of the Brazilian version of the Youth Risk Behavior Survey ( YRBS ) instrument, the authors identified that 91% of the items presented moderate or substantial Kappa values , and these values supported the use of this version in Brazilian adolescents ⁷ . In the analyses discriminated by sex, among 12 variables analyzed in boys, seven variables (commuting to school, alcohol consumption in 30 days; alcohol consumption >5 doses; drunkenness; cigarette consumption; daytime sleepiness, naps) presented Kappa values >0.80. Among girls, this was observed in six variables (alcohol consumption in 30 days; alcohol consumption >5 doses; drunkenness; cigarette consumption; daytime sleepiness and naps). The lowest Kappa values were found for the consumption of sweets (K=0.59) and vegetables (K=0.63) among boys, while among girls it occurred in the variables fruit consumption (K=0.56) and sweets consumption (K=0.58). This information can be corroborated, since the reproducibility indicators can vary widely between studies (r=0.20 and r=0.98) ³ . The divergences observed between the results of the studies analyzed may be related to the different methodologies adopted, both in data collection and treatment. In general, questionnaires demonstrate better test-retest reproducibility than validity indicators ^3,9,10,21 . The correlation values between the application replications of the questionnaireWebcas VI for the EG were r=0.59 and the accuracy values for reproducibility were r=0.99. The correlation value found in this study is lower when analyzing the reproducibility of the PAQ-C and PAQ-A questionnaires ²² (r=0.68 and r=0.88) and when testing the agreement between the replicates of the IPAQ questionnaire in adolescents ²² . Of the eight continuous variables analyzed and considering the entire sample, five (62.5%) presented strong (>0.70) correlation coefficients of agreement, with four variables (50%) among boys. In the analyses related to the C _b indicators , seven variables presented values of substantial or almost perfect accuracy (C _b ≥0.98). However, among boys, these same indicators were observed in six variables, while among girls only in four variables. In the reproducibility analyses of continuous variables, sleep habits and EE (Kcal/day), seven (87.5%) of the eight variables analyzed presented substantial or almost perfect accuracy values (Cb >0.95). On the other hand, approximately 60% of the variables presented correlation values between strong (r >0.70) and moderate (r >0.40 <0.70). The highest correlation values were found when investigating the variables time of waking up and time of sleeping from Monday to Friday (r=0.80; r=0.76). These indicators demonstrate that WebCas VI has a good capacity to assess sleeping habits. However, in relation to the variables time of sleeping yesterday (r=0.17) and time of sleeping on the weekend (r=-0.03), they presented the lowest correlations, demonstrating the instrument’s limitations in identifying and assessing students’ sleeping schedule. In the analyses related to the P indicators , it was observed that three of the eight variables analyzed presented strong correlation values (>0.70). Among boys, these values were observed in four variables, while among girls only in one of the eight variables. The proposal to develop WebCas VI, based on several instruments previously developed and tested in other regions and countries, may have contributed to the good reproducibility indicators observed in this study. Furthermore, the incorporation of variables related to the identification of sleep habits, as well as the use of the list of proposed activities ⁹ , can be considered as highly relevant aspects of this instrument. The intentional sample selection, the imbalance between the sample strata (63.4% boys and 36.6% girls), as well as the fact that 95% of the sample was composed of students from socioeconomic strata C+D+E, can be considered the greatest limitations of the study. Likewise, the scarcity of studies of this nature, especially those that used similar statistical analyses, made it difficult to discuss the results. A reproducibility study with students from high socioeconomic strata is suggested. CONCLUSION The instrument presented satisfactory reproducibility indicators, in both variables (categorical and continuous). However, it is worth noting that in relation to the analyses discriminated by sex, these indicated that boys presented more consistent test and retest reproducibility indicators in relation to girls. Therefore, the authors recommend the use of WebCas in Brazilian students, especially in the South Region.

References

1. Steele MM, Richardson B, Daratha K, . Bindler RC. Multiple Behavioral Factors Related to Weight Status in a Sample of Early Adolescents: Relationships of Sleep, Screen Time, and Physical Activity. Children’s Health Care 2012; 41(4):269-280. 2. Reis TG, Oliveira LCM. Pattern of alcohol consumption and associated factors among adolescents students of public schools in an inner city in Brazil. Brazilian Journal of Epidemiology 2015; 18(1): 13-24. 3. Farias Júnior JC, Lopes ADS, Florindo AA, Hallal PC. Validity and reliability of self-report instruments for measuring physical activity in adolescents: a systematic review. Cad Public Health 2010; 26(9): 1669-1691. 4. Currie C, Zanotti C, Morgan A, Currie D, Looze M, Roberts C, et al. eds. Social determinants of health and well-being among young people. Health Behavior in School-aged Children (HBSC) study: international report from the 2009/2010 survey. Copenhagen, WHO Regional Office for Europe, (Health Policy for Children and Adolescents, No. 6), 2012. 5. Thomas J, Nelson J, Solverman S. Research Methods in Physical activity. (6th ed.): Human Kinetics Publishers, 2010. 6. Legnani E, Legnani RFS, Rech CR, Guimaraes RF, Campos W. Electronic instruments to assess physical activity in children: a systematic review. Motricidade 2013b; 9(4): 90-99. 7. Guedes DP, Lopes CC. Validation of the Brazilian version of the 2007 youth risk behavior survey. Public Health Journal. 2010; 44(5): 840-850. 8. Guedes DP, Guedes JERP. Measurement of Physical Activity in Young Brazilians: Reproducibility and Validity of the PAQ-C and PAQ-A. Brazilian Journal of Sports Medicine 2015; 21(6): 425-432. 9. Ridley K, Ainsworth BE, Olds TS. Development of a compendium of energy expenditures for youth. International Journal of Behavioral Nutrition and Physical Activity 2008; 5(1): 45. 10. Farinatti PTV. Presentation of a Portuguese version of the Compendium of Physical Activities: a contribution to researchers and professionals in Exercise Physiology. Brazilian Journal of Exercise Physiology 2003;2(2):177-208. 11. Fonseca, PHF Promotion and evaluation of physical activity in young Brazilians. Sao Paulo: Phorte, 2012. 12. Henry CJ, Rees DG. New predictive equations for the estimation of basal metabolic rate in tropical peoples. European Journal of Clinical Nutrition 1991; 45(4): 177-185. 13. Institute of medicine of the national academies. The National Academies Press, Washington, DC, 2002. [Accessed: 22/09/2013]. Available at: www.nap.edu 14. CDC – Centers for Disease Control and Prevention. Youth Risk Behavior Surveillance – United States, 2011. MMWR, Surveillance Summaries 2012; 61(4):168. 15. National Cancer Institute (INCA). Vigescola: smoking surveillance in schoolchildren: data and facts from 12 Brazilian capitals. Rio de Janeiro, 2004. [Accessed: 30/05/2015]. Available at: http://controlecancer.bvs.br/lis-search/resource/13743 . 16. Legnani E, Legnani RFS, Dellagrana RA, Silva MP, Barbosa Filho VC, Campos W. Health risk behaviors and excess body weight in schoolchildren from Toledo, Paraná, Brazil. Motricidade 2012; 8(3):59 – 70. 17. ABEP – Brazilian Association of Research Companies. Economic Classification Criteria of Brazil, 2014. [Accessed on: 09/29/2014]. Available at: http://www.abep.org . 18. Bland MJ, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 1986;327(8476): 307- 310. 19. Lawrence I, Lin K. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989: 255-268. 20. Mcbride GB. A proposal for strength-of-agreement criteria for Lin’s Concordance Correlation Coefficient. NIWA Client Report: HAM2005-062, 2005. 21. Nahas MV, Barros MVG, Florindo AA, Hallal PC, Konrad L, Barros SSH, et al. Reproducibility and validity of the health in good health questionnaire to assess physical activity and eating habits in high school students. Brazilian Journal of Physical Activity & Health 2012; 12(3): 12-20. 22. Guedes DP, Lopes CC, Guedes JERP. Reproducibility and validity of the International Physical Activity Questionnaire in adolescents. Brazilian Journal of Sports Medicine 2005; 11(2): 151-158.