Validity and Reliability Analysis of Student Autonomy, Student Engagement, and English Learning Motivation Scales

Caiyun Yu¹, Siti Maziha Mustapha²

^1,2 Faculty of Business, Information & Human Sciences, Infrastructure University Kuala Lumpur, Kajang, Selangor, Malaysia

This pilot study aimed to evaluate the reliability and construct validity of three instruments—Learner Autonomy Scale (LAS), Utrecht Work Engagement Scale for Students (UWES-S), and English Learning Motivation Scale (ELMS)—in the context of higher vocational college EFL students in Shandong Province. A total of 120 valid responses were collected and analyzed using SPSS Statistics 26. Reliability analysis showed high internal consistency for all three instruments, with Cronbach’s alpha values exceeding 0.90. Exploratory Factor Analysis (EFA) was conducted to assess the underlying factor structure of each scale. For the LAS, originally consisting of 32 items, five components were identified. Ten items with factor loadings below 0.6 were removed, resulting in a refined 22-item structure. For the UWES-S (17 items), three components—vitality, dedication, and focus—were identified, with all items retained. The final structure accounted for 82.036% of the variance. For the ELMS (18 items), two components—extrinsic and intrinsic motivation—were confirmed. All items were retained, and the structure explained 79.587% of the variance, with all factor loadings exceeding 0.7. The results demonstrated that the revised instruments are reliable and valid for further large-scale data collection in this research.

KEYWORDS: Construct validity, reliability analysis, Exploratory Factor Analysis

I. INTRODUCTION

In recent years, China has placed increasing emphasis on the development of vocational education to meet the demands of economic transformation and industrial upgrading. As part of this effort, the Ministry of Education and the Ministry of Finance jointly launched the “Double-High Plan” in 2019, aiming to build high-level vocational colleges and promote the construction of high-level programs with Chinese characteristics (Ministry of Education & Ministry of Finance, 2019). There are 15 such “Double-High Plan” higher vocational colleges in Shandong province, which play a critical role in cultivating skilled talent for regional economic growth.

In 2023, a total of 75,686 students were enrolled in these 15 institutions. To ensure the representativeness of the research sample, a proportional random sampling method was adopted in this study. This method ensured that students from each of the 15 colleges had an equal probability of being selected, thereby reflecting the actual composition of the population. Proportional stratified sampling is commonly used in educational research to ensure that each subgroup is adequately represented in the overall sample (Creswell & Creswell, 2018). Using a random number table for each institution, a total of 130 English as a Foreign Language (EFL) students were selected for this pilot study.

The primary purpose of the pilot study was to evaluate the reliability and initial applicability of three research instruments: the Learner Autonomy Scale (LAS), the Utrecht Work Engagement Scale for Students (UWES-S), and the English Learning Motivation Scale (ELMS), consisting of 32, 17, and 18 items respectively. These instruments were selected to explore the relationship between EFL learners’ autonomy, engagement, and motivation within the context of higher vocational education in Shandong province. This pilot study was conducted in preparation for the subsequent field research and is expected to inform the refinement of research tools and provide preliminary insights for the main study.

1. MEASURE OF THE CONSTRUCTS

Likert-type rating scales are among the most widely used tools for capturing perceptions, attitudes, and behavioral tendencies in social science research (Likert, 1932; Boone & Boone, 2012). These instruments allow researchers to assign numerical values to qualitative constructs and statistically analyze relationships within theoretical frameworks.

In this study, a 5-point Likert scale was employed to measure student autonomy, engagement, and learning motivation. Each item was rated on a scale ranging from 1 (strongly disagree) to 5 (strongly agree), with a neutral midpoint included to allow respondents to express uncertainty or indifference. This design helps reduce response bias and supports more accurate self-assessments (Joshi, Kale, Chandel, & Pal, 2015).

The 5-point format is concise, easy to use, and particularly suitable for mobile-device users completing surveys (Revilla, Saris, & Krosnick, 2014). It also enables robust statistical analysis (Privitera & Ahlgrim, 2019) and is widely accepted in educational and psychological research, further supporting its validity and reliability as a measurement tool.

III. REARCH INSTRUMENTS

In this study, the researcher employed a quantitative, non-experimental, correlational survey design to answer the research questions and test the proposed hypotheses (Creswell & Creswell, 2018). This design was selected to investigate the relationships among student autonomy, student engagement, English learning motivation, and English language performance among vocational college students.

In the conceptual framework of this study, English learning motivation was treated as a mediating variable between student autonomy and English language performance, as well as between student engagement and English language performance. English language performance was assessed using students’ slf-reported Grade Point Average (GPA), a commonly used and valid indicator in educational research (Kuncel, Hezlett, & Ones, 2001).

To collect data and address the research objectives, a structured questionnaire was developed for this pilot study. The questionnaire consisted of three standardized instruments: the Learner Autonomy Scale (LAS), containing 32 items (Xu, 2006), the Utrecht Work Engagement Scale-Student version (UWES-S), comprising 17 items (Schaufeli et al., 2002), and the English Learning Motivation Scale (ELMS), including 18 items (Li, 2021).

These instruments were selected based on their proven reliability and validity in second language acquisition and educational psychology research. The pilot study was conducted to examine the reliability of these scales and to prepare for the subsequent large-scale field study.

PILOT DATA COLLECTION PROCEDURES

Before distributing the questionnaire to the 136 randomly selected students from 15 “Double-High Plan” higher vocational colleges in Shandong Province, several preparatory steps were taken. First, the researcher obtained formal permissions from the developers of the three instruments via email. Then, recommendation letters were issued by Infrastructure University Kuala Lumpur and Shandong Vocational College of Science and Technology. These letters introduced the researcher and explained the purpose of the data collection, which facilitated the approval process from the target institutions.

With the support of university faculty contacts, the researcher distributed the questionnaire via the online survey platform Questionnaire Star. Participants received an email containing instructions and a survey link or QR code. Upon completion, the responses were automatically collected through the platform.

A total of 136 questionnaires were distributed, and all were returned. After data screening and cleaning, 120 valid responses were retained, resulting in an effective response rate of 88.23%. As a minimum of 100 responses is generally considered sufficient for a pilot study (Kline, 2016), the sample size was deemed adequate. Reliability analysis and exploratory factor analysis (EFA) were conducted on the cleaned pilot data to assess the internal consistency and construct validity of the instruments prior to the main study.

RELIABILITIES ANALYSIS

The reliability of the data collection instruments (LAS, UWES-S, ELMS) used in this study was assessed using Cronbach’s alpha to measure internal consistency. The analysis was conducted with SPSS Statistics 26 (Statistical Package for Social Sciences, version 26.0) after collecting the pilot data.

As shown in Table 1, the overall Cronbach’s alpha value for all 67 items of the pilot study was 0.963. The reliability coefficients for the individual scales were as follows: student autonomy (0.930), student engagement (0.911), and English learning motivation (0.957).

Table 1 Reliability Analysis of Pilot Data (N = 120)

These findings indicate that the instruments used in this study exhibit high reliability, confirming their suitability for the main study.

EXPLORATORY FACTOR ANALYSIS (EFA)

Exploratory Factor Analysis (EFA) is used to explore the underlying structure of a set of variables and to identify potential redundancy within those variables (Garson, 2012). It is particularly useful in the early stages of research to uncover interrelationships among variables and test the hypothesized factor structure.

In this study, EFA was applied during the pilot phase with three main objectives: (i) to determine the factor structure among the constructs (Bentler & Chou, 1987), (ii) to assess the unidimensionality of the theoretical constructs, and (iii) to reduce the number of variables (Savin & Tombs, 2017; Hair, 2018). Given its suitability, EFA was chosen to examine the items used to measure student autonomy, student engagement, and English learning motivation.

A total of 120 responses were analyzed using SPSS Statistics 26 to conduct the EFA, which provided insights into the factor structure and dimensionality of the instruments.

Exploratory Factor Analysis for Student Autonomy

The Student Autonomy construct, as measured by the Learner Autonomy Scale (LAS), was developed by Chinese scholar Xu Jinfen in 2006. This scale comprises 32 items grouped into five dimensions: (i) awareness of teaching objectives and requirements (ATOR) (5 items), (ii) establishing learning objectives and plans (ELOP) (5 items), (iii) effective use of learning strategies (EULS) (5 items), (iv) monitoring of strategy use (MSU) (7 items), and (v) evaluation of learning process (MELP) (10 items).

As shown in Table 2, the pilot data (N = 120) was suitable for factor analysis. The Kaiser-Meyer-Olkin (KMO) measure for student autonomy was 0.853, which exceeds the recommended threshold of 0.6 (Hair et al., 2018). Additionally, Bartlett’s Test of Sphericity was significant (Chi-Square = 2817.353, p < 0.001), indicating that the data was appropriate for factor analysis. Given the KMO value close to 1.0 and the significant Bartlett’s Test result, the 32 items of student autonomy were deemed suitable for Exploratory Factor Analysis (EFA). The correlation matrix also revealed that many items had correlation coefficients of 0.3 or higher, further supporting the adequacy of the data for factor analysis.

Table 2 KMO and Bartlett’s Test of Student Autonomy

Based on Principal Component Analysis (PCA) with varimax rotation, Table 3 presents the factor extraction results for the 32 items measuring student autonomy. Five components were extracted, all with eigenvalues greater than 1.0, explaining a total variance of 79.733%, which exceeds the minimum acceptable threshold of 60% for construct validity in factor analysis (Hair et al., 2018). Specifically, Factor 1 accounted for 19.36%, Factor 2 for 19.205%, Factor 3 for 13.996%, Factor 4 for 13.944%, and Factor 5 for 13.229% of the total variance.

Table 3 Total Variance Explained for Student Autonomy

Extraction Method: Principal Component Analysis

As illustrated in Figure 1, the scree plot for student autonomy indicated a clear inflection point after the third component, suggesting a three-factor solution.

Figure 1 Scree Plot of Student Autonomy (See in PDF file)

However, based on the rotated component matrix in Table 4, several items had factor loadings below the 0.60 threshold (Garson, 2012), including: ATOR3, ELOP5, MSU2, MSU3, MSU4, MELP1, MELP2, MELP3, MELP5, and MELP10. These items were also found to perform poorly in the earlier reliability analysis, and were thus removed from further analysis.

Table 4 Rotated Component Matrix of Student Autonomy

Extraction Method: Principal Component Analysis.

Rotation Method: Varimax with Kaiser Normalization.

Rotation converged in 6 iterations.

Based on the results of the data analysis, 10 items from the original 32-item LAS were removed due to low factor loadings (< 0.6). An exploratory factor analysis (EFA) was then rerun on the remaining items. The final analysis supported a five-factor structure of student autonomy with 22 items, comprising: ATOR (4 items), ELOP (4 items), EULS (5 items), MSU (4 items), and MELP (5 items). This revised structure was adopted for further data analysis in the main study.

Exploratory Factor Analysis for Student Engagement

The student engagement variable was measured using 17 items from the Utrecht Work Engagement Scale for Students (UWES-S) developed by Schaufeli et al. (2002), comprising three dimensions: Vitality (6 items), Dedication (5 items), and Focus (6 items). This scale has been widely validated across educational contexts.

As shown in Table 5, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was 0.832, which is considered “great” (Hair et al., 2018), and exceeded the acceptable threshold of 0.6. In addition, Bartlett’s Test of Sphericity was statistically significant (χ² = 2862.419, p < .001), indicating that the correlation matrix was not an identity matrix and thus suitable for factor analysis. The item correlation matrix also revealed many coefficients above 0.3, supporting inter-item correlation adequacy.

Based on Principal Component Analysis (PCA) with varimax rotation, Table 6 shows that three components were extracted with eigenvalues greater than 1.0, explaining a total variance of 82.036%, well above the 60% minimum threshold for acceptable construct validity in EFA (Hair et al., 2018). Specifically: component 1 accounted for 31.369% of the variance, component 2 explained 26.686%, and component 3 explained 23.981%.

As visualized in the scree plot (Figure 2), a distinct “elbow” was observed after the third component, indicating a three-factor solution was appropriate.

Figure 2 Scree Plot of Student Engagement (See in PDF file)

Table 7 presents the rotated component matrix, where all 17 items exhibited factor loadings above 0.60, satisfying the criteria for practical significance (Garson, 2012). The items were clearly grouped into their respective dimensions, corresponding to the original structure of the UWES-S.

In conclusion, the exploratory factor analysis confirmed the three-dimensional structure of student engagement in this context. The final model retained all 17 items across three validated sub-constructs: vitality (6 items), dedication (5 items), and focus (6 items). These results align with previous findings in other educational settings, reaffirming the construct validity of UWES-S in the current sample.

Exploratory Factor Analysis for English Learning Motivation

The English learning motivation construct in this study was assessed using an 18-item scale developed by Li Xiaoyan (2012). This instrument was designed specifically for Chinese higher vocational college students and has been used to measure two distinct types of motivation in English language learning: extrinsic motivation and intrinsic motivation. The scale has demonstrated good reliability and validity (Cronbach’s α = 0.832; KMO = 0.931) in prior studies across similar educational contexts in China.

As shown in Table 8, the KMO value for the English learning motivation construct was 0.917, which is considered excellent and well above the minimum threshold of 0.6 (Hair et al., 2018). Bartlett’s Test of Sphericity was also significant (χ² = 2740.255, p < .001), indicating that the correlation matrix was suitable for factor analysis. Furthermore, many of the inter-item correlations exceeded 0.3, further supporting the appropriateness of EFA.

Using Principal Component Analysis (PCA) with varimax rotation, the results in Table 9 showed that two components were extracted, each with an eigenvalue greater than 1.0. The total variance explained was 79.587%, exceeding the commonly accepted threshold of 60% for a valid factor structure. Specifically, Component 1 accounted for 42.925% of the variance, and Component 2 accounted for 36.662%.

Extraction Method: Principal Component Analysis.

As illustrated in Figure 3, the scree plot confirmed the two-factor solution, with a clear inflection point after the second factor.

Figure 3 Scree Plot of English Learning Motivation  (See in PDF file)

According to the rotated component matrix (Table 10), all 18 items had factor loadings greater than 0.70, indicating strong associations with their respective factors. Items were cleanly grouped into the two originally defined sub-constructs without any cross-loadings.

Extraction Method: Principal Component Analysis.

Rotation Method: Varimax with Kaiser Normalization.

Rotation converged in 3 iterations.

The final EFA results confirmed that the two dimensions of the English learning motivation construct aligned with Li Xiaoyan’s (2012) original theoretical model. These dimensions include extrinsic motivation (9 items) and intrinsic motivation (9 items). These results validate the construct structure of the scale and support its use in further analysis in this study.

VI. CONCLUSION

The pilot study was conducted to test the reliability and validity of the research instruments and to ensure the appropriateness of the questionnaire for the main field study. The pilot questionnaire comprised a total of 67 items, including the Learner Autonomy Scale (LAS, 32 items), the Utrecht Work Engagement Scale–Student version (UWES-S, 17 items), and the English Learning Motivation Scale (ELMS, 18 items). Following the pilot study and the results of exploratory factor analysis (EFA), a total of 10 items were removed from the LAS due to low factor loadings or failure to meet reliability thresholds. Besides, no items were removed from the UWES-S and ELMS, as all items met the criteria for construct validity and internal consistency.

Consequently, the final version of the questionnaire used in the field study consisted of 57 items: LAS (22 items), UWES-S (17 items), and ELMS (18 items). The refined instrument demonstrated sound psychometric properties and was deemed suitable for large-scale data collection in the main research phase. In preparation for the subsequent field study, the final questionnaire was reviewed for cultural relevance and clarity. A comprehensive training session was conducted for data collectors to ensure consistent administration and to minimize any potential biases during data gathering. This preparation phase was essential to ensure the validity and reliability of the instrument in the broader sample of participants for the main study.

REFERENCES

1. Bentler, P. M., & Chou, C. P. (1987). Practical issues in structural modeling. Sociological Methods & Research, 16(1), 78–117.
2. Boone, H. N., Jr., & Boone, D. A. (2012). Analyzing Likert data. Journal of Extension, 50(2), Article 2TOT2.
3. Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE Publications.
4. Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Plenum.
5. Field, A. (2018). Discovering statistics using IBM SPSS Statistics (5th ed.). SAGE Publications.
6. Garson, G. D. (2012). Factor analysis: Statnotes. North Carolina State University. Retrieved from
7. Hair, J. F. (2018). Multivariate data analysis (8th ed.). Pearson Education.
8. Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (2018). Multivariate data analysis (8th ed.). Pearson Education.
9. Joshi, A., Kale, S., Chandel, S., & Pal, D. K. (2015). Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4), 396–403.
10. Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.
11. Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162–181.
12. Li, X. Y. (2012). An empirical study on the structure of English learning motivation among vocational college students. Examination Research, (3), 25–29.
13. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.
14. Ministry of Education & Ministry of Finance. (2019). Notice on the implementation plan of the “Double High-level Plan” for vocational colleges.
15. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
16. Privitera, G. J., & Ahlgrim, B. (2019). Research methods for the behavioral sciences (3rd ed.). SAGE Publications.
17. Revilla, M. A., Saris, W. E., & Krosnick, J. A. (2014). Choosing the number of categories in agree–disagree scales. Sociological Methods & Research, 43(1), 73–97.
18. Savin, S., & Tombs, P. (2017). Exploratory factor analysis: A guide to using SPSS and R. Psychological Methods, 22(3), 412–432.
19. Schaufeli, W. B., Martínez, I. M., Pinto, A. M., Salanova, M., & Bakker, A. B. (2002). Burnout and engagement in university students: A cross-national study. Journal of Cross-Cultural Psychology, 33(5), 464–481.
20. Schaufeli, W. B., Salanova, M., González-Romá, V., & Bakker, A. B. (2002). The measurement of engagement and burnout: A two sample confirmatory factor analytic approach. Journal of Happiness Studies, 3, 71–92.
21. Xu, J. F. (2007). Theory and practice of autonomous college foreign language learning. China Social Sciences Press.