ISSN e: 2007-4018 / ISSN print: 2007-4018

English | Español

     

 
 
 
 
 
 
 
 

Vol. XXIII, issue 3 September - December 2017

ISSN: ppub: 2007-3828 epub: 2007-4018

Scientific article

Model of selection and evaluation for graduate applicants in forest sciences

http://dx.doi.org/10.5154/r.rchscfa.2016.12.074

Zamudio-Sánchez, Francisco J. 1 ; Romo-Lozano, José L. * 1 ; Borja-de la Rosa, Amparo 1 ; Martínez-Gómez, Gladys 1 ; Ávalos-Vargas, Adriana 1

  • 1Universidad Autónoma Chapingo. Carretera México-Texcoco km 38.5. C. P. 56230. Chapingo, Texcoco, Estado de México, México.

Corresponding author: joseluisromolozano@yahoo.com.mx, tel.: 52+ (595) 9521787

Received: December 22, 2016; Accepted: June 07, 2017

This is an open-access article distributed under the terms of the Creative Commons Attribution License view the permissions of this license

Abstract

Introduction:

The admission process of students to a postgraduate program is very important for the improvement of indicators of greater importance in the quality of the program. The problem is the large amount of information requested that is not always considered objectively for the selection of applicants with the desired profile.

Objective:

Analyze evaluation models to select postgraduate applicants and, with a metric, choose the most compatible model.

Materials and methods:

We used information from 19 applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo. We applied subjective methods of multi-criteria analysis in the phase of consideration of the criteria (3) and sub-criteria (8): point allocation method and analytical hierarchical process. Values ​​were aggregated using the TOPSIS method and the weighted sum method. The most compatible weighting-aggregation combination was determined with Pareto order.

Results and discussion:

The combination of the weighted sum method and analytical hierarchical process showed a lower average distance to the order of the rest of the combinations and, consequently, generated a selection of applicants more compatible with the selection criteria.

Conclusion:

Multi-criteria methods represent a good option to properly consider the amount of information generated in a selection process.

Keywords:Multi-criteria analysis; weighted sum; TOPSIS; Pareto order

Introduction

The essential purpose of assessing applicants for a postgraduate program is generally to assess the level of satisfaction of the admission profile and to select those who possess characteristics compatible with the program and ensure the success of the program and the individual. “In the strict sense, the aptitude assessments are a diagnostic nature of capacities or potentialities tending to predict educational success” (Ibañez, 2009). The procedures and criteria for admission and selection of students are critical for measuring the organizational quality of the postgraduate program and also have a direct influence on academic quality. “We cannot forget that the quality and credit of a master's degree are intimately related to the quality of the students selected” (Bengoetxea-Castro & Arteaga-Ortiz, 2009).

At the global level, there is a broad agreement in the educational field about the importance of the student selection process for admission for postgraduate studies. In the United States of America, there are many analysis and meta-analysis of admission processes (Kuncel, Credé, & Thomas, 2007; Kuncel, Wee, Serafin, & Hazlett, 2010), and among the most important results the next test predict in an acceptable manner the level of the students, scientific productivity and number of citations: Graduate Record Examinations (GRE-T), Graduate Management Admission Test (GMAT) and Miller Analogies Test (MAT). Similarly, most of the countries of the European Union have made important efforts to homogenize the selection criteria in graduate programs (Davies, 2009; European University Institute [EUI], 2014; Kehm, 2006); standing out themes of equity, quality and capacity to promote mobility. Most European countries operate with a dual system in which universities have some control over admission, but within the framework of government guidelines on selection criteria (McGrath et al., 2014).

In Latin America efforts that express the attention for admission, demanded by the process of selection of applicants for postgraduate studies, have also been made. In Colombia, Colonia-Duque (2010) characterized the profile of graduate students and concluded, among other things, that there is a significant correlation between the average grades obtained in the master's program with variables such as curriculum, average undergraduate grades, interview and skills test. In Chile, they continue testing instruments capable of measuring attributes that complement indicators already used for postgraduate admission processes (Santelices et al., 2010).

In Mexico, a large number of universities use the test known as Examen Nacional de Ingreso (EXANI-III), proposed in 1996 by the Centro Nacional de Evaluación para la Educación Superior (CENEVAL), for the selection of postgraduate students. The thematic structure of the test includes: logical-mathematical reasoning, verbal reasoning, research methodology and skills, information and communication technology (ICT) and English. However, although few studies have been carried out on the subject, it has been found that the predictive capacity of the test to obtain the master’s degree in the desirable times is very low (Álvarez-Montero, Mojardín-Heráldez, & Audelo-López, 2014).

García (1995) emphasized that a characteristic of postgraduate studies in Mexico is that public institutions are much more selective than the private sector. On the other hand, there are a large number of universities that have designed their own test and selection procedure, which integrate the relevant information to select the students that meet the requirements of the admission profile; this is the case of the Master's program in Forest Sciences of the Universidad Autónoma Chapingo (UACh).

The current state of the Mexican postgraduate study system is largely explained by the national policy conducted by the Consejo Nacional de Ciencia y Tecnología (CONACYT) through the program known as Programa Nacional de Posgrados de Calidad (PNPC), which has been executed since 1991 (Adalid & de Urdanivia, 2011). Admission to this program is an important condition for the existence and development of postgraduate programs, especially those in public education. Admission is achieved through participation and approval of a set of indicators included in the three stages of the established evaluation model: self-evaluation, peer evaluation, and results and impact assessment (Consejo Nacional de Ciencia y Tecnología [CONACYT], 2013). In this process, the selection of applicants for postgraduate programs is a stage that can facilitate the achievement of quality indicators if done properly, which are vital to enter or remain in the PNPC

The objective of this research was to determine the combination weighting-aggregation that generates an index for the selection of applicants and is the most compatible with the criteria of the master’s program. For this purpose, we used information from 19 applicants in the year 2015. The combination of different weighting methods with different aggregation methods has been addressed in a very limited way. The literature reports few cases, specifically, within the framework of composite indices (Chakrabarty & Bhattacharjee, 2012). On the other hand, although the issue of selection in decision-making is quite broad and has been approached from different angles, no specific studies are known that analyzes combinations of weighting methods with aggregation methods in decisions related to selection.

Materials and methods

We used two methods of weighting of the criteria and sub-criteria: Point Allocation Method (PAM) and Analytical Hierarchical Process (AHP). The results of both subjective weightings were used to estimate the classification of the 19 students using the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and the Weighted Summarize Method (WSM), resulting in four combinations: TOPSIS-PAM, TOPSIS-AHP, WSM-PAM and WSM-AHP. Subsequently, the Pareto Analysis was used to determine the most compatible combination with the criteria of the program and, consequently, with the best selection of applicants.

Evaluation and selection model for applicants for the Master’s program in Forest Science at the Universidad Autónoma Chapingo

The selection of applicants is conducted by the Coordination of Postgraduate Studies of the program, which is composed by a Coordinator and four members, all of the faculty members. Accepted applicants are those whose overall score is equal to or greater than 80. The selection process looks at three criteria: knowledge, experience and personality. Each criterion comprises different sub-criteria (Figure 1) evaluated on a scale from 0 to 100.

Figure 1. Components of the model of selection for applicants for the Master's Degree in Forest Sciences at Universidad Autónoma Chapingo. JCR: Journal Citation Reports

  • 1.

    Knowledge test. It is a written test consisting of 20 questions; 15 are related to Forest Sciences and five with other areas of interest

  • 2.

    Skills test. Written test to identify aspiring skills in the areas of math, logic, and research. This test has four sections: the first relates to questions related to verbal reasoning, the second deals with mathematical aspects, the third relates to the approach of solving problems and the latter concerns research skills. Each section is composed of 10 questions.

  • 3.

    English language diagnostic test. It is applied to know the English level of the aspirants, in order to, after being accepted, identify the needs of courses for the good performance in the reading and understanding of the literature of support in courses and research. Generally, this test is a version of the Test of English as a Foreign Language (TOEFL) applied by the University's Language Department. The raw score of each section is converted by statistical means into a number; the total score of the test is reported on a scale from 310 to 677. Grades were converted on the scale from 0 to 100, using the technique of rank normalization.

  • 4.

    Curriculum. The background and professional experience of the applicant, are qualified through the documentary evidence.

  • 5.

    Average of the previous grade. The minimum average should be 80 points on the scale from 0 to 100.

  • 6.

    Practical skills for research and exercise of synthesis and understanding. This task consists of searching for five scientific articles published in indexed journals, preferably included in JCR (Journal Citation Reports). Articles must be related to the research topic proposed by the applicant; two in Spanish and three in English. Applicants should summarize on a page the content of each article correlatively with their research topic and deliver within a period of five business days.

  • 7.

    Interview with the Coordinator of the program. The interview is conducted in two phases. In the first one, a questionnaire is given to each applicant, containing 16 questions related to the following subjects: personal objectives to study this Master’s program, research and teaching concerns, knowledge of the academic staff of the program and knowledge of other similar options of study. The second phase consists of direct interview with the members of the Coordination, who ask about aspects addressed in the questionnaire of the first stage and other pertinent questions, to obtain good results in the program.

  • 8.

    Psychometric test. This test is applied by psychology professionals at the university. The results are confidential and are expressed in three categories: fit, moderately fit and unfit, to which the grades of 100, 80 and 66, respectively, are assigned.

Weighting methods

The PAM consists in giving the set of criteria to be weighed to a group of experts and they are asked to distribute 100 points between the set of criteria and subcriterios considered. All this is based on experience and subjective judgment on the importance of each criterion (Organisation for Economic Co-operation and Development [OECD], 2008).

The method AHP uses paired comparisons to analyze alternatives, but also, as in this case, is used to estimate the weightings of the criteria. Comparisons are based on expert judgment to derive priority scales (Saaty & Vargas, 2012). One of the disadvantages is that, due to paired comparisons, inconsistencies can occur in the judgments expressed by the experts when classifying the criteria, especially when they are many criteria (Velásquez & Hester, 2013); however, an ad hoc modification of the Saaty and Vargas method (2012) can avoid them. On the other hand, one of the advantages of the method is the easy use to derive weightings of coefficients and compare the alternatives. Comparisons are made by pairs of criteria, proposing the following questions: Which of the two criteria is the most important? and How many times? The strength of the preference is expressed on a scale of 1 to 9, which allows measurement within the same order of magnitude. The preference of 1 indicates equality of importance between two criteria, while a preference of 9 indicates that a criterion is extremely more important than that with which it is compared (Table 1). The information obtained was processed using the program Expert Choice Desktop 11.5.1860 (Expert Choice Inc., 2014).

Table 1. Saaty and Vargas Scale (2012) to estimate the priority of the criteria used in the selection of postgraduate candidates.

Value Definition Comment
1 Equal importance Criterion 1 is just an important as criterion 2.
3 Moderate importance Experience and judgment slightly favor criterion 1 over 2.
5 Strong importance Experience and judgment strongly favor criterion 1 over 2.
7 Very strong importance Criterion 1 is more important than criterion 2.
9 Extremely important The greater importance of criterion 1 over criterion 2 is beyond doubt.
2, 4, 6 and 8 Intermediate values between the previous ones, when it is necessary to qualify.

Aggregation methods

The TOPSIS method is commonly used in multi-criteria decision analysis. In general, the method consists of estimating a positive ideal point and a negative ideal point. The positive ideal point corresponds to a supposed alternative, from the best values reached in each one of the attributes (criteria) of the set of alternatives analyzed. The negative ideal point, on the other hand, corresponds to another supposed alternative containing the worst values obtained in each of the same criteria. Once these references are developed, the weighted distances of each alternative to the best and worst alternative are estimated and their classification is defined (Pavić & Novoselac, 2013; Wang & Xiao, 2011). The basic principle of the method is that the selected alternative must have the shortest distance of the positive ideal solution and the farthest distance of the non-ideal solution, in a geometrical sense.

The method assumes that each attribute has monotonically increasing or decreasing utility. This facilitates the task of locating ideal solutions (positive and negative). Thus, the preference order of the alternatives is produced by comparing Euclidean distances (San Cristóbal, 2012).

Assuming that there is a set of alternatives A = { a 1 , a 2 , a m ) , qualified by a set of criteria X = { X 1 , X 2 , X n } The decision matrix X = X i j m x n is formed. From n criteria, the first k are assumed positive (higher value, better) and the last n-k are assumed negative (lower value, better). The development of the method is described below.

1. The normalized matrix is constructed by replacing each x ij value by its normalized value (r ij ) using the following formula:

r i j = x i j i = 1 m x i j 2

2. The weighted matrix is constructed by replacing each r ij by its weighted value (a ij ) using the corresponding weightings established (w j ) using the following formula:

a i j = w j r i j = w j x i j i = 1 m x i j 2

3. The best a j + values are selected to establish the positive ideal alternative A + = ( a 1 + ,   a 2 + ,   a n + ) using the following procedure:

a j + = m a x i a i j , j = 1 , . . . , k m i n i a i j , j = k + 1 , . . . , n

The minimum components of this positive ideal alternative correspond to criteria to be minimized.

4. Similarly, the negative ideal alternative A - = ( a 1 - ,   a 2 - ,   a n - ) is established, where each a j - is selected using the following formula:

a j - m i n i a i j , j = 1 . . . , k m a x i a i j , j = k + 1 . . . , n

The maximum components in this alternative correspond to criteria to be maximized.

5. The vector of distances D + = ( d 1 + ,   d 2 + ,   d m + ) T to the alternative A + is estimated by:

d i + = d A i , A + = j = 1 n a - i j a j + 2

6. Similarly, the distances D - = ( d 1 - ,   d 2 - ,   d m - ) T to the alternative A - were estimated with the following formula:

d i - = d A i , A - = j = 1 n a - i j a j - 2

7. Finally, the relative distances D * = ( D 1 * ,   D 2 * ,   D m * ) T of the alternatives A i to the points A + and A - were estimated with the following formula:

D i * = d i - d i + + d i - = d A i , A - d A i , A + + d A i , A -

The highest D i * value indicates the best alternative and the lowest value is the worst alternative

The WSM, also known as a weighted linear combination or classification, shows how the values of the multiple criteria options can be combined into a global value. This is done by multiplying the score of the value of each criterion by the weight of that criterion and then adding all those weighted grades; that is to say:

A i = w j x i j

where x ij is the recorded value of alternative i with respect to criterion j, and w ij is the criterion weighting (Afshari, Mojahed, & Yuseff, 2010). This method clearly demonstrates the main concept of multi-criteria evaluation methods: the integration of the values and weightings of the criteria into a single magnitude (Podvezko, 2011). The condition of application of the method is that the criteria must be independent of each other. This method is one of the most frequently used in the multi-criteria approach (Paracchini, Pacini, Calvo, & Vogth, 2007).

Standardization

The range standardization technique was used in order to convert the data generated by each weighting-aggregation combination to the 0-1 scale using the following equation (Aznar-Bellver & Guijarro-Martínez, 2012):

x i j s t a n d a r d i z e d = x i j - m i n x i j m a x x i j - m i n x i j

The interval of the standardized values is 0 ≤ x ij ≤ 1. In this standardization procedure, the element of minimum qualification always takes value 0 and the one of maximum takes value 1.

Pareto order

The use of Pareto order in the classification of results, via indexes (classifiers), was proposed by Chakrabarty and Bhattacharjee (2012). We consider p = 4 classifiers, which result from the two weighting methods and the two methods of aggregation. In this study it was classified by the estimation of the following expressions:

d i j k = i j - i k 2 j k ,    i = 1 , 2 , . . . , m ; k , j = 1 , 2 , . . . , p ,

d j = i = 1 i = m k = 1 k = p d i j k ,

d j - = d j p - 1

where,

d i j k = square of the difference of the classifications of the applicant (alternative) i, obtained by classifiers j and k (jk; j, k = 1, 2…, p)

ϵ i j = classification of the applicant i (i = 1, 2…, m) by the classifier j (j = 1, 2…, p)

d j = total distance of the classifier j (j = 1, 2…, p)

d - j = average distance of the classifier j (j = 1, 2…, p) with respect to the other methods.

The method that has the lowest mean distance value is the one that is identified as the most compatible for the criteria used.

Results and discussion

The grades of the applicants in the selection and evaluation process, per subcriterion, are shown in Table 2. As expected, the resulting grades express conflict between the different sub-criteria considered; namely, generally, each criterion points out different alternatives (applicants) as the best. Likewise, it is observed that practically all the applicants obtained acceptable grades on the knowledge test, while the lower grades of some applicants were obtained on the skills and English tests.

Table 2. Grades obtained by the applicants for the Master program in Forest Science at Universidad Autónoma Chapingo.

Applicant Knowledge Experience Personality
EC EH EI Curr Average JCR Interv Psychom
1 85 93 0 78 80 60 65 80
2 90 60 9 82 80 90 90 100
3 70 75 50 84 85 85 70 100
4 85 48 25 85 88 80 89 80
5 85 78 17 86 89 70 79 80
6 80 73 26 82 86 50 73 80
7 90 63 71 78 82 65 73 100
8 85 53 79 83 81 92 69 80
9 70 78 39 84 86 74 68 66
10 85 70 99 82 85 88 88 66
11 70 63 70 80 86 90 90 80
12 75 73 100 88 90 60 83 80
13 90 88 50 80 84 76 76 100
14 80 58 100 84 88 75 84 80
15 85 78 91 80 81 88 92 80
16 90 80 57 86 84 40 85 80
17 80 65 82 82 82 70 89 80
18 80 73 97 90 95 78 78 80
19 80 50 100 84 87 90 75 80
EC = knowledge test, EH = skills test, EI = English test, Curr = curriculum, Average = average, JCR = review of article published in a journal recognized in the Journal Citation Reports, Interv = interview, Psicom = psychometric test.

The weighting of criteria and sub-criteria was collected through a survey designed especially for the experts, who have been or are members of the Coordination of Postgraduate Studies in which the selection of applicants was made, which guarantees the experience in the selection process. The results obtained by the AHP and PAM method (Table 3) show significant disparities in the weighting of the criteria and sub-criteria. The case of the sub-criteria knowledge test, to which the weighting method of AHP assigned twice the weight compared to the PAM method. Another case that calls attention is the sub-criteria interview of the Coordination, to which PAM assigns a weight 3.5 times greater than AHP.

Table 3. Weightings of the analytical hierarchical process (AHP) and the point allocation method (PAM) in the selection of applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo.

Criteria Sub-criteria Weighing AHP Weighing PAM
Criteria Sub-criteria Final Criteria Sub-criteria Final
Knowlegde Knowledge test 0.76 56.30 42.49 0.47 45.00 21.38
Skills test 30.70 23.16 31.25 14.84
English test 13.10 9.86 23.75 11.28
Experience Curriculum vitae 0.18 22.50 4.00 0 .25 35.00 8.75
Average grades 65.90 11.72 43.75 10.94
JCR exercise 11.70 2.08 21.25 5.31
Personality Cordination interview 0.07 86.30 5.78 0.27 73.75 20.28
Psychometric test 13.70 0.92 26.25 7.22
JCR: Journal Citation Reports

The application of the two aggregation methods, TOPSIS and WSM, using in each case the two weightings, PAM and AHP, resulted in four weighting-aggregation combinations. Table 4 shows the values for the 19 candidates evaluated.

Table 4. Values obtained in the four combinations of weighting-aggregation methods used in the selection of applicants for the Master's Program in Forest Sciences of the Universidad Autónoma Chapingo.

Applicant TOPSIS-AHP TOPSIS-PAM WSM-AHP WSM-PAM
1 0.5323(48) 0.3670(4) 0.0511(36) 0.0465(0)
2 0.4030(18) 0.3740(6) 0.0491(17) 0.0499(29)
3 0.4539(30) 0.4814(29) 0.0494(20) 0.0505(33)
4 0.3239(0) 0.3625(3) 0.0473(0) 0.0487(18)
5 0.4973(40) 0.3873(9) 0.0515(39) 0.0497(27)
6 0.4337(25) 0.3482(0) 0.0491(17) 0.0473(7)
7 0.5831(60) 0.5871(53) 0.0539(62) 0.0530(54)
8 0.4953(40) 0.5544(45) 0.0516(40) 0.0517(43)
9 0.4405(27) 0.4095(14) 0.0485(12) 0.0475(9)
10 0.6894(85) 0.7530(89) 0.0572(93) 0.0575(92)
11 0.4307(25) 0.5856(52) 0.0495(21) 0.0528(52)
12 0.6196(68) 0.7234(83) 0.0550(73) 0.0562(80)
13 0.7098(89) 0.5954(54) 0.0570(92) 0.0549(70)
14 0.5541(53) 0.6773(72) 0.0538(62) 0.0556(75)
15 0.7556(100) 0.8022(100) 0.0579(100) 0.0585(100)
16 0.6900(85) 0.6103(58) 0.0561(83) 0.0539(62)
17 0.5640(56) 0.6701(71) 0.0532(56) 0.0546(67)
18 0.6727(81) 0.7342(85) 0.0567(89) 0.0571(88)
19 0.5031(42) 0.6190(60) 0.0522(47) 0.0540(62)
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method. Values in parentheses converted to scale 0-100.

The values generated by each combination of methods were converted to scale 0-100, using the standardization technique by ranking (Table 4, values in parentheses), which allowed to identify the applicants that reached or surpassed 80 points in the global qualification. In this way, we find that combinations TOPSIS-AHP and WSM-AHP assign to five applicants a score greater than or equal to 80, while with TOPSIS-PAM and WSM-PAM only to four applicants were assigned (Table 5).

In this analysis it is emphasized that the four combinations include three applicants in their selection; 10, 15 and 18. The applicant number 15 is the one that best satisfies the admission profile since he appears in the first place of the four selections. The second applicant is the number 10, as he appears in second place in three selections and fourth in the other, while applicant number 18 is included in different orders (Table 5). Another important result is the fact that combinations TOPSIS-AHP and WSM-AHP select the same applicants, but in different order. Similarly, combinations TOPSIS-PAM and WSM-PAM select the same applicants, but in exactly the same order.

Table 5. Selection of the best candidates for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo, according to a combination of weighting-aggregation methods.

Classification Applicant/method
TOPSIS-AHP TOPSIS-PAM WSM-AHP WSM-PAM
1 15 15 15 15
2 13 10 10 10
3 16 18 13 18
4 10 12 18 12
5 18 - 16 -
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method.

Pareto order as a technique to identify the weighting-aggregation combination most compatible with the criteria used, according to the total distance or calculated average distance, shows that the aggregation method WSM-AHP has the minimum average distance (Table 6). This method should be considered as the most compatible with the other three built based on the criteria included.

Table 6. Total distances and averages, according to the weighting-aggregation method, for the identification of the combination most compatible with the criteria evaluated in the selection of applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo.

Combination-Methods Total distance Averge distance
TOPSIS-AHP 548 182.66
TOPSIS-PAM 428 142.66
WSM-AHP 336 112
WSM-PAM 432 144
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method.

The fact that the resulting combination is WSM-AHP has the advantage that the aggregation method WSM is easy to apply. On the other hand, the weighting method AHP could present difficulties of operation if the number of criteria to be weighed is increased; however, this possible limitation has not affected the results obtained in the present study.

The order of Pareto determines that the index that generates an order whose distance is the minimum to the ordinances of the other indexes, is the one of greater compatibility with the criteria of the program. This is because each of the indices formed with different combinations, considers the criteria of the program differently, but they are always the same. The association of these ideas come from the following. On the one hand, the allocation of weights via AHP or PAM is done according to the interpretation that the experts have of the criteria of the program and, being participants of the same, they are the ones who know them best and establish through its academy. On the other hand, the aggregation methods, incorporates the assessments of the criteria used in the selection, so that, in this respect, the index also includes them. In other words, each index treats differently the criteria it uses and the difference is given by how they are weighted and aggregated, taking into account the knowledge of the experts and the algebraic properties of the aggregation methods. In this way, each index represents a different way of treating the information, producing a different order.

When you select the index whose order has a minimum distance with the order of the rest of the indexes tested, it is certainly the most compatible among those tested with the criteria of the program. Of course, if more weighting and aggregation methods were tested, the more compatible index would be more robust; however, it is important to keep the options between the most frequent and simple, which are usually the ones that have produced better results, as well as limiting the number of them.

Conclusions

Multi-criteria methods represent a good option to properly consider the large amount of information generated in a selection process. The weighting method determined the applicants to choose, since regardless of the aggregation used, the selected students were the same if the weights of the criteria were the same. The combination of WSM-AHP methods was the most compatible with the other combinations used, as it induces a classification whose distance to other classifications is minimal. This indicates that the considerations of the experts to weight (AHP) and the technique used to add (WSM) are more compatible than the rest of the combinations. Since some of the criteria considered are qualitative and susceptible to be evaluated by linguistic tags, a possible direction in later studies is the use of fuzzy logic. During the last 10 years, students in the Master’s program in Forest Science were chosen using the point allocation method (PAM) in the criterion weighting phase, and the weighted sum method (WSM) in the aggregation. The results obtained through this selection were very good, so they add certainty and offer the opportunity to specify the procedure used so far.

References

Adalid, C. M., & de Urdanivia, D. (2011). Conacyt y el posgrado: Políticas de evaluación y calidad. Gestión y Estrategia, 40, 87-98. Retrieved from http://zaloamati.azc.uam.mx/bitstream/handle/11191/2972/conacyt-y-el-posgrado-politicas-de-evaluacion-y-calidad.pdf?sequence=1

Álvarez-Montero, F., Mojardín-Heráldez, A., & Audelo-López, C. (2014). Criterios e instrumentos para la admisión en estudios de doctorado. Journal of Research in Educational Psychology, 12(3), 853-886. doi: 10.14204/ejrep.34.13138

Afshari, A., Mojahed, M., & Yuseff, R. M. (2010). Simple additive weighting approach to personnel selection problem. International Journal of Innovation, Management and Technology, 1(5), 511-515. doi: 10.7763/IJIMT.2010.V1.89

Aznar-Bellver, J., & Guijarro-Martínez, F. (2012). Nuevos métodos de valoración: Modelos multicriterio. España: Editorial Universitat Politècnica de València. Retrieved from http://personales.upv.es/~fraguima/Modelos%20multicriterio.pdf

Bengoetxea-Castro, E., & Arteaga-Ortiz, J. (2009). La evaluación de postgrados internacionales en la Unión Europea. Ejemplos de buenas prácticas de programas europeos. Revista de Universidad y Sociedad del Conocimiento, 6(2), 60-68. Retrieved from http://openaccess.uoc.edu/webapps/o2/bitstream/10609/2984/1/v6n2_bengoetxea_arteaga.pdf

Chakrabarty, N., & Bhattacharjee, D. (2012). Aggregation and weightage issues concerning composite index development: Experience with digital divide in Asian countries. Asian Pacific Journal of Library and Information Science, 2(2), 24-37. Retrieved from https://apjlis.msu.ac.th/index.php/APJLIS/article/viewFile/87/93

Colonia-Duque, S. M. (2010). Caracterización del perfil de los estudiantes de posgrado a partir de la información de admisión y el desempeño académico. Aplicación a la maestría en ingeniería administrativa de la Universidad Nacional Sede Medellín. Tesis doctoral, Universidad Nacional de Colombia, Medellín, Colombia. Retrieved from http://www.bdigital.unal.edu.co/3601/1/42144324.2010.pdf

Consejo Nacional de Ciencia y Tecnología (CONACYT). (2013). Marco de referencia para la evaluación y seguimiento de programas de posgrado. Retrieved from http://dsia.uv.mx/sipo/Material_apoyo/Marco_Referencia_PNPC_2013-3.pdf

Davies, H. (2009). Survey of master degrees in Europe. Brussels: European University Association. Retrieved from http://www.eua.be/Libraries/publications-homepage-list/EUA_Survey_Of_Master_Degrees_In_Europe_FINAL_www.pdf?sfvrsn=2

European University Institute (EUI). (2014). Academic rules and regulations for the doctoral and master’s programmes. Retrieved from http://www.eui.eu/Documents/ServicesAdmin/DeanOfStudies/EUI-RulesRegs.pdf

Expert Choice Inc. (2014). Expert Choice Desktop 11.5.1860. Airlington, VA, USA: Author.

García, J. M. (1995). El desarrollo del posgrado en México: El caso de los sectores público y privado. Revista Latinoamericana de Estudios Educativos (México), 20(1), 107-130. Retrieved from http://www.cee.edu.mx/revista/r1981_1990/r_texto/t_1990_1_05.pdf

Ibañez, B. C. (2009). Evaluación diagnóstica y admisión de aspirantes a la educación superior: Algunas notas metodológicas. Tecnociencia Chihuahua, 3(1), 39-46. Retrieved from http://tecnociencia.uach.mx/numeros/v3n1/data/EvaluacionDiagnosticayAdmisiondeAspirantes.pdf

Kehm, B. M. (2006). Doctoral education in Europe and North America: A comparative analysis. Wenner Gren International Series, 67-78. Retrieved from http://www.portlandpress.com/pp/books/online/fyos/083/0067/0830067.pdf

Kuncel, N. R., Wee, S., Serafin, L., & Hazlett, S. A. (2010). The validity of Graduate Record Examination for master´s and doctoral programs: A meta-analytic investigation. Educational and Psychological Measurement, 70(2), 340-353. doi: 10.1177/0013164409344508

Kuncel, N. R., Credé, M., & Thomas, L. L. (2007). A Meta-Analysis of the predictive validity of the Graduate Management Admission Test (GMAT) and Undergraduate Grade Point Average (UGPA) for graduate student academic performance. Academy of Management Learning & Education, 6(1), 51-68. doi: 10.5465/AMLE.2007.24401702

McGrath, C. H., Henham, M. L., Corbett, A., Durazzi, N., Frearson, M., Janta, B., ...Schweppenstedde, D. (2014). Higher education entrance qualifications and exams in Europe. POLICY DEPARTMENT B: STRUCTURAL AND COHESION POLICIES. Retrieved from http://www.europarl.europa.eu/RegData/etudes/etudes/join/2014/529057/IPOL-CULT_ET(2014)529057_EN.pdf

Organisation for Economic Cooperation and Development (OECD). (2008). Handbook on constructing composite indicators: Methodology and user guide. publishing. France: Author. Retrieved from http://www.oecd.org/std/leading-indicators/42495745.pdf

Paracchini, M. L., Pacini, C., Calvo, S., & Vogt, J. (2007). Weighting and aggregation of indicators for sustainability impact assessment in the SENSOR context. In K. Helming, M. Pérez-Soba, & P. Tabbush (Eds.), Sustainability impact assessment of land changes (pp. 349-372). New York: Springer Berlin Heidelberg.

Pavić, Z., & Novoselac, V. (2013). Notes on TOPSIS method. International Journal of Research in Engineering and Science, 1(2), 5-12. Retrieved from http://www.ijres.org/papers/v1-i2/B120512.pdf

Podvezko, V. (2011). The comparative anlaysis of MCDA methods SAW and COPRAS. Inzinerine Ekonomika-Engineering Economics, 22(2), 134-146. doi: 10.5755/j01.ee.22.2.310

Saaty, T. L., & Vargas, L. G. (2012). Models, methods, concepts & applications of the analytic hierarchy process. USA: Springer Science & Business Media. doi: 10.1007/978-1-4614-3597-6

San Cristóbal, J. R. (2012). Multi-criteria analysis in the renewable energy industry. New York: Springer London Dordrecht Heidelberg.

Santelices, M. V., Ugarte, J. J., Flotts, M. P., Radovic, D., Catalán, X., & Kyllonen, P. (2010). Medición de atributos no cognitivos para el sistema de admisión a la educación superior en Chile. Revista Iberoamericana de Evaluación Educativa, 3(2), 49-75. Retrieved from https://repositorio.uam.es/xmlui/bitstream/handle/10486/661612/RIEE_3_2_3.pdf?sequence=1

Velásquez, M., & Hester, P. (2013). An analysis of multi-criteria decision making methods. International Journal of Operation Research, 10(2), 56-66. Retrieved from http://www.orstw.org.tw/ijor/vol10no2/ijor_vol10_no2_p56_p66.pdf

Wang, X. C., & Xiao, X. Y. (2011). On weights determination in ideal point multiattribute decision-making model. In S. W. Wei, S. Xu, Ch. Wuan, Y. Zhong, W. Wojtkowski, G. Wojtkowski, & H. Linger (Eds.), Information Systems Development (pp. 475-481). New York: Springer

Figures:

Figure 1. Components of the model of selection for applicants for the Master's Degree in Forest Sciences at Universidad Autónoma Chapingo. JCR: Journal Citation Reports

Tables:

Table 1. Saaty and Vargas Scale (2012) to estimate the priority of the criteria used in the selection of postgraduate candidates.
Value Definition Comment
1 Equal importance Criterion 1 is just an important as criterion 2.
3 Moderate importance Experience and judgment slightly favor criterion 1 over 2.
5 Strong importance Experience and judgment strongly favor criterion 1 over 2.
7 Very strong importance Criterion 1 is more important than criterion 2.
9 Extremely important The greater importance of criterion 1 over criterion 2 is beyond doubt.
2, 4, 6 and 8 Intermediate values between the previous ones, when it is necessary to qualify.
Table 2. Grades obtained by the applicants for the Master program in Forest Science at Universidad Autónoma Chapingo.
Applicant Knowledge Experience Personality
EC EH EI Curr Average JCR Interv Psychom
1 85 93 0 78 80 60 65 80
2 90 60 9 82 80 90 90 100
3 70 75 50 84 85 85 70 100
4 85 48 25 85 88 80 89 80
5 85 78 17 86 89 70 79 80
6 80 73 26 82 86 50 73 80
7 90 63 71 78 82 65 73 100
8 85 53 79 83 81 92 69 80
9 70 78 39 84 86 74 68 66
10 85 70 99 82 85 88 88 66
11 70 63 70 80 86 90 90 80
12 75 73 100 88 90 60 83 80
13 90 88 50 80 84 76 76 100
14 80 58 100 84 88 75 84 80
15 85 78 91 80 81 88 92 80
16 90 80 57 86 84 40 85 80
17 80 65 82 82 82 70 89 80
18 80 73 97 90 95 78 78 80
19 80 50 100 84 87 90 75 80
EC = knowledge test, EH = skills test, EI = English test, Curr = curriculum, Average = average, JCR = review of article published in a journal recognized in the Journal Citation Reports, Interv = interview, Psicom = psychometric test.
Table 3. Weightings of the analytical hierarchical process (AHP) and the point allocation method (PAM) in the selection of applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo.
Criteria Sub-criteria Weighing AHP Weighing PAM
Criteria Sub-criteria Final Criteria Sub-criteria Final
Knowlegde Knowledge test 0.76 56.30 42.49 0.47 45.00 21.38
Skills test 30.70 23.16 31.25 14.84
English test 13.10 9.86 23.75 11.28
Experience Curriculum vitae 0.18 22.50 4.00 0 .25 35.00 8.75
Average grades 65.90 11.72 43.75 10.94
JCR exercise 11.70 2.08 21.25 5.31
Personality Cordination interview 0.07 86.30 5.78 0.27 73.75 20.28
Psychometric test 13.70 0.92 26.25 7.22
JCR: Journal Citation Reports
Table 4. Values obtained in the four combinations of weighting-aggregation methods used in the selection of applicants for the Master's Program in Forest Sciences of the Universidad Autónoma Chapingo.
Applicant TOPSIS-AHP TOPSIS-PAM WSM-AHP WSM-PAM
1 0.5323(48) 0.3670(4) 0.0511(36) 0.0465(0)
2 0.4030(18) 0.3740(6) 0.0491(17) 0.0499(29)
3 0.4539(30) 0.4814(29) 0.0494(20) 0.0505(33)
4 0.3239(0) 0.3625(3) 0.0473(0) 0.0487(18)
5 0.4973(40) 0.3873(9) 0.0515(39) 0.0497(27)
6 0.4337(25) 0.3482(0) 0.0491(17) 0.0473(7)
7 0.5831(60) 0.5871(53) 0.0539(62) 0.0530(54)
8 0.4953(40) 0.5544(45) 0.0516(40) 0.0517(43)
9 0.4405(27) 0.4095(14) 0.0485(12) 0.0475(9)
10 0.6894(85) 0.7530(89) 0.0572(93) 0.0575(92)
11 0.4307(25) 0.5856(52) 0.0495(21) 0.0528(52)
12 0.6196(68) 0.7234(83) 0.0550(73) 0.0562(80)
13 0.7098(89) 0.5954(54) 0.0570(92) 0.0549(70)
14 0.5541(53) 0.6773(72) 0.0538(62) 0.0556(75)
15 0.7556(100) 0.8022(100) 0.0579(100) 0.0585(100)
16 0.6900(85) 0.6103(58) 0.0561(83) 0.0539(62)
17 0.5640(56) 0.6701(71) 0.0532(56) 0.0546(67)
18 0.6727(81) 0.7342(85) 0.0567(89) 0.0571(88)
19 0.5031(42) 0.6190(60) 0.0522(47) 0.0540(62)
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method. Values in parentheses converted to scale 0-100.
Table 5. Selection of the best candidates for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo, according to a combination of weighting-aggregation methods.
Classification Applicant/method
TOPSIS-AHP TOPSIS-PAM WSM-AHP WSM-PAM
1 15 15 15 15
2 13 10 10 10
3 16 18 13 18
4 10 12 18 12
5 18 - 16 -
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method.
Table 6. Total distances and averages, according to the weighting-aggregation method, for the identification of the combination most compatible with the criteria evaluated in the selection of applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo.
Combination-Methods Total distance Averge distance
TOPSIS-AHP 548 182.66
TOPSIS-PAM 428 142.66
WSM-AHP 336 112
WSM-PAM 432 144
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method.