MCEWRMulti-Criteria Evaluation Model for Water Resources

1. Weighted Average Method (WAM) audited

Family: additive value. Best alternative: the largest ranking value. References: [1].

The simplest and oldest idea: multiply how well an alternative does on each criterion by the importance of that criterion, and add. The alternative with the largest total wins. The method assumes full compensation: a poor value on one criterion can always be offset by good values elsewhere, in the proportion the weights dictate. That is reasonable for costs and benefits that trade against each other, and questionable for safety limits or irreversible losses, which is why the outranking methods below exist. WAM is the reference against which every other method is read.

Options in the Method step (step 6): none of its own.

2. Compromise Programming (CP) audited

Family: distance to the ideal. Best alternative: the smallest ranking value. References: [2] [3] [20].

Imagine the alternative that has the best value of every criterion at once – the ideal point. It usually does not exist. Compromise Programming measures how far each real alternative is from it and prefers the nearest. The ideal \(f_j^{*}\) and the anti-ideal \(f_{j}^{-}\) of each criterion are either taken from the alternatives present or, better, fixed in advance in the input file (the Upper and Lower limits): fixed reference points do not move when an alternative is added or removed, so the ranking cannot reverse. The exponent \(s\) says how much a single large shortfall should hurt: with \(s = 1\) shortfalls are added, with \(s = 2\) they are squared first, so one serious weakness costs more than several small ones.

The classical form takes the \(s\)-th root of \(L_s\); the 1995 program does not, and the root does not change the order. The switch CP: the s-th root of the distance adds it. With the limits fixed in advance and \(s = 1\), CP is term for term the SPOTIS method of Dezert et al. (2020) [20], whose ranking is free of rank reversal.

Options in the Method step (step 6): cp_power (1 or 2) and cp_use_provided_limits (use the Upper and Lower values of the file as the ideal and the anti-ideal).

3. PROMETHEE II (and PROMETHEE I) audited

Family: outranking (PROMETHEE family). Best alternative: the largest ranking value. References: [4].

Rather than scoring each alternative, PROMETHEE compares them two at a time. For each ordered pair \((a, b)\) and each criterion, a preference function \(P_j\) converts the difference \(x_j(a) - x_j(b)\) into a degree of preference between 0 and 1. Six functions are available: the usual criterion (any difference counts fully), the U-shape, V-shape, level, linear and Gaussian criteria, which need an indifference threshold \(q_j\) below which a difference is ignored and a preference threshold \(p_j\) above which the preference is complete. The weighted sum of the degrees is the preference index \(\pi(a,b)\); the flows measure how much an alternative dominates the others (leaving flow \(\phi^{+}\)) and is dominated by them (entering flow \(\phi^{-}\)). PROMETHEE II ranks by the net flow. PROMETHEE I keeps the two flows apart and reports, for each pair, preference, indifference or incomparability – the honest answer when one alternative dominates more while the other is dominated less.

The 1995 program divides the flows by \(m\); the modern convention divides by \(m - 1\) (the switch PROMETHEE flows divided by n − 1). The order is the same.

Options in the Method step (step 6): the preference function (1-6), p and q per criterion in the Criteria step (step 3); the switch promethee_partial in step 6.

4. ELECTRE II audited

Family: outranking (ELECTRE family). Best alternative: the smallest ranking value. References: [5] [6] [15].

ELECTRE also compares pairs, and adds a veto. For a pair \((a, b)\) the concordance \(C(a,b)\) is the share of the weight on which \(a\) is at least as good as \(b\); the discordance \(D(a,b)\) is the largest difference against \(a\), on the criteria where \(b\) is better, measured on the scale of the criterion. \(a\) outranks \(b\) when enough of the weight agrees and no single criterion objects too strongly. Nothing is added up: a very poor performance cannot be compensated, only vetoed. ELECTRE II builds two outranking graphs – a strong one with the severe levels \(p^{*}, q^{*}\) and a weak one with the milder \(p^{0}\) (or \(p^{-}\)) and \(q^{0}\) – ranks the alternatives in each, and takes the median of the two positions.

The five levels \(p^{*} \geq p^{0} \geq p^{-}\) and \(q^{*} \geq q^{0}\) belong to the problem and are read from the input file. If the strong relations form a cycle the graph cannot be ranked (error 201); raise the concordance levels, lower the discordance levels, or use ELECTRE III.

Options in the Method step (step 6): the five levels in the file's group line; electre_min_discordance_zero for unstandardized data.

5. Analytic Hierarchy Process (AHP) audited

Family: pairwise hierarchy. Best alternative: the largest ranking value. References: [7] [8] [19].

AHP is built for the case where nobody can put a number on a criterion but everybody can compare two things at a time. Items are compared in pairs on Saaty's 1-to-9 scale (1 equal, 3 slightly better, 5 clearly, 7 much, 9 extremely better; the reciprocal for the reverse judgement). The comparison matrix \(A\) is reciprocal with ones on the diagonal, and the priorities are its principal eigenvector, normalized to add to 1. The same procedure gives the weights of the criteria from pairwise comparisons between them, and the priorities are combined level by level through the hierarchy. A consistency ratio tells whether the judgements contradict one another; above 0.10 they should be revised. With ordinary measured values instead of judgements, AHP's aggregation reduces to a weighted sum.

\(RI_n\) is Saaty's random index for a matrix of size \(n\) (0.58 for \(n = 3\), 0.90 for 4, 1.12 for 5, 1.24 for 6, 1.32 for 7). The final priority of an alternative is \(\sum_j w_j \, p_j(a)\).

Options in the Method step (step 6): ahp_min_method: how a criterion to be minimized is inverted before the comparison.

6. Expected Value (EV) provisional

Family: ordinal weights and scores. Best alternative: the largest ranking value. References: [9] [10] [1].

Expected Value takes only the order of importance of the criteria seriously – criterion 1 matters more than criterion 2 – and replaces the unknown weights by their expected values over all weight vectors that respect that order. The result is the rank-order-centroid vector of Barron and Barrett [10]. The scores of the alternatives are treated the same way: only their rank under each criterion is used, replaced by its expected value. The two are combined by a weighted sum. Tied criteria (equal declared weights) share the average of the positions they occupy. The method is honest about ordinal knowledge: it neither invents numbers nor discards the order.

Options in the Method step (step 6): ev_weights and ev_scores: ordinal (the method proper) or cardinal (use the numbers as given; both cardinal reproduces WAM).

7. Regime Analysis (RA) provisional

Family: ordinal information. Best alternative: the largest ranking value. References: [11] [1].

Regime Analysis asks: given only the order of importance of the criteria, how confident can we be that \(a\) is better than \(b\)? For each pair it considers every weight vector consistent with the stated order, and measures the share of them for which the weight on \(a\)'s side exceeds the weight on \(b\)'s side. That share is read as the probability that \(a\) is preferred; the ranking value of an alternative is the average of its probabilities against the others. The result is a statement of robustness – \(a\) beats \(b\) under 96 % of the admissible weightings – rather than a bare order. The probabilities are estimated by sampling the admissible weight set; the seed and the number of draws are recorded so that a published number can be reproduced exactly.

\(\Omega\) is the set of weight vectors that respect the declared order of importance, with tied criteria treated as equal.

Options in the Method step (step 6): ra_weights (ordinal or cardinal), ra_samples, ra_seed.

8. TOPSIS provisional

Family: distance to the ideal and the anti-ideal. Best alternative: the largest ranking value. References: [12] [13].

TOPSIS – Technique for Order Preference by Similarity to the Ideal Solution – measures two distances: the best alternative should be near the ideal (the best value of every criterion) and far from the anti-ideal (the worst value of every criterion). Each column is first normalized by its Euclidean length and weighted; the ideal \(v_j^{+}\) and anti-ideal \(v_j^{-}\) are read from the weighted table; the closeness \(C\) balances the two distances between 0 and 1. There is nothing to tune, which explains the method's popularity, but both reference points are recomputed from whichever alternatives are present, so TOPSIS is the method most often caught by the rank-reversal test.

Options in the Method step (step 6): none of its own; with standardization 0 it is the textbook method on the raw data.

9. VIKOR provisional

Family: compromise ranking. Best alternative: the smallest ranking value. References: [13].

VIKOR looks for a compromise: an alternative that is good for the group of criteria as a whole and not badly unfair to any single one. It computes the total weighted shortfall \(S\) (the group utility) and the worst single shortfall \(R\) (the individual regret), and blends them into the index \(Q\) with a strategy weight \(v\) (0.5 by default; \(v = 1\) counts the total only, \(v = 0\) the worst criterion only). The smallest \(Q\) wins – but VIKOR then tests the winner: the acceptable advantage condition \(Q(a_2) - Q(a_1) \geq 1/(m - 1)\) and the acceptable stability condition (the first by \(Q\) must also be first by \(S\) or by \(R\)). If either fails, a set of compromise solutions is reported instead of a single winner.

Options in the Method step (step 6): vikor_v, between 0 and 1.

10. ELECTRE III provisional

Family: outranking with pseudo-criteria. Best alternative: the smallest ranking value. References: [14] [15].

ELECTRE III is ELECTRE with a memory of how imprecise measurements are. Each criterion carries an indifference threshold \(q_j\), below which a difference is noise, and a preference threshold \(p_j\), above which the preference is complete; between them the partial concordance \(c_j\) rises linearly. An optional veto threshold \(v_j\) lets a single criterion destroy the credibility of an outranking when the difference against \(a\) is too large. The comprehensive concordance and the discordances combine into a credibility \(\sigma(a,b)\) between 0 and 1. The ranking is obtained by two distillations – descending, which repeatedly extracts the best alternatives, and ascending, which extracts the worst – and the two positions are averaged; pairs on which the distillations disagree are reported as incomparable. With \(q = p = 0\) and no veto the credibility reduces to the plain concordance, the standard check of an implementation.

The distillations use the discrimination threshold \(s(\lambda) = \alpha + \beta \lambda\) with \(\alpha = 0.30\) and \(\beta = -0.15\) by default.

Options in the Method step (step 6): q, p and veto per criterion in the Criteria step (step 3); electre3_alpha, electre3_beta.

Given weights

The weights of the criteria table, divided by their sum within each group of sibling criteria (the switch divide by the group sum), so that every group sums to 1: \(w_j = W_j / \sum_k W_k\). With the switch off the numbers are used exactly as entered, which only makes sense when they already are shares.

Equal weights

Every evaluated criterion of a group receives \(w_j = 1/n\). Not a method but the neutral starting point of a study, and the natural companion of the sensitivity analysis: a ranking that changes as soon as the weights move away from equal deserves a second look.

Best-Worst Method (BWM), linear model provisional

The Best-Worst Method of Rezaei [24] asks the decision maker for far fewer comparisons than a full pairwise matrix – \(2n - 3\) instead of \(n(n-1)/2\) – and in a form that is easy to answer consistently. Within a group of criteria the decision maker names the best (most important) criterion \(B\) and the worst (least important) criterion \(W\), then gives on the 1–9 scale of Saaty how much more important the best is than every other criterion (the Best-to-Others vector \(a_{Bj}\)) and how much more important every criterion is than the worst (the Others-to-Worst vector \(a_{jW}\)); \(a_{BB} = a_{WW} = 1\), and \(a_{BW}\) appears in both vectors. Ideal weights would satisfy \(w_B / w_j = a_{Bj}\) and \(w_j / w_W = a_{jW}\) exactly; real judgements do not, so the weights are chosen to make the largest violation as small as possible. The original model minimizes the largest ratio error and may have several optimal weight vectors; the linear model of Rezaei [25] minimizes the largest difference error and has a unique optimum. MCEWR solves it as a linear program – \(n + 1\) unknowns, \(4n\) inequalities – with its own simplex routine (no third-party package, so that the same code runs in the browser and on the server):

The optimum \(\xi^{L*}\) is the largest violation of a comparison by the optimal weights: 0 when the judgements are fully consistent (\(a_{Bj}\, a_{jW} = a_{BW}\) for every \(j\), and then \(w_j = a_{jW} / \sum_k a_{kW}\) exactly), and the larger the more the judgements contradict one another. Two consistency indicators accompany the weights: the consistency ratio \(CR = \xi^{L*} / CI(n, a_{BW})\), where \(CI\) is the largest \(\xi^{L*}\) any comparison system with \(n\) criteria and that best-to-worst value can produce (the definition of Rezaei [24] with the index of the linear model derived by Ratandhara and Kumar [27]), so that \(CR = 0\) is perfect consistency and \(CR = 1\) the worst possible; and the input-based ratio of Liang, Brunelli and Rezaei [26], \(\max_j |a_{Bj} a_{jW} - a_{BW}| / (a_{BW}^2 - a_{BW})\), which needs no optimization and can be read before the weights are computed. Neither has a universal threshold; a ratio well above 0.1–0.2 is a signal to look at the judgements again. Judgements are given per group of sibling criteria (a hierarchy has one comparison system per group); groups without judgements keep their given weights.

Verification: the solver reproduces the five worked examples of Ratandhara and Kumar [27] (weights and \(\xi^{L*}\) to four decimals), their table of consistency indices, the closed-form optimum on random comparison systems, and an exact-arithmetic simplex. The judgements are saved in the project file (.mcewr) and travel with it.

Entropy weights

The share \(p_j(a)\) of each alternative in the total of criterion \(j\) is treated as a probability, and the entropy of Shannon [16] measures how evenly the values are spread. A criterion whose values differ much has low entropy and carries much information \(d_j\); the weights are proportional to the information.

CRITIC weights

CRITIC – Criteria Importance Through Intercriteria Correlation [17] – rewards a criterion that is dispersed (its contrast, the standard deviation \(\sigma_j\) of its column standardized to 0-1) and little correlated with the others (its conflict, the sum of \(1 - r_{jk}\) over the other criteria): it says something the others do not. Because every column is rescaled to 0-1 first, CRITIC responds to the shape of the spread, not to its physical size.

MEREC weights

MEREC – Method based on the Removal Effects of Criteria [18] – measures how much the overall performance of the alternatives changes when a criterion is removed. Values are normalized so that the best alternative gets the smallest number; the overall performance \(S(a)\) is a logarithmic aggregate; the effect of removing criterion \(j\) is the total absolute change of the performances, and the weights are proportional to those effects.

\(\tilde{x}_j(a) = \min_j / x_j(a)\) for a criterion to maximize and \(x_j(a) / \max_j\) for one to minimize.

The weight sweep and the critical share

The share \(t\) of one criterion is varied from 0 to 1 while the other criteria keep their proportions to one another, and the winner is recorded along the way. Where the winner changes, a bisection finds the exact share – the critical share \(t_j^{c}\) – and the relative change \(|t_j^{c} - w_j| / w_j\) it represents. The criterion with the smallest change is the weak point of the decision. For an additive method the crossing can be verified by hand, because the ranking value is linear in the weights.

The decision-value sweep and the critical value

The same question for the data: how much must one value \(x_{aj}\) of the decision matrix change before the winner changes? The value of the chosen alternative (by default the base winner) on each criterion is moved from \(x_{aj} - R_j\) to \(x_{aj} + R_j\), where \(R_j\) is the range of the criterion over the alternatives, the method is re-run at every step, and a bisection finds the nearest critical value \(x_{aj}^{c}\). The change is reported as a share of the range, \(|x_{aj}^{c} - x_{aj}| / R_j\), so that values near zero and values in different units can be compared; the cell with the smallest share is the most sensitive value of the study. Values given as Saaty comparisons are not varied. (One-at-a-time sensitivity of the performance values: Triantaphyllou and Sánchez [28].)

The settings of the method

One numeric setting – the standardization code, or an option of the method such as the VIKOR \(v\), the Compromise Programming power \(p\) or the ELECTRE III thresholds \(\alpha, \beta\) – is given several values and the method is re-run for each. For every value the winner, the whole order and Spearman's \(\rho\) of the positions against the base run are reported; a value the engine refuses (outside the option's range) is listed as not evaluated. It answers, for instance, whether the VIKOR compromise depends on the strategy weight \(v\), or whether the standardization alone changes the ranking.

Monte Carlo robustness: rank acceptability under random perturbation

Every value of the decision matrix, and if you wish every free weight, is multiplied by an independent factor \(1 + u\); \(N\) such samples are drawn, the alternatives are re-ranked each time, and the share of samples that put each alternative at each position is reported – the rank acceptability in the spirit of SMAA-2 [29] – with its mean rank, the share of samples in which it keeps its base position, and the stability of the base winner. \(u\) is drawn uniformly in \([-\delta, \delta]\), or from the symmetric triangular distribution on that interval with mode 0 when small errors are thought more likely than large ones. Weights are renormalized by the method as usual; values given as Saaty comparisons and weights given as Saaty priorities are not perturbed; a sample the engine refuses (for example a negative value under an objective weight scheme) is counted and reported. The random seed is recorded so that the table can be reproduced exactly.