Calculating IPAC values, and consequently individual marks, with the IPAC tool can be done through a variety of methods, most of which including normalisation. Users can choose between methods that offer IPAC values in the form of a % versus normalised values around 1. In addition, bias correction is offered which is a means of softening, if not entirely removing, the effects of bias in students' ratings of their peers. Whilst this may result is some marks exceeding the maximum mark available, this will be capped in the system so a student cannot gain over 100% on a score. There are other methods included that allow the tool to provide a robust selection of methods to determine the students' IPAC values. Since there will be preferences on which factors are more important, the tool will provide a range of selections from which the tutor can choose their desired normalisation method. This can be done via the Settings page (see Menu > Data analysis with IPAC software > Settings ).

Ratings Table for a Single Criteria

In the case that self-assessment is not included, these calculations will omit the ratings where i = j.

For multiple criteria, this 2-dimensional matrix will become 3-dimensional and all equations involving sums will involve accounting for the inclusion of other categories.

The row shows the marks given by the student.

The column shows the marks received by that student.

Definitions

r _i,j = rating giving by student i to student j

N = number of students in a group

CW = criteria weighting

CW _percentage = % weight of a single criterion

B = bias factor

M = bias factor inverse

x* = bias corrected variable

IPAC = individual peer assessed contribution value

SD = standard deviation of a rater's received ratings

IAF = individual agreement factor

α = parameter for scaling parabolic mapping (recommended α = 1.2)

Simple Percentage

The simple percentage method involves taking the sum of a student's received ratings and dividing that by the total possible rating. The user will be required to define the number of assessment levels' range of the rating scale utilised.

The IPAC value for student j can be calculated as shown:

$$\text{IPAC}_{j} = \frac{\text{sum of marks received}}{\text{maximum possible marks for all values received}} $$

Simple Percentage + Bias Correction

The bias corrected simple percentage method involves first correcting the students' ratings by examining the degree of leniency of each rater. If a student is overly generous and gives out higher ratings than the average received rating in the group, then their outgoing ratings will be scaled down. The opposite is true for students who are too harsh or stingy; their outgoing ratings will be scaled up. Following the correction of ratings for bias, the percentages are then calculated by the sum of a student's received bias corrected ratings and dividing that by the total possible rating (Li, 2001). The user will be required to define the number of assessment levels range of the rating scale utilised.

The IPAC value for student j can be calculated as shown:

$$\text{IPAC}_{j} = \frac{\text{sum of bias corrected marks received}}{\text{maximum possible bias correct marks}} $$

for each mark

$$\text{bias corrected mark} = \text{original marks} * M$$ $$M = \frac{1}{B}$$ $$B_{\text{for student } i, \text{criteria } k} = \frac{\text{average mark given by } i \text{ for criteria } k * CW_{k}}{\text{average mark given by team for criteria } k \text{ for all values received}}$$ $$CW_{k} = \frac{CW_{\text{percentage}, k}}{\text{average } CW_{\text{percentage}}}$$

Normalised Linear

The normalised linear method involves defining the IPAC value as a sum of a student's received ratings divided by the average sum of received ratings in the group (Conway et al, 1993). It is the average of the average mark received for each criteria, multiplied by the group mark.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$\text{IPAC}_\text{j} = \text{average (normalised mark received for each criteria * CW}_\text{k})$$ $$\text{CW}_\text{k}=\frac{\text{CW}_\text{percentage,k}}{\text{average CW}_\text{percentage}}$$ $$\text{normalised mark}=\frac{\text{mark received}}{\text{average team mark for criteria for all values greater than 0}}$$

When using the Normalised Linear settings (or any of its variations below), the average IPAC score for each student group is equal to 1. This means that if one member of the team has contributed very little, meaning s/he has a low IPAC score, the scores for the rest of the team will be higher to show they had to compensate for this disengaged student. When using the IPAC software for data analysis, the tutor can choose between two options in this case. As an example, let's assume a group of 5 students, all working equally except for one student that only did 40% of what was the average contribution on the team. The tutor can choose among these two options within the IPAC software (see relevant documentation page for more details on setting and using the IPAC software):

• ‘Exclude students from their teams’ is disabled. In this case, the IPAC marks of the students would be 0.4, 1.15, 1.15, 1.15, 1.15. The tutor acknowledges that the students had to work more than their share to compensate for the weak student, and boosts their marks by giving them IPAC scores higher than 1. This is particularly useful when all the teams have the same number of students, so it compensates students in the groups where one of the members was not participative.
• ‘Exclude students from their teams’ is enabled. In this case, the IPAC scores will be calculated as if the disengaged student was not in the team at all, and the tutor can mark her/him using some other method. The IPAC software only considers the “working” members of the team for the calculations. In this case, the IPAC marks of the students would be 0, 1, 1, 1, 1.

Normalised Linear + Bias Correction

The normalised linear method with bias correction involves first correcting the students' ratings by examining the degree of leniency of each rater (the bias). If a student is overly generous and gives out higher ratings than the average received rating in the group, then their outgoing ratings will be scaled down. The opposite is true for students who are overly harsh or stingy; their outgoing ratings will be scaled up. Following this, the bias corrected ratings are then used to create individual contribution factors and from there calculate IPAC values.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$\text{IPAC}_\text{j} = \text{average (normalised mark received for each criteria * CW}_\text{k})$$ $$\text{CW}_\text{k}=\frac{\text{CW}_\text{percentage,k}}{\text{average CW}_\text{percentage}}$$ $$\text{normalised and biased corrected mark}=\frac{\text{bias corrected mark received}}{\text{bias corrected average team mark for criteria for all values received}}$$ $$\text{bias corrected mark} = \text{original mark} * M$$ $$M = \frac{1}{B}$$ $$B_{\text{for student } i, \text{criteria } k} = \frac{\text{average mark given by } i \text{ for criteria } k * CW_{k}}{\text{average mark given by team for criteria } k \text{ for all values received}}$$

Normalised Linear + Bias Correction with Agreement Scaling

The normalised linear with bias correction and agreement scaling involves first correcting the students' ratings by examining the degree of leniency of each rater (the bias). If a student is overly generous and gives out higher ratings than the average received rating in the group, then their outgoing ratings will be scaled down. The opposite is true for students who are overly harsh or stingy; their outgoing ratings will be scaled up. In addition to this, the IPAC values are then scaled by the degree to which the standard deviation of a student's received ratings compares to the maximum standard deviation of received ratings in the group (Neus, 2011).

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$\text{IPAC}_\text{j} = \text{(IAF * (normalised and biased corrected IPAC}_j \text{ - 1)) + 1 }$$ $$\text{IAF}=1-\frac{\text{average(SD for j using bias corrected and normalised marks * CW)}}{\text{max(average SD using bias corrected and normalised marks * CW)}}$$ $$\text{CW}_\text{k}=\frac{\text{CW}_\text{percentage,k}}{\text{average CW}_\text{percentage}}$$

Rescaled Normalised Linear

The rescaled normalised linear method involves applying an arbitrary scaling factor, p, that remaps the IPAC values along a straight line with a non-zero origin (meaning that a student with a hypothetical IPAC value of 0 now has a mark) and an altered gradient (Goldfinch, 2004). The user will be required to define the parameter, p, with increasing p resulting in a steeper gradient. This parameter should only be defined between 0 and 1, with 1 recovering the Normalised Linear method.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$\text{IPAC}_j = \text{(p * (normalised linear IPAC}_j \text{ - 1)) + 1}$$

Rescaled Normalised Linear + Bias Correction

The rescaled normalised linear with bias correction method involves first correcting the students' ratings by examining the degree of leniency of each rater (the bias). If a student is overly generous and gives out higher ratings than the average received rating in the group, then their outgoing ratings will be scaled down. The opposite is true for students who are too harsh or stingy; their outgoing ratings will be scaled up. Following this, an arbitrary scaling parameter, p, is used to remap the IPAC values onto a straight line with a non-zero origin (meaning that a student with a hypothetical IPAC value of 0 now has a mark) and an altered gradient. The user will be required to define the parameter, p, with increasing p resulting in a steeper gradient. This parameter should only be defined between 0 and 1, with 1 recovering thethe Normalised Linear + Bias Correction method.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$\text{IPAC}_j = \text{(p * (normalised and bias corrected linear IPAC}_j \text{ - 1)) + 1}$$

Normalised Linear-Parabolic

The normalised linear-parabolic involves defining the IPAC value as a sum of a student's received ratings divided by the average sum of received ratings in the group. Following that, IPAC values above 1 are mapped to a parabolic function with a limit at which the IPAC value is capped that can be modified by the user (Nepal, 2012). The parameter that controls the position of this limit is α and increasing α increases the cap.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$ \text{IPAC}_j = \begin{cases} \text{normalised linear IPAC}_j - \frac{(\text{normalised linear IPAC}_j \text{ - 1)} ^2}{2\beta},& \text{if normalised linear IPAC}_j > 1 \text{ and } 1 + \frac{\beta}{2} > \text{ normalised linear IPAC}_j\\ \text{normalised linear IPAC}_j, & \text{ if normalised linear IPAC}_j \leq 1 \\ 1 + \frac{\beta}{2}, & \text{if normalised linear IPAC}_j \geq 1 + \frac{\beta}{2} \end{cases}$$ $$\beta=\alpha(1-\frac{\text{group mark}}{100})$$

Normalised Linear-Parabolic + Bias Correction

The normalised linear-parabolic with bias correction method involves first correcting the students ratings by examining the degree of leniency of each rater (the bias). If a student is overly generous and gives out higher ratings than the average received rating in the group, then their outgoing ratings will be scaled down. The opposite is true for students who are too harsh or stingy; their outgoing ratings will be scaled up. Following this, the IPAC value is defined as the sum of a student's received ratings divided by the average sum of received ratings in the group. IPAC values above 1 are mapped to a parabolic function with a limit at which the IPAC value is capped that can be modified by the user. The parameter that controls the position of this limit is α. Increasing α increases the cap on the IPAC value.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$ \text{IPAC}_j = \begin{cases} \text{normalised and bias corrected linear IPAC}_j - \frac{(\text{normalised and bias corrected linear IPAC}_j \text{ - 1)} ^2}{2\beta},& \text{if normalised and bias corrected linear IPAC}_j > 1 \text{ and } 1 + \frac{\beta}{2} > \text{ normalised linear IPAC}_j\\ \text{normalised and bias corrected linear IPAC}_j, & \text{ if normalised and bias corrected linear IPAC}_j \leq 1 \\ 1 + \frac{\beta}{2}, & \text{if normalised and bias corrected linear IPAC}_j \geq 1 + \frac{\beta}{2} \end{cases}$$ $$\beta=\alpha(1-\frac{\text{group mark}}{100})$$

Normalised Linear-Parabolic + Bias Correction with Agreement Scaling (Spatar et al, 2015)

The normalised linear-parabolic with bias correction and agreement scaling involves first correcting the students' ratings by examining the degree of leniency of each rater (the bias). If a student is overly generous and gives out higher ratings than the average received rating in the group, then their outgoing ratings will be scaled down. The opposite is true for students who are too harsh or stingy; their outgoing ratings will be scaled up. Following this, the IPAC value is defined as the sum of a student's received ratings divided by the average sum of received ratings in the group. The IPAC values are then scaled by the degree to which the standard deviation of a student's received ratings compares to the maximum standard deviation of received ratings in the group. There is an additional factor of 2 used in the calculation of the SIAF in comparison to the IAF to ensure IPAC values do not result in marks greater than 100% when applied to the group mark. IPAC values above 1 are mapped to a parabolic function with a limit at which the IPAC value is capped that can be modified by the user. The parameter that controls the position of this limit is α. Increasing α increases the cap on the IPAC value.

$$\text{final mark} = \text{IPAC}_\text{j} * \text{group mark}$$ $$ \text{IPAC}_j = \begin{cases} \text{NBS IPAC}_j - \frac{(\text{NBS IPAC}_j \text{ - 1)} ^2}{2\beta^*},& \text{if NBS IPAC}_j > 1 \text{ and } 1 + \frac{\beta^*}{2} > \text{ NBS IPAC}_j\\ \text{NBS IPAC}_j, & \text{ if NBS IPAC}_j \leq 1 \\ 1 + \frac{\beta^*}{2}, & \text{if NBS IPAC}_j \geq 1 + \frac{\beta^*}{2} \end{cases}$$ $$\text{nb: NBS IPAC}_j \text{ = normalised and bias corrected linear with scaled agreement scaling IPAC}_j$$ $$\beta^*=\alpha^*(1-\frac{\text{group mark}}{100})$$

References

Goldfinch, J (1994) Further developments in peer assessment of group projects. Assessment and Evaluation in Higher Education 19(1): 29-35

Nepal K.P. (2012) An approach to assign individual marks from a team mark: the case of Australian grading system at universities. Assessment & Evaluation in Higher Education 37(5): 555-562

Neus J.L. (2011) Peer assessment accounting for student agreement. Assessment & Evaluation in Higher Education 36(3): 301-314

Spatar C.; Penna N.; Mills H.; et al. (2015) A robust approach for mapping group marks to individual marks using peer assessment. Assessment & Evaluation In Higher Education 40(3): 371-389