To derive the average bit error probability for this system, we need to calculate the probability of error for each possible transmitted symbol and then take the average.

Letâ€™s consider the case when the transmitted symbol is +sqrt(Eb). The received signal can be written as:

rn = 0.8*sn + 0.5*sn-1 + zn
= 0.8*sqrt(Eb) + 0.5*sqrt(Eb) + zn
= 1.3*sqrt(Eb) + zn

The decision threshold for the correlator receiver is 0. If the received signal is greater than 0, the receiver decides that the transmitted symbol is +sqrt(Eb), otherwise it decides that the transmitted symbol is -sqrt(Eb).

The probability of error for the case when the transmitted symbol is +sqrt(Eb) can be calculated as:

P(error|+sqrt(Eb)) = P(rn < 0 | +sqrt(Eb))
= P(1.3*sqrt(Eb) + zn < 0 | +sqrt(Eb))
= P(zn < -1.3*sqrt(Eb))

Since zn is a zero-mean complex Gaussian random variable with variance 0, its real and imaginary parts are independent and normally distributed with zero mean and variance 0. Therefore, P(zn < -1.3*sqrt(Eb)) can be calculated as:

P(zn < -1.3*sqrt(Eb)) = P(Re(zn) < -1.3*sqrt(Eb)) * P(Im(zn) < -1.3*sqrt(Eb))
= P(Re(zn) < -1.3*sqrt(Eb))^2
= (1/2)*[1 - erf(1.3*sqrt(Eb)/sqrt(2))]^2

Similarly, the probability of error for the case when the transmitted symbol is -sqrt(Eb) can be calculated as:

P(error|-sqrt(Eb)) = P(rn > 0 | -sqrt(Eb))
= P(1.3*sqrt(Eb) + zn > 0 | -sqrt(Eb))
= P(zn > -1.3*sqrt(Eb))

P(zn > -1.3*sqrt(Eb)) = P(Re(zn) > -1.3*sqrt(Eb)) * P(Im(zn) > -1.3*sqrt(Eb))
= P(Re(zn) > -1.3*sqrt(Eb))^2
= (1/2)*[1 - erf(1.3*sqrt(Eb)/sqrt(2))]^2

The average bit error probability can be calculated as the average of the two error probabilities:

P(error) = (1/2)*[P(error|+sqrt(Eb)) + P(error|-sqrt(Eb))]
= (1/2)*[(1/2)*[1 - erf(1.3*sqrt(Eb)/sqrt(2))]^2 + (1/2)*[1 - erf(1.3*sqrt(Eb)/sqrt(2))]^2]
= [1 - erf(1.3*sqrt(Eb)/sqrt(2))]^2

Therefore, the average bit error probability for this system as a function of Eb/N0 is:

P(error) = [1 - erf(1.3*sqrt(Eb/N0)/sqrt(2))]^2