In the end RTFM was the solution…. and a bit of simple algebra..

7.2 Polygon Antialiasing

Antialiasing the edges of filled polygons is similar to antialiasing points and lines. However, antialiasing polygons in color index mode isn’t practical since object intersections are more prevalent and you really need to use OpenGL blending to get decent results.

To enable polygon antialiasing call glEnable() with GL_POLYGON_SMOOTH. This causes pixels on the edges of the polygon to be assigned fractional alpha values based on their coverage. Also, if you want, you can supply a value for GL_POLYGON_SMOOTH_HINT.

In order to get the polygons blended correctly when they overlap, you need to sort the polygons in front to back order. Before rendering, disable depth testing, enable blending and set the blending factors to GL_SRC_ALPHA_SATURATE (source) and GL_ONE (destination). The final color will be the sum of the destination color and the scaled source color; the scale factor is the smaller of either the incoming source alpha value or one minus the destination alpha value. This means that for a pixel with a large alpha value, successive incoming pixels have little effect on the final color because one minus the destination alpha is almost zero.

Since the accumulated coverage is stored in the color buffer, destination alpha is required for this algorithm to work. Thus you must request a visual or pixel format with destination alpha. OpenGL does not require implementations to support a destination alpha buffer so visual selection may fail.

Ok so the cognitive leap I failed to make was the whole background thing. if you draw your background – all your final colors are blended with it – this is what I battled against tonight. To end up with the colors you want, you need draw your polygons in inverse order. Nearest first. Thats the counter intuitive part. The background gets drawn last. So if you want a non-black background you need to draw a Quad to the screen last. The reason is obvious if you think of the math of the blend function.

Source and destination scale factors are referred to as (S_R ,S_G ,S_B ,S_A) and (D_R ,D_G ,D_B ,D_A).

Source and destination color components are referred to as (R_S ,G_S ,B_S ,A_S) and (R_D ,G_D ,B_D ,A_D).

They are understood to have integer values between zero and (K_R ,K_G ,K_R ,K_A), where K_R = 2^mR 1, K_G = 2^mG 1, K_B = 2^mB 1, K_A = 2^mA 1 and (m_R ,m_G ,m_B ,m_A) is the number of red, green, blue, and alpha bit planes. This is a fancy way of saying 8 bits = maximum value of 256, 16 bits = maximum value of 65536 and so on. For our example below we can assume that k_? = 1.0 as it simplifies the math.

Colours are blended by combining the defined source and destination blend factors and source and destination pixels using the following equation.

R (d) = min(K_R,R_SS_R+R_Dd_R)

G (d) = min(K_G,G_SS_G+G_Dd_G)

B (d) = min(K_B,B_SS_B+B_Dd_B)

A (d) = min(K_A,A_SS_A+A_Dd_A)

A_S = Alpha source which is material opacity, ranging from 1.0 (K_A), representing complete opacity, to 0.0 (0), representing complete transparency.

A_D = Alpha destination

The essential components of the solution is as follows

glBlendFunc ( GL_SRC_ALPHA_SATURATE , GL_ONE );

 in the source scale factor signifies (S_R ,S_G ,S_B ,S_A) =  (i , i , i, 1)  where i = min (A_S , k_A – A_D) / k_A

in the destination scale factor signifies (D_R ,D_G ,D_B ,D_A) = (1 ,1 , 1, 1)

Assume a destination pixel starts with 0,0,0,0 (so A_d = 0)

We write 1,0,0,0 to the display at a pixel location

We can assume that K_A is 1

so

(R_S , G_S, B_S, A_S)  = (1,0,0,1)

(R_D ,G_D ,B_D ,A_D) = (0,0,0,0)

min (A_A , K_A – A_D) / K_A -> min (1 , 1 – 0) ->i = 1

so

(S_R ,S_G ,S_B ,S_A)  = (1, 1, 1, 1)

(D_R ,D_G ,D_B ,D_A) = (1 ,1 , 1, 1)

R (d) = min(1,1*1+0*1) = 1

G (d) = min(1,0*1+0*1) = 0

B (d) = min(1,0*1+0*1) = 0

A (d) = min(1,1*1+0*1) = 1

So we have written a 100% opaque red pixel

Now lets write a green over the top of that…

so

(R_S ,G_S ,B_S ,A_S)  = (0,1,0,1)

(R_D ,G_D ,B_D ,A_D) = (1,0,0,1)

min (A_S , K_A – A_d) / K_A -> min (1 , 1 – 1)/1 -> i = 0

(S_R ,S_G ,S_B ,S_A)  = (0, 0, 0, 1)

(D_R ,D_G ,D_B ,D_A) = (1 ,1 , 1, 1)

so

R (d) = min(1,0*0+1*1) = 1

G (d) = min(1,1*0+0*1) = 0

B (d) = min(1,0*0+0*1) = 0

A (d) = min(1,1*1+1*1) = 1

And thats still Red – not Green because of the destination alpha. As soon as a pixel gets to an alpha of 1 – its is no longer changed by subsequent writes. As long as you write the pixels closest first they will naturally clip. All the tessellated polygon artifact alphas will saturate instead of cancel out – so you will get no artifacts.

The explanation of why it works is all in the maths.

GL