I think a reasonable transform to apply would be the odds ratio, p / (1 - p), where p is in this case the probability of a correct answer. To use some sample numbers from Gary's post, 54% correct gives on odds ratio of ~1.17; 85% gives ~5.7, which is more than 4x better; but then 87% gives ~6.7, which is only a silght further improvement…
I think a reasonable transform to apply would be the odds ratio, p / (1 - p), where p is in this case the probability of a correct answer. To use some sample numbers from Gary's post, 54% correct gives on odds ratio of ~1.17; 85% gives ~5.7, which is more than 4x better; but then 87% gives ~6.7, which is only a silght further improvement. But if we got to, say, 93%, the odds ratio would be ~13.3 -- almost another doubling. I think it would be fair to say that 93% is about twice as good as 87%.
Yeah, I was thinking of error rate reduction, which could go on halving forever, very similarly to odds ratio doubling - basically the dynamics you would expect from logistic growth which rises exponentially initially but then turns to inverse-exponentially approaching an asymptote as resource constraints set in. But still, I don't think the hype guy who said the vague phrase "doubling of capabilities" had any such rigorous definition in mind.
I think a reasonable transform to apply would be the odds ratio, p / (1 - p), where p is in this case the probability of a correct answer. To use some sample numbers from Gary's post, 54% correct gives on odds ratio of ~1.17; 85% gives ~5.7, which is more than 4x better; but then 87% gives ~6.7, which is only a silght further improvement. But if we got to, say, 93%, the odds ratio would be ~13.3 -- almost another doubling. I think it would be fair to say that 93% is about twice as good as 87%.
Yeah, I was thinking of error rate reduction, which could go on halving forever, very similarly to odds ratio doubling - basically the dynamics you would expect from logistic growth which rises exponentially initially but then turns to inverse-exponentially approaching an asymptote as resource constraints set in. But still, I don't think the hype guy who said the vague phrase "doubling of capabilities" had any such rigorous definition in mind.