In "Precision-Recall-Gain Curves: PR Analysis Done Right" by Flach and Kull (2015), they argue that the standard way of measuring the area under the precion-recall curve (the AUPRC metric) is flawed. They show that this metric does not have the same properties as the area under an ROC curve. In particular, the scores calculated for random models shifts as you change class balance, straight lines between two points are not valid because the scale is curved, and the AUPRC is hard to interpret.

Let $\pi$ be the fraction of positives (or class balance). Their fix is to move the origin to $(\pi ,\pi )$ and stretch the axes:

$\text{precision\_gain}=\frac{\text{precision}-\pi }{(1-\pi )\text{precision}}$
$\text{recall\_gain}=\frac{\text{recall}-\pi }{(1-\pi )\text{recall}}$

Now, random models score zero, lines between points are straight, and the Pareto frontier of models is a convex hull.

Intuition on the origin shift

Let $\pi$ be the fraction of real positives in the data. A "coin-flip" model that guess positive with chance $\pi$ has a precision score of $\pi$ and recall of $\pi$ at the point where its threshold keeps only $\pi$ of the items. Also, a model which always predicts positive also has a precision score of $\pi$. Therefore, the authors argue that $\pi$ is the "natural zero": the score you get for no skill beyond chance.

So the performance score of a random baseline depends on the class distribution, which is not good. If you want to measure your "gain" over random chance, you should put that chance point at the origin. That's what subtracting $\pi$ in the numerator does.

Intuition on the stretching

The distances in the the raw PR curve are deceptive. Moving from 0.02 to 0.04 precision is a 100% lift when $\pi$ = 0.01, but the same absolute jump from 0.52 to 0.54 is tiny. So we want a scale where "halfway to perfect" means the same everywhere: a model with a score of 0.5 should have the same relative ability whether the data are imbalanced or not.

Vanilla precision spans $\pi$ (random chance) to 1, so the middle absolute value can mean wildy different things as you vary $\pi$. Rescaling this so that the minimum is $\pi$.

However, the paper explains that this rescaling needs to be done in the harmonic scale (not linear scale) which is why the equation is a bit different. In essence, the vanilla AUPRC metric is taking the arithmetic mean of the precision scores, which is not appropriate in the linear coordinate system.

Calculating precision-recall-gain in Python

There's official implementation of PRG in Matlab, R, and Python here, but the Python implementation was very broken. Someone had opened a pull request to Sklearn, but unfortunately it looks like it won't get merged. So, I copied their implementation into a stand-alone Python library precision-recall-gain. You can find it here: https://github.com/crypdick/precision-recall-gain.