This is because the phase angles are almost 90 degrees apart between the voltage and the current. Since the power is the product of these two almost 90 degrees out of phase sinusoidal waveforms - any phase angle error will result in a much higher error than two sinusoidal signals that are shifted by a much smaller angle. Therefore the phase angle errors in all the instrumentation (Voltage and Current transformers) will contribute far more to errors for low power factors.
If you want to visualize this - multiply two sinusoids in Excel or another spreadsheet for maths package (Matlab) and look at the Power Product for a range of phase angles. It will be well illustrated like this. Of course if the waveforms have harmonics - the situation is further complicated.
The low power factor instantaneous power has a large positive AND negative component - with only a small difference or net power flow towards the load. That component is the measured real power.
The high power factor instantaneous power has almost all positive power flow, making it much easier - and more accurate - to determine the net flow in the positive direction.
The real power is the difference between the area under the power trace above and below the zero axis. For zero power factor - the areas are equal. For very low power factor - they are close to equal with only a small difference between them.
The error is proportional to the ratio of the total instantaneous power to the DIFFERENCE between the positive and negative power. Quite obviously - there is much higher error for the low power factors.
One does of course need to visualize these instantaneous power traces to appreciate this. Excel does it easily, but other methods can be used. Some texts illustrate this but I don't have any at hand.