What happens in current transformer core during CT saturation?

The core saturation is a physical phenomenon. It happens when the coupled magnetic flux is so intense that all magnetic domains on a ferromagnetic material are already aligned and thus does not respond to any further increase in the flux. The implications on the current transformer secondary current may be diverse. Having a saturated core does not mean the current on the current transformer secondary will be high or even constant as the flux increases.

Once saturated, the inductance of the circuit drops dramatically. Taking account other phenomena like hysteresis, the resultant waveform on the secondary circuit coupled by a saturated core is highly distorted and full of harmonics.

Depending on the level of saturation and the capability of the measuring device to process the distorted waveform (harmonics) the measured current on the secondary is much smaller than the corresponding RMS value present in the primary.

But things can be worse:
A basic principle of electromechanical conversion states that the output on the secondary is related to the variation of the coupled magnetic flux. It explains why a regular current transformer and transformers can't operate on DC. Although the DC current is not converted into secondary current, it does generates magnetic flux and thus contribute to core saturation. Theoretically a core fully saturated by a DC current has no output (DC current causes no variation of coupled magnetic flux through time).

Asymmetrical faults (like phase-phase) may generate DC components contributing to current transformer saturation (That´s where the 20x criteria on current transformer sizing came from).

Although also resulting in non-linear output, the saturation of a current transformer magnetic core has a different characteristic from other saturation phenomena observed on systems like audio amplifiers.

The bottom line is that the current waveform on the secondary of a saturated current transformer may be diverse depending on what components are present on the primary current and also on the burden wired to the secondary circuit of the current transformer. Current transformer saturation leads to protection devices mis-operation as the measured current does not correspond to the current on the system.