That's the way motors work, at low loads you're supplying the magnetizing current for the windings/iron combination. The power factor improves as the motor does some real work as it approaches synchronous speed. A variable frequency drive (VFD) is not a device that changes the design characteristics of a motor.
Assuming it is an induction motor and the VFD is used only for starting and the motor runs DOL once it is up to full speed. If it operates on the VFD, then the power factor seen by the network depends on the type of VFD. If it has a diode rectifier section, the network will see it as constant power factor, regardless of load.
The VFD output to the motor may exhibit low power if that's where you are measuring it, it will be the vector sum of both displacement and distortion power factors - however the power factor at the source side will be high as it is the VFD DC bus that supplies the motor magnetizing current, the supply does not 'see' this reactive component - so the power you are charged for will be high, hence low cost.
With a VFD we can control the speed of the motor. A VFD can sense the characteristics of a motor and then it can supply the required electrical energy to the motor. So that we can save electrical energy through the VFD, with soft starter we can't achieve this energy optimization feature and also with soft starter we can't control the motor by giving Analog signals, PLC signals & Modbus protocol and with other kind of communications. VFD can displays the actual current rating, voltage, speed, frequency, regenerative currents, and lot more including with acceleration and deceleration time settings.