The resistor breaking in an AC motor variable frequency drive (VFD) consists of a switching power converter where the back EMF of the motor acts as the voltage source and the converter is controlled by the duty-cycle to produce a variable voltage across the breaking resistor. The voltage is increased to the breaking resistor by increasing the control duty-cycle and this produces an increasing armature current with a torque that resists the mechanical shaft torque resulting in breaking. The mechanical energy is converted to heat in the resistor. In some applications, an active front end rectifier can be used as an inverter to regenerate back into the AC power source (grid) and use this to act as the dynamic break. Since the energy is returned to the grid, it is typically more energy efficient.
Energy to be dissipated in electrical resistance braking shall be equal to momentum of the system or product of moment of inertia of the system and angular velocity. The power to be dissipated would depend upon the desired deceleration (i.e., time from full speed to stand still).
As far as I know there is no simple calculation of these designs as there are many aspects to the design of the power stage converter, power semiconductor devices and the switching controls, since you should consider many parameter, ex. motor characteristic, VFD characteristic, etc. Usually designer or manufacturer already calculate this thing and give recommended value for the resistor. If you in the condition doubting the resistor value from the manufacturer, you can check if it is suitable or not by checking the temperature in normal operation situation and make it sure that it's not reach the limit. But for some VFD suppliers give manual to choose the optimum value for certain variable frequency drive product.
Actually Siemens supplies the entire such system where input source is AC 3-phase and regenerative braking power is fed back to grid. It is very useful for saving substantial energy in application that requires large VFD and application calls for frequent braking.