Thanks, that last comment may be the key to the issue. Could you double check your betaflight issues to check if you have dynamic idle set, and if so what the values are, as increasing this may help to give the motors more authority to correct itself when your throttle is low:
For more info, please see:
Otherwise let me know when you can get the logs so we can look into that abit more.