It's not so much your kick is off, but imbalance, hips are very low - and why you swim easier with a pool buoy. Pull buoy lifts the back end like picking up a wheelbarrow from the handles. Thing of your legs as the handles and your lungs as wheel of wheelbarrow. We all pivot about our lungs. Excessive and/or poorly timed kicking can drive hips even lower too.
Learning balance and core stability, making your legs and hips light without the aid of artificial buoyancy (pull buoy, wetsuit, neoprene pants) is your #1 priority.
Ditch the pull buoy, it's only masking a balance issue you need to address.