thanks for your feedback. For the record, I have tried to swim some lengths backstroke with the pull buoy, but I didn't notice any dramatic improvement in speed, I was only slightly faster. So I think balance is not my main problem.
Another funny thing to try (I did it today) is swimming backstroke with an ankle band. I'm already used to swim freestyle with an ankle band and I'm quite comfortable with it (can keep my body balanced and taut without much effort).
What I instead discovered by using the band in backstroke is that my body made a lot of zig zag in every stroke (especially in the 2nd half of the length, when the pushoff effect has vanished), it was far less stable than it is in freestyle. That tells me that maybe what holds me back more in backstroke is that I push water in wrong directions during the catch/pull/push path.
It can be a good drill to do, trying to keep the body as stable as possible.