Well, more than 1 minute over 500m sounds quite a lot. It means more than 12s per 100m on average, huge. And we are talking about a relaxed flutter kick, not an all out sprint kick, right?
I should wear a pair of long fins to gain more than 1 minute over 500m. More likely your flutter kick masks some balance issues.
Are you faster with a pull buoy than you are with 2bk? If so, that's the confirm.
Hope that helps,