"breathing heavily the entire way"
I am not sure what your original problems were, but I bet this is holding you back a lot. Your excellent aerobic capacity is allowing you to do this, but it points out that something is wrong. I am mildly asthmatic and in mediocre aerobic shape so I can't get away with poor breathing. The great thing about breathing, is that most of the time if you fix your breathing, your breathing will be fixed.
Relax especially in your chest. Don't force the breath in or out excessively. Still breathing hard? Relax even more. Etc. Strive to breathe as if you were going on a moderate walk. Sure, your time might temporarily suffer. But you can't move forward without fixing this.
Actually, your times aren't really that bad. I bet you would enjoy the whole experience more if your breathing were well controlled. And you might find yourself able to concentrate on the other things that are holding you back. (For example, increased breathing effort usually causes you to drop your hips or your shoulders and wreck your streamlining) Remember, as Terry has said, 90% (or more) of the issue is streamlining.