I have been tentative to add my two cents here, because I have never swum your speeds and only coached one person who reached your speeds. But here are a couple things I believe relevant.
In conversations I have had with Coach Kredich of University of Tennessee, I learned that some types of training, specifically training the nervous system to failure (his example set was: 1x100 from the blocks, 2x25 at 100 Tempo, 2x50 Fast...2 Times the goal is to hold your goal race 100 for all of it) can take as much as 6 weeks to recover the nervous system. The muscles recovered long before that, but the nervous system recovers more slowly. So they did other short sprints and a lot of holding technique as tempo increases, but this was considered really hard and stopped 6 weeks out from conferences. This, I believe, fits in with what Terry was saying about the need to rest the nervous system.
Second, I saw an interview with, I believe it was Bill Boomer, about training cats and dogs. Some swimmers, even sprinters, do really well on very long training days (15K a day type sessions). Boomer called these athletes the dogs. They just need to keep going. Janet Evans, while not a sprinter, was a classic dog in his mind. Some, need very limited, very quality training. These he called cats. Gary Hall, Jr was his example there. The point is that every body is different. Our job as athletes is to find where our body fits in. So when you asked about too much aerobic work, the answer cannot be generalized. It sounds like it is too much for your body. Many of the USA elite coaches say the best strength of USA swimming is the lack of unity in our approach. If you are a cat, you can find a cat coach. If you are a dog, you can find a dog coach. Most of us are in between and most coaches are in between. I believe you find the best success when you are aware of your body and fit your coach to your physical and mental needs.
Finally, Terry has written a few times that the racing strategy that worked for his athletes at the Army Sprint group was 25% set your stroke with as little effort as possible, 50% build, 25% hold on. This seems like it might fit you because it helps you finish well. I have had success using this strategy (although success for me means 1:10/100yds).
I spend my practice sessions doing just that, practicing for each phase. I spend time trying to maximize my top speed for the final 25%. I spend time trying to maximize the speed I can hold and minimize my effort as I set my stroke. I try to get the easy speed as close to my max as possible. Then I work on building from one to the other. Every swim in practice is trying to match some portion of my race. I don't believe doing 20 x 100s on 1:10 matches any portion of your race.
I am spending a lot of time thinking right now about the program that I will set up when I get back into team coaching. I appreciate your post as it gave me a chance to think about what it takes to swim much faster than I ever will.