For me, it was more about having to memorize things. I hated that. Once I figured out the difference between Math and Arithmetic, I realized I only hated doing arithmetic. That discovery started with my discovery of a little book about polynomials in the town library (wish I could remember the title).

Agreed that time pressure can destroy the pleasure in anything. Back when I was professionally writing software, I used to describe it as "The most fun job in the world turned into misery through the application of time pressure."