But is actually counterintuitive in the actual levels since it just means you'll jump into a pit or slide down the mountain
Not usually. It usually doesn't matter either way whether you do it or not. Sometimes it's better, sometimes it's not. But people will do it anyways because it's fun. Have you ever watched someone play SM64 for the first time? Even when all they have to do is run forward and drop into a cannon, they'll hop, dive and slide their way there, because it's fun. And sometimes doing those stunts actually is the most efficient way of progressing, it's just not always obvious/intended. For this reason, people will try when given the opportunity, and if it doesn't work they'll give up once they get bored and just progress the safe way. No biggie.
I don't care about self-imposed challenges in a game design discussion, especially speedrunning
If the game forced you to be a parkour god then only autists would like the game. People like having that freedom of movement, they don't like being told they have to be a speedtranny just to beat the game. A game is fun because you hit buttons and see your character do the thing you want them to do, not because the game gives you a star for overcoming a challenge. This is why fighting games aren't popular, you have to beat the shit out of the controller to just get your character to move and then you throw out a punch or kick you didn't mean to and get stunlocked. That's not fun. This is also why 99% of gamers play I Want To Be The Guy for 5 minutes and then go, "Heh, that's funny," and never touch it again.
That reason being the fact it was the only game a lot of autists had to play for a good chunk of their childhood
Probably not since the N64 wasn't that popular and there's a ton of popular games from the 90s that aren't popular speedrun games.
nothing in the game itself is built with that in mind
It doesn't matter.