That Sugar Town line bowled me over the first time I heard it, though it didn't really fade after that. The whole song is pretty amazing, one of the more genuinely moving songs he has written.
PS--
I generally prefer Dylan's versions to the covers, but
here's a cover that I might prefer to Dylan's, and it's in a more standard and smooth voice.