Robert Ford, surely.

https://youtu.be/DYBE-F8FIfk

I think you're right about Robert Ford. And all these years I thought it was Frost, who did have a memorable face.

It seems I'm not the only one who heard Robert Frost.

Here's the blogger Ann Althouse, in 2018:

"For half a century, I've been listening to that song and hearing 'I might look like Robert Frost.'"

From a Jesse James Archive in Wikipedia: "I thought that the line Dylan (in Outlaw Blues) said 'I might look like Robert Frost.'"

Robert Pattison in "The Triumph of Vulgarity," his book on rock music, writes "...Bob Dylan, one of whose lyrical characters sings, 'I might look like Robert Frost...'"

I still hear Frost when I listen, but what made change my mind and deny the evidence of my senses is the story that Ford shot James in the back while he, James, was hanging a picture on the wall. But that may be too pat an explanation. Frost is more in keeping with Dylan's subversive sense of humor.