Julie: Clive's point about "coherence of image and argument" struck a chord with me, since on more than one occasion (one of them last week!) I've unsuccessfully attempted to Shanghai just part of an allusion to serve my purposes. Sadly, in each case I've ended up with the full allusion, and all its unwanted baggage, along for the ride. - Allusion first - sometimes a poet will use a word/symbol that has a strong meaning that they don't want the reader to dwell on, and a weak meaning that they do. Even though they support the weaker meaning in the rest of the poem, the other meaning interferes. Were there an alternative word that didn't have this problem, maybe the poet would use it. But readers have choices too - if the stronger meaning doesn't fit, they can (maybe should) forget about it.
And as you point out, audiences vary in what they consider strong and weak meanings. Comedians and politicians fall foul of this as well as poets. Allusions are a risky business, and you can't please everyone all the time - I don't even think one should try to. See
Allusions, FWIW.
As for coherence - well again it depends on the market and the risks you're prepared to take. My stuff usually has a centre (or 2) because I usually play a percentage game.
Problems and poetics of the nonaristotelian novel goes into this a bit, looking into the history of judging the quality of a text by assessing its unity.