Avoiding Spiders - Page 2

Rose Kelleher · #11 01-31-2009, 04:49 PM

I still don't understand why the critique forums can't simply be restricted to logged-in members only, which would solve the problem without all this fuss. Why on earth should people's in-progress drafts need to be made available to the general public?

Harrumph!

Golias · #12 01-31-2009, 10:33 PM

Drat! My mention of the title I intended to use for the test poem at TDE has brought the accursed spider to this thread again, though not to the Test thread. So the tag seems to work, so far, for the purpose intended, but as was pointed out by MaryAnn, we still have to watch out for repeating the true title elsewhere on the net.

G/W

Clive Watkins · #13 02-01-2009, 03:34 AM

Good point, Rose!

Clive

Stephen Collington · #14 02-01-2009, 04:05 AM

Aaargh. What Rose said, in spades.

The changeover to the new format is wonderful, but it has meant that all sorts of tricks that worked under the old system don't seem to work now. (And yes, Clive, the ROBOTS tags did actually work until the switch, even if in theory they weren't supposed to. That didn't mean that things didn't sometimes get snagged anyway. There were backdoor vulnerabilities. If someone used the "Print Thread" or "Quote" button, and left the resulting window open long enough when the robots were around, then the text would indeed be scooped--the only thing protected was the main thread page as seen on the board. And no, there were never any illusions on that score. All of this has been discussed, in detail, at various times over the last year.)

Anyway, bloody hell, nothing seems to work now. Check out, for example, December's "Deck the Halls" threads on Distinguished Guest, and look at what has happened to all of Sharon's beautiful decorative efforts. Gone. And worse, the ROBOTS tags which protected the texts are gone too. This means that people who submitted their work for the event on the understanding that it would be kept away from the spiders' eyes are out of luck. All is now wide open.

So, yep, there's a problem. And really, the only way to fix that problem is for Alex to insert ROBOTS tags into the <head> coding of the board pages themselves. That is, after all, how it's supposed to be done. Our quick'n'dirty cheat method may have worked in the past, but it really doesn't now. (Shaun, the META tags coding should be invisible; if you can see it on the "finished" posted page--like in your post above--that means it isn't working. And hiding it by colouring it white won't make a lick of difference to the search engines.)

Can we, collectively, come to some kind of agreement about this? The whole board doesn't have to be made invisible, but the workshop forums shouldn't be vulnerable to prying editors, and it should always be an option for anyone convening a special event (a Bakeoff, a Deck the Halls, etc.) to make its pages invisible too. In fact, it's even more vital for the special events; a workshop thread is eventually deleted, and when the search engines return after a few months for a refresher and fail to find it, they unlist the page and delete the cache. (Otherwise every other link on Google would be a 404 error eventually.) But special events threads are up to stay--and worse, the contributors have no editing access to their own material, since it is posted under someone else's name. Unless we can fix this problem, I can see it having a real chilling/killing effect on many of the events that make the Sphere such a lively place. If posting a poem here means rendering it ineligible for publication or prizes elsewhere down the road, then people will simply stop participating. The better the poem, the more the poet will hesitate to share it. That's got it exactly backwards, no?

Posting a poem on Eratosphere for critique or conversation is not publication. But if Google and Co. can find it and make it accessible across the Web with a simple keyword search, well, that won't make much difference, will it. Eratosphere is open to all, and that's a good thing. By all means, let's keep the main page available to the search engines. There, they can google up such useful keywords as: workshop, poetry, verse, critique, metrical, non-metrical, serious writing, translation, literary criticism etc. etc. etc. More than enough to advertise what we do here so that people looking for a great online workshop experience can find us. Everything else, the search engines can keep their noses out of.

Can we please fix this problem?

Cally Conan-Davies · #15 02-01-2009, 04:41 AM

Thank you for your excellent summary of the problem, Steve. I do hope something can be done about it. We will surely lose great poems to critique and from which to learn.

Cally

Clive Watkins · #16 02-01-2009, 05:07 AM

Indeed. Thanks for this handy analysis, Stephen.

I’d just add that, checking my files, I find that my failed experiment with the blocker-text to which I referred above dates from before Alex redesigned the site. That is to say, it did not seem to work – reliably, at least – even then. There were, as you say, “backdoor vulnerabilities”, though I never use (have never used) the Print Thread and Quote facilities.

As to posting my own stuff – that is, stuff I feel reasonably confident about – well, I post little anyway and send out less; but the vulnerability of the site to robots is certainly one factor that weighs with me somewhat. If Alex could institute some overall remedy, either along the lines you suggest or following Rose’s hint – or in some other way, I do think that would be advantageous.

Though there are apparently hundreds of members on the books, the number of active participants is relatively small. In my experience this has always been the case. Running the site as if it were a kind of reality-TV show open to the passing eyes of the world, may not be such a good idea.

Clive

Allen Tice · #17 02-01-2009, 09:54 AM

Stephen, thanks for an excellent wail! I've added emphases in the citations below.

Quote:

Originally Posted by Golias

... In the recent past I've had four poems declined, with apologies, because of their prior appearance online. G/W

Quote:

Originally Posted by Maryann Corbett

...it's also important that you not put the real title of the poem in the title field. If you do, the bots will find that.

If anyone else's post refers to the poem's real title without protecting it with tags, that too will mess things up.

Quote:

Originally Posted by Maryann Corbett

....And as an example of how long things stay in the cache, consider this. My poem "Rose Catalogue in January" was pruned at whatever time I got the January pruning to work correctly, at least a couple of weeks ago. It still shows up on a Google search.

Please pardon the emphasis below. It may be justified.

There are sure to be Eratosphere members who want to learn which other journals search the net and reject on the basis of Eratosphere trials. I know I am one of them. Would others share their experiences, or at least name the problem publications?

Rose Kelleher · #18 02-01-2009, 09:54 AM

There are plenty of bad spiders out there that ignore robot tags. The only foolproof way to keep them out is to require a login.

David Rosenthal · #19 02-01-2009, 10:19 AM

Quote:

Originally Posted by Stephen Collington

Check out, for example, December's "Deck the Halls" threads on Distinguished Guest, and look at what has happened to all of Sharon's beautiful decorative efforts. Gone. And worse, the ROBOTS tags which protected the texts are gone too. This means that people who submitted their work for the event on the understanding that it would be kept away from the spiders' eyes are out of luck. All is now wide open.

This is a little alarming. I just googled my poem and it was right there at the top of the list.

Quote:

Originally Posted by Stephen Collington

So, yep, there's a problem. And really, the only way to fix that problem is for Alex to insert ROBOTS tags into the <head> coding of the board pages themselves.

I vote for this and for making the workshop forums accessible only to registered members.

David R.

Roger Slater · #20 02-01-2009, 10:43 AM

As I independently suggested it on the other thread, I naturally agree with the idea of making the site invisible unless you are logged in. I repeat my suggestion that the home page provide "guest" log-in information for people who do not want to register, and do not want to post, but just want to check us out. That would keep us from losing potentially good members because they did not want to register before finding out more about us.

In the makeshift Erato in Exile site, I had a separate forum for people who wanted their poems to be invisible to those who were not logged in. The feature worked perfectly, near as I can tell. And as I mentioned in the other thread, I am a member of a very, very active site for the Society of Children's Book Writers & Illustrtors. It is busier site than Eratosphere for sure. But it's only visible to logged in members (paid membership, in fact), and it never shows up in search engines. In fact, children's book editors are notoriously fussy about prior publiction, yet I have seen manuscripts at the SCBWI site that later went on to be published as books. The invisibility option works. I strongly favor it.