Google

Ooo, paranoia! There are still other search engines. Also, lot of links are found in newsgroups, blogs and forums such as this one before they are ever indexed. Removing Google records isn't enough to remove information from the world - you have to stop word-of-mouth, too. Admittedly, word-of-mouth is much slower. And admittedly, censoring Google records would effectively hide information for a lot of the masses who can't be bothered to look anywhere else. But does it really matter, given the sort of [things] the masses tend to search for? - MoonShadow

PeterTaylor, amusing himself by scrolling through those logs, was startled to find that if one Googles CICCU one of his quotes pages is the 16th hit.

It has to be said that Google is very very big and much used.  But no, not total.  Still [El'Reg] does like to point out how it is very large, very much a central service - and totally opaque in operation.  At the moment, it seems google is a good thing, but the old adage about too much of a good thing does seem to apply.  --Vitenka

Also, its popularity is an effect of its practices, and to that extent it finds itself in a feedback loop. If it was found to be censoring interesting things (e.g. if interesting things were mentioned in newsgroups etc. yet remained conspicuously absent from Google indexes), Google's popularity would wane and people would look for and/or produce alternatives. Google itself is quite recent and became popular simply because its indices had fewer omissions than those of other search engines. - MoonShadow

That's wrong: even after Google was quite popular, often searching on altavista would produce results when Google knew nothing. Google's popularity was due to its mechanisms for separating wheat from chaff: its number of sites indexed actually rose quite slowly.

True 'nuff.  But who is going to set up a rival SearchEngine in this environment?  When the web haemorrages money and Google owns the space?  Also - we're not talking total censorship - just a chilling effect.  If google does list the page, but only on an incredibly specific search, or otherwise just ranks it very low - then few if any blogs will find it in the first place to link to it (using blogs since usenet is a wilderness) and so it just won't spread as fast as a competing idea.  The recent hijacking of the phrase 'SecondSuperpower' is a good example.  --Vitenka

It is usually due to a website admin's actions that a site gets indexed in the first place. Those actions could be to place the site URL in a blog or on slashdot instead. - MoonShadow

(PeterTaylor) Really? I didn't do anything to get my sites indexed.

No, but you did things that resulted in your site getting indexed, without which it would not have been. Which was part of my point. What you say you in fact did is actually a good example of the other part (i.e. you linked to it from two forums expecting the forum visitors to come to it, which is a wonderful demonstration of ways of drawing traffic to your site that aren't search engines! - MoonShadow

Well, Google has to find your site somehow - either you submit the URL directly, or a page that's already indexed links to you and Google follows the link.  I had to explain this quite carefully to my dad a few weeks ago. "But the pages are on the web now, won't SearchEngines just come across them anyway?" "No." --M-A

I said that I didn't do anything to get my sites indexed.  I did put two external links to them (one in my JDC forum profile, and one in my 5k competition profile).

... so Thing You Did was to link to them.  That counts.

I think we may be disagreeing on interpretation of "to get". I was interpreting it as "with the intent of getting". You seem to be interpreting it as "which could possibly result in getting".

Oh, fair enough.  Yes, you don't have to intend to get your site indexed, but nonetheless it was as a result of your actions that the sites got indexed.  That was all.

Except that the correct answer is yes.  It will take them a bit longer, because they'll have to come to it via someone elses link to you - but they will do it.  Assuming someone links to you.  But since your site is hosted somewhere, that host almost certainly does.  --Vitenka

Really? Why on earth would Demon link to my parents' humble homepage?  In fact, I don't know of any ISPs that link to their users' homepages... --M-A

SRCF.

MoonShadow, remind me - did freeparking link to us? I don't think EasyDSL? do. How did toothycat.net first get indexed on Google? - SunKitten

Freeparking (people who hold our DNS tags for us) don't link to us. Neither do EasyDSL?. Google first came to our site from a link on my old site on tripod.com. Google first indexed that site when I told it it existed; it had been completely unindexed until then.

Sections of toothycat.net are not linked from the main site or external sites, and have never been entered into Google. In particular, user directories other than [Tsunami's] exist. While robots are not actively banned from these sections, they do not exist in Google's directory, as a search reveals. In addition, a lot of sites and sections of sites that do ban robots and hence don't get indexed still get lots of traffic (typically sites generated by CGI scripts, like bulletin boards, large reference sites and shops). This shows that both the argument that you don't need to do anything for your site to get indexed (telling the URL to a friend who gets crawled so they can link to you counts as doing something) and the argument that removing a site's records from Google will effectively remove that site from the Internet are not universally true.

In which case this doesn't contradict what I'm trying to say. My points basically are that if you need to do something to get indexed that is outside your normal activities, then if Google start to censor you, you could do the thing that falls outside your normal activities in Slashdot or on Usenet instead of in the Google submission form, and draw people that way; whereas if you do *not* need to do anything outside your normal activities for Google to index you, Google censoring you won't wipe you off the face of the net since people can still find you the same way Google would have, and people would probably have been finding you for some while by that means before Google got around to indexing you in the first place. - MoonShadow

What do you mean by "direct hit", as opposed to any other kind of hit? About 80% of toothycat.net hits come from Google. However, over 70% of the visitors that go to more than one page do not come from search engines (yes, I can and do track individual visits to build statistics, and have raw logs starting from september last year). The people that stay tend to be visitors from friends' sites, visitors that followed the link in my or SunKitten's signature from a usenet posting I made, visitors from one of a number of bulletin boards some ToothyWikizens frequent and have linked to the wiki from, or visitors from one of several online  comics directories that PhoenixFeathers is listed in (http://www.onlinecomics.net seems to attract a very hardcore sort of visitor who read through the entire archive from the start! We get lots of hits from them and a large proportion do that.) Of the rest that stay, about 1 in 5 are searching for keywords such as "toothycat", "toothywiki" or my or SunKitten's real names, which implies they already know about our site through some channel other than Google; 1 in 3 are searching for very specific recipes and acronym definitions. The multiple political, religious etc rants on the wiki have attracted very few hits from Google, and (so far) none at all that went on to visit any other pages on the site. Thus I find that my experience suggests that Google has relatively low importance when it comes to "getting a message out to the Internet". ((all percentages rounded down to nearest 10%)- MoonShadow

Suggestion - fire a pre-emptive strike at anyone coming to the ScriptKiddie page that way :)

(PeterTaylor) I participate in some Java fora, as I expect most ToothyWikizens have gathered by now, and I have two stock answers which I use an awful lot: "You see that search box in the top-right corner?" and "Google is your friend". (Well, I have a third, which is to tell them to read the API documentation, but that's less relevant in this context). I really don't understand people who post a question to a forum and sit back for someone who knows the answer before even trying Google (and when you get the solution as first hit on the first search you try, you can be pretty confident they didn't try).

People do need to learn how to use google though.  It's not obvious which search terms are effective, or how to narrow (or, even more important and less obvious to learn; how to usefully broaden) a search.  Perhaps this will aid someone:  Use nouns often, verbs rarely - unless you are certain that that particular verb will have ben used in either the question or the answer.  Don't try to 'hint' with category type words - "computers programming web javascript html php counter script" will get you there less quickly than just "php counter script".  However, words that are associated with the solution help a lot.  "php counter script" will get you a lot of those counters on peoples pages, if you wish to write one then you could try "php counter script download".  Although, amusingly, I got a hit first tie without it (just tried, thatere's the toruble with contrived examples.)  "Howto" will get you linux documentation far more quickly than "linux documentation" will.  One other wonderful thing is googles ability to fix a broken download.  If you did some complex set of searches and finally found a page which claims to have the file that you are looking for, but it 404s - simply paste the filename into google and get a list of alternate download sites.  One is sure to work.  --Vitenka

(PeterTaylor) It also does complex numbers.

I can't make it do that... it'll do "4 in roman numerals" and it will evaluate that sum in hex or decimal... it does find this page when you search for it though -- Naath

Ooooo! Edith likes. This will come in useful with my literatutre review. It seems to have turned out a few papers WebOfScience? didn't and I'm impressed by the relevance ordering. I'll withhold final judgement until it screws me over horribly but for the moment: Shiney O_O

I don't think TragedyOfTheCommons? is quite the right term, but it's certainly an interesting problem - if it is happening.  I suspect that enough people are running link aggregator sites that it's not a problem yet, but I can see how it might be.  Maybe people will have to start manually submitting their sites to google again, and google will use more of its collected data to determine relevancy.  --Vitenka