ToothyWikiInternals/BackLinks

All the important stuff is basically done. Things seem happy and performing reasonably well. People please tell me if I've broken anything, as always.

For people who want the source:

[Search.pm] - indexing back end, maintains its own database, provides query, update and delete functionality; database implementation hidden from wiki, so changing the implementation the next time the wiki outgrows the current method should be rather easier

[searchtest.pl] - commandline test module, hacky, contains hunks of raw and bleeding wiki code, used to manually query and regenerate the index (in particular, perl searchtest.pl reindex 1 causes every single wiki page to be parsed and the index generated from the results).

changes to wiki.pl - call update() with extracted links when a page is saved (implemented), call fetch() for a backlink search (implemented), call delete() when a MaintainMe/Delete is performed (todo). Will update the copy of the source that's on web when I get a RoundTuit. The changes are all trivial, anyway.

The whole thing appears to perform extremely well in terms of speed. The index for the current wiki is ~25Mb, ~3k is read on average per search, ~60k is read/written on average per update; this is all bearable (the wiki itself is ~144Mb).

AlexChurchill is very impressed with the speed of title-clicking now. One question: it seems that clicking the title of a single-word Wiki page will only turn up pages that explicitly link to that page (by double square brackets or suchlike). Thus, the trick of finding all pages mentioning the word "Japanese" by clicking the title of page Japanese won't work: there are 150 of the former, and only 57 of the latter. Pages like CUCAS (which mention Japanese without making it a link) aren't included in the latter group. Is this deliberate, or a known side-effect, or accidental?

Aargh. The example has now been destroyed. There are 92 others. Please let's not go thru editing each one thus forcing MoonShadow and me to find a different single-word wikiword... it's currently just serving as an explanatory example... --AC

: Well, it'd be nicer if we could automate the process ;) The point stands though - shouldn't we go and wikify everything (slowly, as and when a page is edited) - or perhaps it'd jus be better to avoid single word WikiWords? --Vitenka
: There is a different meaning to Japanese and Japanese. The latter returns almost all pages whose names are specifically Japanese phrases or refer to the language. The non-WikiWorded word refers to technology, people, language, food, or smilies. As for converting them all - consider the case of Go... ;) --AlexChurchill

TBH, I prefer it this way. Title-clicking gives you backlinks; typing things in the search box searches for that word. Before, there was no way of finding a definitive list of backlinks. "Go" is a very good example - would you really want all instances of the word "go" to turn up if you clicked on the title of the "Go" page? --M-A

: The backlink system only indexes links (duh). I haven't done anything about any other sort of indexing, and don't know what I'll do when I do yet. The system I'm currently using for backlinks won't scale to general-purpose word indexing (it'd be fine performace-wise, but not index-size-wise, unless someone cares to donate a ~100-120Gb hard drive). I hadn't considered the effect on single-word categories like Japanese - the old ability to backlink-search for them without explicitly making them into backlinks was a useful side-effect of the fact that the wiki didn't distinguish between different types of search in the past, and is not likely to remain; at best, I will improve the performance of the whole-word search. I would side with Vitenka on this and say that in the long term, things that should be visible in the backlink search for a page should be made into links to that page, as the term "backlink search" would, in fact, suggest. - MoonShadow

: Well, LongTerm? then, we should refactor the language and make Japanese and Japanese be different word in CommonUsage?. That may be slightly longer term than you want though. How about adding a link on the BackLink? SearchResults? page which would have the same effect as pasting the word into the search box and hitting search? --Vitenka

LOL! I thoroughly agree the EnglishLanguage needs a good Refactoring, altho I probably wouldn't start with "Japanese". I don't object to the new behaviour, I was just wondering whether this consequence was something you were aware of and had considered. I do agree that most references to Go (the game) should be double-square-bracketed, as should most references to logic or coffee. Japanese I'm not so sure about. Vitenka's link idea is a good one - it would also turn up times people have typed "webcomic" (uncapitalised) if clicked on from a backlink-search from page WebComic, and suchlike. --AC

Actually, thinking about it - it would probably be enough if text could carry intonation. Oh, and it might be better to just pre-fill the search box and let the user just hit the button rather than add another link. Though a bit of text to explain why they might want to (seemingly) run the same search again could help. --Vitenka

: Too late - already added the link :) - MoonShadow

As to the full search thing - doesn't perl already have a module to help with a SparseMatrix?? --Vitenka (sure, number of words * number of pages is a lot of bytes, but most of those bytes are zero)

: Can [you] see one? - MoonShadow, presently googling for "sparse matrix"

Ah, no. Sorry - the one I was thinking about was python. --Vitenka

: I *have*, however, just come across [SDBM_File]. Which looks pretty useful. - MoonShadow