tag:blogger.com,1999:blog-6905541471605961414.post2191665263655089749..comments2024-03-28T17:25:17.827+07:00Comments on Thai 101: RID online version 0.9: a closer lookRikkerhttp://www.blogger.com/profile/17196282287835224940noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-6905541471605961414.post-78829051790496179122008-02-09T21:02:00.000+07:002008-02-09T21:02:00.000+07:00Well I'd thought I'd better make sure it did pass ...Well I'd thought I'd better make sure it did pass them all before posting, I didn't think it would look too impressive otherwise! I'd be interested to know if you managed to catch it out using any other words though. There's a bit of a weakness in it for catching ร / ล mistakes, but that should be fixed by tomorrow. <BR/><BR/>The เหรียน typo is the most interesting example of the difficulties involved I think. Both เรียน and เหรียญ have a Levenshtein distance of one and เรียน is far and away the more common word, but yet there's no doubt someone typing เหรียน is much more likely to mean เหรียญ . The way I did is by converting to phonetic form first and ranking that higher than Levenshtein distance matches, but I'm sure there's other ways of going about it. Trying Google I noticed they get it right too, and I wonder what their method is. I doubt they do a phonetic conversion, instead my guess would be with their huge traffic they can track click counts on the 'Did you mean....?' suggestion links to get an idea of which is the most accurate when there's a couple of possibilities. That would have the additional advantage of working for every language rather than just Thai too.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-27548028867038172232008-02-09T16:03:00.000+07:002008-02-09T16:03:00.000+07:00Well, I'm embarrassed by my last comment. I was so...Well, I'm embarrassed by my last comment. I was so busy tripping over myself to play with your spelling checker that I didn't the part where you'd already tested it against my examples. What can I say, I think I might be <A HREF="http://en.wikipedia.org/wiki/A_moron_in_a_hurry" REL="nofollow">that guy</A> they're always referring to in legal cases... :PRikkerhttps://www.blogger.com/profile/17196282287835224940noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-65937150524638088932008-02-09T13:25:00.000+07:002008-02-09T13:25:00.000+07:00Mike, thanks for sharing the link. It scored 11/11...Mike, thanks for sharing the link. It scored 11/11 on my simple tests, compared to RID's 1/11. But then I have to assume you already knew it would. :)<BR/><BR/>As for Levenshtein distance, I'm familiar with the idea, though I'd forgotten the name. It comes in particularly handy for typos like ประเสิรฐ, where all the characters are right, there's just an error of placement. Among others typo types.Rikkerhttps://www.blogger.com/profile/17196282287835224940noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-60880392294156186982008-02-09T13:08:00.000+07:002008-02-09T13:08:00.000+07:00"I wonder how hard it would be to implement someth..."I wonder how hard it would be to implement something like Soundex for Thai."<BR/><BR/>I've actually been working on something similar to this myself, you can <A HREF="http://www.thai2english.com/beta/spelling/" REL="nofollow">try it out if you like</A>. You'll have to forgive the lack of design on the page though, as it was just a page I was using for testing how it works, not one I intended for public consumption.<BR/><BR/>It works by using a combination of phonetic matching and a <A HREF="http://en.wikipedia.org/wiki/Levenshtein_distance" REL="nofollow">fancy mathematical similarity rating</A> :). I'm still testing it out at the moment, but in its current state it gets all your examples above correct Rikker. With ปรอท , if you'd forgotten the exact spelling but knew the pronunciation was ปะหฺรอด, you could enter that in (with or without the pinthu) and it comes up with ปรอท as the first match.Unknownhttps://www.blogger.com/profile/15670708340521404404noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-28932737772835176192008-02-08T15:26:00.000+07:002008-02-08T15:26:00.000+07:00As Mike, or Glenn at Thai-Language.com, can attest...As Mike, or Glenn at Thai-Language.com, can attest, it's true what the adage says: the last 10% takes 90% of the effort.<BR/><BR/>I wonder how hard it would be to implement something like Soundex for Thai. A quick Google search shows I'm not the first person to think of this. See <A HREF="http://www.eric.ed.gov/ERICWebPortal/custom/portlets/recordDetails/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValue_0=EJ286911&ERICExtSearch_SearchType_0=no&accno=EJ286911" REL="nofollow">here </A>and <A HREF="http://citeseer.ist.psu.edu/350658.html" REL="nofollow">here </A> for mentions of Thai Soundex, though you can't access the actual articles.Rikkerhttps://www.blogger.com/profile/17196282287835224940noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-90655297432794467402008-02-08T15:16:00.000+07:002008-02-08T15:16:00.000+07:00"Mechanically converting Thai words into phonetic ..."Mechanically converting Thai words into phonetic forms is not difficult. "<BR/><BR/>I'd really have to differ on that one. It's not that difficult for about 80% of Thai words, but once you get into the remainder with the implicit vowels, silent letters and vowels, implicit 'linker' syllables and the like it gets a lot more difficult very quickly!Unknownhttps://www.blogger.com/profile/15670708340521404404noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-22812220401853921942008-02-08T14:27:00.000+07:002008-02-08T14:27:00.000+07:00ํYep, I'm with you on that one. And yet I prefer t...ํYep, I'm with you on that one. And yet I prefer the simplicity of this site to the equally clueless busy-ness of their main site.<BR/><BR/>The unfortunate situation we find ourselves in is that they've set the bar so low that this is a welcome improvement.<BR/><BR/>Before this fix, when you still got 40 "not found" messages drowning out your found results, was a truly awful low. To think it stayed that way for half of a year...Rikkerhttps://www.blogger.com/profile/17196282287835224940noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-4077239569754875102008-02-08T14:22:00.000+07:002008-02-08T14:22:00.000+07:00I have to admit, the site is still disappointing o...I have to admit, the site is still disappointing on the whole. Hello! 1995 called. They want their web developer back.<BR/><BR/>The webmasters can't even bother to set the content-type of the page, which encourages people to set their browser to use TIS-620 by default, which is the cause of so many "font" (actually character set) rendering problems in the greater Thai IT community.<BR/><BR/>Next, a simple search for "*" throws an error, indicating that the developer couldn't even be bothered to do any work at all. It's really awful and shameful. I would expect better from a high school student.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-48685188882829472302008-02-08T14:17:00.000+07:002008-02-08T14:17:00.000+07:00Not at all, not at all. It's just a good example o...Not at all, not at all. It's just a good example of words that one would want to find the full set of.<BR/><BR/>Also, phonetic conversion is one way of going about it, but it can be tricky. The old RID online contained the full phonetic for all words, not just words with ambiguous orthography, so they should have a good set of data to start from. <BR/><BR/>Another way of doing basically the same thing would be query expansion. Searching for, say, ประทาน would search for anything that matched ป[รล]ะ[ทธฒฑ]า[นณญลร]. You could define the set of characters based on whether it was in initial or final position. So น would expand to [นณ] syllable-initially, and to [นณญลร] syllable finally.<BR/><BR/>I'd like to know the specifics of what they're doing now. Can anyone reverse engineer it based on the results we get? Look at a search for ปรา, for example. It appears to do cluster simplification, add tone marks, and match all one-character-off words (e.g. ปราศ and เปรา).Rikkerhttps://www.blogger.com/profile/17196282287835224940noreply@blogger.comtag:blogger.com,1999:blog-6905541471605961414.post-39818396888362373232008-02-08T13:53:00.000+07:002008-02-08T13:53:00.000+07:00Thanks for the update, Rikker. Is the ใจ* example...Thanks for the update, Rikker. Is the ใจ* example a subtle jab at me indicating that I should have done more thorough research for my ใจ post last week?<BR/><BR/>I agree that many misspellings should be caught. Mechanically converting Thai words into phonetic forms is not difficult. Then you just compare the word given with all other words that match the phonetic form. In fact, that would be a great resource for people to look up words are unsure of. (In particular, foreigners and young students could benefit from that.)Anonymousnoreply@blogger.com