Project Gutenberg Thailand: Beta Testers Wanted

[Read the PGT 2014 Update!]

[Update: Signup instructions and basic proofreading guidelines can now be found here. A big thank you to everyone who has shown interest in PGT.]

I'm looking for a few beta testers to help me work out kinks in a proofreading web application for Project Gutenberg Thailand.

I wrote a few months ago (see part 1 and part 2)  about my long-held goal to setup a website dedicated to digitizing and disseminating public domain Thai ebooks. This new site would be called Project Gutenberg Thailand (PGT), an independent sister site to Project Gutenberg. The focus of PGT would be both books in Thai and books about Thailand whose copyrights have expired.

At the time I didn't have a web developer to help me get the ball rolling. Shortly after making those posts, my friend Ben Crowder (who has actually helped me since the earliest planning stages of the project back in 2007) stepped up to the plate and created a web application for proofreading or typing old books. The goal was to keep it simple.

Right now I need two types of beta testers:

1) Thai typists (native or advanced Thai reading/typing skill).

[EDIT: I've now uploaded PGT's first book for Thai OCR proofreading, using a newer printing of the epic poem ขุนช้างขุนแผน Khun Chang Khun Phaen. Volunteers working on Thai can now choose between a typing project or a proofreading project (or do both).]

2) English proofreaders (native or advanced English reading skill).

Due to the limits of Thai OCR, the text of older Thai books needs to be typed by hand. OCR for English is quite good, however, and needs only to be proofread and corrected against the original page image.

You do not need a large amount of free time. That's the beauty of distributed typing/proofreading. You do a page here, a page there, as time allows. So if you have some free time and are interested in volunteering some time to support this project, contact me by email (rdockum at gmail) or on Twitter (@thai101). As a beta tester I'll expect you to report bugs if you encounter them, suggest features you wish to see added, and so forth. There is no commitment, it's strictly voluntary. You know, sabai-sabai.

Project Gutenberg Thailand: Some nitty gritty (and a sample Thai ebook)

[Read the PGT 2014 Update!]

In my previous post on Project Gutenberg Thailand, I avoided getting too much into the technical details. Here are more of my thoughts on the steps of the book digitization process.

Scanning:

I have many public domain Thai books in my personal collection, including works by the authors I mentioned in my previous post, as well as numerous literary works by Rama VI, books by Prince Damrong, and so forth.

Many older books have had recent printings, and so they are neither rare nor expensive. In such cases my preferred method for scanning is deconstruct the binding of the book and scan each page using a flatbed scanner. This produces the best quality image, with no shadow or distortion.

This is not always possible, of course, if the book is old or rare, or doesn't belong to me. For such cases I use an OpticBook scanner, which is a special book scanner that allows you to scan one page at a time without destroying the book or breaking its spine. It looks like this, by the way:

In my free time I've scanned several public domain Thai books, and have many more ready and waiting to be scanned.

Photocopying books, say from the library, also works, but scanning is preferable because OCR software works best with grayscale images of 300 dpi resolution or higher. A photocopier is black and white, and ultimately you have to scan the photocopies into a computer anyway. The main benefit of photocopying is that Thai libraries offer cheap photocopy services, so you can have a pro do the heavy lifting for you, so to speak.

In addition, there are excellent resources like the Thammasat Electronic Rare Books site, which have black and white PDF files of many old books. Not all of these are actually public domain, but sites like this provide another avenue for public domain source material that can be digitized as text.

OCR software:

To date the best OCR software for Thai is ABBYY FineReader Professional, either version 9 or 10. In fact, ABBYY is the only one that's any good. I have used it extensively.

A program called ArnThai was released a few years ago by Thailand's National Electronics and Computer Technology Center (NECTEC), but unfortunately ArnThai is rather terrible. The quality of the OCR is not very good, but also it has no batch processing of any kind, supports limited input and output filetypes, and has no mechanism for training characters at all. ABBYY FineReader, while not perfect, has sophisticated tools for all of these things.

OCR accuracy for Thai is well above 90%, but it still has problems. ABBYY has trouble with older typefaces, for instance. And even on newer books it still has some trouble accurately detecting all superscript and subscript characters, or differentiating very similar characters. This is par for the course. Since it has a training mechanism, though, a human can teach the software how to properly recognize difficult typefaces when needed, which greatly improves the quality.

Proofreading:

The basic mechanism for crowdsourced proofreading is to show the user the original page image side-by-side with the text output from the OCR software. (Or, if the text was manually typed, with that.) Here's an example taken from pgdp.net:

The user make corrections and submits the page when it is completed. The process that pgdp.net uses is sometimes overly complicated. Every page is checked about a dozen times, each time focusing on different things (basic text accuracy, formatting, etc.). They have the luxury of plentiful volunteers.

To start with, at least, Project Gutenberg Thailand will not have this luxury. The method I propose is like this:

Each page is proofread two times, once each by two different human proofreaders. They make their corrections and submit the page. Ideally, the two versions would be identical, but there will of course be some errors. To identify the errors, the two versions are then compared programmatically, to find discrepancies between them. Any place where they differ can be assumed to be an error made by one proofreader or the other. The points of discrepancy are highlighted and shown to a third person, who can quickly correct only the highlighted points, thereby fixing mistakes made by the original two proofreaders. At that point the error rate on the page will be extremely low.

Distribution:

Once books are turned into digital text, they can be formatted into the various ebook formats for reading on computer, cell phone, or ebook reader, as well as being available on the website as text and HTML files. I believe public domain works should be distributed far and wide, of course, so I would be pleased to see any output of Project Gutenberg Thailand also posted to Thai Wikisource, as well as any other website that wished to host it.

As a test, I took the electronic text of the short book Letters from Jangwangram จดหมายจางวางหร่ำ, a 1905 epistolary novel, and created an EPUB file. I then transferred it onto my iPhone and loaded it into Stanza, a free ebook reading app. I'm happy to say it is quite readable, even though the lack of word breaks in Thai creates some odd spacing. I also loaded it onto my Kindle 3, and though readable, the default font was not ideal, and there were similar spacing issues.

You can download this EPUB on your own devices if you'd like to test it out:

จดหมายจางวางหร่ำ
โดย น.ม.ส.
Letters from Jangwangram
by N.M.S. (pen name of Prince Bidyalankarana)
jotmai-jangwangram.epub

(Note: This ebook is not in Unicode. It uses HTML entities because some ebook formats are not Unicode compatible yet. I'm still learning the particulars of the several popular ebook formats.)

Project Gutenberg Thailand: Liberating public domain Thai literature

[Read the PGT 2014 Update!]

For a few years now it's been one of my goals--it's been so long I should probably say 'dreams'--to start Project Gutenberg Thailand, a repository for public domain literature in Thai and about Thailand. The founder of the original Project Gutenberg, Michael Hart, encourages such spin-off sites, and was enthusiastic when I contacted him about the idea back in 2007.

The closest thing that currently exists is Thai Wikisource, but it has little in the way of modern literature, instead having mostly selections of classical Thai verse and public domain government documents. As far as I know, there is no existing movement to identify and disseminate more recent public domain Thai works.

Owing to a number of reasons, however, not the least of which being my own lack of sufficient free time, nothing has ever gotten off the ground.

Last year, frustrated with my inability to make any headway on this project, I began compiling a list of Thai authors whose works are in the public domain. To put it simply, under Thai law a book is copyrighted until 50 years after the author's death.

With such a relatively short copyright term, the works of many well-known 20th century authors have entered the public domain. Unfortunately it too often seems to be those authors who died young. Yakob ยาขอบ, author of the immortal Conqueror of the Ten Directions ผู้ชนะสิบทิศ, died in 1956 at age 48. And two early novelists born in 1905 failed to reach middle age -- Prince Akartdamkoeng มจ.เจ้าอากาศดำเกิง penned such well-remembered tomes as The Circus of Life ละครแห่งชีวิต before killing himself at 25; while Mai Mueangdoem ไม้ เมืองเดิม, of The Old Wound แผลเก่า fame, met his fate at 37. Such authors are no less significant for modern Thai culture than the Fitzgeralds and Steinbecks of American culture, and while they can be still be found in print, it is a shame that such works aren't yet available as free ebooks.

I hear you asking, "So what needs to be done to set these works free?" (I have excellent hearing.) "And how can I help?"

Well, we need to put in place the process for taking the paper books and turning them into electronic text. The basic steps are as follows:

1. Get the book and scan each page, to create images.
2a. Use OCR [optical character recognition] software to turn the images into digital text.
OR
2b. Have humans type out the text contained in the images.
3. Have humans proofread the digital text by comparing it side-by-side with the original image.

This process is well-established for English. Project Gutenberg has a sister website, called Distributed Proofreaders (pgdp.net) that crowdsources this work for books in Latin-alphabet languages. OCR for English is extremely accurate. Until 2009, OCR for Thai was miserably poor, but nowadays it's rather good. In other words, the time is finally right to start digitizing books for Project Gutenberg Thailand.

With the original Project Gutenberg, anyone can sign up to help via the Distributed Proofreaders site. For Project Gutenberg Thailand, we must build a similar community of people willing to contribute a little bit of time here and there to help liberate public domain Thai books from their paper prisons.

So here I am, writing this blog post, hoping to drum up help to get the ball rolling.

The most immediate need is a sympathetic soul with web programming chops to help create the website for crowdsourced Thai proofreading and/or typing. I've put a fair amount of thought into the core features needed, but I lack the programming and design skills needed to make it a reality.

If you are interested in helping with this effort, come join the Google Group, let me know on Twitter at @thai101, or email me at rdockum [at] gmail [dot] com.