[phpBB Debug] PHP Warning: in file [ROOT]/phpbb/session.php on line 590: sizeof(): Parameter must be an array or an object that implements Countable
[phpBB Debug] PHP Warning: in file [ROOT]/phpbb/session.php on line 646: sizeof(): Parameter must be an array or an object that implements Countable
[phpBB Debug] PHP Warning: in file [ROOT]/phpbb/session.php on line 1068: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3839)
[phpBB Debug] PHP Warning: in file [ROOT]/phpbb/session.php on line 1068: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3839)
[phpBB Debug] PHP Warning: in file [ROOT]/phpbb/session.php on line 1068: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3839)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 472: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 113: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5129: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3839)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5129: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3839)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5129: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3839)
Toki pona corpus project needs your help! - Toki Pona Forums

Toki pona corpus project needs your help!

Discuss any other topic in here.
Diskutu ĉiujn aliajn temojn ĉi tie.
janMato
Posts: 1545
Joined: Wed Dec 02, 2009 12:21 pm
Location: Takoma Park, MD
Contact:

Toki pona corpus project needs your help!

Postby janMato » Mon Feb 01, 2010 7:30 pm

I've been working on doing some corpus linguistics for toki pona. In the last hour, I've downloaded 220KB of long (as in paragraph or longer) texts of toki pona. I've substantial samples from Japanese, English, Russian, French, and German speakers.

Much of it is public domain or CC, so I'll repost a zip file shortly. The rest is unknown license, but I'll make it available to anyone who wants it for computational linguistics work, contact me directly and we'll arrange an exchange in a dark alley where copyright doesn't apply. If this was all on a generous license, we could republish it all as a comprehensive toki pona anthology on lulu.

As you might know, toki pona had a burning-of-the-libraries-of-Alexandria moment when Geocities went down. All of the following fan sites are gone.
geocities.com/stephentoddpope/tokiponabible.html
geocities.com/yohsweb/
geocities.com/yves_prudhomme/toki_pona/
geocities.com/stephentoddpope/tokiponahome
geocities.com/girlinside123/toki.html

Not only did we lose some irreplaceable works of early toki pona culture, but we've lost the "tenpo ale la mi pali e lipu ni. lipu ni li pini ala." animated gifs. tan ni kin la mi pana anpa e telo oko.

Corpus Research Resources and how you can help
Stand and be counted. Please post medium or long texts you have written to the net, preferably with at least one version without interlinear-gloss (ie. translations between each sentence)

Set your work free. If you do post somewhere on the web, please post with a generous license, public domain, CC or GPL documentation so that we can copy your works willy nilly without a guilty conscience. Otherwise we have to buy the linguistic corpus data from our pot dealer in dark alleys.

Post your work to the forum. When you post to this forum, it becomes covered by a CC license per your TOS with jan Sonja.

alasa tawa nimi pi toki pona. I usually can find most toki pona on the net by using toki pona key words, many of them do not exist in any language. There are significant gaps though in finding toki pona from Asian countries, probably because the search engines I know of aren't indexing hangul, chinese, etc.

http://www.suburbandestiny.com/?p=639 <-- Have I missed any toki pona pages on the net?
Search engine restricted to site listed above: http://www.google.com/cse/home?cx=01053 ... rmsq7cfp8g
Last edited by janMato on Sun Sep 05, 2010 5:04 pm, edited 2 times in total.

janKipo
Posts: 3064
Joined: Fri Oct 09, 2009 2:20 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1107: count(): Parameter must be an array or an object that implements Countable

Re: Toki pona corpus project needs your help!

Postby janKipo » Mon Feb 01, 2010 8:20 pm

Thank you. I have fallen way behind on this and never reached your coverage. I do , however, have a lot of shorter items, casual realtime (well, almost) exchanges and the like (I don't have the IRC logs). This does include a few items from the lost sites (I think -- the names are familiar), but not complete sets. Shall I ZIP what I have to you? I haven't the time nor patience at the moment to set up various searches and counts, all of which need doing. as well as all the indexing that will be of use to Sonja as she works on The Book.

janMato
Posts: 1545
Joined: Wed Dec 02, 2009 12:21 pm
Location: Takoma Park, MD
Contact:

Re: Toki pona corpus project needs your help!

Postby janMato » Mon Feb 01, 2010 9:31 pm


janMato
Posts: 1545
Joined: Wed Dec 02, 2009 12:21 pm
Location: Takoma Park, MD
Contact:

Re: Toki pona corpus project needs your help!

Postby janMato » Tue Feb 02, 2010 11:51 pm


janMato
Posts: 1545
Joined: Wed Dec 02, 2009 12:21 pm
Location: Takoma Park, MD
Contact:

Re: Toki pona corpus project needs your help!

Postby janMato » Wed Feb 03, 2010 1:03 am

Here is a zip file of all the current redistributable toki pona text I can find. So far 50% of the collected corpus is redistributable.

Shout outs to the following awesome people who have published with a license compatible with republication.
Michael Freedman - BY-NC-ND
Everyone who posted to wikia - CC-BY-SA
John Clifford - (by forum post implying he was willing to contribute, but not specific license yet)
Joop Kiefte - Public Domain
Bryant Knight - Public Domain
Sonja Kisa - BY-NC-SA
Dave Raftery - Creative Commons
Rowa Giso (not sure about their name) - AFAIK, these texts are a derivative of B Knight's works, which is now public domain.

Posts to this forum are covered by CC per your TOS with jan Sonja (thanks for pointing that out jan Ote!) I suspect that re-licensing magic only works when the original copyright holder posts, though.

However, as far as I can tell, tokilili, yahoo, live journal, etc are all under copyright of the original contributors, the sites that post TOS usually say that the site gets limited license to run the mailing list or what have you, but there's no redistribution rights for folk like me...but like any copyright, it only matters to the extent that some can afford to enforce. But that is a story for another day.

I excluded anything that was a translation of a copyrighted work and wasn't small enough to be considered fair use.

Once I get my toki pona website up again, I'll publish it there, too.

The next step is to rummage through this and start correcting the grammar and removing extraneous English, although the No-Deriv's licenses on some of the texts worry me about if I can do that.
Attachments
Redistributable Toki Pona Corpus.zip
(53.29 KiB) Downloaded 327 times
Last edited by janMato on Wed Feb 03, 2010 9:17 am, edited 1 time in total.

User avatar
jan Ote
Posts: 424
Joined: Thu Oct 08, 2009 1:15 am
Location: ma Posuka
Contact:

Re: Toki pona corpus project needs your help!

Postby jan Ote » Wed Feb 03, 2010 6:24 am


janMato
Posts: 1545
Joined: Wed Dec 02, 2009 12:21 pm
Location: Takoma Park, MD
Contact:

Re: Toki pona corpus project needs your help!

Postby janMato » Wed Feb 03, 2010 9:35 am


User avatar
jan Ote
Posts: 424
Joined: Thu Oct 08, 2009 1:15 am
Location: ma Posuka
Contact:

Re: Toki pona corpus project needs your help!

Postby jan Ote » Wed Feb 03, 2010 11:20 am


User avatar
jan Ote
Posts: 424
Joined: Thu Oct 08, 2009 1:15 am
Location: ma Posuka
Contact:

Re: Toki pona corpus project needs your help!

Postby jan Ote » Tue Mar 23, 2010 6:19 am

Attachments
suno-pona-corpus.zip
9 tp text (short, medium, long), Creative Commons BY-NC-SA
(9.2 KiB) Downloaded 271 times

linguafrakka
Posts: 3
Joined: Mon Apr 16, 2018 1:53 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1107: count(): Parameter must be an array or an object that implements Countable

Re: Toki pona corpus project needs your help!

Postby linguafrakka » Mon Apr 16, 2018 2:05 pm

Thought it would be worth mentioning here that I've recently brought the Toki Pona language to life over on Glosbe.com. The site originally had only 24 translated words, but I've now beefed the site up with over 1,200 English words and their corresponding word/phrase in Toki Pona. I made sure to provide thorough English definitions for all translated words to ensure there is no semantic ambiguity regarding English words with multiple corresponding Toki Pona words (ex. "cause" could be "kama", "pana", or "tan" based on context). Totally recommend checking out the site as a handy reference and contribution-hub.

https://glosbe.com/en/mis_tok


(Sorry if my post was badly formatted. Created an account here just to spread the word about Glosbe.)


[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1107: count(): Parameter must be an array or an object that implements Countable

Return to “ijo ante | miscellaneous | diversaj”

Who is online

Users browsing this forum: No registered users and 2 guests