| Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
|
|
I removed the initAtIndex functions from the public vocabulary, because
the last couple of days of sweat and blood prove that it's hard to use
correctly.
That's probably it for WordBreak, now to fix the overlong bug on v0.14
and get this integrated with the new reverse grapheme iterator.
|
|
This also adds helper functions for initializing iterators at an index
within the string. Not that user code should do that necessarily, but
`wordAtIndex` definitely should, and there's no reason not to make it
available to others. With an appropriate warning at least.
|
|
While of only occasional use in real programs, one thing these are good
for is reliably retrieving the word at a given index. Which turns out
to be.. tricky is the best word.
|
|
Ended up needing a clone of the codepoint iterator, so that WB4 can
ignore points in a matter compatible with backward search.
So I created a special SneakIterator which can return WBPs directly,
so as to skip ignorables. This is also needed for flag emoji, since
the odd-number case must be handle immediately. So we count back in
a WB4 compatible way, then store the count on the word iterator, and
voila.
|
|
The handling of ignorables is really different, because they 'adhere'
to the future of the iteration, not the past.
|
|
Next up I hook it to the tests.
|
|
This is not actually the way to do it, and can break on some crafted
strings. The way to actually do it: implement a reverse word search
iterator, then do next() to find a word break, prev() to find a
_valid_ word start, then next() again to find the valid end of said
word. Maybe 2+, 2-, 1+ actually.
I can probably write a test to see if the cursor spot is ambiguous,
and apply an extra round if so. Need to mull the rules over before
making any rash moves.
|
|
After fixing a bug in Runicode which was fenceposting codepoints off the
end of ranges. As one does.
|
|
I'm not sure the details of this strategy can actually be made to work.
But, something can.
|
|
A by-the-book implmentation of the word break rules from tr29. This is
superficially inefficient, but compilers are more than able to handle
the common subexpression folding ignored by this approach.
Now to port the WordBreakPropertyTests, and clean up the inevitable bugs
in the implementation.
|
|
Passes some simple lookup tests.
|
Sam Atman