Part of the slowdown of RBBI::handleNext/handlePrevious on some processors is
due to additional overhead of various tests for uncommon cases (lookahead, BOF).
If the rules are known not to use any of these features, the loop can be vastly
simplified. If the rules compiler set a flag in the rules,
handleNext/handlePrevious could test that flag in an if at the very beginning of
the function, and use two different forms of the entire function. Another way to
do this would be via a handleNext and a handleNextSimple, and use a
pointer-to-member function that would be set at the time the rules are loaded.
Another optimization is in the way the result position is recorded. Right now
UTEXT_GETNATIVEINDX and utext_setNativeIndex are used. If instead a counter were
kept, the loop could keep track of how much it needs to back up, and then just
call UTEXT_PREV32 (or UTEXT_NEXT32) repeatedly at the end. That might be faster
than utext_setNativeIndex, which is currently showing up as about 8% of the time
for word break on a PowerPC 970 (G5).
You can assign this to Andy and myself. Apple needs this as a patch on top of