From 85a194c94516daecda74096e05b6dedf1b63e7b0 Mon Sep 17 00:00:00 2001 From: John MacFarlane <jgm@berkeley.edu> Date: Thu, 20 Nov 2014 12:01:04 -0800 Subject: Added utf8proc_is_space. --- src/utf8.c | 124 +++++++++++++++++++++++++++++++++++++++++++------------------ src/utf8.h | 1 + 2 files changed, 89 insertions(+), 36 deletions(-) diff --git a/src/utf8.c b/src/utf8.c index c798981..e144c72 100644 --- a/src/utf8.c +++ b/src/utf8.c @@ -190,14 +190,32 @@ void utf8proc_case_fold(strbuf *dest, const uint8_t *str, int len) } } +// matches anything in the Zs class, plus LF, CR, TAB, FF. +int utf8proc_is_space(int32_t uc) +{ + return (uc == 9 || + uc == 10 || + uc == 12 || + uc == 13 || + uc == 32 || + uc == 160 || + uc == 5760 || + (uc >= 8192 && uc <= 8202) || + uc == 8239 || + uc == 8287 || + uc == 12288); +} + // matches anything in the P[cdefios] classes. int utf8proc_is_punctuation(int32_t uc) { return ((uc >= 33 && uc <= 35) || (uc >= 37 && uc <= 42) || (uc >= 44 && uc <= 47) || - (uc >= 58 && uc <= 59) || - (uc >= 63 && uc <= 64) || + uc == 58 || + uc == 59 || + uc == 63 || + uc == 64 || (uc >= 91 && uc <= 93) || uc == 95 || uc == 123 || @@ -205,77 +223,98 @@ int utf8proc_is_punctuation(int32_t uc) uc == 161 || uc == 167 || uc == 171 || - (uc >= 182 && uc <= 183) || + uc == 182 || + uc == 183 || uc == 187 || uc == 191 || uc == 894 || uc == 903 || (uc >= 1370 && uc <= 1375) || - (uc >= 1417 && uc <= 1418) || + uc == 1417 || + uc == 1418 || uc == 1470 || uc == 1472 || uc == 1475 || uc == 1478 || - (uc >= 1523 && uc <= 1524) || - (uc >= 1545 && uc <= 1546) || - (uc >= 1548 && uc <= 1549) || + uc == 1523 || + uc == 1524 || + uc == 1545 || + uc == 1546 || + uc == 1548 || + uc == 1549 || uc == 1563 || - (uc >= 1566 && uc <= 1567) || + uc == 1566 || + uc == 1567 || (uc >= 1642 && uc <= 1645) || uc == 1748 || (uc >= 1792 && uc <= 1805) || (uc >= 2039 && uc <= 2041) || (uc >= 2096 && uc <= 2110) || uc == 2142 || - (uc >= 2404 && uc <= 2405) || + uc == 2404 || + uc == 2405 || uc == 2416 || uc == 2800 || uc == 3572 || uc == 3663 || - (uc >= 3674 && uc <= 3675) || + uc == 3674 || + uc == 3675 || (uc >= 3844 && uc <= 3858) || uc == 3860 || (uc >= 3898 && uc <= 3901) || uc == 3973 || (uc >= 4048 && uc <= 4052) || - (uc >= 4057 && uc <= 4058) || + uc == 4057 || + uc == 4058 || (uc >= 4170 && uc <= 4175) || uc == 4347 || (uc >= 4960 && uc <= 4968) || uc == 5120 || - (uc >= 5741 && uc <= 5742) || - (uc >= 5787 && uc <= 5788) || + uc == 5741 || + uc == 5742 || + uc == 5787 || + uc == 5788 || (uc >= 5867 && uc <= 5869) || - (uc >= 5941 && uc <= 5942) || + uc == 5941 || + uc == 5942 || (uc >= 6100 && uc <= 6102) || (uc >= 6104 && uc <= 6106) || (uc >= 6144 && uc <= 6154) || - (uc >= 6468 && uc <= 6469) || - (uc >= 6686 && uc <= 6687) || + uc == 6468 || + uc == 6469 || + uc == 6686 || + uc == 6687 || (uc >= 6816 && uc <= 6822) || (uc >= 6824 && uc <= 6829) || (uc >= 7002 && uc <= 7008) || (uc >= 7164 && uc <= 7167) || (uc >= 7227 && uc <= 7231) || - (uc >= 7294 && uc <= 7295) || + uc == 7294 || + uc == 7295 || (uc >= 7360 && uc <= 7367) || uc == 7379 || (uc >= 8208 && uc <= 8231) || (uc >= 8240 && uc <= 8259) || (uc >= 8261 && uc <= 8273) || (uc >= 8275 && uc <= 8286) || - (uc >= 8317 && uc <= 8318) || - (uc >= 8333 && uc <= 8334) || + uc == 8317 || + uc == 8318 || + uc == 8333 || + uc == 8334 || (uc >= 8968 && uc <= 8971) || - (uc >= 9001 && uc <= 9002) || + uc == 9001 || + uc == 9002 || (uc >= 10088 && uc <= 10101) || - (uc >= 10181 && uc <= 10182) || + uc == 10181 || + uc == 10182 || (uc >= 10214 && uc <= 10223) || (uc >= 10627 && uc <= 10648) || (uc >= 10712 && uc <= 10715) || - (uc >= 10748 && uc <= 10749) || + uc == 10748 || + uc == 10749 || (uc >= 11513 && uc <= 11516) || - (uc >= 11518 && uc <= 11519) || + uc == 11518 || + uc == 11519 || uc == 11632 || (uc >= 11776 && uc <= 11822) || (uc >= 11824 && uc <= 11842) || @@ -286,34 +325,44 @@ int utf8proc_is_punctuation(int32_t uc) uc == 12349 || uc == 12448 || uc == 12539 || - (uc >= 42238 && uc <= 42239) || + uc == 42238 || + uc == 42239 || (uc >= 42509 && uc <= 42511) || uc == 42611 || uc == 42622 || (uc >= 42738 && uc <= 42743) || (uc >= 43124 && uc <= 43127) || - (uc >= 43214 && uc <= 43215) || + uc == 43214 || + uc == 43215 || (uc >= 43256 && uc <= 43258) || - (uc >= 43310 && uc <= 43311) || + uc == 43310 || + uc == 43311 || uc == 43359 || (uc >= 43457 && uc <= 43469) || - (uc >= 43486 && uc <= 43487) || + uc == 43486 || + uc == 43487 || (uc >= 43612 && uc <= 43615) || - (uc >= 43742 && uc <= 43743) || - (uc >= 43760 && uc <= 43761) || + uc == 43742 || + uc == 43743 || + uc == 43760 || + uc == 43761 || uc == 44011 || - (uc >= 64830 && uc <= 64831) || + uc == 64830 || + uc == 64831 || (uc >= 65040 && uc <= 65049) || (uc >= 65072 && uc <= 65106) || (uc >= 65108 && uc <= 65121) || uc == 65123 || uc == 65128 || - (uc >= 65130 && uc <= 65131) || + uc == 65130 || + uc == 65131 || (uc >= 65281 && uc <= 65283) || (uc >= 65285 && uc <= 65290) || (uc >= 65292 && uc <= 65295) || - (uc >= 65306 && uc <= 65307) || - (uc >= 65311 && uc <= 65312) || + uc == 65306 || + uc == 65307 || + uc == 65311 || + uc == 65312 || (uc >= 65339 && uc <= 65341) || uc == 65343 || uc == 65371 || @@ -332,10 +381,12 @@ int utf8proc_is_punctuation(int32_t uc) (uc >= 68409 && uc <= 68415) || (uc >= 68505 && uc <= 68508) || (uc >= 69703 && uc <= 69709) || - (uc >= 69819 && uc <= 69820) || + uc == 69819 || + uc == 69820 || (uc >= 69822 && uc <= 69825) || (uc >= 69952 && uc <= 69955) || - (uc >= 70004 && uc <= 70005) || + uc == 70004 || + uc == 70005 || (uc >= 70085 && uc <= 70088) || uc == 70093 || (uc >= 70200 && uc <= 70205) || @@ -343,7 +394,8 @@ int utf8proc_is_punctuation(int32_t uc) (uc >= 71105 && uc <= 71113) || (uc >= 71233 && uc <= 71235) || (uc >= 74864 && uc <= 74868) || - (uc >= 92782 && uc <= 92783) || + uc == 92782 || + uc == 92783 || uc == 92917 || (uc >= 92983 && uc <= 92987) || uc == 92996 || diff --git a/src/utf8.h b/src/utf8.h index 2474e07..319e39a 100644 --- a/src/utf8.h +++ b/src/utf8.h @@ -13,6 +13,7 @@ void utf8proc_encode_char(int32_t uc, cmark_strbuf *buf); int utf8proc_iterate(const uint8_t *str, int str_len, int32_t *dst); int utf8proc_charlen(const uint8_t *str, int str_len); void utf8proc_detab(cmark_strbuf *dest, const uint8_t *line, size_t size); +int utf8proc_is_space(int32_t uc); int utf8proc_is_punctuation(int32_t uc); #ifdef __cplusplus -- cgit v1.2.3