mbox series

[v12,0/2] C.UTF-8

Message ID 20210906154336.610973-1-carlos@redhat.com
Headers show
Series C.UTF-8 | expand

Message

Carlos O'Donell Sept. 6, 2021, 3:43 p.m. UTC 
The following changes implement a minimally sized C.UTF-8.
First we implement the 'codepoint_collation' directive.
Then we implement C.UTF-8 with an LC_COLLATE that uses the
'codepoint_collation' directive to support using strcmp
or wcscmp for collation i.e. code point sorting. The final
C.UTF-8 is only ~396KiB with the largest ~346KiB in
LC_CTYPE for all of Unicode.

v12 fixes commit message to match NEWS.

v11 fixes a defect in the tst-regex.c test. All modified
tests were reviewed for similar defects and none were
found. v11 also removes an obsolete Contributed-by line
in C-collate-seq.c.

v10 fixes a defect in the transbug.c test.

v9 is rebased against the changes to remove ISO-8859-1
characters from the bug-regex1.c test
(69623c0db0a540f26ee537bae09446d3dcdf1f80).

v8 includes a NEWS entry for the updated C.UTF-8.

v7 fixed the regressions detected in Fedora Rawhide
here: https://bugzilla.redhat.com/show_bug.cgi?id=1986421,
but does so by generating identity tables for 
_NL_COLLATE_COLLSEQMB, and _NL_COLLATE_COLLSEQWC to provide
mappings for ASCII characters. This ensures that static
applications using the new C.UTF-8 have a functioning
fnmatch, regcomp, and regexec for ASCII ranges. This raises
the size of LC_COLLATE from 92 to 1406 bytes. Valgrind
reports no errors using the tables with C.UTF-8 under
tst-fnmatch. v7 also corrected collation sequence byte
ordering on BE targets, and I verified this by building
crossed locales with localedef --big-endian and confirming
that s390x built native C.UTF-8 is the same as an x86_64
C.UTF-8 built wtih --big-endian.

The fixes that were in v4 for nrules == 0 will be included
in the next release of glibc, and when those are proven
correct they can be backported to provide dyanmic
or newly compiled static applications with the ability
to use all code points in ranges.

Carlos O'Donell (2):
  Add 'codepoint_collation' support for LC_COLLATE.
  Add generic C.UTF-8 locale (Bug 17318)

 NEWS                             |  10 +-
 iconv/Makefile                   |  22 +-
 iconv/tst-iconv9.c               |  87 +++++
 locale/C-collate-seq.c           | 100 ++++++
 locale/C-collate.c               |  78 +----
 locale/programs/ld-collate.c     |  36 +-
 locale/programs/locfile-kw.gperf |   1 +
 locale/programs/locfile-kw.h     | 299 ++++++++---------
 locale/programs/locfile-token.h  |   1 +
 localedata/C.UTF-8.in            | 157 +++++++++
 localedata/Makefile              |   2 +
 localedata/SUPPORTED             |   1 +
 localedata/locales/C             | 194 +++++++++++
 posix/Makefile                   |  16 +-
 posix/bug-regex1.c               |  20 ++
 posix/bug-regex19.c              |  22 +-
 posix/bug-regex4.c               |  25 ++
 posix/bug-regex6.c               |   2 +-
 posix/transbug.c                 |  24 +-
 posix/tst-fnmatch.input          | 549 ++++++++++++++++++++++++++++++-
 posix/tst-regcomp-truncated.c    |   1 +
 posix/tst-regex.c                |  33 +-
 22 files changed, 1417 insertions(+), 263 deletions(-)
 create mode 100644 iconv/tst-iconv9.c
 create mode 100644 locale/C-collate-seq.c
 create mode 100644 localedata/C.UTF-8.in
 create mode 100644 localedata/locales/C