From 98fc0505155575df4170c863107153df2a751a13 Mon Sep 17 00:00:00 2001 From: Yuri Pankov Date: Thu, 11 Oct 2018 18:30:12 +0000 Subject: [PATCH] Restore some of the ctype definitions reported in the PR from pre-CLDR data, namely 0xE000-0xF8FF private use area, and 0xFF00-0xFFF half- and fullwidth punctuation. While here, update tools/tools/locale/README based on my experience rebuilding the locale data. PR: 225692 Reviewed by: bapt, cem (previous version) Approved by: re (gjb), kib (mentor) Differential Revision: https://reviews.freebsd.org/D17471 --- share/ctypedef/en_US.UTF-8.src | 17 +++++++++ tools/tools/locale/README | 46 +++++++++++++++-------- tools/tools/locale/etc/common.UTF-8.src | 17 +++++++++ tools/tools/locale/etc/manual-input.UTF-8 | 17 +++++++++ 4 files changed, 81 insertions(+), 16 deletions(-) diff --git a/share/ctypedef/en_US.UTF-8.src b/share/ctypedef/en_US.UTF-8.src index bebdb9180b56..836e22cfde51 100644 --- a/share/ctypedef/en_US.UTF-8.src +++ b/share/ctypedef/en_US.UTF-8.src @@ -6240,6 +6240,12 @@ graph ;...; graph ;...; digit ;...; +********************************************************************** +* 0xE000 - 0xF8FF Private Use Area (from pre-CLDR data) +********************************************************************** + +graph ;...; + ********************************************************************** * 0xFB50 - 0xFDFF Arabic Presentation Forms (differential) ********************************************************************** @@ -6278,6 +6284,17 @@ punct ;...; blank +********************************************************************** +* 0xFF00 - 0xFFFF Half- and Fullwidth Punctuation (from pre-CLDR data) +********************************************************************** + +punct ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...; + ********************************************************************** * 0x10300 - 0x1032F Old Italic ********************************************************************** diff --git a/tools/tools/locale/README b/tools/tools/locale/README index 35830fa173ff..1874b750315c 100644 --- a/tools/tools/locale/README +++ b/tools/tools/locale/README @@ -2,23 +2,37 @@ To generate the locales: -Tools needed: java, perl, devel/p5-Tie-IxHash, converters/p5-Text-Iconv and -textproc/p5-XML-Parser +Tools needed: + java (openjdk >= 8) + perl + converters/p5-Text-Iconv + devel/p5-Tie-IxHash + textproc/p5-XML-Parser -fetch cldr data from: http://cldr.unicode.org -extract in a directory ~/unicode/cldr/v30.0.3 for example -fetch unidata from http://www.unicode.org/Public/zipped/ (latest version) -extract in a directory ~/unicode/UNIDATA/9.0.0 for example +Fetch CLDR data from: http://unicode.org/Public/cldr/. You need all of the +core.zip, keyboards.zip, and tools.zip. -Note that the prebuilt cldr tools are not working on freebsd, it needs to -be rebuilt: -cd $CLDRDIR/tools/java -ant build +Extract: + mkdir -p ~/unicode/cldr/v33.0 + cd ~/unicode/cldr/v33.0 + unzip ~/core.zip ~/keyboards.zip ~/tools.zip -either modify tools/tools/locales/etc/unicode.conf or export variables: -CLDRDIR="~/unicode/cldr/v30.0.3" -UNIDATADIR="~/unicode/UNIDATA/9.0.0" +Fetch unidata (UCD.zip) from http://www.unicode.org/Public/zipped/latest. -run: -make POSIX -make install +Extract: + mkdir -p ~/unicode/UNIDATA/11.0.0 + cd ~/unicode/UNIDATA/11.0.0 + unzip ~/UCD.zip + +Either modify tools/tools/locales/etc/unicode.conf or export variables: + CLDRDIR=~/unicode/cldr/v33.0; export CLDRDIR + UNIDATADIR=~/unicode/UNIDATA/9.0.0; export UNIDATADIR + +Build the CLDR tools: + cd $CLDRDIR/tools/java + ant jar + +Run: + make POSIX + make + make install diff --git a/tools/tools/locale/etc/common.UTF-8.src b/tools/tools/locale/etc/common.UTF-8.src index bebdb9180b56..836e22cfde51 100644 --- a/tools/tools/locale/etc/common.UTF-8.src +++ b/tools/tools/locale/etc/common.UTF-8.src @@ -6240,6 +6240,12 @@ graph ;...; graph ;...; digit ;...; +********************************************************************** +* 0xE000 - 0xF8FF Private Use Area (from pre-CLDR data) +********************************************************************** + +graph ;...; + ********************************************************************** * 0xFB50 - 0xFDFF Arabic Presentation Forms (differential) ********************************************************************** @@ -6278,6 +6284,17 @@ punct ;...; blank +********************************************************************** +* 0xFF00 - 0xFFFF Half- and Fullwidth Punctuation (from pre-CLDR data) +********************************************************************** + +punct ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...; + ********************************************************************** * 0x10300 - 0x1032F Old Italic ********************************************************************** diff --git a/tools/tools/locale/etc/manual-input.UTF-8 b/tools/tools/locale/etc/manual-input.UTF-8 index f234a9b711d8..c856ccbe513e 100644 --- a/tools/tools/locale/etc/manual-input.UTF-8 +++ b/tools/tools/locale/etc/manual-input.UTF-8 @@ -876,6 +876,12 @@ graph ;...; graph ;...; digit ;...; +********************************************************************** +* 0xE000 - 0xF8FF Private Use Area (from pre-CLDR data) +********************************************************************** + +graph ;...; + ********************************************************************** * 0xFB50 - 0xFDFF Arabic Presentation Forms (differential) ********************************************************************** @@ -914,6 +920,17 @@ punct ;...; blank +********************************************************************** +* 0xFF00 - 0xFFFF Half- and Fullwidth Punctuation (from pre-CLDR data) +********************************************************************** + +punct ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...; + ********************************************************************** * 0x10300 - 0x1032F Old Italic **********************************************************************