Big5

Jump to navigation Jump to search
Big5
Language(s)Traditional Chinese
ClassificationExtended ASCII,[a][b] Variable-width encoding, DBCS, CJK encoding
ExtendsASCII[b]
ExtensionsWindows-950, Big5-HKSCS, numerous others
Other related encoding(s)CNS 11643
  1. ^ Not in the strictest sense of the term, as ASCII bytes can appear as trail bytes.
  2. ^ a b Big5 does not specify a single-byte component; however, ASCII (or an extension) is used in practice.

Big-5 or Big5 is a Chinese character encoding method used in Taiwan, Hong Kong, and Macau for traditional Chinese characters.

The People's Republic of China (PRC), which uses simplified Chinese characters, uses the GB 18030 character set instead.

Big5 gets its name from the consortium of five companies in Taiwan that developed it.[1]

Organization[edit]

The original Big5 character set is sorted first by usage frequency, second by stroke count, lastly by Kangxi radical.

The original Big5 character set lacked many commonly used characters. To solve this problem, each vendor developed its own extension. The ETen extension became part of the current Big5 standard through popularity.

The structure of Big5 does not conform to the ISO 2022 standard, but rather bears a certain similarity to the Shift JIS encoding. It is a double-byte character set (DBCS) with the following structure:

First byte ("lead byte") 0x81 to 0xfe (or 0xa1 to 0xf9 for non-user-defined characters)
Second byte 0x40 to 0x7e, 0xa1 to 0xfe

(the prefix 0x signifying hexadecimal numbers).

Standard assignments (excluding vendor or user-defined extensions) do not use the bytes 0x7F through 0xA0, nor 0xFF, as either lead (first) or trail (second) bytes. Bytes 0xA1 through 0xFE are used for both lead and trail bytes for double-byte (Big5) codes. Bytes 0x40 through 0x7E are used as trail bytes following a lead byte, or for single-byte codes otherwise. If the second byte is not in either range, behaviour is undefined (i.e., varies from system to system). Additionally, certain variants of the Big5 character set, for example the HKSCS, use an expanded range for the lead byte, including values in the 0x81 to 0xA0 range (similar to Shift JIS), whereas others use reduced lead byte ranges (for instance, the Apple Macintosh variant uses 0xFD through 0xFF as single-byte codes, limiting the lead byte range to 0xA1 through 0xFC).[2]

The numerical value of individual Big5 codes are frequently given as a 4-digit hexadecimal number, which describes the two bytes that comprise the Big5 code as if the two bytes were a big endian representation of a 16-bit number. For example, the Big5 code for a full-width space, which are the bytes 0xa1 0x40, is usually written as 0xa140 or just A140.

Strictly speaking, the Big5 encoding contains only DBCS characters. However, in practice, the Big5 codes are always used together with an unspecified, system-dependent single-byte character set (ASCII, or an 8-bit character set such as code page 437), so that you will find a mix of DBCS characters and single-byte characters in Big5-encoded text. Bytes in the range 0x00 to 0x7f that are not part of a double-byte character are assumed to be single-byte characters. (For a more detailed description of this problem, please see the discussion on "The Matching SBCS" below.)

The meaning of non-ASCII single bytes outside the permitted values that are not part of a double-byte character varies from system to system. In old MSDOS-based systems, they are likely to be displayed as 8-bit characters; in modern systems, they are likely to either give unpredictable results or generate an error.

A more detailed look at the organization[edit]

In the original Big5, the encoding is compartmentalized into different zones:

0x8140 to 0xa0fe Reserved for user-defined characters 造字
0xa140 to 0xa3bf "Graphical characters" 圖形碼
0xa3c0 to 0xa3fe Reserved, not for user-defined characters
0xa440 to 0xc67e Frequently used characters 常用字
0xc6a1 to 0xc8fe Reserved for user-defined characters
0xc940 to 0xf9d5 Less frequently used characters 次常用字
0xf9d6 to 0xfefe Reserved for user-defined characters

The "graphical characters" actually comprise punctuation marks, partial punctuation marks (e.g., half of a dash, half of an ellipsis; see below), dingbats, foreign characters, and other special characters (e.g., presentational "full width" forms, digits for Suzhou numerals, zhuyin fuhao, etc.)

In most vendor extensions, extended characters are placed in the various zones reserved for user-defined characters, each of which are normally regarded as associated with the preceding zone. For example, additional "graphical characters" (e.g., punctuation marks) would be expected to be placed in the 0xa3c0–0xa3fe range, and additional logograms would be placed in either the 0xc6a1–0xc8fe or the 0xf9d6–0xfefe range. Sometimes, this is not possible due to the large number of extended characters to be added; for example, Cyrillic letters and Japanese kana have been placed in the zone associated with "frequently-used characters".

What a Big5 code actually encodes[edit]

An individual Big5 code does not always represent a complete semantic unit. The Big5 codes of logograms are always logograms, but codes in the "graphical characters" section are not always complete "graphical characters". What Big5 encodes are particular graphical representations of characters or part of characters that happen to fit in the space taken by two monospaced ASCII characters. This is a property of double-byte character sets as normally used in CJK (Chinese, Japanese, and Korean) computing, and is not a unique problem of Big5.

(The above might need some explanation by putting it in historical perspective, as it is theoretically incorrect: Back when text mode personal computing was still the norm, characters were normally represented as single bytes and each character takes one position on the screen. There was therefore a practical reason to insist that double-byte characters must take up two positions on the screen, namely that off-the-shelf, American-made software would then be usable without modification in a DBCS-based system. If a character can take an arbitrary number of screen positions, software that assumes that one byte of text takes one screen position would produce incorrect output. Of course, if a computer never had to deal with the text screen, the manufacturer would not enforce this artificial restriction; the Apple Macintosh is an example. Nevertheless, the encoding itself must be designed so that it works correctly on text-screen-based systems.)

To illustrate this point, consider the Big5 code 0xa14b (…). To English speakers this looks like an ellipsis and the Unicode standard identifies it as such; however, in Chinese, the ellipsis consists of six dots that fit in the space of two Chinese characters (……), so in fact there is no Big5 code for the Chinese ellipsis, and the Big5 code 0xa14b just represents half of a Chinese ellipsis. It represents only half of an ellipsis because the whole ellipsis should take the space of two Chinese characters, and in many DBCS systems one DBCS character must take exactly the space of one Chinese character.

Characters encoded in Big5 do not always represent things that can be readily used in plain text files; an example is "citation mark" (0xa1ca, ﹋), which is, when used, required to be typeset under the title of literary works. Another example is the Suzhou numerals, which is a form of scientific notation that requires the number to be laid out in a 2-D form consisting of at least two rows.

The Matching SBCS[edit]

In practice, Big5 cannot be used without a matching Single Byte Character Set (SBCS); this is mostly to do with a compatibility reason. However, as in the case of other CJK DBCS character sets, the SBCS to use has never been specified. Big5 has always been defined as a DBCS, though when used it must be paired with a suitable, unspecified SBCS and therefore used as what some people call a MBCS; nevertheless, Big5 by itself, as defined, is strictly a DBCS.

The SBCS to use being unspecified implies that the SBCS used can theoretically vary from system to system. Nowadays, ASCII is the only possible SBCS one would use. However, in old DOS-based systems, Code Page 437—with its extra special symbols in the control code area including position 127—was much more common. Yet, on a Macintosh system with the Chinese Language Kit, or on a Unix system running the cxterm terminal emulator, the SBCS paired with Big5 would not be Code Page 437.

Outside the valid range of Big5, the old DOS-based systems would routinely interpret things according to the SBCS that is paired with Big5 on that system. In such systems, characters 127 to 160, for example, were very likely not avoided because they would produce invalid Big5, but used because they would be valid characters in Code Page 437.

The modern characterization of Big5 as an MBCS consisting of the DBCS of Big5 plus the SBCS of ASCII is therefore historically incorrect and potentially flawed, as the choice of the matching SBCS was, and theoretically still is, quite independent of the flavour of Big5 being used.

History[edit]

The inability of ASCII to support large character sets such as used for Chinese, Japanese and Korean led to governments and industry to find creative solutions to enable their languages to be rendered on computers. A variety of ad-hoc and usually proprietary input methods led to efforts to develop a standard system. As a result, Big5 encoding was defined by the Institute for Information Industry of Taiwan in 1984. The name "Big5" is in recognition that the standard emerged from collaboration of five of Taiwan's largest IT firms: Acer (宏碁); MiTAC (神通); JiaJia (佳佳), ZERO ONE Technology (零壹 or 01tech); and, First International Computer (FIC) (大眾).

Big5 was rapidly popularized in Taiwan and worldwide among Chinese who used the traditional Chinese character set through its adoption in several commercial software packages, notably the E-TEN Chinese DOS input system (ETen Chinese System). The Republic of China government declared Big5 as their standard in mid-1980s since it was, by then, the de facto standard for using traditional Chinese on computers.

Extensions[edit]

The original Big-5 only include CJK logograms from two lists "常用國字標準字體表; cháng yòng gúo zì bīao zhǔn zì tĭ bǐao" (4808 characters) and "次常用國字標準字體表; cì cháng yòng gúo zì bīao zhǔn zì tĭ bǐao" (6343 characters), but not letters from people's names, place names, dialects, chemistry, biology, Japanese kana. As a result, many Big-5 supporting software include extensions to address the problems.

The plethora of variations make UTF-8 or UTF-16 a more consistent code page for modern use.

Vendor Extensions[edit]

ETEN extensions[edit]

In ETEN(倚天) Chinese operating system, the following code points are added to make it compliant with IBM5550 code page:

  • A3C0-A3E0: 33 control characters.
  • C6A1-C875: circle 1-10, bracket 1-10, Roman numerals 1-9 (i-ix), CJK radical glyphs, Japanese hiragana, Japanese katakana, Cyrillic characters
  • F9D6-F9FE: '碁', '銹', '恒', '裏', '墻', '粧', '嫺', and 34 extra symbols.

In some versions of Eten, there are extra graphical symbols and Simplified Chinese characters.

Microsoft code pages[edit]

Microsoft (微軟) created its own version of Big5 extension as Code page 950 for use with Microsoft Windows, which supports ETEN's extensions, but only the F9D6-F9FE code points. In Windows ME, the euro currency symbol was mapped to Big-5 code point A3E1, but not in later versions of the operating system.

After installing Microsoft's HKSCS patch on top of traditional Chinese Windows (or any version of Windows 2000 and above with proper language pack), applications using code page 950 automatically use a hidden code page 951 table. The table supports all code points in HKSCS-2001, except for the compatibility code points specified by the standard.[3]

Code page 950 used by Windows 2000 and Windows XP maps hiragana and katakana characters to Unicode private use area block when exporting to Unicode, but to the proper hiragana and katakana Unicode blocks in Windows Vista.[citation needed][clarification needed]

ChinaSea font[edit]

ChinaSea fonts (中國海字集)[4] are Traditional Chinese fonts made by ChinaSea. The fonts are rarely sold separately, but are bundled with other products, such as the Chinese version of Microsoft Office 97. The fonts support Japanese kana, kokuji, and other characters missing in Big-5. As a result, the ChinaSea extensions have become more popular than the government-supported extensions.[when?] Some Hong Kong BBSes had used encodings in ChinaSea fonts before the introduction of HKSCS.

'Sakura' font[edit]

The 'Sakura' font (日和字集 Sakura Version) is developed in Hong Kong and is designed to be compatible with HKSCS. It adds support for kokuji and proprietary dingbats (including Doraemon) not found in HKSCS.

Unicode-at-on[edit]

Unicode-at-on (Unicode補完計畫), formerly BIG5 Extension, extends BIG-5 by altering code page tables, but uses the ChinaSea extensions starting with version 2. However, with the bankruptcy of ChinaSea, late development, and the increasing popularity of HKSCS and Unicode (the project is not compatible with HKSCS), the success of this extension is limited at best.

Despite the problems, characters previously mapped to Unicode Private Use Area are remapped to the standardized equivalents when exporting characters to Unicode format.

OPG[edit]

The web sites of the Oriental Daily News and Sun Daily, belonging to the Oriental Press Group Limited(東方報業集團有限公司) in Hong Kong, used a downloadable font with a different Big-5 extension coding than the HKSCS.

Official Extensions[edit]

Taiwan Ministry of Education font[edit]

The Taiwan Ministry of Education supplied its own font, the Taiwan Ministry of Education font(臺灣教育部造字檔) for use internally.

Taiwan Council of Agriculture font[edit]

Taiwan's Council of Agriculture font, Executive Yuan introduced a 133-character custom font, the Taiwan Council of Agriculture font(臺灣農委會常用中文外字集) that includes 84 characters from the 'fish' radical and 7 from the 'bird' radical.

Big5+[edit]

The Chinese Foundation for Digitization Technology(中文數位化技術推廣委員會) introduced Big5+ in 1997, which used over 20000 code points to incorporate all CJK logograms in Unicode 1.1. However, the extra code points exceeded the original Big-5 definition (Big5+ uses high byte values 81-FE and low byte values 40-7E and 80-FE), preventing it from being installed on Microsoft Windows without new codepage files.

Big-5E[edit]

To allow Windows users to use custom fonts, the Chinese Foundation for Digitization Technology introduced Big-5E, which added 3954 characters (in three blocks of code points: 8E40-A0FE, 8140-86DF, 86E0-875C) and removed the Japanese kana from the ETEN extension. Unlike Big-5+, Big5E extends Big-5 within its original definition. Mac OS X 10.3 and later supports Big-5E in the fonts LiHei Pro (儷黑 Pro.ttf) and LiSong Pro (儷宋 Pro.ttf).

Big5-2003[edit]

The Chinese Foundation for Digitization Technology made a Big5 definition and put it into CNS 11643 in note form, making it part of the official standard in Taiwan.

Big5-2003 incorporates all Big-5 characters introduced in the 1984 ETEN extensions (code points A3C0-A3E0, C6A1-C7F2, and F9D6-F9FE) and the Euro symbol. Cyrillic characters were not included because the authority claimed CNS 11643 does not include such characters.

CDP[edit]

The Academia Sinica made a Chinese Data Processing font (漢字構形資料庫) in late 1990s, which the latest release version 2.5 included 112,533 characters, some less than the Mojikyo fonts.

HKSCS[edit]

Hong Kong also adopted Big5 for character encoding. However, written Cantonese has its own characters not available in the normal Big5 character set. To solve this problem, the Hong Kong Government created the Big5 extensions Government Chinese Character Set (GCCS) in 1995 and Hong Kong Supplementary Character Set in 1999. The Hong Kong extensions were commonly distributed as a patch. It is still being distributed as a patch by Microsoft, but a full Unicode font is also available from the Hong Kong Government's web site.

There are two encoding schemes of HKSCS: one encoding scheme is for the Big-5 coding standard and the other is for the ISO 10646 standard. Subsequent to the initial release, there are also HKSCS-2001 and HKSCS-2004. The HKSCS-2004 is aligned technically with the ISO/IEC 10646:2003 and its Amendment 1 published in April 2004 by the International Organization for Standardization (ISO).

HKSCS includes all the characters from the common ETEN extension, plus some characters from Simplified Chinese, place names, people's names, and Cantonese phrases (including profanity).

MSCS[edit]

Similar to Hong Kong's situation, there are also characters that are needed by Macao but is neither included in Big5 nor HKSCS. Therefore, the Macao Supplementary Character Set have been released to public in Macao for information exchange.[5]

Kana and Cyrillic[edit]

There are two major Big5 extension layouts for encoding kana, Russian Cyrillic and list markers in the range 0xC6A1 through 0xC875. These are not compatible with one another.[6] They are compared in the table below.

Big5 code BIG5.TXT layout[7] ETEN layout (also HKSCS, HTML5)[8]
0xC6A1
0xC6A2
0xC6A3
0xC6A4
0xC6A5
0xC6A6
0xC6A7
0xC6A8
0xC6A9
0xC6AA
0xC6AB
0xC6AC
0xC6AD
0xC6AE
0xC6AF
0xC6B0
0xC6B1
0xC6B2
0xC6B3
0xC6B4
0xC6B5
0xC6B6
0xC6B7
0xC6B8
0xC6B9
0xC6BA
0xC6BB
0xC6BC
0xC6BD
0xC6BE
0xC6BF
0xC6C0 丿
0xC6C1
0xC6C2
0xC6C3
0xC6C4
0xC6C5
0xC6C6
0xC6C7
0xC6C8
0xC6C9
0xC6CA
0xC6CB
0xC6CC
0xC6CD
0xC6CE 广
0xC6CF
0xC6D0
0xC6D1
0xC6D2
0xC6D3
0xC6D4
0xC6D5
0xC6D6
0xC6D7
0xC6D8 ¨
0xC6D9 ˆ
0xC6DA
0xC6DB
0xC6DC
0xC6DD
0xC6DE
0xC6DF
0xC6E0
0xC6E1
0xC6E2
0xC6E3
0xC6E4
0xC6E5
0xC6E6
0xC6E7
0xC6E8
0xC6E9
0xC6EA
0xC6EB
0xC6EC
0xC6ED
0xC6EE
0xC6EF
0xC6F0
0xC6F1
0xC6F2
0xC6F3
0xC6F4
0xC6F5
0xC6F6
0xC6F7
0xC6F8
0xC6F9
0xC6FA
0xC6FB
0xC6FC
0xC6FD
0xC6FE
0xC740
0xC741
0xC742
0xC743
0xC744
0xC745
0xC746
0xC747
0xC748
0xC749
0xC74A
0xC74B
0xC74C
0xC74D
0xC74E
0xC74F
0xC750
0xC751
0xC752
0xC753
0xC754
0xC755
0xC756
0xC757
0xC758
0xC759
0xC75A
0xC75B
0xC75C
0xC75D
0xC75E
0xC75F
0xC760
0xC761
0xC762
0xC763
0xC764
0xC765
0xC766
0xC767
0xC768
0xC769
0xC76A
0xC76B
0xC76C
0xC76D
0xC76E
0xC76F
0xC770
0xC771
0xC772
0xC773
0xC774
0xC775
0xC776
0xC777
0xC778
0xC779
0xC77A
0xC77B
0xC77C
0xC77D
0xC77E
0xC7A1
0xC7A2
0xC7A3
0xC7A4
0xC7A5
0xC7A6
0xC7A7
0xC7A8
0xC7A9
0xC7AA
0xC7AB
0xC7AC
0xC7AD
0xC7AE
0xC7AF
0xC7B0
0xC7B1 Д
0xC7B2 Е
0xC7B3 Ё
0xC7B4 Ж
0xC7B5 З
0xC7B6 И
0xC7B7 Й
0xC7B8 К
0xC7B9 Л
0xC7BA М
0xC7BB У
0xC7BC Ф
0xC7BD Х
0xC7BE Ц
0xC7BF Ч
0xC7C0 Ш
0xC7C1 Щ
0xC7C2 Ъ
0xC7C3 Ы
0xC7C4 Ь
0xC7C5 Э
0xC7C6 Ю
0xC7C7 Я
0xC7C8 а
0xC7C9 б
0xC7CA в
0xC7CB г
0xC7CC д
0xC7CD е
0xC7CE ё
0xC7CF ж
0xC7D0 з
0xC7D1 и
0xC7D2 й
0xC7D3 к
0xC7D4 л
0xC7D5 м
0xC7D6 н
0xC7D7 о
0xC7D8 п
0xC7D9 р
0xC7DA с
0xC7DB т
0xC7DC у
0xC7DD ф
0xC7DE х
0xC7DF ц
0xC7E0 ч
0xC7E1 ш
0xC7E2 щ
0xC7E3 ъ
0xC7E4 ы
0xC7E5 ь
0xC7E6 э
0xC7E7 ю
0xC7E8 я
0xC7E9
0xC7EA
0xC7EB
0xC7EC
0xC7ED
0xC7EE
0xC7EF
0xC7F0
0xC7F1
0xC7F2
0xC7F3 А
0xC7F4 Б
0xC7F5 В
0xC7F6 Г
0xC7F7 Д
0xC7F8 Е
0xC7F9 Ё
0xC7FA Ж
0xC7FB З
0xC7FC И
0xC7FD (not used) Й
0xC7FE (not used) К
0xC840 (not used) Л
0xC841 (not used) М
0xC842 (not used) Н
0xC843 (not used) О
0xC844 (not used) П
0xC845 (not used) Р
0xC846 (not used) С
0xC847 (not used) Т
0xC848 (not used) У
0xC849 (not used) Ф
0xC84A (not used) Х
0xC84B (not used) Ц
0xC84C (not used) Ч
0xC84D (not used) Ш
0xC84E (not used) Щ
0xC84F (not used) Ъ
0xC850 (not used) Ы
0xC851 (not used) Ь
0xC852 (not used) Э
0xC853 (not used) Ю
0xC854 (not used) Я
0xC855 (not used) а
0xC856 (not used) б
0xC857 (not used) в
0xC858 (not used) г
0xC859 (not used) д
0xC85A (not used) е
0xC85B (not used) ё
0xC85C (not used) ж
0xC85D (not used) з
0xC85E (not used) и
0xC85F (not used) й
0xC860 (not used) к
0xC861 (not used) л
0xC862 (not used) м
0xC863 (not used) н
0xC864 (not used) о
0xC865 (not used) п
0xC866 (not used) р
0xC867 (not used) с
0xC868 (not used) т
0xC869 (not used) у
0xC86A (not used) ф
0xC86B (not used) х
0xC86C (not used) ц
0xC86D (not used) ч
0xC86E (not used) ш
0xC86F (not used) щ
0xC870 (not used) ъ
0xC871 (not used) ы
0xC872 (not used) ь
0xC873 (not used) э
0xC874 (not used) ю
0xC875 (not used) я

See also[edit]

References[edit]

  1. ^ chinese mac Character Sets
  2. ^ Apple, Inc (2005-04-04) [1996-06-31]. Map (external version) from Mac OS Chinese Traditional encoding to Unicode 3.0 and later. Unicode Consortium.
  3. ^ "狗爺語錄 » Blog Archive » What is Code Page 951 (CP951)?". Archived from the original on 2007-02-22. Retrieved 2006-09-27.
  4. ^ 黃國書. "Chinasea 1.0 中國海字集". ISU FTP. Archived from the original on 2005-03-19. Retrieved 2016-12-05.
  5. ^ [http://web.archive.org/web/20150104014324/http://appsrv.cse.cuhk.edu.hk/~irg/irg/irg32/IRGN1580MacaoCharsFromMISCS.pdf Submission of Characters from Macao Information Systems Character Set ]
  6. ^ Lunde, Ken (1996-07-12). "2.3.1: BIG FIVE". CJK.INF Version 2.1.
  7. ^ Unicode Consortium (2015-12-02) [1994-02-11]. BIG5 to Unicode table (complete).
  8. ^ van Kesteren, Anne. "big5". Encoding Standard. WHATWG.

External links[edit]