Chapter 4 Overview of en_US.UTF-8 Locale Support

en_US.UTF-8 Locale Support Overview

en_US.UTF-8 locale is the flagship Unicode locale in the Solaris 7 product that supports and provides multi-scripts processing capability by using UTF-8 as its codeset.

UTF-8 is a file system safe Universal Character Set Transformation Format of Unicode / ISO/IEC 10646-1 formulated by XoJIG of X/Open in 1992 and approved by ISO and IEC as Amendment 2 to ISO/IEC 10646-1:1993 in 1996.

The locale supports computation of all characters defined in Unicode 2.1 / ISO/IEC 10646-1:1993 with Amendment 1 to 5 in the locale. However, due to a limited set of font resources and the fact that few users intend to use all of the code point values, users of the en_US.UTF-8 locale will see only character glyphs from the following character sets:

• ISO 8859-1 (Latin-1)

• ISO 8859-2 (Latin-2)

• ISO 8859-4 (Latin-4)

• ISO 8859-5 (Latin/Cyrillic)

• ISO 8859-6 (Arabic)

• ISO 8859-7 (Latin/Greek)

• ISO 8859-8 (Hebrew)

• ISO 8859-9 (Latin-5)

• BIG5 (Traditional Chinese)

• GB 2312-1980 (Simplified Chinese)

• JIS X0201-1976, JIS X0208-1983 (Japanese)

• KS C 5601-1992 Annex 3 (Korean)

• ISO 8859-8 (Hebrew)

• ISO 8859-11 TIS 620.7573 (Thai)

The above coverage of the scripts has significantly increased in the Solaris 7 product, which now also supports the following range of scripts:

• Western/Eastern/Northern European

• Greek

• Turkish

• Cyrillic

• Simplified Chinese

• Traditional Chinese

• Japanese

• Korean

• Arabic

• Hebrew

• Thai

Since this locale is primarily for developers, it belongs to the developer's cluster of the Solaris 7 product. Thus, when you install the Solaris 7 product, you should choose the developer's cluster to install the locale on your system.

Exactly the same level of en_US.UTF-8 locale support is provided for both 32-bit and 64-bit Solaris systems.

Motif and CDE desktop applications and libraries support the en_US.UTF-8 locale. However, OpenWindows, XView, and, OPENLOOK DeskSet applications and libraries do not support the en_US.UTF-8 locale.

System Environment

Locale Environment Variable

To use the en_US.UTF-8 locale environment, make sure to choose the locale first. Be sure you have the en_US.UTF-8 locale installed on your system.

To use the en_US.UTF-8 locale environment

1. In a TTY environment, choose the locale first, by setting the LANG environment variable to en_US.UTF-8, as in the following C-shell example:

   system% setenv LANG en_US.UTF-8

2. Make sure that other categories are not set (or are set to en_US.UTF-8) since the LANG environment variable has a lower priority than other environment variables such as LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_NUMERIC, LC_MONETARY and LC_TIME at setting the locale. See the setlocale(3C) man page for more details about the hierarchy of environment variables.

3. To check current locale settings in various categories, use the locale(1) utility. locale(1)

system% locale 
			LANG=en_US.UTF-8
			LC_CTYPE="en_US.UTF-8"
			LC_NUMERIC="en_US.UTF-8"
			LC_TIME="en_US.UTF-8"
			LC_COLLATE="en_US.UTF-8"
			LC_MONETARY="en_US.UTF-8"
			LC_MESSAGES="en_US.UTF-8"
			LC_ALL=

You can also start the en_US.UTF-8 environment from the CDE desktop. At the CDE login screen's Options -> Language menu, choose en_US.UTF-8.

TTY Environment Setup

To ensure correct text edit operation by a terminal or by a terminal emulator such as dtterm(1), users should push certain locale-specific STREAMS modules onto their Streams.

For more information on STREAMS modules and streams in general, see the STREAMS Programming Guide.

Table 4-1 shows STREAMS modules supported by the en_US.UTF-8 locale in the terminal environment:

Table 4-2 lists the 64-bit STREAMS Modules Supported by en_US.UTF-8.

Loading a STREAMS Module at Kernel

To load a STREAMS module at kernel, first become root:

system% su
Password:
system#

To determine whether you are running a 32-bit Solaris or 64-bit Solaris system, use the isainfo(1) utility as follows:

system# isainfo -v
	64-bit sparcv9 applications
	32-bit sparc applications
	system#

If the command returns this information, you are running the 64-bit Solaris system. If you are running the 32-bit Solaris system, the utility shows the following:

system# isainfo -v
	32-bit sparc applications
	system#

Use modinfo(1M) to be certain that your system has not already loaded the STREAMS module:

system# modinfo | grep eucu8modulename

If the STREAMS module, such as eucu8, is already installed, the output will look as follows:

system# modinfo | grep eucu8
89 ff798000  4b13  18   1  eucu8 (eucu8 module)
system#

If the module is already installed, you don't need to load it. However, if the module has not yet been loaded, use modload(1M)as follows:

system# modload /usr/kernel/strmod/eucu8modulename

This loads the 32-bit eucu8 STREAMS module at the kernel, so you can push it onto a Stream. If you are running the 64-bit Solaris product, use modload(1M) as follows:

system# modload /usr/kernel/strmod/sparcv9/eucu8

The STREAMS module is installed at the kernel, and you can now push it onto a Stream.

To unload a module from the kernel, use modunload(1M), as shown below. In this example, the eucu8 module is being unloaded.

system# modinfo | grep eucu8
89 ff798000  4b13  18   1  eucu8 (eucu8 module)
system# modunload -i 89

dtterm and Terminals Capable of Input and Output of UTF-8 Characters

The dtterm(1) and any terminal that supports input and output of the UTF-8 codeset should have the following STREAMS configuration:

head <-> ttcompat <-> u8euc <->  ldterm <-> eucu8 <-> pseudo-TTY

In this example, u8euc and eucu8 are the modules supported by the en_US.UTF-8 locale. Make sure you already loaded the STREAMS modules into the kernel as specified in the previous section.

To set up the above STREAMS configuration, use,strchg(1) as shown below:

system% cat > /tmp/mystreams 
ttcompat
u8euc
ldterm
eucu8
ptem
^D
system% strchg -f /tmp/mystreams

When using strchg(1) , be sure you are either root or the owner of the device. To see the current configuration of STREAMS, use strconf(1) as shown below:

system% strconf
ttcompat
u8euc
ldterm
eucu8
ptem
pts
system%

To revert to the original configuration, set the STREAMS configuration again as shown below:

system% cat > /tmp/orgstreams
ttcompat
ldterm
ptem
^D
system% strchg -f/tmp/orgstreams

Terminal Support for Latin-1, Latin-2, or KOI8-R

For terminals that support only Latin-1 (ISO 8859-1), Latin-2 (ISO 8859-2), or KOI8-R, you should have the following STREAMS configuration:

head <-> ttcompat <-> u8euc  <-> ldterm <-> eucu8 <-> u8lat1 <-> TTY

This configuration is only for terminals that support Latin-1. For Latin-2 terminals, replace the STREAMS module u8lat1 with u8lat2. For KOI8-R terminals, replace the module with u8koi8.

To set up the STREAMS configuration shown above, use strchg(1), as follows:

system% cat > tmp/mystreams
ttcompat
u8euc
ldterm
eucu8
u8lat1
ptem
^D
system% strchg -f /tmp/mystreams

Be sure that you are either root or the owner of the device when you use strchg(1). To see the current configuration, usestrchg(1) , as follows:

system% strconf
ttcompat
u8euc
ldterm
eucu8
u8lat1
ptem
pts
system%

To revert to the original configuration, set the STREAMS configuration as follows:

system% cat > /tmp/orgstreams
ttcompat
ldterm
ptem
^D
system% strchg -f  /tmp/orgstreams

Setting Terminal Options

To set up the UTF-8 text edit behavior on TTY, you must first set some terminal options using stty(1) as follows:

system% /bin/stty cs8 -istrip defeucw

Since /usr/ucb/stty is not yet internationalized, you should use /bin/stty instead.

You can also query the current settings using stty(1) with the -a option, as shown below:

system% /bin/stty -a

Saving the Settings in ~/.cshrc

Assuming the necessary STREAMS modules are already loaded with the kernel, you can save the following lines in your .cshrc file (C shell example) for convenience:

setenv LANG en_US.UTF-8
if ($?USER != 0 && $?prompt != 0) then
				 cat >! /tmp/mystreams$$ << _EOF
				 ttcompat
				 u8euc
				 ldtterm
				 eucu8
				 ptem
_EOF
				 /bin/strchg -f /tmp/mystream$$
				 /bin/rm -f /tmp/mystream$$
				 /bin/stty cs8 -istrip defeucw
endif

With these lines in your.cshrc file, you do not have to type all of the commands each time. Note that the second _EOF should be in the first column of the file. You can also create a file called mystreams and save it so the .cshrc references to mystreams instead of creating it whenever you start a C shell.

Code Conversions

The en_US.UTF-8 locale supports various code conversions among major codesets of several countries through iconv(1) and iconv(1).

The available fromcode and tocode names that can be applied to iconv(1) and iconv_open(3)are shown in Table 4-3.

For more details on iconv code conversion, see the , iconv(1) and iconv_open(3), iconv(3), and iconv_close(3) man pages. For more information on available code conversions, see iconv_en_US.UTF-8(5).

Script Selection and Input Modes

The en_US.UTF-8 locale supports multiple scripts. There are a total of eight input modes in the en_US.UTF-8 locale:

• English/European

• Cyrillic

• Greek

• Arabic

• Hebrew

• Thai

• Unicode Hexadecimal code input method

• Table lookup input method

English/European Input Mode

The English/European input mode includes not only the English alphabet but also characters with diacritical marks (for example, á, è, î, õ, and ü) and special characters (such as ¡, §, ¿) from European scripts.

This input mode is the default mode for any application. The input mode is displayed at the bottom left corner of the GUI application.

To insert characters with diacritical marks or special characters from Latin-1, Latin-2, Latin-4, Latin-5 and Latin-9, you must type a Compose sequence, as shown in the following examples:

• For Ä, press and release Compose, then A, and then "

• For ¿, press and release Compose, then +, and then -

When there is no <Compose> key available on your keyboard, you can substitute for the <Compose> key by simultaneously pressing the <Control> key and <shift-T> together.

For the input of the Euro currency symbol (Unicode value U+20AC) from the locale, you can use any one of following input sequences:

• <AltGraph> and <e> together

• <AltGraph> and <4> together, or

• <AltGraph> and <5> together

These input sequences mean that you press both keys simultaneously. If there is no <AltGraph> key available on your keyboard, you can substitute the <Alt> key for the <AltGraph> key.

The following tables are the most commonly used Compose sequences in Latin-1, Latin-2, Latin-4,Latin--5 and Latin-9 script input for Sparc.

To start these sequences, type <Compose> key and release it.

Table 4-4 lists the Common Latin-1 Compose Sequences.

Compose sequences defined in Table 4-3 are not included in Table 4-4..

Table 4-5 lists the Common Latin-2 Compose Sequences.

Table 4-5 contains the Latin-2 compose sequences.

Compose sequences defined in Table 4-3 or Table 4-4 are not included in Table 4-5.

Table 4-6 lists the Common Latin-4 Compose Sequences.

Compose sequences defined in Table 4-3 or Table 4-4 or Table 4-5 are not included in Table 4-6.

Table 4-7 lists the Common Latin-5 Compose Sequences.

Any compose sequences already described do not re-appear in this table.

Table 4-8 lists the Common Latin-9 Compose Sequences.

Cyrillic Input Mode

To switch to Cyrillic input mode from English input mode, press Compose c c. If you are currently in Greek input mode, first return to English input mode, then switch to Cyrillic mode.

The input mode is displayed at the bottom left corner of your GUI application.

After you switch to Cyrillic input mode, you cannot enter English text. To switch back to English input mode, type Control-Space. The Russian keyboard layout appears in Figure 4-1.

Figure 4-1 Cyrillic Keyboard

Greek Input Mode

To switch to Greek input mode from English input mode, press Compose g g. If you are currently in Cyrillic input mode, first return to English input mode and then switch to Greek mode.

The input mode is displayed at the left bottom corner of your GUI application .

After you switch to Greek input mode, you cannot enter English text. To switch back to English input mode, type Control-Space. The Greek keyboard layouts appear in Figures 4-2 and 4-3.

Figure 4-2 Greek Euro Keyboard

Figure 4-3 Greek UNIX Keyboard

Arabic Input Mode

To switch to Arabic input mode, type <Compose> + <g> + <g> from your current input mode. The input mode is displayed at the left bottom corner of your GUI application Once you switch to the Arabic input mode, you have to switch back to English/European input mode to enter English/European characters by typing <Control> and <Space> together.

Figure 4-4 Arabic Keyboard

Hebrew Input Mode

To switch into Hebrew input mode, type <Compose> <h> <h> from your current input mode. The input mode is displayed at the left bottom corner of your GUI application.

Once you switched into the Hebrew input mode, you have to switch back to the English/European input mode to enter English/European characters by typing <Control> and <Space> and together. You can also switch into other input modes by typing the corresponding input mode switch key sequence. The Hebrew keyboard layout is shown at following figure:

Figure 4-5 Hebrew Keyboard

Thai Input Mode

To switch into Thai input mode, type <Compose> <t> <t> from your current input mode. The input mode will be displayed at the left bottom corner of your GUI application.

Once you have switched into the Thai input mode, you have to switch back to English/European input mode to enter English/European characters by typing <Control> and <Space> together. You can also switch into other input modes by typing the corresponding input mode switch key sequence. The Thai keyboard layout is shown at following figure:

Figure 4-6 Thai Keyboard

Unicode Hexadecimal Code Input Method Input Mode

To switch into the Unicode hexadecimal code input method input mode, type <Compose> <l> <l> from your current input mode. The input mode is displayed at the left bottom corner of your GUI application:

To use this input mode, you need to know about the hexadecimal code point values of the characters. Refer to The Unicode Standard, Version 2.0 for the mapping between code point values and characters. To input a character, type four hexadecimal digits, for instance, 00a1 for Inverted Exclamation Mark, 03b2 for Greek Small Letter Beta, ac00 for a Korean Hangul Syllable KA, 30a2 for Japanese Katakana Letter A, 4e58 for a Unified Han character and so on. Users can use both uppercase and lowercase letters of A, B, C, D, E, and, F for hexadecimal digits. If you mistype a digit or two, you can delete the digits by using the <Delete> key or the <Backspace> key.

Table Lookup Input Method Input Mode

To switch into table lookup input method input mode, type <Compose> <l> <l> from your current input mode. The input mode is displayed at the left bottom corner of your GUI application.

Once you turn on the input mode, there is a lookup window showing multiple candidates of Unicode characters. You can choose any one of the candidates by moving your pointer and clicking the left button on your mouse. You can also select any one of the candidates by choosing a left-hand- side letter associated with each of the candidates.

Once you are finished using the current input mode, you can switch into other input mode by typing corresponding input mode switch key sequence.

Input Mode Switch Key Sequence Summary

Starting in the Solaris 7 environment, , users can switch from one input mode to another without any restrictions. The following table shows the input mode switch key sequences for each input mode.

Printing

The en_US.UTF-8 locale provides a printing utility, xutops(1). This utility can print flat text files written in UTF-8 using X11 bitmap fonts available on the system. Because the output from the utility is standard PostScript, the output can be sent to any PostScript printer.

To use the utility, type the following:

system% xutops filename | lp

You can also use the utility as a filter since the utility accepts stdin stream:

system% lpr filename | xutops | lp

You can set the utility as a printing filter for a line printer. For example, the following command sequence tells the printer service LP that the printer lp1 accepts only xutops format files. This command line also installs the printer lp1 on port/dev/ttya. See the lpadmin(1M) man page for more details.

system# lpadmin -p lp1 -v /dev/ttya -I XUTOPS 
system# accept lp1
system# enable lp1

Using lpfilter(1M), you can add the utility as a filter as follows:

system# lpfilter -f filtername -F pathname

The command tells the printer that a converter (in this case, xutops) is available through the filter description file named pathname. Pathname can be as follows:

Input types: simple
Output types: XUTOPS
Command: /usr/openwin/bin/xutops

The filter converts the default type file input to PostScript output using /usr/openwin/bin/xutops.

To print a UTF-8 text file, use the following command:

system% lp -T XUTOPS UTF-8-file

DtMail

As a result of increased coverage in scripts, Solaris 7 DtMail running in the en_US.UTF-8 locale supports various MIME character sets shown below.

• US-ASCII (7-bit US ASCII)

• UTF-8 (UCS Transmission Format 8 of Unicode)

• UTF-7 (UCS Transmission Format 7 of Unicode)

• ISO-8859-1 (Latin-1)

• ISO-8859-2 (Latin-2)

• ISO-8859-3 (Latin-3)

• ISO-8859-4 (Latin-4)

• ISO-8859-5 (Latin/Cyrillic)

• ISO-8859-6 (Latin/Arabic)

• ISO-8859-7 (Latin/Greek)

• ISO-8859-8 (Latin/Hebrew)

• ISO-8859-9 (Latin-5)

• ISO-8859-10 (Latin-6)

• ISO-8859-15 (Latin-9)

• KOI8-R (Cyrillic)

• ISO-2022-JP (Japanese)

• ISO-2022-KR and EUC-KR (Korean)

• ISO-2022-CN (Simplified Chinese)

• ISO-2022-TW (Traditional Chinese)

This support allows users to view virtually any kind of email encoded in various MIME character sets from any region of the world in a single instance of DtMail. The decoding of received email is done by DtMail, which looks at the MIME character set and content transfer encoding provided with the email. However, in case of sending, you need to specify a MIME character set that is understood by the recipient mail user agent (in other words, mail client), unless you want to use the default MIME character set provided by the en_US.UTF-8 locale. To switch the character set of out-going email, at the 'New Message' window, type either <CONTROL> + <Y> or click the "Format" menu button and then again click on the "Change Char Set" button by using your mouse. The next available character set name will be displayed at left bottom corner on top of the Send button. If your email message header or message body contains characters that cannot be represented by the MIME charset specified, the system automatically switches the MIME character set to the UTF-8 that can represent any characters.

If your message contains characters from the 7-bit US-ASCII character set only, your email's default MIME character set is US-ASCII. Any mail user agent can interpret such email message without any loss of characters or information.

If your message contains characters from a mixture of scripts, your email's default MIME character set is UTF-8 and any 8-bit characters of UTF-8 is encoded with Quoted-Printable encoding.. For more detail on MIME, registered MIME charsets and Quoted-Printable encoding, refer to RFC 2045, 2046, 2047, 2048, 2049, 2279, 2152, 2237, 1922, 1557, 1555, and, 1489.

Programming Environment

Appropriately, internationalized applications should automatically enable the en_US.UTF-8 locale, but proper FontSet/XmFontList definitions in the application's resource file are required.

For information on internationalized applications, see Creating Worldwide Software: Solaris International Developer's Guide, 2nd edition.

Font Set Used with X Applications

The en_US.UTF-8 locale in the Solaris 7 environment supports fonts for the following character sets.

• ISO 8859-1

• ISO 8859-2

• ISO 8859-4

• ISO 8859-5

• ISO 8859-7

• ISO 8859-9

• ISO 8859-15

• BIG5

• GB 2312-1980

• JISX 0201.1976

• JISX 0208.1983

• KSC 5601.1992-3

• ISO 8859-6-1

• ISO 8859-8

• TIS 620.2533-1

Because the Solaris 7 environment supports the CDE desktop environment, each character set has a guaranteed sets of fonts.

The following is a list of the Latin-1 fonts that are supported in the Solaris 7 product.

• -dt-interface system-medium-r-normal-xxs sans utf-10-100-72-72-p-59-iso8859-1
• -dt-interface system-medium-r-normal-xs sans utf-12-120-72-72-p-71-iso8859-1
• -dt-interface system-medium-r-normal-s sans utf-14-140-72-72-p-82-iso8859-1
• -dt-interface system-medium-r-normal-m sans utf-17-170-72-72-p-97-iso8859-1
• -dt-interface system-medium-r-normal-l sans utf-18-180-72-72-p-106-iso8859-1
• -dt-interface system-medium-r-normal-xl sans utf-20-200-72-72-p-114-iso8859-1
• -dt-interface system-medium-r-normal-xxl sans utf-24-240-72-72-p-137-iso8859-1

For information on CDE common font aliases, including -dt-interface user-* and -dt-application-* aliases, see Common Desktop Environment: Internationalization Programmer's Guide.

In the en_US.UTF-8 locale, utf is also supported as a common font alias. A font set for an application should have a collection of fonts that contains each of the character sets, as in the following example.

fs = XCreateFontSet(display,
	"-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-1,
	 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-2,
	 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-5,
	 -dt-interface system-medium-r-normal-s*-utf*-*-*-*-*-*-*-iso8859-6,
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-7,
	 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-8,
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-9",
	 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-15",
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-big5-1",
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-gb2312.1980-0",
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-jisx0201.1976-0",
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-jisx0208.1983-0",
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-kcs5601.1992-3",
 -dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-tis620.2533-0",
    &missing_ptr, &missing_count, &def_string);

XmFontList Definition as CDE/Motif Applications

As with FontSet definition, the XmFontList resource definition of an application should also include each font of the character sets that the locale supports.

Example 4-1 XmNFontList definition for the en_US.UTF-8 locale

*fontList:\
-dt-interface system-medium-r-normal-s*-*-*-*-*-*-*-*-iso8859-1;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-2;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-4;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-5;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-7;\ 
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-8;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-9;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-iso8859-15;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-big5-1;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-gb2312.1980-0;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-jisx0201.1976-0;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-jisx0208.1983-0;\
-dt-interface system-medium-r-normal-s*utf-*-*-*-*-*-*-*-tis620.2533-0: 

For more details on the XmFontList and the XmNFontList, refer to the XmFontList(3X) man page, OSF/Motif Programmer's Guide and the resource section of each Motif widget in the OSF/Motif Programmer's Reference Manual.