|Summary:||Kochi Gothic, Mincho aren't used in English Unicode locales|
|Product:||[Fedora] Fedora||Reporter:||John Thacker <johnthacker>|
|Component:||fontconfig||Assignee:||Owen Taylor <otaylor>|
|Status:||CLOSED RAWHIDE||QA Contact:|
|Version:||rawhide||CC:||byte, eng-i18n-bugs, fedora-ja-list, petersen, wtogami, ynakai|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2005-10-23 17:00:12 UTC||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
|Bug Depends On:||111781, 150430|
Description John Thacker 2003-10-24 20:48:38 UTC
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031016 Description of problem: If the locale is set to, say, en_US.UTF-8, then Kochi Gothic and Kochi Mincho are never used to display Unicode documents. On systems without bitmap-fonts-cjk installed, this means something like MiscFixed will be used for hiragana and katakana, which is extremely ugly. It works fine with LANG=ja_JP.UTF-8 is set, however. The problem occurs when viewing files both in Mozilla and in other programs, like gedit or gucharmap, when the default "Sans" alias is selected as the font. (Mozilla also works fine when viewing Shift-JIS or EUC-JP files.) I'm not sure if this is a fontconfig problem or a font problem, but I do know that the fonts.cache-1 file for Kochi Gothic and Kochi Mincho doesn't list en (among others) as a supported language, whereas Microsoft's MS Gothic and MS Mincho files list lots of supported languages and work fine as well. The fangsong ti font in bitmap-fonts-cjk also lists lots of supported languages, and works well also. As long as your locale is set to some variation of en_US, including en_US.UTF-8, fontconfig will refuse to use Kochi Gothic and Kochi Mincho to display Japanese characters. This is bad, since those are the best Japanese fonts, and it ruins the point of using Unicode locales in the first place. Version-Release number of selected component (if applicable): ttfonts-ja-1.2-29 How reproducible: Always Steps to Reproduce: 1. export LANG=en_US.UTF-8 2. Set default font to Sans or Serif 3. View a webpage containing Unicode Japanese with mozilla, open a file with Unicode encoded Japanese in gedit, or run gucharmap and look at what font is used for Hiragana and Katakana for Sans Actual Results: Kochi Gothic and Kochi Mincho aren't used to display Japanese characters. Fangsong ti, if installed from bitmap-fonts-cjk, MS Gothic or MS Mincho, if installed, or MiscFixed will be used instead. Expected Results: Kochi Gothic and Kochi Mincho, as the preferred fonts for Japanese display, should be used. Additional info:
Comment 1 John Thacker 2003-10-25 04:29:47 UTC
To be more precise, fontconfig will select Kochi Gothic and Kochi Mincho, but only if no other fonts that can provide the glyphs and claim to support language "en" are available. However, since the bitmap-fonts RPM must always be installed for vte and hence for gnome-terminal, this means that the terrible (for Japanese) MiscFixed font will be preferred for katakana and hiragana over Kochi Gothic and Kochi Mincho. (Similarly, Fangsong ti will be preferred, as will the MS Japanese fonts if installed.)
Comment 2 John Thacker 2003-10-25 04:33:07 UTC
That is, most applications will eventually fallback to Kochi Gothic and Kochi Mincho, if no other fonts (including MiscFixed) are available. Mozilla is even worse-- it always completely refuses to use an "out of language" font, regardless of glyph presence, and thus won't use Kochi Gothic and Kochi Mincho for Unicode at all.
Comment 3 Leon Ho 2003-12-08 00:44:41 UTC
Maybe it is because Kochi Gothic's lang support does not listed en_US as one of the supported locale. MiscFixed on the other hand, support most of the langs (includes en_US). Is it the main reason, Owen?
Comment 4 John Thacker 2003-12-10 06:07:38 UTC
That is why, yes. There's been some discussion on the fontconfig list about the right way to handle this. The problem is that it's very natural to want to have a preferred font used by the Sans alias for viewing Japanese (and similarly for other languages), and there seems to be no particular reason to insist that the font used for viewing Japanese be able to handle English as a supported locale, if it's only used for Japanese characters not provided in English fonts. The problem is figuring out the right way for fontconfig to handle it. It's even worse when dealing with files in Unicode, which doesn't have a language associated with the encoding. Owen and Keith Packard have discussed it some. In the short run, adding broader coverage to Kochi Gothic and Kochi Mincho would help. In the long run, fontconfig's behavior is clearly broken. (It's unreasonble to expect that all fonts for all languages contain all necessary Japanese glyphs so that fontconfig will be happy when selecting them for users in Japanese locales, for example.)
Comment 5 John Thacker 2004-01-29 17:41:11 UTC
pango-1.3.2, freetype-2.1.7 and other updates from rawhide/development seem to go a long way towards resolving this problem. I've also been running fontconfig-2.2.92, but that didn't seem to really resolve it until the pango and freetype updates. In any case, closing as RAWHIDE, since it looks resolved using development for FC2 right now.
Comment 6 Giovanni Glass 2004-10-06 12:05:17 UTC
Created attachment 104832 [details] Screenshot of kana not using the same font as kanji This is from Firefox PR10 and Fedora Core 3 Test2 (patched to current as of Oct. 6, 2004) This is a screenshot of google using English unicode with Japanese search results. I increased the font size to emphasize how all the kana are not using the same font as the kanji and how they don't scale or become anti-aliased. This is NOT fixed and it has been happening since Fedora Core 1 (I run this at work). SJIS and ISO-2022-JP doesn't seem to have this problem. I can click into any of these Japaense search results and they don't have the messed up kana like in the screenshot.
Comment 7 John Thacker 2004-10-06 13:28:04 UTC
Try putting the following in your /etc/fonts/local.conf file. It will cause fontconfig (at least on 2.2.96, but probably on the FC included version as well) to match on Kochi Gothic and Kochi Mincho ignoring language. ("Strong" binding matches ignoring language.) <match target="pattern"> <test name="family"> <string>sans-serif</string> </test> <edit name="family" binding="strong" mode="prepend"> <string>Luxi Sans</string> <string>Bitstream Vera Sans</string> <string>Verdana</string> <string>Kochi Gothic</string> </edit> </match> <match target="pattern"> <test name="family"> <string>serif</string> </test> <edit name="family" binding="strong" mode="prepend"> <string>Bitstream Vera Serif</string> <string>Times New Roman</string> <string>Kochi Mincho</string> </edit> </match> <match target="pattern"> <test name="family"> <string>monospace</string> </test> <edit name="family" binding="strong" mode="prepend"> <string>Luxi Mono</string> <string>Bitstream Vera Sans Mono</string> <string>Andale Mono</string> <string>Kochi Gothic</string> </edit> </match>
Comment 8 Giovanni Glass 2004-10-07 05:39:02 UTC
I tried that out. It works! Is this a hack or something that can be commited into RAWHIDE CVS?
Comment 9 Giovanni Glass 2004-10-07 05:48:54 UTC
Created attachment 104882 [details] After the change suggested to the local.conf file Looks good and it scales correctly.
Comment 10 John Thacker 2004-10-07 13:04:14 UTC
Well, it is a bit of a hack. It's very much a "Western user who also uses Japanese"-centric solution, and it also means that the language of one's locale (or the language associated with a charset in Mozilla) will not be used to choose the proper font. It will result in the Japanese Kochi fonts always being preferred over Chinese (and Korean) fonts for displaying Han characters. For Unicode, especially for English-based Unicode locales, there's no way around this. This results in Japanese fonts being used for specifically Chinese webpages and Chinese charsets as well, from what I remember. This does create problems with the glyphs which have different representations in the two languages. Also, since characters which are rare in Japanese and don't exist in the Kochi fonts will be selected from the Chinese fonts, a somewhat inconsistent character style sometimes results as well. Also, this results in purely Japanese pages with Japanese charsets using Luxi Sans (for Sans, and so on for Serif) for the (non-doublewidth) Roman characters and Kochi fonts for other things, rather than using Kochi fonts for everything when possible. Furthermore, regardless of one's locale, Luxi Sans will be used for all characters it contains, rather than a language-specific font. When language-specific fonts have Latin characters designed to blend in with their other glyphs better, this will mean slightly worse looking characters.
Comment 11 Owen Taylor 2004-10-07 15:59:13 UTC
Certainly not something that would go into the default fonts.conf. It's conceivable we could tighten up the set of required characters to support 'en' if someone wanted to investigate what the Kochi fonts are missing. This problem will go away with the change of Mozilla to use Pango, since Pango is smarter about selecting fonts. (It will realize that an 'en' language makes no sense for Han/Kana characters and subsitute it with 'ja' for Kana or 'xx' for Han characters)
Comment 12 John Thacker 2004-10-08 14:27:34 UTC
The required characters to support 'en' according to fontconfig which are missing are all accented characters sometimes used (but sometimes dropped) for foreign words which have been adopted by English. They are (thanks to fontconfig source and gucharmap): 0x00C0 (Capital Letter A with Grave) 0x00C7 (Capital Letter C with Cedilla) 0x00C8 (Capital Letter E with Grave) 0x00C9 (Capital Letter E with Acute) 0x00CA (Capital Letter E with Circumflex) 0x00CB (Capital Letter E with Diaeresis) 0x00CF (Capital Letter I with Diaeresis) 0x00D1 (Capital Letter N with Tilde) 0x00D4 (Capital Letter O with Circumflex) 0x00D6 (Capital Letter O with Diaeresis) 0x00E0 (Small Letter A with Grave) 0x00E7 (Small Letter C with Cedilla) 0x00E8 (Small Letter E with Grave) 0x00E9 (Small Letter E with Acute) 0x00EA (Small Letter E with Circumflex) 0x00EB (Small Letter E with Diaeresis) 0x00EF (Small Letter I with Diaeresis) 0x00F1 (Small Letter N with Tilde) 0x00F4 (Small Letter O with Circumflex) 0x00F6 (Small Letter O with Diaeresis) If these characters could somehow be added to the Kochi fonts, the fonts would support English according to fontconfig.
Comment 13 Owen Taylor 2004-11-18 18:14:06 UTC
*** Bug 138783 has been marked as a duplicate of this bug. ***
Comment 14 Need Real Name 2004-12-14 00:09:31 UTC
I've hacked at fontconfig today to try and solve this bug. What I did was to ammend the strict sort order used by FcFontSort so that it 'satisfies' the language specified in the pattern by locating the best matching font supporting each pattern language and then ignores language in the remaining fonts for purposes of matching. So, when asking for 'sans:lang=en', you'll get an English font first, and then the remaining fonts sorted with respect to the 'sans' alias alone -- pushing Kochi fonts ahead of other English-supporting Han fonts. This change is in Fontconfig HEAD CVS; if someone would like to play and see if it fixes the problems for them, that would be great.
Comment 15 John Thacker 2004-12-28 22:06:21 UTC
I've been home for the holidays since the change was made, and have finally got a chance to test it. IMO, this solves all the problems that can be rationally and consistently solved in a default configuration, and solves everything that is fontconfig's problem. Most remaining issues I can think of are clearly issues between pango, user applications, documents, and individual configuration. (For example, in an English Unicode locale, view a UTF-8 document that isn't labeled for language but is mostly Japanese. Should the Latin characters be rendered with the Japanese font used for everything else, or with the preferred English-supporting font? Similarly problems of preferring a Japanese vs. Chinese font for Han characters in an unspecified document when in a non-CJK locale are up to individual users.) Now we just need a fontconfig release, and inclusion of that release into Fedora Core and RHEL. I know RH prefers to ship stable versions over development versions, so even though I find the 2.2.9x series very stable, I'd be surprised if RH included it. What's preventing a 2.3.0 release?
Comment 16 Jens Petersen 2005-03-01 11:05:39 UTC
I tried fontconfig-2.2.99 and it seems to fix bug 138783 too . :)
Comment 17 John Thacker 2005-03-03 16:43:31 UTC
So, since 2.3.0 is released, I assume that Owen is working hard on editing the various packages and default configuration, and we'll see it in RawHide soon, right? ;)
Comment 18 John Thacker 2005-03-10 16:28:31 UTC
Can we *please* update to 2.3.0 or 2.3.1 ( http://fontconfig.org/release/ ) in time for FC4? It fixes this (and some other important i18n and font rendering problems).
Comment 19 Jens Petersen 2005-04-19 07:35:01 UTC
Still appears to be fontconfig-2.2.3 in fc devel...
Comment 20 David Zeuthen 2005-04-19 20:44:53 UTC
Owen told me he wanted to do this upgrade so I'm reassigning the bug to him.
Comment 21 Warren Togami 2005-05-16 09:14:59 UTC
If Sopwith does not accept the version upgrade (which is highly unlikely now given that regressions in fontconfig will affect many people and we have little time left before final), can the fix in 2.3.x be backported into our current fontconfig? Then that'll be our short-term solution. Adding FC5Blocker for the long term solution, upgrade fontconfig.
Comment 22 Owen Taylor 2005-05-16 19:52:23 UTC
The code portion is big enough that a backport, in my opinion, would be riskier than a straight upgrade, and create significant deviation from upstream, something that we want to avoid in Fedora.
Comment 23 Mike A. Harris 2005-06-02 21:28:15 UTC
Moving to FC5Target, as FC4 is 'done', barring release blockers that pop up.
Comment 24 Akira TAGOH 2005-09-28 08:58:17 UTC
This bug could be changed to MODIFIED since we got fontconfig 2.3.2-1 on rawhide now.
Comment 25 Jens Petersen 2005-09-28 09:09:13 UTC
Light testing looks good to me.
Comment 26 John Thacker 2005-10-23 17:00:12 UTC
Yes, this works for me too. I'll close it as fixed in Rawhide.