Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 162852 - Character handling in squirrelmail
Summary: Character handling in squirrelmail
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: squirrelmail
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Warren Togami
QA Contact:
URL: https://sourceforge.net/tracker/?func...
Whiteboard:
: 167877 (view as bug list)
Depends On:
Blocks: 171491 FC5Blocker
TreeView+ depends on / blocked
 
Reported: 2005-07-10 13:56 UTC by Nicolas Mailhot
Modified: 2007-11-30 22:11 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2006-03-04 13:44:56 UTC


Attachments (Terms of Use)
Problem screenshot (deleted)
2005-09-14 06:13 UTC, Nicolas Mailhot
no flags Details
In case this one still applies... (deleted)
2006-02-28 22:51 UTC, Nicolas Mailhot
no flags Details | Diff

Description Nicolas Mailhot 2005-07-10 13:56:02 UTC
Description of problem:

It seems squirrelmail is still stuck in the bad-old-days of multiple charsets,
with the main developpers lacking the will to enforce UTF-8 accross translators.

As a result every translation forces a single encoding and the default charset
parameter is ignored for everything except en_US.

Of course that also means if you reply to a mail someone sent you and your UI
translation forces a charset different from the one in this mail your quotes
will be b0rked.

Since upstream seems decided to let the mess perdure, the Fedora package should
convert all translations to UTF-8 so FC squirrelmail interracts properly with
the world (and not just the language zone the UI translation uses)

(See squirrelmail bug #1235345 for upstream position)

Comment 1 Warren Togami 2005-09-04 09:56:42 UTC
Closing this as this is work required upstream and not Fedora's sole responsibility.

Comment 2 David Woodhouse 2005-09-04 10:06:00 UTC
Do you have a reference to the squirrelmail bug (apparently #1235345)? I don't
seem to be able to find it.

Comment 3 Warren Togami 2005-09-09 22:30:02 UTC
*** Bug 167877 has been marked as a duplicate of this bug. ***

Comment 4 Nicolas Mailhot 2005-09-11 20:20:23 UTC
Fedora is the party that cares about software being UTF-8 compatible.
Not handling UTF-8 well was sufficient ground for removal in the past.

Unless this Red Hat policy changed (which I doubt) if no one steps up to do the
UTF-8 work, suirrelmail should leave FC IMHO.

Which would be a pity as I use and like it.

Comment 6 Warren Togami 2005-09-11 21:51:21 UTC
> Unless this Red Hat policy changed (which I doubt) if no one steps up to do 
> the UTF-8 work, squirrelmail should leave FC IMHO.

Agreed.

> Which would be a pity as I use and like it.

I know. =(

Comment 7 jørgen nørgaard 2005-09-12 07:26:23 UTC
From http://www.squirrelmail.org/wiki/en_US/WishList, where it is stated:

"From 1.5.1cvs and 1.4.4cvs squirrelmail allows setting default charset when default language is en_US"

Sounds like there should be a solution, though I still have to figure out how it is actually done, just saw it 
and I dont have time right now.

I surely would miss squirrelmail if it went away from FC. It understandable if it is removed from a Redhat 
product on the other hand.



Comment 8 Warren Togami 2005-09-12 08:26:27 UTC
It is sure a pity because I used squirrelmail for years, but it would probably
move to Extras if it does become removed.  If you are aware of a standards
compliant replacement that is secure and proven mature then please chime in here.

Comment 9 David Woodhouse 2005-09-12 08:49:36 UTC
Shouldn't be too hard just to convert all the locales to UTF-8 while we're
building the package. That ensures that their charset is a superset of the
charset in any mail they'll read and reply to, and was probably the right thing
to do anyway. 

Now, if I could just work out what bloody charset the ko_KR help files are
actually in at the moment, since iconv claims they're not valid euc-kr...

Comment 10 Fredrik Tolf 2005-09-12 09:24:23 UTC
#8: May I point you to Dolda Webmail (<http://doldawebmail.sf.net/>)? It is not
quite as feature complete or mature as Squirrelmail, but on the other hand, it
is perfectly compliant with UTF-8, and has some other advantages over
Squirrelmail (the most significant of which are listed on the home page).

I know that it is kind of shameless for me to advertise my own project like this
(especially since it doesn't even have an RPM specfile), but I thought it could
still be a good thing since it has precisely that which Squirrelmail seems to be
lacking. Internationalization was something that I kept a high priority while
writing it.


Comment 11 David Woodhouse 2005-09-12 10:13:07 UTC
squirrelmail-1.4.6-0.cvs20050812.2.fc5 has everything converted to UTF-8 (apart
from ko_KR as discussed). Please test.

Comment 12 Nicolas Mailhot 2005-09-13 20:18:44 UTC
I've lost the French localisation with this version and accented letters are
borked in my mail folders (�l�ments envoy�s for example)

I can't test the rest - without locale it's reverted to en_US which is already
UTF-8 compliant

Comment 13 Rahul Sundaram 2005-09-13 20:33:52 UTC
(In reply to comment #10)
> #8: May I point you to Dolda Webmail (<http://doldawebmail.sf.net/>)? It is not
> quite as feature complete or mature as Squirrelmail, but on the other hand, it
> is perfectly compliant with UTF-8, and has some other advantages over
> Squirrelmail (the most significant of which are listed on the home page).
> 
> I know that it is kind of shameless for me to advertise my own project like this
> (especially since it doesn't even have an RPM specfile), but I thought it could
> still be a good thing since it has precisely that which Squirrelmail seems to be
> lacking. Internationalization was something that I kept a high priority while
> writing it.
> 

(Not relevant to the report)

Good to know. Nothing shameful about being proud of your own work ;-). Might
want to take a look at packaging this for Fedora Extras repository

http://fedoraproject.org/wiki/Extras

Then do more evangelisation and let the user choose

Comment 14 David Woodhouse 2005-09-13 21:27:16 UTC
Hm, with squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm if I hit shift-reload
on a mail page it alternates between French and English. With the new one, it's
always English. 

Comment 15 David Woodhouse 2005-09-13 21:42:40 UTC
Hm. Confused now. First the previous version started working every time instead
of every other time, then I updated parts of it individually so see when it
stopped working.... and then I updated en masse to the new version and it's
_still_ showing me French every time.


Comment 16 David Woodhouse 2005-09-13 22:08:29 UTC
Definitely seems to be working in the new package now, with utf-8

http://david.woodhou.se/squirrelmail-view.jpeg
http://david.woodhou.se/squirrelmail-reply.jpeg

Comment 17 Nicolas Mailhot 2005-09-14 06:11:58 UTC
Did you try the mock-built FE package or your own one ?

[root@rousalka nim]# rpm -Uvh --force
/var/cache/yum/development/packages/squirrelmail-1.4.6-0.cvs20050812.2.fc5.noarch.rpm
Préparation...              ########################################### [100%]
   1:squirrelmail           ########################################### [100%]
[root@rousalka nim]# exit
exit
[nim@rousalka ~]$ rpm -Vq squirrelmail
S.5....T. c /etc/httpd/conf.d/squirrelmail.conf
S.?....T. c /etc/squirrelmail/config.php
..?...... c /etc/squirrelmail/config_local.php
..?...... c /etc/squirrelmail/default_pref
missing     /var/lib/squirrelmail/prefs/default_pref
[nim@rousalka ~]$ su
Password:
$[root@rousalka nim]# /etc/init.d/httpd restart
Arrêt de httpd :                                           [  OK  ]
Démarrage de httpd :                                       [  OK  ]


It's still not working there -> see the attached screenshot

Comment 18 Nicolas Mailhot 2005-09-14 06:13:24 UTC
Created attachment 118788 [details]
Problem screenshot

Comment 19 David Woodhouse 2005-09-14 14:10:23 UTC
It's not built in mock -- it's a Fedora Core package. I was trying my local
version. However, even with the beehive-built
squirrelmail-1.4.6-0.cvs20050812.1.fc5 I have bizarre problems with fr_FR locale. 

I've changed both my own preferences and /etc/squirrelmail/config.php to
'French' and fr_FR respectively.

If I view a mailbox index and keep clicking 'Previous' (or 
'Précédent' to go to another page, precisely every eighth page is in French; the
other seven are in English. If I middle-click on a mail to show it in a new tab,
then keep hitting shift-reload, then one in every _four_ pages is French. 

Was this not happening for you in squirrelmail-1.4.6-0.cvs20050812.1.fc5? If
not, please could you try moving selected parts of the old package into the new
until it stops working, and let me know? The interesting bits are
/usr/share/squirrelmail/functions/i18n.php,
/usr/share/squirrelmail/locale/fr_FR, and /usr/share/squirrelmail/help/fr_FR

Comment 20 Nicolas Mailhot 2005-09-14 14:19:47 UTC
I never had local switching problems before. I'll try to test what you ask this
evening, if I find the time and the previous SM package.

If I can't do it this evening though it'll have to wait for monday

Comment 21 David Woodhouse 2005-09-14 14:21:01 UTC
It's probably gone from the rawhide mirrors by now.
http://david.woodhou.se/squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm

Comment 22 Nicolas Mailhot 2005-09-14 17:38:37 UTC
Well, as soon as I inject the new files I loose the locale.
There is probably some magic to be done there - don't anyone know a sympathetic
SM developper ?

Comment 23 David Woodhouse 2005-09-14 19:45:00 UTC
Which new files? functions/i18n.php, locale/fr_FR/setup.php, the .mo files, the
help files?

Comment 24 Nicolas Mailhot 2005-09-14 20:52:05 UTC
just injecting the squirrelmail-1.4.6-0.cvs20050812.2.fc5.noarch.rpm
functions/i18n.php and locale dir in
squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm seems to remove access to the
french locale

Comment 25 David Woodhouse 2005-09-14 20:54:04 UTC
What about _only_ i18n.php? 
What about _only_ locale/fr_FR/setup.php?
What about _only_ the contents of locale/fr_FR/LC_MESSAGES/ ?

Comment 26 Nicolas Mailhot 2005-09-14 21:04:16 UTC
I'll try the various permutations, but not this night and probably not till
monday. I have a guest that's eating much of my free time

Comment 27 Nicolas Mailhot 2005-10-27 10:16:34 UTC
Ok I've found some time to look at this problem.
I've found one thing: during the UTF-8 change this line was added to the spec:
find -name '*.mo' |xargs rm

Turns out SM is using mofiles not pofiles so this line kills all translations
(rebuilt a package with this line commendted out and translations were restored)

If someone is ready to patch SM to use pofiles this line can be removed (do not
forget to fix the local test in configtest) but in the meanwhile it must be kept

Comment 28 Nicolas Mailhot 2005-10-27 10:18:14 UTC
I still have string problems with this change BTW (conversion of folder names
with accented letters) but 80% of the problems are fixed with this simple revert

Comment 29 Nicolas Mailhot 2005-10-27 10:30:19 UTC
Folder handling seems badly b0rked WRT folders with non 7bit ascii names

Just create a folder in a sane client that uses characters in the rest of the
UTF-8 range and SM will go mad when trying to display it. Unfortunately many
languages have some default folders with non ascii-letters, SM got by so far by
assuming ISO 8bit encodings

Comment 31 Nicolas Mailhot 2005-10-27 13:24:22 UTC
Upstream says mbstrings is needed for UTF-8 folders and testing confirms this.

So a full package fix is :
1. do not remove mo files 
2. add a require on php-mbstring

With this SM is a happy camper on my box. (French UTF-8 locale)

-> adding to FC5 tracker

Comment 32 Nicolas Mailhot 2005-10-27 13:26:35 UTC
And upstream adds this warning :

« One more thing. Be careful  in switching Japanese to utf-8.
It is not enough to convert translation files to utf-8. Some
Japanese specific code uses euc-jp and iso-2022-jp charsets. »

Comment 33 Nicolas Mailhot 2005-11-12 18:56:44 UTC
Escalading to FC5Blocker since the fix is known

Comment 34 Nicolas Mailhot 2006-01-17 22:16:01 UTC
Ok, the problem is fixed now thanks

However i've quickly reverted to my own SM package since it seems the old FC5
snapshot does not like rawhide dovecot very much (I'm using
squirrelmail-1.4.6-0.cvs20051204 today). Some error about imap atoms

Will try to do a proper bug report this week -> bedtime

Comment 35 Nicolas Mailhot 2006-01-17 22:16:56 UTC
Ok, the problem is fixed now thanks

However I've quickly reverted to my own SM package since it seems the old FC5
snapshot does not like rawhide dovecot very much (I'm using
squirrelmail-1.4.6-0.cvs20051204 today). Some error about imap atoms

Will try to do a proper bug report this week -> bedtime

Comment 36 Warren Togami 2006-02-28 21:43:22 UTC
This is causing a bit of a mess as I attempt to upgrade it to 1.4.6 in order to
gain security fixes.  dwmw2 can you help redo this for 1.4.6 and the new locale
package?

*WHY* has upstream not done this themselves?  pain...



Comment 37 David Woodhouse 2006-02-28 21:49:30 UTC
It was just a global search and replace and iconv -- for each locale file,
change it to claim to be UTF-8, and use iconv to make that true. Except for the
Korean ones, iirc, which didn't seem to be in any valid character set that iconv
liked.

I'll take a look -- what's the problem?

Comment 38 Nicolas Mailhot 2006-02-28 22:44:56 UTC
The problem is upstream is slowly converting to utf-8 one at a time so you have
to re-do the locale patches every time

Warren, I have a more current SRPM if you want (cvs20060118)
I needed it since the last openssl changes broke imaps with older squirrelmail
versions (I know I should have reporter it way back)

Comment 39 Nicolas Mailhot 2006-02-28 22:51:29 UTC
Created attachment 125428 [details]
In case this one still applies...

Comment 40 Nicolas Mailhot 2006-02-28 22:53:59 UTC
(Upstream considers each locale maintainer is free to choose his preferred
encoding, that's why it's a big changing mess. Fortunately some sanity seems to
begin to prevail and locales are being converted to UTF-8)

Comment 41 David Woodhouse 2006-02-28 23:05:51 UTC
(In reply to comment #38)
> The problem is upstream is slowly converting to utf-8 one at a time so you have
> to re-do the locale patches every time

Ah. In that case we should probably just script that change too instead of using
a patch for it?


Comment 42 Warren Togami 2006-03-01 04:06:47 UTC
If you know exactly how to do it, then please help me on this.  I'm wanting to
just dump squirrelmail entirely.

And yes, scripting the change (excluding Korean) is ideal instead of a patch
that will repeatedly break with upstream changes.

Comment 43 David Woodhouse 2006-03-01 10:11:55 UTC
OK, I just checked something in -- take a look and see if you like it.

Comment 44 Warren Togami 2006-03-02 22:58:49 UTC
Seems good in English and Japanese.  It will be in FC5 tomorrow, and will
probably push FC4 update tomorrow too.


Comment 45 Nicolas Mailhot 2006-03-03 21:36:57 UTC
I'm afraid this release (1.4.6-1.fc5) is very broken.
A good test procedure (the one I use) is the following :

1. install dejavu from FE to get good cyrillic & greek coverage
2. configure the gnome keyboard switching applet to have an alternate greek or
russian keyboard (additionnaly people are currently trying to nail down the last
boogs there - more testers will help)
3. go into sm, send yourself a mail with cyrillic or greek glyphs in the title
and text
4. check sm displays it all right (you don't need to understand the text, just
recognize what you typed)
5. use another MUA to check it agrees with SM (at least evo and firefox) <-
crucial step, sm often does double mistakes wich result in gibberish only the
broken version can read

6. get an IMAP account somewhere
7. try to create a folder with greek or cyrillic letters in it
8. check the result in SM (refresh)
9. check the result in SM and thunderbird

Your release fails 5. and 9. My old jan 18 2006 cvs snapshot build OTOH succeeds
- I'll try now to isolate the difference 

Comment 46 David Woodhouse 2006-03-03 21:54:26 UTC
Thanks for the testing. Could you describe the failure? What was wrong with the
mail in #5? Was it actually valid UTF-8 with an invalid charset header, or was
something else wrong? Can you send me an example? Did your web browser actually
operate in UTF-8 mode while you were doing this? Replying to a known-good mail
containing non-ASCII characters, so that those characters get quoted, is also a
good test.

What was the name of the directory you tried to create, and what _actually_ got
created (the horrid IMAP UTF-7 version)?

Comment 47 Nicolas Mailhot 2006-03-03 22:09:53 UTC
I've already identified one problem in the spec file: function/i18n.php is
munged *after* a copy has already been "installed", so the munging has no effect

This basic error is made possible by the fact the spec does not cleanly separate
file processing from file installation, and mixes all in %install. Moving
processing to a %build fwould certainly bring some sanity there

I'll now test a rebuilt rpm to see if it was the core problem

Comment 48 David Woodhouse 2006-03-03 22:28:18 UTC
OK, I just checked in http://david.woodhou.se/squirrelmail.spec

Better?

Comment 49 Nicolas Mailhot 2006-03-03 22:48:21 UTC
looks better, I'll test it

With the previous spec I can confirm the ordering of the i18n.php sed-ing was
the problem (on my box, with my test procedures)

Comment 50 Nicolas Mailhot 2006-03-03 23:01:45 UTC
Ok, your spec works for me
Not that it proves a lot of things
But at least it's not broken the same way as the previous FC SM packages


Note You need to log in before you can comment on or make changes to this bug.