Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 158665 - ppmquant fails with "rc=11" because of segfault in pnmcolormap
Summary: ppmquant fails with "rc=11" because of segfault in pnmcolormap
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: netpbm
Version: 3
Hardware: i386
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Jindrich Novy
QA Contact: Ben Levenson
URL:
Whiteboard:
: 139111 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2005-05-24 17:56 UTC by Josh Parsons
Modified: 2013-07-02 23:07 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-06-01 11:53:27 UTC


Attachments (Terms of Use)
Patch to fix the pnmcolormap segfault (deleted)
2005-05-31 13:06 UTC, Jindrich Novy
no flags Details | Diff

Description Josh Parsons 2005-05-24 17:56:39 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Fedora/1.0.4-1.3.1 Firefox/1.0.4

Description of problem:
ppmquant fails with "rc=11" on every input I've been able to give it.

Version-Release number of selected component (if applicable):
netpbm-progs-10.27-3.FC3

How reproducible:
Always

Steps to Reproduce:
ppmpat -madras 16 16 |/usr/bin/ppmquant 2 >/dev/null

Actual Results:  pnmcolormap: making histogram...
pnmcolormap failed, rc=11


Expected Results:  pnmcolormap: making histogram...
pnmcolormap: 3 colors found
pnmcolormap: choosing 2 colors...
pnmremap: 2 colors found in colormap


Additional info:

Comment 1 Jindrich Novy 2005-05-25 08:55:52 UTC
This is weird. This just works for me with netppm-10.27-3. My output is:

[jnovy@obelix FC-3]$ ppmpat -madras 16 16 |/usr/bin/ppmquant 2 >/dev/null
pnmcolormap: making histogram...
pnmcolormap: 3 colors found
pnmcolormap: choosing 2 colors...
pnmremap: 2 colors found in colormap
[jnovy@obelix FC-3]$

It looks like it fails in computeHistogram() in pnmcolormap.c. From a brief look
into the code I see it might fail in seek/tell file operations called via the
libpam interface. However I cannot reproduce it. Have you checked free space on
your disc?

Comment 2 Ben Webb 2005-05-26 00:06:39 UTC
Same problem here on two FC3 boxes: with netpbm-10.25-2 it works:
[ben@synth doc_INT]$ ppmpat -madras 16 16 | /usr/bin/ppmquant 2 > /dev/null
pnmcolormap: making histogram...
pnmcolormap: 3 colors found
pnmcolormap: choosing 2 colors...
pnmremap: 2 colors found in colormap

while with netpbm-10.27-3.FC3 it fails:
[ben@synth doc_INT]$ ppmpat -madras 16 16 | /usr/bin/ppmquant 2 > /dev/null
pnmcolormap: making histogram...
pnmcolormap failed, rc=11

Simply downgrading to 10.25-2 again makes the problem go away, so it can't be a
disk space problem.

Comment 3 Josh Parsons 2005-05-26 22:48:46 UTC
I have plenty of free disk space - but the problem is getting worse.  Now
pnmcolormap is dumping core:

$ ppmpat -madras 16 16 |pnmcolormap all >/dev/null
pnmcolormap: making histogram...
Segmentation fault

$ ppmpat -madras 16 16 >test.ppm
$ gdb --args /usr/bin/pnmcolormap all test.ppm

...

Program received signal SIGSEGV, Segmentation fault.
0x414a4513 in pnm_freetupletable () from /usr/lib/libnetpbm.so.10
(gdb) bt
#0  0x414a4513 in pnm_freetupletable () from /usr/lib/libnetpbm.so.10
#1  0x414a46a0 in pnm_computetuplefreqtable2 () from /usr/lib/libnetpbm.so.10
#2  0x08048ecb in ?? ()
#3  0xbfffeb10 in ?? ()
#4  0x00000000 in ?? ()


Comment 4 Jindrich Novy 2005-05-30 14:32:30 UTC
Yes, this error is caused by the segfault in pnmcolormap call to tuplehashtotable():

Program received signal SIGSEGV, Segmentation fault.
0x0000002a9559e93e in tuplehashtotable (pamP=0x7fbffff690,
tuplehash=0x2a95559010, allocsize=32767) at libpammap.c:431

431                     tupletable[j]->value = p->tupleint.value;
(gdb) bt
#0  0x0000002a9559e93e in tuplehashtotable (pamP=0x7fbffff690,
tuplehash=0x2a95559010, allocsize=32767) at libpammap.c:431
#1  0x0000002a9559eafb in pnm_computetuplefreqtable2 (pamP=0x7fbffff690,
tupleArray=0x0, maxsize=32767, newMaxval=255, countP=0x7fbffff4e4)
    at libpammap.c:538
#2  0x000000000040254c in computeHistogram (pamP=0x7fbffff690,
colorfreqtableP=0x7fbffff4e8, colorsP=0x7fbffff4e4, freqtableMaxvalP=0x7fbffff4d8)
    at pnmcolormap.c:655
#3  0x00000000004025fb in computeColorMapFromInput (pamP=0x7fbffff690,
allColors=1, reqColors=0, methodForLargest=LARGE_NORM, methodForRep=REP_CENTER_BOX,
    colormapP=0x7fbffff538, colorsP=0x7fbffff544, colormapMaxvalP=0x7fbffff530)
at pnmcolormap.c:713
#4  0x0000000000402c78 in main (argc=3, argv=0x7fbffff8f8) at pnmcolormap.c:907



Comment 5 Jindrich Novy 2005-05-30 14:41:22 UTC
*** Bug 139111 has been marked as a duplicate of this bug. ***

Comment 6 Jacques Rodary 2005-05-31 00:06:12 UTC
sorry: bug 139111 is definetly not a duplicate of this one: I had no problem with
netpbm-10.26.1, which isn't available anymore.   

Comment 7 Jacques Rodary 2005-05-31 08:02:39 UTC
and downgrading to 10.25-2 doesn't work for my problem of image conversion in
latex2html: I had to downgrade to netpbm-9.24-10.90.1

Comment 8 Jindrich Novy 2005-05-31 13:06:28 UTC
Created attachment 114988 [details]
Patch to fix the pnmcolormap segfault

The segfault is caused by the fact that variable "retval" is declared twice in
pnm_alloctupletable(), first in the global function context and secondly in a
separate block, where the memory is allocated. The culprit is the global
"retval" which is returned uninitialized. This can be seen from here:

tupletable
pnm_alloctupletable(const struct pam * const pamP,
		    unsigned int       const size) {

    tupletable retval;

    if (UINT_MAX / sizeof(struct tupleint) < size)
	pm_error("size %u is too big for arithmetic", size);
    else {
	unsigned int const mainTableSize = size * sizeof(struct tupleint *);
	unsigned int const tupleIntSize =
	    sizeof(struct tupleint) - sizeof(sample)
	    + pamP->depth * sizeof(sample);

	/* To save the enormous amount of time it could take to allocate
	   each individual tuple, we do a trick here and allocate everything
	   as a single malloc block and suballocate internally.
	*/
	if ((UINT_MAX - mainTableSize) / tupleIntSize < size)
	    pm_error("size %u is too big for arithmetic", size);
	else {
	    void * pool;
	    tupletable retval;
	    unsigned int i;

	    pool = malloc(mainTableSize + size * tupleIntSize);

	    retval = (tupletable) pool;

	    for (i = 0; i < size; ++i)
		retval[i] = (struct tupleint *)
		    ((char*)pool + mainTableSize + i * tupleIntSize);
	}
    }
    return retval;
}

The attached one-liner fixes this. It seems this mistake was for ages in the
upstream netpbm as I can see from Jacques Rodary's comment.

Comment 9 Jindrich Novy 2005-05-31 13:08:10 UTC
This is now fixed in devel and FC3 branch.

I'll close this ERRATA when FC3 update is released.

Comment 10 Jacques Rodary 2005-05-31 14:56:53 UTC
It's OK now with netpbm-10.27-4.FC3 already available with FC3-updates. Many
thanks. 


Note You need to log in before you can comment on or make changes to this bug.