Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1064825 - gcc address sanitizer reports global-buffer-overflow when compiled with optimization
Summary: gcc address sanitizer reports global-buffer-overflow when compiled with optim...
Alias: None
Product: Fedora
Classification: Fedora
Component: gcc
Version: 20
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: Jakub Jelinek
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2014-02-13 11:32 UTC by Thomas Sondergaard
Modified: 2014-02-13 17:29 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2014-02-13 17:29:35 UTC

Attachments (Terms of Use)
Example that shows the problem (deleted)
2014-02-13 11:32 UTC, Thomas Sondergaard
no flags Details

System ID Priority Status Summary Last Updated
GNU Compiler Collection 60183 None None None Never

Description Thomas Sondergaard 2014-02-13 11:32:45 UTC
Created attachment 862746 [details]
Example that shows the problem

Description of problem:
asan_symbolizer reports global-buffer-overflow in asan_bug/d3des.c:376, but only when compiled with optimization

Version-Release number of selected component (if applicable):
gcc (GCC) 4.8.2 20131212 (Red Hat 4.8.2-7)

Also present in gcc (GCC) 4.8.1 20130715 (Red Hat 4.8.1-4) from rhel6 devtoolset-2.

How reproducible:
Fully reproducible

Steps to Reproduce:
1. untar attachment
2. make sure you have from llvm/clang in path
3. run make
4. Modify Makefile to disable optimization (remove -O2)
5. run make

Actual results:
asan reports global-buffer-overflow in asan_bug/d3des.c:376 when optimization is enabled and doesn't when optimization is disabled

Expected results:
asan results should be consistent independent of optimization levels. The d3des.{h,c} files I have from realvnc, but tigervnc and several other realvnc derived projects use the same. As far as I can see the code does not overflow the buffer.

The code in d3des.c around line 376 looks like this:
    unsigned long keys = KnL; // via function parameter
    for( round = 0; round < 8; round++ ) {
      work ^= *keys++;
      work  = right ^ *keys++;
      work ^= *keys++;
      work  = leftt ^ *keys++;  // *** d3des.c Line 376 ***
In the eight iteration of the loop immediately after line 376, keys will have been incremented 32 times and is thus one past the end of KnL, but while post-increment (++) has higher precedence than dereference (*) it is a post-increment, so it is not dereferenced after being past the end.

So it looks to me like the address sanitizer is reporting a false positive or the compiler is generating incorrect code.

Comment 1 Thomas Sondergaard 2014-02-13 11:39:09 UTC
Problem is not reproducible when compiled with clang from Fedora 20, regardless of optimization.

clang version 3.3 (tags/RELEASE_33/final)

Comment 2 Jakub Jelinek 2014-02-13 17:29:35 UTC
Tracking this upstream as .

Note You need to log in before you can comment on or make changes to this bug.