Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1064825

Summary: gcc address sanitizer reports global-buffer-overflow when compiled with optimization
Product: [Fedora] Fedora Reporter: Thomas Sondergaard <thomas>
Component: gccAssignee: Jakub Jelinek <jakub>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 20CC: jakub, law
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-13 17:29:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
Example that shows the problem none

Description Thomas Sondergaard 2014-02-13 11:32:45 UTC
Created attachment 862746 [details]
Example that shows the problem

Description of problem:
asan_symbolizer reports global-buffer-overflow in asan_bug/d3des.c:376, but only when compiled with optimization

Version-Release number of selected component (if applicable):
gcc (GCC) 4.8.2 20131212 (Red Hat 4.8.2-7)

Also present in gcc (GCC) 4.8.1 20130715 (Red Hat 4.8.1-4) from rhel6 devtoolset-2.

How reproducible:
Fully reproducible

Steps to Reproduce:
1. untar attachment
2. make sure you have from llvm/clang in path
3. run make
4. Modify Makefile to disable optimization (remove -O2)
5. run make

Actual results:
asan reports global-buffer-overflow in asan_bug/d3des.c:376 when optimization is enabled and doesn't when optimization is disabled

Expected results:
asan results should be consistent independent of optimization levels. The d3des.{h,c} files I have from realvnc, but tigervnc and several other realvnc derived projects use the same. As far as I can see the code does not overflow the buffer.

The code in d3des.c around line 376 looks like this:
    unsigned long keys = KnL; // via function parameter
    for( round = 0; round < 8; round++ ) {
      work ^= *keys++;
      work  = right ^ *keys++;
      work ^= *keys++;
      work  = leftt ^ *keys++;  // *** d3des.c Line 376 ***
In the eight iteration of the loop immediately after line 376, keys will have been incremented 32 times and is thus one past the end of KnL, but while post-increment (++) has higher precedence than dereference (*) it is a post-increment, so it is not dereferenced after being past the end.

So it looks to me like the address sanitizer is reporting a false positive or the compiler is generating incorrect code.

Comment 1 Thomas Sondergaard 2014-02-13 11:39:09 UTC
Problem is not reproducible when compiled with clang from Fedora 20, regardless of optimization.

clang version 3.3 (tags/RELEASE_33/final)

Comment 2 Jakub Jelinek 2014-02-13 17:29:35 UTC
Tracking this upstream as .