Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1362014 - 2.0: rbd bench-write: segmentation fault when value of "--io-size" is greater than or equal to image size
Summary: 2.0: rbd bench-write: segmentation fault when value of "--io-size" is greater...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 2.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: 3.0
Assignee: Jason Dillaman
QA Contact: Vasishta
Bara Ancincova
URL:
Whiteboard:
Depends On:
Blocks: 1322504 1383917 1412948 1494421
TreeView+ depends on / blocked
 
Reported: 2016-08-01 07:54 UTC by Harish NV Rao
Modified: 2017-12-05 23:31 UTC (History)
7 users (show)

Fixed In Version: RHEL: ceph-12.1.2-1.el7cp Ubuntu: ceph_12.1.2-2redhat1xenial
Doc Type: Bug Fix
Doc Text:
.The `rbd bench write` command no longer fails when `--io-size` is equal to the image size Previously, the `rbd bench-write --io-size <size> <image>` command failed with a segmentation fault if the size specified by the `--io-size` option was greater than 4 GB. With this update, the option is restricted from being too large.
Clone Of:
Environment:
Last Closed: 2017-12-05 23:31:14 UTC


Attachments (Terms of Use)
bt (deleted)
2016-08-01 07:54 UTC, Harish NV Rao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3387 normal SHIPPED_LIVE Red Hat Ceph Storage 3.0 bug fix and enhancement update 2017-12-06 03:03:45 UTC
Ceph Project Bug Tracker 16967 None None None 2016-08-09 00:28:18 UTC
Ceph Project Bug Tracker 18422 None None None 2017-01-04 21:36:36 UTC

Description Harish NV Rao 2016-08-01 07:54:14 UTC
Created attachment 1186276 [details]
bt

Description of problem:

 rbd bench-write: segmentation fault when value of "--io-size" is greater than or equal to image size.

Version-Release number of selected component (if applicable): ceph version 10.2.2-26.el7cp 


How reproducible: always


Steps to Reproduce:
1. create an image of size 100M
  --cmd: rbd  create test_rbd/cephRBD --size 100M

2. start 'rbd bench-write' on this image with '--io-size' as 101M
  --cmd: rbd --cluster ceph  bench-write test_rbd/cephRBD --io-size 101M --io-threads 3 --io-total 1M --io-pattern rand
  --result: segmentation fault seen

3. create an image of size 4096M

4. start 'rbd bench-write' on this image with '--io-size' as 4096M
  --cmd: rbd --cluster ceph  bench-write -p test_rbd --image test0 --io-size 4096M --io-threads 3 --io-total 1M --io-pattern rand
  --result: segmentation fault seen

Actual results:
segmentation fault observed after executing step 2 and 4.

Expected results:


Additional info:
1) Please see the attachment with bt for both the cases.
2) Image sizes:
[root@magna105 ubuntu]# rbd info test_rbd/test0
rbd image 'test0':
	size 4096 MB in 1024 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.658322ae8944a
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags: 
[root@magna105 ubuntu]# rbd info test_rbd/cephRBD
rbd image 'cephRBD':
	size 102400 kB in 25 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.6bb7e238e1f29
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags:

Comment 4 Jason Dillaman 2016-08-12 12:29:30 UTC
Upstream pull request: https://github.com/ceph/ceph/pull/10708

Comment 9 Vasishta 2016-10-07 12:32:13 UTC
Still facing this issue whenever io size and image size are greater than or equal to 4096 and io size is lesser than or equal to chosen image size. 
( 4096 =< io_size =< image_size )

Regards,
Vasishta

Comment 10 Jason Dillaman 2016-10-07 12:50:53 UTC
Please provide the exact command you executed and the exact result you witnessed. What you are describing doesn't match the bug (which was a crash when io_size was *greater* than the image size).

Comment 11 Vasishta 2016-10-07 14:13:10 UTC
Hi Jason,

As it is mentioned in Comment 0 that
'segmentation fault when value of "--io-size" is greater than or equal to image size', both cases (1. greater than and 2.equal) have been tried.

1. When io_size is greater than image_size - 

Error message is getting displayed, saying this is not possible


2. When io_size is equal to image_size -

   a. If image_size & io_size is lesser than 4096


sudo rbd --cluster ceph  bench-write images/img2 --io-size 4096M --io-threads 3 --io-total 100M --io-pattern rand

Comment 12 Vasishta 2016-10-07 14:36:27 UTC
Sorry Jason, 

Please ignore Comment 11. It is incomplete.
I'll provide full info shortly.


Regards,
Vasishta

Comment 13 Jason Dillaman 2016-10-07 16:51:37 UTC
Ack -- so comment 9 should read something along the lines of the following:

rbd bench-write will still crash if the io_size is *equal to* the image size so long as the image_size/io_size is less than X.

The original wording stated that it was crashing as long as io_size was less than the image size, which is a major issue if true.

Given that this is a small corner case, I am going to move this to the next release.

Comment 15 Jason Dillaman 2017-01-04 21:38:20 UTC
The actual issue is that the "--io-size=4096M" (i.e. 4GB io size) is leading to memory corruption due to overflow. The "rbd bench-write" command actually works properly if "--io-size" is equal to the image size so long as it doesn't pass the 4GB size boundary.

Comment 17 Vasishta 2017-09-12 15:38:11 UTC
Hi all, 

Couldn't complete Comment 11, Sorry for that.

As Jason has mentioned in Comment 15, Seg Fault was occurring whenever io-size was >= 4096, if image-size was >= 4096.
i,e image_size >= io-size >= 4096 (After the first fix) 

Now io-size has been capped to 4095M, irrespective of image size. 

$ sudo rbd bench  data/im2 --io-type write --io-size 4096M --io-threads 3 --io-total 100M --io-pattern rand --cluster 12_luminous
rbd: io-size should be less than 4G
bench failed: (22) Invalid argument

$ sudo rbd bench  data/im2 --io-type write --io-size 10240M --io-threads 3 --io-total 100M --io-pattern rand --cluster 12_luminous
rbd: io-size should be less than 4G
bench failed: (22) Invalid argument

$ sudo rbd bench  data/im3 --io-type write --io-size 5000M --io-threads 3 --io-total 100M --io-pattern rand --cluster 12_luminous
rbd: io-size 5000 MB larger than image size 102400 kB
bench failed: (22) Invalid argument

Moving to VERIFIED state.

Regards,
Vasishta

Comment 24 errata-xmlrpc 2017-12-05 23:31:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387


Note You need to log in before you can comment on or make changes to this bug.