Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 709325

Summary: Possibility to create queue files dynamically and resizing queues without deleting them
Product: Red Hat Enterprise MRG Reporter: Siddhesh Poyarekar <spoyarek>
Component: qpid-cppAssignee: Kim van der Riet <kim.vdriet>
Status: CLOSED ERRATA QA Contact: Zdenek Kraus <zkraus>
Severity: medium Docs Contact:
Priority: high    
Version: 1.3CC: esammons, freznice, gsim, iboverma, jmorgan, jross, jwulf, kim.vdriet, lzhaldyb, mnewsome, mtessun, pematous, rbinkhor, sauchter, zkraus
Target Milestone: 3.0Keywords: FutureFeature, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: qpid-cpp-0.22-27 Doc Type: Enhancement
Doc Text:
A new store implementation named Linearstore replaces Legacystore in this release. Linearstore addresses limitations imposed by Legacystore, specifically its use of fixed circular file buffers for queue journals. Linearstore utilizes a pool of fixed-size empty journal files and appends the journal files to a queue journal in a linear manner. The file is returned to the pool once a journal file is cleared of messages, and has no preceding journal files with messages waiting. There are a number of changes between Legacystore and Linearstore users must be aware of. See Comment 33 in the original Bugzilla Enhancement for further usage caveats and migration limitations. This information will be migrated to formal user documentation shortly after the 3.0 Release.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-24 15:02:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1036026, 1039522, 1039525, 1039949, 1051097, 1051924, 1052445, 1052518, 1052727, 1052775, 1053701, 1053749, 1054448, 1063700, 1064181, 1064230, 1078142, 1098118    
Bug Blocks: 785156, 803771    

Comment 3 J.H.M. Dassen (Ray) 2011-07-22 09:22:31 UTC
The customer sees another potential benefit to this feature:

"The synchronization of the broker stores in cluster takes quite a long
time. It seems that the broker is transferring all the journals to the other
cluster nodes. With our huge stores this will be significant time. During
this time, clients can log into the broker, but they cannot execute any
operations (i.e. this is not some background synchronization which causes
just some lower performance, it basically stops the brokers from serving the

This can be a problem in case of broker failovers and subsequent restarts

If the store would not take 1TB of disk space but maybe only 50GB (due to
the queues which are growing dynamically), the synchronization of the store
to new cluster members would be significantly faster and thus increase the
operational time in case of broker restarts."

Comment 17 Justin Ross 2013-07-22 14:52:00 UTC
*** Bug 453523 has been marked as a duplicate of this bug. ***

Comment 18 Justin Ross 2013-07-22 14:58:22 UTC
Setting conditional nack.  This work is coming in late for 3.0 and is at risk.

Comment 21 Kim van der Riet 2013-11-20 16:22:11 UTC
Added first version of linearstore to the qpid source tree. The source tree is located at cpp/src/qpid/linearstore. 

Added linearstore: b7f38df731735ee35cb5d5558b19fbe4d56033e1
Merge branch:      1f2dc49884b1f712151fe70608b1cca78e7d70c7

The store is basically functional, but not broadly tested. See the ISSUES file for a list of outstanding known issues with the store.

Basic Idea:

Unlike the legacystore module which has circular disk journals of fixed size, the linearstore adds files to each queue as needed from a pool of empty files called the Empty File Pool (EFP), and returns files which no longer contain records to the EFP. This allows for arbitrary sized journals, and are currently without a size limit.

Best performance will be obtained if the EFP is populated with empty files that can be used for journals in advance. However, if the EFP is not present or is empty, the store will create and format the files on the fly (with a performance penalty). The used files will be returned to the EFP.

The old store geometry parameters are no longer valid (file-size, num-jfiles). The EFP uses a default file size of 2MB per file, but pools of other sizes can be configured. In addition, the pools may exist on different partitions, allowing for some pools to exist on fast media (eg flash disks) while others may exist on standard media. It is possible to set the EFP identity (partition and empty file size within that partition) when creating a queue.

Theoretically, it should also be possible to switch file size within a partition on the fly through management (eg move from using 2MB files to 32MB files if the file number is increasing too rapidly for a particular queue), but this functionality is not fully implemented yet.

LINEAR STORE JOURNALS ARE INCOMPATIBLE WITH LEGACYSTORE JOURNALS. There is no tool or mechanism to upgrade or transfer files from existing legacystore journals. The format of the record and file headers has changed, although they are still very similar.

It is not possible to run both linearstore and legacystore at the same time. Beware that default behaviour may cause the broker to attempt to load all available modules, so if both these stores are present, problems may arise.

This is still a work in progress, but will benefit from some simple initial testing of transactional and non-transactional testing. Long tests are not possible as the mechanism to recycle empty journal files to the EFP while the borker is running is not currently working, and journal files will accumulate. Note, however, that on recovery, empty files on the trailing edge of the journal will be repatriated into the correct EFP.

--store-dir is still valid, and the store will create a "qls" (Qpid linear store) directory under the specified store dir which contains the EFP, the db4 database and the journals. If --store-dir is not specified, --data-dir will be used.

Comment 25 Zdenek Kraus 2014-08-08 08:22:44 UTC
Since all dependent bugs are VERIFIED moving to VERIFIED.

Comment 30 Zdenek Kraus 2014-09-17 17:36:20 UTC
Still waiting for DOC, please provide doc text. reraising NEEDINFO(kvdr,jross)

Comment 31 Kim van der Riet 2014-09-19 17:19:35 UTC
Added doc text.

Comment 33 Jared MORGAN 2014-09-23 03:30:36 UTC
Feature: Linearstore

Reason: The original Legacystore has several serious limitations, one of which is the fact that it uses fixed circular file buffers for queue journals. This system is optimized for speed in systems where messages are consumed strictly in-order, but has the following disadvantages:

1. If messages are consumed out-of-order or a message is orphaned (abandoned), then it will cause the buffer to eventually fill, and will cuase the queue to become inoperable for adding messages until blocking message(s) are consumed.

2. It is difficult to size a fixed buffer correctly. The user must estimate the worst-case scenario for queue depth and size the buffer accordingly. This can be a wasteful use of disk space and makes broker recovery unnecessarily long. If there are many queues, then this can quickly drain the available disk space.

Result: A new store has been developed which is largely based on the previous legacystore. However, it keeps a pool of empty files of fixed size from which it takes journal files as needed and appends them to a queue journal in a linear manner. Once a journal file has all its messages consumed and there are no preceding journal files still with messages, the file is returned to the pool. This store is known as the linearstore.

1. There is no compatibility between legacystore and linearstore journals. A user which upgrades from legacystore to linearstore must do so with a clean store.

2. The Empty File Pool (EFP) contains empty files of the correct size and format from which journal files are taken. If the EFP is empty when the linearstore needs a file, it will create and format one automatically and use it in the queue. However, this causes the store to run a little more slowly than if old files are re-used.

3. Ideally, the EFP should be maintained by an external process to keep the number of empty formatted files topped up for immediate use by the store. However, currenlty there is no such utility. Users will need to rely on the automatic creation ability of the store for the time being. For a fixed number of queues, this create an initial burden of file creation when queues are first created and used, but over time the EFP will fill up with used files and will be circulated as needed.

4. There are currently no limits on the maximum size of any journal. Currently, a queue can contain an unlimited number of journal files. This will be addressed in a future enhancement of the linearstore. Under normal usage patterns, a queue will maintain a reasonable number of journal files. However, under some unusual circumstances, such as orphaned persistent mesages, the number of files in the queue will grow. It is up to the user to monitor file usage.

5. There are currently no QMF abilities in linearstore. This will be addressed in a later enhancement.

6. The legacystore geometry parameters --jfile-size-pgs and --num-jfiles (and the TPL equivalents --tpl-jfile-size-pgs and --tpl-num-jfiles) are no longer used in linearstore. Journal files are now a fixed size and the number is automatically adjusted to the number of messages theat must be stored.

7. The linearstore --partition parameter is not fully functional, and should be ignored for all practical purposes. A future enhancement will add the ability to specify a disk partition for a queue journal, thus allowing the user to place some queues on high-performance media (such as solid state drives) while others may be on regular magnetic disk. Currently the --partition parameter will select a partition directory from the store tree structure, but lacks the ability to actually span several real partitions. The default partition is p001, and this should be the only one used for linearstore until the enhancement is complete.

8. The --overwrite-before-return parameter will force each journal file to be completely overwritten before it is returned to the EFP. This parameter should only be used where security concerns dictate that old message data should not be present in the EFP. Using this parameter comes with an as-yet undetermined performance penalty owing to the additional write activity needed for each file as it is reutnred to the EFP.

9. The current directory structure of the store below the --store-dir parameter is as follows:

            +-dat <BDB database>
            |  +-efp
            |  |  +-2048k
            |  |  +-8192k
            |  |  +- <other file sizes>
            |  |
            |  +-jrnl
            |     +-queue_1
            |     +-queue_2
            |     ...
            |     +-queue_n
            +-p002 <dir structure as above>
            +-pnnn <dir structure as above>

Comment 34 Zdenek Kraus 2014-09-23 07:36:08 UTC
release note is in place


Comment 36 errata-xmlrpc 2014-09-24 15:02:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.