Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1693218 - Add some additional MongoDB information to pulp sos plugin (collection size)
Summary: Add some additional MongoDB information to pulp sos plugin (collection size)
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sos
Version: 7.7
Hardware: All
OS: All
unspecified
high
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-27 11:21 UTC by Waldirio M Pinheiro
Modified: 2019-04-15 20:42 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github sosreport sos pull 1645 None None None 2019-04-10 20:39:40 UTC

Description Waldirio M Pinheiro 2019-03-27 11:21:26 UTC
Description of problem:
When doing the MongoDB troubleshooting, there is an output very useful that we don't have already on foreman sos plugin.


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Steps below.
2.
3.

Actual results:
We don't have this information on sosreport

Expected results:
See the collection size

Additional info:

Comment 2 Waldirio M Pinheiro 2019-03-27 11:22:56 UTC
Hi

When running the command below, we collect the info and redirect to "/tmp/mongo_log.log"
---
cat << EOF | mongo pulp_database | tee /tmp/mongo_log.log
function getReadableFileSizeString(fileSizeInBytes) {

    var i = -1;
    var byteUnits = [' kB', ' MB', ' GB', ' TB', 'PB', 'EB', 'ZB', 'YB'];
    do {
        fileSizeInBytes = fileSizeInBytes / 1024;
        i++;
    } while (fileSizeInBytes > 1024);

    return Math.max(fileSizeInBytes, 0.1).toFixed(1) + byteUnits[i];
};
var collectionNames = db.getCollectionNames(), stats = [];
collectionNames.forEach(function (n) { stats.push(db[n].stats()); });
stats = stats.sort(function(a, b) { return b['size'] - a['size']; });
for (var c in stats) { print(stats[c]['ns'] + ": " + getReadableFileSizeString(stats[c]['size']) + " (" + getReadableFileSizeString(stats[c]['storageSize']) + ")"); }
EOF
---

Now, let's take a look at the content
---
...
pulp_database.units_rpm: 1.8 GB (6.7 GB)
pulp_database.lazy_content_catalog: 38.9 MB (82.3 MB)
pulp_database.repo_content_units: 27.4 MB (36.0 MB)
pulp_database.units_erratum: 16.4 MB (36.0 MB)
pulp_database.erratum_pkglists: 14.4 MB (21.3 MB)
pulp_database.units_package_group: 4.4 MB (5.3 MB)
pulp_database.task_status: 3.1 MB (85.3 MB)
pulp_database.repo_importers: 1.5 MB (5.0 MB)
pulp_database.consumer_unit_profiles: 639.9 kB (17.7 MB)
pulp_database.repo_profile_applicability: 447.9 kB (5.0 MB)
pulp_database.consumer_bindings: 205.9 kB (680.0 kB)
pulp_database.units_docker_tag: 138.0 kB (680.0 kB)
pulp_database.units_package_category: 119.7 kB (648.0 kB)
pulp_database.units_docker_manifest: 88.7 kB (168.0 kB)
pulp_database.units_package_environment: 87.8 kB (648.0 kB)
pulp_database.repo_distributors: 66.9 kB (680.0 kB)
pulp_database.units_docker_blob: 51.8 kB (168.0 kB)
pulp_database.repo_publish_results: 44.8 kB (1.3 MB)
pulp_database.system.indexes: 27.6 kB (40.0 kB)
pulp_database.repos: 19.3 kB (40.0 kB)
pulp_database.consumer_history: 15.7 kB (10.7 MB)
pulp_database.repo_sync_results: 13.9 kB (168.0 kB)
pulp_database.units_yum_repo_metadata_file: 9.8 kB (40.0 kB)
pulp_database.content_types: 6.2 kB (8.0 kB)
pulp_database.units_package_langpacks: 3.7 kB (40.0 kB)
pulp_database.units_distribution: 2.0 kB (8.0 kB)
pulp_database.permissions: 0.9 kB (8.0 kB)
pulp_database.consumers: 0.7 kB (168.0 kB)
pulp_database.roles: 0.5 kB (8.0 kB)
pulp_database.event_listeners: 0.5 kB (8.0 kB)
pulp_database.migration_trackers: 0.4 kB (8.0 kB)
pulp_database.workers: 0.4 kB (8.0 kB)
pulp_database.resource_manager_lock: 0.2 kB (8.0 kB)
pulp_database.users: 0.2 kB (8.0 kB)
pulp_database.celery_beat_lock: 0.2 kB (8.0 kB)
pulp_database.orphaned_profile_hash: 0.2 kB (8.0 kB)
pulp_database.repo_group_distributors: 0.1 kB (8.0 kB)
pulp_database.scheduled_calls: 0.1 kB (8.0 kB)
pulp_database.units_iso: 0.1 kB (8.0 kB)
pulp_database.units_ostree: 0.1 kB (8.0 kB)
pulp_database.repo_groups: 0.1 kB (8.0 kB)
pulp_database.repo_group_publish_results: 0.1 kB (40.0 kB)
pulp_database.units_drpm: 0.1 kB (8.0 kB)
pulp_database.units_docker_image: 0.1 kB (8.0 kB)
pulp_database.units_puppet_module: 0.1 kB (8.0 kB)
pulp_database.deferred_download: 0.1 kB (168.0 kB)
pulp_database.units_srpm: 0.1 kB (679.2 MB)
pulp_database.content_catalog: 0.1 kB (8.0 kB)
pulp_database.consumer_groups: 0.1 kB (8.0 kB)
pulp_database.reserved_resources: 0.1 kB (8.0 kB)
bye
---

Comment 3 Bryn M. Reeves 2019-03-27 11:25:10 UTC
OK, but you're going to have to tell us what that data is so that we can add it.

Please provide any commands that should be run, or existing files that should be collected, as well as an indication of whether any of this data could contain secrets (that would need to be removed by sos during processing).

Comment 4 Waldirio M Pinheiro 2019-03-27 11:57:09 UTC
Hello Bryn

This is the command

---
cat << EOF | mongo pulp_database | tee /tmp/mongo_log.log
function getReadableFileSizeString(fileSizeInBytes) {

    var i = -1;
    var byteUnits = [' kB', ' MB', ' GB', ' TB', 'PB', 'EB', 'ZB', 'YB'];
    do {
        fileSizeInBytes = fileSizeInBytes / 1024;
        i++;
    } while (fileSizeInBytes > 1024);

    return Math.max(fileSizeInBytes, 0.1).toFixed(1) + byteUnits[i];
};
var collectionNames = db.getCollectionNames(), stats = [];
collectionNames.forEach(function (n) { stats.push(db[n].stats()); });
stats = stats.sort(function(a, b) { return b['size'] - a['size']; });
for (var c in stats) { print(stats[c]['ns'] + ": " + getReadableFileSizeString(stats[c]['size']) + " (" + getReadableFileSizeString(stats[c]['storageSize']) + ")"); }
EOF
---

Copying/paste the complete block above, will be generated the file "/tmp/mongo_log.log" and this is what we need on the sosreport.

Let me know if you have any additional question or concern.

Thanks

Comment 6 Pavel Moravec 2019-04-10 20:39:40 UTC
ACK for collecting that data, useful to know sizes of pulp collections esp. for performance issues. Some minor comments:

- too late to 7.7, shall be in 7.8
- pulp plugin should collect that, not foreman
- PR raised, bit simplified (try to squeeze the loong commands to 80 chars ;-))

Comment 7 Kenny Tordeurs 2019-04-11 13:24:14 UTC
For external mongodb this will need to include some more parameters:

~~~
# cat << EOF | mongo --host ktordeur-sat65-mongo.sysmgmt.lan -u pulp_user -p pulp_password --port 27017  pulp_database | tee /tmp/mongo_log.log
function getReadableFileSizeString(fileSizeInBytes) {

    var i = -1;
    var byteUnits = [' kB', ' MB', ' GB', ' TB', 'PB', 'EB', 'ZB', 'YB'];
    do {
        fileSizeInBytes = fileSizeInBytes / 1024;
        i++;
    } while (fileSizeInBytes > 1024);

    return Math.max(fileSizeInBytes, 0.1).toFixed(1) + byteUnits[i];
};
var collectionNames = db.getCollectionNames(), stats = [];
collectionNames.forEach(function (n) { stats.push(db[n].stats()); });
stats = stats.sort(function(a, b) { return b['size'] - a['size']; });
for (var c in stats) { print(stats[c]['ns'] + ": " + getReadableFileSizeString(stats[c]['size']) + " (" + getReadableFileSizeString(stats[c]['storageSize']) + ")"); }
EOF
~~~

Comment 10 Pavel Moravec 2019-04-15 20:42:37 UTC
upstream sos PR 1645 updated to support external DBs.


Note You need to log in before you can comment on or make changes to this bug.