Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1041749 - rhevm-log-collector garbles the tty when multiple hosts are gathered and ssh is called with the "-t" flag
Summary: rhevm-log-collector garbles the tty when multiple hosts are gathered and ssh ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-log-collector
Version: 3.3.0
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: 3.4.0
Assignee: Sandro Bonazzola
QA Contact: Jiri Belka
URL:
Whiteboard: integration
Depends On: 1059692
Blocks: 1058744 rhev3.4beta 1142926
TreeView+ depends on / blocked
 
Reported: 2013-12-12 19:09 UTC by James W. Mills
Modified: 2018-12-03 20:56 UTC (History)
9 users (show)

Fixed In Version: ovirt-3.4.0-beta2
Doc Type: Bug Fix
Doc Text:
Previously, when the rhevm-log-collector utility was run for multiple hypervisor hosts, it resulted in a garbled TTY and "stty sane" had to be run to recover the TTY. Now, the TTY is not garbled.
Clone Of:
: 1049759 1058744 (view as bug list)
Environment:
Last Closed: 2014-06-09 14:06:16 UTC
oVirt Team: ---
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 23426 None None None Never
oVirt gerrit 23796 None None None Never
Red Hat Knowledge Base (Solution) 489813 None None None Never
Red Hat Product Errata RHBA-2014:0668 normal SHIPPED_LIVE rhevm-log-collector bug fix and enhancement update 2014-06-09 18:03:59 UTC

Description James W. Mills 2013-12-12 19:09:46 UTC
Description of problem:

When rhevm-log-collector is run for multiple hypervisor hosts and invokes ssh with the "-t" flag, it results in a garbled TTY requiring "stty sane" to be run to get it back.

Version-Release number of selected component (if applicable):

rhevm-log-collector-3.2.2-4.el6ev.noarch
rhevm-log-collector-3.2.2-6.el6ev.noarch
rhevm-log-collector-3.1.0-10.el6ev.noarch

How reproducible:

I was able to reproduce this 100% of the time if gathering more than one host.  A single host does not exhibit this problem.

Steps to Reproduce:
1.  This problem depends on bug 1010472, which adds the "-t" flag to ssh.  The patch looks like this:

--- rhevm-log-collector.orig    2013-10-07 10:40:57.000000000 -0400
+++ rhevm-log-collector 2013-12-12 12:59:03.552000942 -0500
@@ -503,6 +503,10 @@
         def format_ssh_command(self, cmd="ssh"):
             cmd = "/usr/bin/%s " % cmd

+       # add a controlling tty
+       if cmd.startswith("/usr/bin/ssh"):
+           cmd += "-t "
+
             if "ssh_port" in self.configuration:
                 port_flag = "-p" if cmd.startswith("/usr/bin/ssh") else "-P"
                 cmd += port_flag + " %(ssh_port)s " % self.configuration

2.  With that patch in place, run 'rhevm-log-collector collect' in an environment with more than one host

Actual results:

The command will succeed, but the TTY will become garbled during or just after the collection of data from the hosts

The only recovery is to run 'stty sane' or 'reset' on the TTY where the 'rhevm-log-collector' was invoked.

Expected results:

The command succeeds and the TTY is sane.

Additional info:

This appears to be something to be a combination of the "-t" flag and specifying multiple hosts.  If you remove the "-t" flag the behavior goes away, and if you only specify one host the behavior goes away.  It might have something to do with the logging module being used in multiple threads simultaneously.

I was able to completely alleviate the problem with this patch:

--- rhevm-log-collector.orig    2013-10-07 10:40:57.000000000 -0400
+++ rhevm-log-collector 2013-12-12 12:59:03.552000942 -0500
@@ -626,6 +630,7 @@
             )
             stdout = self.sosreport()
             self.parse_sosreport_stdout(stdout)
+            self.caller.call('stty sane')
             self.configuration["hypervisor_dir"] = os.path.join(
                 self.configuration.get("local_scratch_dir"),
                 self.configuration.get("hostname")

Comment 2 Sandro Bonazzola 2014-01-20 09:01:40 UTC
Hi James, thanks for your analysis of the issue.


(In reply to James W. Mills from comment #0)

> Additional info:
> 
> This appears to be something to be a combination of the "-t" flag and
> specifying multiple hosts.  If you remove the "-t" flag the behavior goes
> away, and if you only specify one host the behavior goes away.  It might
> have something to do with the logging module being used in multiple threads
> simultaneously.

Well logging module is supposed to be thread safe:
http://docs.python.org/2/library/logging.html#thread-safety
I don't think it's due to logging module.

> 
> I was able to completely alleviate the problem with this patch:
> 
> --- rhevm-log-collector.orig    2013-10-07 10:40:57.000000000 -0400
> +++ rhevm-log-collector 2013-12-12 12:59:03.552000942 -0500
> @@ -626,6 +630,7 @@
>              )
>              stdout = self.sosreport()
>              self.parse_sosreport_stdout(stdout)
> +            self.caller.call('stty sane')
>              self.configuration["hypervisor_dir"] = os.path.join(
>                  self.configuration.get("local_scratch_dir"),
>                  self.configuration.get("hostname")

There are other ssh calls after that one so better move the call to stty sane after them.

Comment 3 Sandro Bonazzola 2014-01-28 11:22:17 UTC
merged on upstream master for 3.5.0, pushed to 3.4.0 branch.

Comment 4 Sandro Bonazzola 2014-01-28 13:16:14 UTC
merged on 3.4.0 branch

Comment 6 Jiri Belka 2014-02-25 09:27:18 UTC
ok, beta3 with 2 hosts...

# engine-log-collector
INFO: Gathering oVirt Engine information...
INFO: Gathering PostgreSQL the oVirt Engine database and log files from localhost...
Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to skip): 
About to collect information from 2 hypervisors. Continue? (Y/n): Y
INFO: Gathering information from selected hypervisors...
INFO: collecting information from 10.34.63.222
INFO: collecting information from 10.34.66.81
INFO: finished collecting information from 10.34.63.222
INFO: finished collecting information from 10.34.66.81
Creating compressed archive...
INFO: Log files have been collected and placed in /tmp/sosreport-LogCollector-20140225092540.tar.xz.
The MD5 for this file is 24c9bcc187d96d7ef818005f4d1ed055 and its size is 36.2M

Comment 8 errata-xmlrpc 2014-06-09 14:06:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0668.html


Note You need to log in before you can comment on or make changes to this bug.