Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1649054 - glustereventsd does not start on Ubuntu 16.04 LTS
Summary: glustereventsd does not start on Ubuntu 16.04 LTS
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: eventsapi
Version: 5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Aravinda VK
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: Gluster_5_Affecting_oVirt_4.3 glusterfs-5.4 1679406 1683008
TreeView+ depends on / blocked
 
Reported: 2018-11-12 19:32 UTC by Beat Jörg
Modified: 2019-03-27 13:44 UTC (History)
6 users (show)

Fixed In Version: glusterfs-5.5
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1679406 (view as bug list)
Environment:
Last Closed: 2019-02-26 13:52:48 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
glustereventsd.py and utils.py fixup (deleted)
2018-12-28 01:27 UTC, Arano-kai
no flags Details | Diff
glustereventsd.py and utils.py fixup (deleted)
2018-12-28 01:49 UTC, Arano-kai
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Gluster.org Gerrit 22262 None Merged eventsapi: Fix Python3 compatibility issues 2019-02-26 13:52:47 UTC

Description Beat Jörg 2018-11-12 19:32:40 UTC
Description of problem:
glustereventsd does not start due to an import error in utils.py

Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/sbin/glustereventsd", line 24, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     import handlers
Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/lib/x86_64-linux-gnu/glusterfs/events/handlers.py", line 12, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     import utils
Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     from .eventsapiconf import (LOG_FILE,
Nov 12 20:21:04 s002 glustereventsd[28140]: SystemError: Parent module '' not loaded, cannot perform relative import

Installed Pyython version is 3.5.2

Comment 1 Mrten 2018-12-04 17:12:09 UTC
same as Bug 1650017

Comment 2 Mrten 2018-12-04 17:25:31 UTC
This is a fix for the first bug:

--- utils.py.old	2018-12-04 18:19:18.846463181 +0100
+++ utils.py	2018-12-04 18:21:09.830149998 +0100
@@ -26,13 +26,13 @@
 from hashlib import sha256
 from calendar import timegm

-from .eventsapiconf import (LOG_FILE,
+from eventsapiconf import (LOG_FILE,
                            WEBHOOKS_FILE,
                            DEFAULT_CONFIG_FILE,
                            CUSTOM_CONFIG_FILE,
                            UUID_FILE,
                            CERTS_DIR)
-from . import eventtypes
+import eventtypes


and you need this one as well:

--- glustereventsd.py.old	2018-12-04 18:25:06.841481182 +0100
+++ glustereventsd.py	2018-12-04 18:22:51.529863014 +0100
@@ -28,7 +28,7 @@
 from utils import logger, PidFile, PidFileLockFailed, boolify


-class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
+class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):

     def handle(self):
         data = self.request[0].strip()
@@ -98,7 +98,7 @@

     # Start the Eventing Server, UDP Server
     try:
-        server = SocketServer.ThreadingUDPServer(
+        server = socketserver.ThreadingUDPServer(
             (SERVER_ADDRESS, port),
             GlusterEventsRequestHandler)
     except socket.error as e:

Comment 3 Mrten 2018-12-04 17:28:51 UTC
but you still have "glustereventsd[20980]: Unable to get Port details from Config"

in the log :(

Comment 4 Mrten 2018-12-04 17:36:28 UTC
default config file does not exist:

root@gluster-3:# less /etc/glusterfs/eventsconfig.json
/etc/glusterfs/eventsconfig.json: No such file or directory

Comment 5 Mrten 2018-12-04 17:38:39 UTC
inserting this:

https://github.com/gluster/glusterfs/blob/master/events/src/eventsconfig.json

into /etc/glusterfs/eventsconfig.json makes it start without problems

Comment 6 Mrten 2018-12-05 09:24:13 UTC
haha, but now geo-replication is broken

[2018-12-05 08:52:49.556853] E [syncdutils(monitor):338:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap
    tf(*aargs)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 339, in wmon
    slave_host, master, suuid, slavenodes)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 325, in monitor
    brick_path=w[0]['dir'])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 602, in gf_event
    from events.gf_event import gf_event as gfevent
  File "/usr/lib/x86_64-linux-gnu/glusterfs/events/gf_event.py", line 19, in <module>
    from .utils import logger, setup_logger, get_config
  File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module>
    from eventsapiconf import (LOG_FILE,
ImportError: No module named 'eventsapiconf'

Comment 7 Mrten 2018-12-05 09:31:22 UTC
better patch:

--- utils.py.old	2018-12-04 18:19:18.846463181 +0100
+++ utils.py	2018-12-05 10:30:03.306848363 +0100
@@ -26,13 +26,25 @@
 from hashlib import sha256
 from calendar import timegm

-from .eventsapiconf import (LOG_FILE,
-                           WEBHOOKS_FILE,
-                           DEFAULT_CONFIG_FILE,
-                           CUSTOM_CONFIG_FILE,
-                           UUID_FILE,
-                           CERTS_DIR)
-from . import eventtypes
+try:
+    from .eventsapiconf import (LOG_FILE,
+                                WEBHOOKS_FILE,
+                                DEFAULT_CONFIG_FILE,
+                                CUSTOM_CONFIG_FILE,
+                                UUID_FILE,
+                                CERTS_DIR)
+except ImportError:
+    from eventsapiconf import (LOG_FILE,
+                               WEBHOOKS_FILE,
+                               DEFAULT_CONFIG_FILE,
+                               CUSTOM_CONFIG_FILE,
+                               UUID_FILE,
+                               CERTS_DIR)
+
+try:
+  from . import eventtypes
+except ImportError:
+  import eventtypes

Comment 8 dchang0 2018-12-12 08:06:26 UTC
Hi, Mrten--

I still have the exact same bug AFTER applying the patches you provided (the patch in Comment 2 for glustereventsd.py and the patch in Comment 7 for utils.py).

I'm running Raspbian Stretch Lite (not Ubuntu 16.04 like the original bug reporter) on a Raspberry Pi Zero W and built glusterfs 5.1 from source code. The patches appear to have been applied correctly but attempting to start glustereventsd still gets this error:

Dec 11 16:48:14 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'.
Dec 11 16:50:53 datastore2c systemd[1]: Starting LSB: Gluster Events Server...
-- Subject: Unit glustereventsd.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has begun starting up.
Dec 11 16:50:55 datastore2c glustereventsd[710]: Starting glustereventsd service: glustereventsdTraceback (most recent call last):
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File "/usr/local/sbin/glustereventsd", line 24, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     import handlers
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     import utils
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     from .eventsapiconf import (LOG_FILE,
Dec 11 16:50:55 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import
Dec 11 16:50:55 datastore2c glustereventsd[710]:  failed!
Dec 11 16:50:56 datastore2c glustereventsd[710]: Traceback (most recent call last):
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File "/usr/local/sbin/glustereventsd", line 24, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     import handlers
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     import utils
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     from .eventsapiconf import (LOG_FILE,
Dec 11 16:50:56 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1
Dec 11 16:50:56 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server.
-- Subject: Unit glustereventsd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has failed.
-- 
-- The result is failed.
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state.
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'.


I can provide whatever info you need to help debug this. Thanks!

Comment 9 Arano-kai 2018-12-28 01:27:27 UTC
Created attachment 1517163 [details]
glustereventsd.py and utils.py fixup

Hi dchang0.
According to your logs, patch is not apply.
Here the compiled from Mrten's answers patch that you need apply with following cmd:
`patch -p 0 -d [libdir] < glustereventsd.patch`
Where [libdir]:
- Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/"
- Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs)
Also, there is missing file that you can get by `wget 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/eventsconfig.json' -O /etc/glusterfs/eventsconfig.json`

Comment 10 dchang0 2018-12-28 01:44:44 UTC
(In reply to Arano-kai from comment #9)
> Created attachment 1517163 [details]
> glustereventsd.py and utils.py fixup
> 
> Hi dchang0.
> According to your logs, patch is not apply.
> Here the compiled from Mrten's answers patch that you need apply with
> following cmd:
> `patch -p 0 -d [libdir] < glustereventsd.patch`
> Where [libdir]:
> - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/"
> - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs)
> Also, there is missing file that you can get by `wget
> 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/
> eventsconfig.json' -O /etc/glusterfs/eventsconfig.json`

Thanks. I will check the files in question manually and reapply them if needed and report back.

eventsconfig.json was not missing for me (I built completely from scratch, and that file was included and installed in the proper location).

Comment 11 Arano-kai 2018-12-28 01:49:11 UTC
Created attachment 1517165 [details]
glustereventsd.py and utils.py fixup

Woops, forget to add 'SystemError' to catcher.
Here the fixed version.

Comment 12 dchang0 2018-12-28 04:14:37 UTC
This doesn't matter now, since you've changed at least the glustereventsd.py patch, but I'm posting these anyway for diagnostic purposes.

The source code was built in /root/glusterfs-5.1. The install with the patches applied is in /usr/local...

-----


root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py
31c31
< class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
---
> class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):
101c101
<         server = SocketServer.ThreadingUDPServer(
---
>         server = socketserver.ThreadingUDPServer(


-----


root@datastore2c:/home/pi# diff  /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py
29,35c29,42
< from .eventsapiconf import (LOG_FILE,
<                            WEBHOOKS_FILE,
<                            DEFAULT_CONFIG_FILE,
<                            CUSTOM_CONFIG_FILE,
<                            UUID_FILE,
<                            CERTS_DIR)
< from . import eventtypes
---
> try:
>     from .eventsapiconf import (LOG_FILE,
>                                 WEBHOOKS_FILE,
>                                 DEFAULT_CONFIG_FILE,
>                                 CUSTOM_CONFIG_FILE,
>                                 UUID_FILE,
>                                 CERTS_DIR)
> except ImportError:
>     from eventsapiconf import (LOG_FILE,
>                                WEBHOOKS_FILE,
>                                DEFAULT_CONFIG_FILE,
>                                CUSTOM_CONFIG_FILE,
>                                UUID_FILE,
>                                CERTS_DIR)
36a44,47
> try:
>     from . import eventtypes
> except ImportError:
>     import eventtypes


-----

So it looks like I did apply the patches before, but they were probably bad/broken patches.

Anyway, I will replace the patched files with the original source, then apply your new patches and see what happens and report back.

Comment 13 dchang0 2018-12-28 04:52:57 UTC
Okay, I just put the original files from source back (using make install) and then applied your new patches from Attachment #1517165 [details]. I'm getting a different error (probably an easier one to solve).

Strangely, the patch command does not create the .orig files. Perhaps my patch command is broken or non-standard...


Here are the diffs:

-----

root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py
31c31
< class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
---
> class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):
101c101
<         server = SocketServer.ThreadingUDPServer(
---
>         server = socketserver.ThreadingUDPServer(

------

root@datastore2c:/usr/local/libexec# diff  /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py
29,35c29,46
< from .eventsapiconf import (LOG_FILE,
<                            WEBHOOKS_FILE,
<                            DEFAULT_CONFIG_FILE,
<                            CUSTOM_CONFIG_FILE,
<                            UUID_FILE,
<                            CERTS_DIR)
< from . import eventtypes
---
> try:
>     from .eventsapiconf import (LOG_FILE,
>                                 WEBHOOKS_FILE,
>                                 DEFAULT_CONFIG_FILE,
>                                 CUSTOM_CONFIG_FILE,
>                                 UUID_FILE,
>                                 CERTS_DIR)
> except (ImportError, SystemError):
>     from eventsapiconf import (LOG_FILE,
>                                WEBHOOKS_FILE,
>                                DEFAULT_CONFIG_FILE,
>                                CUSTOM_CONFIG_FILE,
>                                UUID_FILE,
>                                CERTS_DIR)
> try:
>     from . import eventtypes
> except (ImportError, SystemError):
>     import eventtypes


-----

Here is the new error when attempting to run:

service glustereventsd start

It appears to be a simple path error (can't find the gluster command).



root@datastore2c:/usr/local/libexec# journalctl -xe
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     import utils
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/local/libexec/glusterfs/events/utils.py", line 37, in <module>
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     from eventsapiconf import (LOG_FILE,
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 33, in <modul
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     CUSTOM_CONFIG_FILE = get_glusterd_workdir() + CUSTOM_CONFIG_FILE_TO_SYNC
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 22, in get_gl
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     universal_newlines = True)
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     restore_signals, start_new_session)
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     raise child_exception_type(errno_num, err_msg)
Dec 27 20:46:15 datastore2c glustereventsd[14033]: FileNotFoundError: [Errno 2] No such file or directory: 'gluster'
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1
Dec 27 20:46:15 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server.
-- Subject: Unit glustereventsd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has failed.
-- 
-- The result is failed.
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state.
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'.


-----

The gluster command is at this path:

root@datastore2c:/usr/local/libexec# which gluster
/usr/local/sbin/gluster


I confirm that it runs insofar as it can put me into the interactive command line mode and respond to simple commands like help.

Thanks for your help so far. Let me know what I can do to help.

Comment 14 dchang0 2018-12-28 08:20:18 UTC
BTW, if what I'm reporting in Comment 13 is now a new bug, please let me know and I will start a different bug report for it. Unless it is actually better to treat these two errors as one bug.  Thanks!

Comment 15 Arano-kai 2018-12-28 10:28:31 UTC
It is probably you hit by https://stackoverflow.com/q/5658622, that is some $PATH inconsistency from python view and shell view.
Since systemd used and units run in isolated env, you can try to explicitly set $PATH by overriding glustereventsd.service:
# echo $PATH
<current shell PATH>
# systemctl edit glustereventsd.service

[Service]
Environment=PATH=<current shell PATH>

Comment 16 Arano-kai 2018-12-28 11:18:07 UTC
As for Comment 12, patch is right and applied. That is not right is python version:
# python3 -V
Python 3.5.3
# /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
...
>>>SystemError<<<: Parent module '' not loaded, cannot perform relative import

# python3 -V
Python 3.7.2rc1
# /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
...
>>>ImportError<<<: attempted relative import with no known parent package

So we need catch SystemError for older python.

Comment 17 dchang0 2018-12-29 05:14:01 UTC
(In reply to Arano-kai from comment #15)
> It is probably you hit by https://stackoverflow.com/q/5658622, that is some
> $PATH inconsistency from python view and shell view.
> Since systemd used and units run in isolated env, you can try to explicitly
> set $PATH by overriding glustereventsd.service:
> # echo $PATH
> <current shell PATH>
> # systemctl edit glustereventsd.service
> 
> [Service]
> Environment=PATH=<current shell PATH>

Thanks for the help! I ended up editing the PATH= line in the file /etc/init.d/glustereventsd

It seems to work now (starts without errors). I have not tested the actual functionality yet but will do so soon enough.

I would say that with the patches you provided in attachment 1517165 [details], this bug is fixed completely for me. Thanks again!

Comment 18 Sahina Bose 2019-02-14 07:10:46 UTC
Aravinda, we have multiple oVirt users hitting this issue. Can this be targeted to a 5.x release?

Comment 19 Worker Ant 2019-02-26 05:12:53 UTC
REVIEW: https://review.gluster.org/22262 (eventsapi: Fix Python3 compatibility issues) posted (#1) for review on release-5 by Aravinda VK

Comment 20 Worker Ant 2019-02-26 13:52:48 UTC
REVIEW: https://review.gluster.org/22262 (eventsapi: Fix Python3 compatibility issues) merged (#1) on release-5 by Aravinda VK

Comment 21 Shyamsundar 2019-03-27 13:44:02 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.5, please open a new bug report.

glusterfs-5.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2019-March/000119.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.