| tommylikehu | bnemec: thanks | 00:36 |
|---|---|---|
| openstackgerrit | wangxiyuan proposed openstack/oslo.limit master: Add limit check func https://review.openstack.org/596520 | 02:13 |
| *** phuongnh has joined #openstack-oslo | 03:37 | |
| openstackgerrit | wangxiyuan proposed openstack/oslo.limit master: Add limit check func https://review.openstack.org/596520 | 04:03 |
| *** bnemec has quit IRC | 05:30 | |
| *** bnemec has joined #openstack-oslo | 05:31 | |
| *** ykarel has joined #openstack-oslo | 06:01 | |
| ykarel | Hi i am seeing a strange error with python3 + keystonemiddleware and oslo.config >= 6.3.0, detected in glance. Don't know if it's already known, i reported https://bugs.launchpad.net/keystonemiddleware/+bug/1789351 | 06:03 |
| openstack | Launchpad bug 1789351 in oslo.config "Glance deployment with python3 + "keystone" paste_deploy flavor Fails" [Undecided,New] | 06:03 |
| ykarel | can someone check, dhellmann bnemec ^^ | 06:03 |
| *** links has joined #openstack-oslo | 06:16 | |
| *** rha has joined #openstack-oslo | 06:32 | |
| *** pcaruana has joined #openstack-oslo | 06:39 | |
| *** mattgo has joined #openstack-oslo | 07:00 | |
| *** rcernin has quit IRC | 07:02 | |
| *** dims has quit IRC | 07:08 | |
| *** dims has joined #openstack-oslo | 07:10 | |
| *** mattgo has quit IRC | 07:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack/devstack-plugin-kafka stable/queens: import zuul job settings from project-config https://review.openstack.org/591801 | 07:36 |
| *** cdent has joined #openstack-oslo | 07:39 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** mattgo has joined #openstack-oslo | 07:45 | |
| *** shardy has joined #openstack-oslo | 08:04 | |
| openstackgerrit | Merged openstack/devstack-plugin-kafka stable/queens: import zuul job settings from project-config https://review.openstack.org/591801 | 08:28 |
| *** ykarel|lunch is now known as ykarel | 08:40 | |
| *** e0ne has joined #openstack-oslo | 08:43 | |
| *** pbourke has quit IRC | 08:47 | |
| *** pbourke has joined #openstack-oslo | 08:49 | |
| *** hoangcx has joined #openstack-oslo | 08:56 | |
| *** rcernin has joined #openstack-oslo | 08:59 | |
| *** sambetts|afk has quit IRC | 09:03 | |
| *** sambetts_ has joined #openstack-oslo | 09:06 | |
| *** takamatsu has joined #openstack-oslo | 09:15 | |
| *** hoonetorg has quit IRC | 09:20 | |
| *** cdent has quit IRC | 09:45 | |
| *** rcernin has quit IRC | 09:47 | |
| *** cdent has joined #openstack-oslo | 10:20 | |
| *** nicolasbock has joined #openstack-oslo | 10:20 | |
| *** stephenfin has quit IRC | 10:35 | |
| *** stephenfin has joined #openstack-oslo | 10:36 | |
| *** mmethot has joined #openstack-oslo | 11:48 | |
| *** mattgo has quit IRC | 11:49 | |
| *** giblet_off is now known as gibi | 11:49 | |
| openstackgerrit | Merged openstack/oslo.log master: Filter args dict in JSONFormatter https://review.openstack.org/596850 | 12:02 |
| *** mattgo has joined #openstack-oslo | 12:03 | |
| openstackgerrit | Merged openstack/oslo.service stable/rocky: Add Python 3.5 jobs https://review.openstack.org/596897 | 12:04 |
| openstackgerrit | Merged openstack/oslo.service stable/rocky: Skips signal handling on Windows https://review.openstack.org/596837 | 12:04 |
| *** raildo has joined #openstack-oslo | 12:08 | |
| *** sambetts_ is now known as sambetts | 12:26 | |
| *** ansmith has joined #openstack-oslo | 12:27 | |
| *** cdent has quit IRC | 12:46 | |
| openstackgerrit | Eric Fried proposed openstack/oslo.utils master: UUID sentinel https://review.openstack.org/594179 | 12:55 |
| *** kgiusti has joined #openstack-oslo | 12:56 | |
| *** r-mibu has joined #openstack-oslo | 13:02 | |
| *** phuongnh has quit IRC | 13:15 | |
| *** cdent has joined #openstack-oslo | 13:29 | |
| *** stephenfin has quit IRC | 13:33 | |
| *** stephenfin has joined #openstack-oslo | 13:34 | |
| *** purplerbot has joined #openstack-oslo | 13:34 | |
| *** moguimar has quit IRC | 13:35 | |
| *** moguimar has joined #openstack-oslo | 13:44 | |
| openstackgerrit | Andy Smith proposed openstack/oslo.messaging master: Remove the deprecated ZeroMQ driver https://review.openstack.org/596882 | 14:01 |
| *** links has quit IRC | 14:27 | |
| *** pcaruana has quit IRC | 14:29 | |
| *** pcaruana has joined #openstack-oslo | 14:30 | |
| *** pbourke has quit IRC | 14:55 | |
| *** e0ne has quit IRC | 14:56 | |
| *** nicolasbock has quit IRC | 14:56 | |
| *** nicolasbock has joined #openstack-oslo | 14:59 | |
| *** pcaruana has quit IRC | 15:00 | |
| *** pbourke has joined #openstack-oslo | 15:01 | |
| *** jroll has quit IRC | 15:05 | |
| *** jroll has joined #openstack-oslo | 15:06 | |
| *** dkehn has joined #openstack-oslo | 15:09 | |
| *** ykarel is now known as ykarel|away | 15:19 | |
| *** ducnv_ has joined #openstack-oslo | 15:22 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.utils master: Fix docstring formatting nit in uuidsentinel https://review.openstack.org/597157 | 15:26 |
| *** ducnv_ has quit IRC | 15:33 | |
| *** ducnv_ has joined #openstack-oslo | 15:33 | |
| *** ykarel|away has quit IRC | 15:36 | |
| *** ykarel|away has joined #openstack-oslo | 15:37 | |
| *** mattgo has quit IRC | 15:39 | |
| *** jhesketh has quit IRC | 15:41 | |
| *** jhesketh has joined #openstack-oslo | 15:41 | |
| *** ykarel|away has quit IRC | 15:42 | |
| *** Tahvok has joined #openstack-oslo | 16:00 | |
| *** raildo has quit IRC | 16:03 | |
| *** e0ne has joined #openstack-oslo | 16:06 | |
| *** snapiri has quit IRC | 16:08 | |
| *** dave-mccowan has joined #openstack-oslo | 16:12 | |
| *** raildo has joined #openstack-oslo | 16:13 | |
| *** raildo has quit IRC | 16:15 | |
| *** raildo has joined #openstack-oslo | 16:16 | |
| Tahvok | Hey guys! I was referred here from nova guys.. I'm having some nova-conductor issues, when it's consuming a lot of cpu (more than it should have), and is probably the cause for the high cpu usage of rabbit as well. | 16:16 |
| Tahvok | When doing a debug=true, we see lots of reply messages: http://paste.openstack.org/show/nqYc4ghBYpLAFvinYYAp/ | 16:17 |
| Tahvok | But when trying oslo.messaging=DEBUG, we see this: http://paste.openstack.org/show/728981/ | 16:18 |
| Tahvok | We're not sure what this unknown messages are.. Is there a way to understand where they're coming from? | 16:18 |
| kgiusti | Tahvok: give me a few minutes I'll take a look at the logs | 16:20 |
| *** raildo has quit IRC | 16:23 | |
| *** raildo has joined #openstack-oslo | 16:23 | |
| *** njohnston_ has joined #openstack-oslo | 16:38 | |
| *** njohnston has quit IRC | 16:39 | |
| *** njohnston_ is now known as njohnston | 16:39 | |
| *** sambetts is now known as sambetts|afk | 16:41 | |
| kgiusti | Tahvok: sorry for the delay. The first log simply show several server processes getting requests and sending the corresponding responses - lots of traffic but nothing unusual as far as I can tell. | 16:54 |
| kgiusti | Tahvok: the second log - not much to tell since the exception is being raised w/o passing in the topic, etc. | 16:55 |
| Tahvok | kgiusti: so is there a way to debug what is going on in the second log? | 16:55 |
| kgiusti | Tahvok: AFAIKT the exception is thrown when no messages arrive during a poll | 16:56 |
| Tahvok | kgiusti: oh... so it's polling for messages, doesn't get anything, and then throw the exception? | 16:56 |
| kgiusti | Tahvok: yes, IIUC | 16:57 |
| Tahvok | So the question is who should answer the message it sent? Is there a way to know what the message contains that it's sending and awaiting reply to? | 16:58 |
| kgiusti | Tahvok: are any of the nova clients logging timeout errors? | 16:59 |
| Tahvok | kgiusti: I've been checking them, but didn't find any with a problem | 17:00 |
| Tahvok | nova is working as it should, I can create instances and all. I'm just concerned of the high cpu usage of rabbit (100%-400% on 32 core machine), and spikes of nova-conductor services: http://paste.openstack.org/show/728925/ | 17:01 |
| Tahvok | Also we see about ~100 messages a second of nova-conductor, which isn't supposed to happen to my understanding | 17:02 |
| kgiusti | Tahvok: yeah that first log showed a _lot_ of incoming messages coming in very fast. | 17:04 |
| kgiusti | Tahvok: but sadly the logs don't give us a clue about the addresses... | 17:05 |
| kgiusti | Tahvok: does rabbitmqctl show any queues with lots message activity? | 17:07 |
| kgiusti | Tahvok: I think there's also a logging problem in oslo.messaging where you have to add both "oslo.messaging=DEBUG,oslo_messaging=DEBUG" in order to see all logging (note the '.' and the '_') | 17:10 |
| Tahvok | kgiusti: not that I see anything: http://paste.openstack.org/show/HblZ62II4qKg77UrhbQX/ | 17:10 |
| Tahvok | kgiusti: will try that now | 17:10 |
| *** r-mibu has quit IRC | 17:12 | |
| openstackgerrit | Merged openstack/oslo.utils master: UUID sentinel https://review.openstack.org/594179 | 17:13 |
| Tahvok | kgiusti: yep, that fix gave me much more: http://paste.openstack.org/show/zkThdO4ukCb5fqvvGLoI/ | 17:13 |
| Tahvok | I think it's now showing me the first log and the second together | 17:14 |
| kgiusti | Tahvok: I'm on the phone at the moment.... | 17:17 |
| *** ducnv_ has quit IRC | 17:18 | |
| Tahvok | So, I've reduced the number of workers to 1, so the log would be only from one worker.. And now it's more organized: http://paste.openstack.org/show/Iz2EukH6oxZnmdK3OKDX/ | 17:21 |
| *** raildo has quit IRC | 17:35 | |
| *** raildo has joined #openstack-oslo | 17:38 | |
| *** raildo_ has joined #openstack-oslo | 17:42 | |
| *** raildo has quit IRC | 17:45 | |
| kgiusti | Tahvok: sorry - back now. Right now the flow is consistent: 1) recv req, 2) poll timeout (spurious) 3) reply sent - you can see how the msg_ids correlate | 17:46 |
| *** raildo_ is now known as raildo | 17:46 | |
| kgiusti | Tahvok: that's logging for an RPC server. I don't see any client requests in that log | 17:46 |
| Tahvok | kgiusti: yep, except I don't see where the messages are coming from | 17:46 |
| kgiusti | Tahvok: yeah the logs not giving us that | 17:47 |
| * kgiusti shakes tiny fist at screen | 17:47 | |
| Tahvok | kgiusti: Well, I will try to investigate more tomorrow. Thanks a lot! | 17:54 |
| Tahvok | dansmith: sean-k-mooney : thanks a lot as well! Will continue tomorrow | 17:54 |
| openstackgerrit | Andy Smith proposed openstack/oslo.messaging master: Remove the deprecated ZeroMQ driver https://review.openstack.org/596882 | 18:01 |
| *** shardy has quit IRC | 18:06 | |
| dhellmann | moguimar: I don't suppose you have any idea why this castellan patch is failing, do you? https://review.openstack.org/#/c/588780/ | 18:13 |
| *** dave-mccowan has quit IRC | 18:36 | |
| *** AJaeger has joined #openstack-oslo | 18:37 | |
| *** dave-mccowan has joined #openstack-oslo | 18:46 | |
| AJaeger | dhellmann: shall I split up https://review.openstack.org/#/c/588780/ so that we can move forward with python3-first? | 19:00 |
| *** kgiusti has quit IRC | 19:05 | |
| *** e0ne has quit IRC | 19:08 | |
| AJaeger | dhellmann: done it quickly, see https://review.openstack.org/#/c/588780/ and https://review.openstack.org/597214 | 19:18 |
| *** hoonetorg has joined #openstack-oslo | 19:22 | |
| *** e0ne has joined #openstack-oslo | 19:23 | |
| *** hoonetorg has quit IRC | 19:24 | |
| *** dave-mccowan has quit IRC | 19:26 | |
| dhellmann | AJaeger : thanks, sorry, I'm working on something else this afternoon | 19:34 |
| AJaeger | dhellmann: no worries... | 19:34 |
| *** njohnston has quit IRC | 19:42 | |
| *** njohnston has joined #openstack-oslo | 19:43 | |
| AJaeger | dhellmann, bnemec, https://review.openstack.org/#/c/588780/ passed tests - will either of you +A? | 19:47 |
| AJaeger | wow, https://review.openstack.org/597214 passed - the job works now suddenly? | 19:54 |
| bnemec | AJaeger: Your first patch failed in a different way from the previous ones. Maybe your fetch-subunit-output fix got it working and it failed on a heisenbug instead? | 19:57 |
| AJaeger | maybe... | 20:00 |
| openstackgerrit | Andy Smith proposed openstack/oslo.messaging master: Remove the deprecated ZeroMQ driver https://review.openstack.org/596882 | 20:01 |
| *** raildo has quit IRC | 20:05 | |
| *** ansmith has quit IRC | 20:30 | |
| openstackgerrit | Merged openstack/oslo.utils master: Fix docstring formatting nit in uuidsentinel https://review.openstack.org/597157 | 20:53 |
| *** e0ne has quit IRC | 20:58 | |
| *** bnemec has quit IRC | 21:12 | |
| *** bnemec has joined #openstack-oslo | 21:13 | |
| *** ansmith has joined #openstack-oslo | 21:28 | |
| *** hoonetorg has joined #openstack-oslo | 21:36 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Ignore --find-links in requirements file https://review.openstack.org/597290 | 21:42 |
| *** rcernin has joined #openstack-oslo | 21:46 | |
| *** cdent has quit IRC | 22:16 | |
| *** sambetts|afk has quit IRC | 23:01 | |
| *** sambetts_ has joined #openstack-oslo | 23:05 | |
| *** threestrands has joined #openstack-oslo | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!