| mattoliverau | oh wow. Lucky you :) | 00:08 |
|---|---|---|
| mattoliverau | timburke: I'll chair, especially if it's just be an update meeting. Just ping me when you know :) | 00:09 |
| *** zaitcev_ has quit IRC | 00:17 | |
| *** zaitcev_ has joined #openstack-swift | 00:31 | |
| *** ChanServ sets mode: +v zaitcev_ | 00:31 | |
| *** gyee has quit IRC | 00:36 | |
| *** altlogbot_3 has quit IRC | 00:46 | |
| *** altlogbot_0 has joined #openstack-swift | 00:48 | |
| *** tdasilva has quit IRC | 01:30 | |
| *** tdasilva has joined #openstack-swift | 01:30 | |
| *** ChanServ sets mode: +v tdasilva | 01:30 | |
| *** BjoernT has joined #openstack-swift | 02:25 | |
| *** tdasilva has quit IRC | 02:31 | |
| *** tdasilva has joined #openstack-swift | 02:31 | |
| *** ChanServ sets mode: +v tdasilva | 02:31 | |
| *** diablo_rojo has joined #openstack-swift | 02:43 | |
| *** spsurya has joined #openstack-swift | 03:03 | |
| timburke | thanks mattoliverau! | 03:19 |
| *** tdasilva has quit IRC | 03:32 | |
| *** tdasilva has joined #openstack-swift | 03:33 | |
| *** gkadam has joined #openstack-swift | 03:33 | |
| *** ChanServ sets mode: +v tdasilva | 03:33 | |
| *** gkadam has quit IRC | 03:40 | |
| *** BjoernT has quit IRC | 03:42 | |
| *** psachin has joined #openstack-swift | 03:43 | |
| *** tkajinam has quit IRC | 04:09 | |
| *** tkajinam has joined #openstack-swift | 04:16 | |
| *** tkajinam has quit IRC | 04:17 | |
| *** tkajinam has joined #openstack-swift | 04:18 | |
| *** tdasilva has quit IRC | 04:34 | |
| *** tdasilva has joined #openstack-swift | 04:34 | |
| *** ChanServ sets mode: +v tdasilva | 04:34 | |
| openstackgerrit | Merged openstack/swift master: py3: fix up swift-orphans https://review.opendev.org/670932 | 04:57 |
| openstackgerrit | Merged openstack/swift master: Fix test_parse_get_node_args https://review.opendev.org/670894 | 04:57 |
| *** tdasilva has quit IRC | 04:58 | |
| *** diablo_rojo has quit IRC | 05:27 | |
| *** persia has joined #openstack-swift | 06:00 | |
| *** persia has quit IRC | 06:03 | |
| *** persia has joined #openstack-swift | 06:05 | |
| *** persia has quit IRC | 06:05 | |
| *** persia has joined #openstack-swift | 06:05 | |
| openstackgerrit | Merged openstack/swift master: Allow "static symlinks" https://review.opendev.org/633094 | 06:20 |
| *** zaitcev_ has quit IRC | 06:21 | |
| *** persia has left #openstack-swift | 06:27 | |
| *** rcernin has quit IRC | 06:35 | |
| *** zaitcev_ has joined #openstack-swift | 06:35 | |
| *** ChanServ sets mode: +v zaitcev_ | 06:35 | |
| mattoliverau | congrats clayg and timburke, static symlinks have merged!! \o/ | 06:44 |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: sharder: Keep cleaving on empty shard ranges https://review.opendev.org/675820 | 07:05 |
| *** psachin has quit IRC | 07:25 | |
| *** ianychoi has quit IRC | 07:46 | |
| *** ianychoi has joined #openstack-swift | 07:47 | |
| *** e0ne has joined #openstack-swift | 08:26 | |
| *** tkajinam has quit IRC | 08:32 | |
| openstackgerrit | Merged openstack/swift master: Update docker image to latest Alpine 3.10.1 https://review.opendev.org/677293 | 08:47 |
| *** tdasilva has joined #openstack-swift | 08:54 | |
| *** ChanServ sets mode: +v tdasilva | 08:54 | |
| openstackgerrit | Merged openstack/swift master: Add Dockerfile to build a py3 swift docker image https://review.opendev.org/677294 | 08:59 |
| *** rcernin has joined #openstack-swift | 09:15 | |
| *** zaitcev_ has quit IRC | 09:17 | |
| *** zaitcev_ has joined #openstack-swift | 09:31 | |
| *** ChanServ sets mode: +v zaitcev_ | 09:31 | |
| *** zaitcev_ has quit IRC | 09:48 | |
| *** zaitcev_ has joined #openstack-swift | 10:02 | |
| *** ChanServ sets mode: +v zaitcev_ | 10:02 | |
| *** rcernin has quit IRC | 10:28 | |
| *** zaitcev_ has quit IRC | 10:37 | |
| *** gkadam has joined #openstack-swift | 10:38 | |
| *** dosaboy has quit IRC | 10:42 | |
| *** aluria has quit IRC | 10:42 | |
| *** zaitcev_ has joined #openstack-swift | 10:51 | |
| *** ChanServ sets mode: +v zaitcev_ | 10:51 | |
| *** gkadam is now known as gkadam-afk | 11:00 | |
| *** tesseract has joined #openstack-swift | 11:08 | |
| *** tomha has joined #openstack-swift | 11:23 | |
| *** tomha has quit IRC | 11:24 | |
| *** dosaboy has joined #openstack-swift | 11:28 | |
| *** gkadam-afk is now known as gkadam | 11:42 | |
| *** henriqueof has quit IRC | 12:09 | |
| *** gkadam has quit IRC | 12:32 | |
| *** zaitcev_ has quit IRC | 12:53 | |
| *** henriqueof has joined #openstack-swift | 12:56 | |
| *** BjoernT has joined #openstack-swift | 13:05 | |
| *** zaitcev_ has joined #openstack-swift | 13:07 | |
| *** ChanServ sets mode: +v zaitcev_ | 13:07 | |
| openstackgerrit | Merged openstack/swift master: Rename symlink method https://review.opendev.org/677247 | 13:12 |
| *** tomha has joined #openstack-swift | 13:18 | |
| *** tomha has quit IRC | 13:20 | |
| *** zaitcev__ has joined #openstack-swift | 13:25 | |
| *** ChanServ sets mode: +v zaitcev__ | 13:25 | |
| *** zaitcev_ has quit IRC | 13:29 | |
| openstackgerrit | Merged openstack/swift stable/rocky: Remove openSUSE 42.3 job https://review.opendev.org/677159 | 13:54 |
| *** henriqueof has quit IRC | 14:03 | |
| *** zaitcev_ has joined #openstack-swift | 14:14 | |
| *** ChanServ sets mode: +v zaitcev_ | 14:14 | |
| clayg | rledisez: on lp bug #1840780 are you running with eventlet debug on? | 14:15 |
| openstack | Launchpad bug 1840780 in OpenStack Object Storage (swift) "Eventlet exception "do not call blocking functions from the mainloop"" [Undecided,New] https://launchpad.net/bugs/1840780 | 14:15 |
| clayg | the traceback reminded me of There's https://eventlet.net/doc/modules/debug.html#eventlet.debug.hub_blocking_detection which i've played with before | 14:15 |
| clayg | maybe some call that should be tpool'd isn't? | 14:15 |
| rledisez | clayg: no, debug is not enabled | 14:17 |
| *** zaitcev__ has quit IRC | 14:18 | |
| rledisez | we didn't have that with our previous deployement (swift 2.18 / eventlet 0.21). not which component is generating this error | 14:18 |
| rledisez | *not sure | 14:18 |
| openstackgerrit | Merged openstack/swift master: Fix symlink docstring https://review.opendev.org/677244 | 14:24 |
| *** openstackstatus has quit IRC | 14:58 | |
| *** openstack has joined #openstack-swift | 15:01 | |
| *** ChanServ sets mode: +o openstack | 15:01 | |
| *** pvradu has joined #openstack-swift | 15:03 | |
| *** baojg has joined #openstack-swift | 15:36 | |
| *** baojg has quit IRC | 15:36 | |
| *** baojg has joined #openstack-swift | 15:37 | |
| *** baojg has quit IRC | 15:37 | |
| *** baojg has joined #openstack-swift | 15:38 | |
| *** baojg has quit IRC | 15:38 | |
| *** baojg has joined #openstack-swift | 15:39 | |
| *** baojg has quit IRC | 15:40 | |
| *** gyee has joined #openstack-swift | 15:45 | |
| *** pvradu has quit IRC | 15:50 | |
| *** zaitcev_ has quit IRC | 15:59 | |
| timburke | i'm reminded of a comment on https://review.opendev.org/#/c/493636/: "I'm not sure how to solve it generally ... I guess next time we find a lockup under load at scale we just ask sam for help? It totally makes sense to me after he explained it!" | 16:05 |
| patchbot | patch 493636 - swift - Fix deadlock when logging from a tpool thread. (MERGED) - 7 patch sets | 16:05 |
| timburke | though i might replace "a lockup under load at scale" with "an eventlet-related bug" :P | 16:05 |
| *** tdasilva has quit IRC | 16:11 | |
| timburke | huh. why/how is traceback.print_exception() writing stuff to a swift.common.utils.LoggerFileObject? | 16:13 |
| *** zaitcev_ has joined #openstack-swift | 16:13 | |
| *** ChanServ sets mode: +v zaitcev_ | 16:13 | |
| clayg | cause we grab stdour? | 16:19 |
| timburke | but like, why is it running through a *logger* at all? | 16:21 |
| timburke | 'cause that seems like the crux of the issue | 16:21 |
| timburke | there's some sort of an exception in the callback at https://github.com/eventlet/eventlet/blob/v0.25.0/eventlet/hubs/poll.py#L110-L116 so we go to squelch it | 16:23 |
| timburke | first thing, right off the bat, is to print the traceback: https://github.com/eventlet/eventlet/blob/v0.25.0/eventlet/hubs/hub.py#L300-L301 | 16:23 |
| timburke | but if that gets us down into some type of logging... we gotta go acquire the logging lock | 16:24 |
| *** e0ne has quit IRC | 16:25 | |
| timburke | if anybody else is trying to do logging at the same time, that trampolines and we get this traceback :-/ | 16:25 |
| timburke | (i wonder what the error was that we were trying to log...) | 16:26 |
| *** spsurya has quit IRC | 16:36 | |
| *** diablo_rojo has quit IRC | 16:58 | |
| *** diablo_rojo has joined #openstack-swift | 16:59 | |
| *** gyee has quit IRC | 17:09 | |
| *** tesseract has quit IRC | 17:10 | |
| *** gyee has joined #openstack-swift | 17:22 | |
| *** zaitcev_ has quit IRC | 17:45 | |
| *** tesseract has joined #openstack-swift | 17:45 | |
| *** zaitcev_ has joined #openstack-swift | 17:58 | |
| *** ChanServ sets mode: +v zaitcev_ | 17:58 | |
| *** zaitcev_ is now known as zaitcev | 17:58 | |
| rledisez | timburke: could it be related to the fact we are using the log_custom_handlers feature? | 18:11 |
| rledisez | (but still we never saw that stack in swift 2.18/eventlet 0.21 and for now we have only 20 nodes on 2.22/0.25 | 18:12 |
| timburke | ...maybe? i've never used that... i feel like it'll likely depend on what the handler's doing, if that's it | 18:18 |
| timburke | have you managed to get a feel for what would cause the error on accept()? at least, it *seems like* there must've been an error in accept()... | 18:20 |
| rledisez | timburke: no. i can't corellate a trafic increase or load or something else | 18:31 |
| *** zaitcev has quit IRC | 19:11 | |
| *** zaitcev has joined #openstack-swift | 19:25 | |
| *** ChanServ sets mode: +v zaitcev | 19:25 | |
| *** e0ne has joined #openstack-swift | 19:32 | |
| *** diablo_rojo has quit IRC | 20:10 | |
| *** diablo_rojo has joined #openstack-swift | 20:27 | |
| *** zaitcev_ has joined #openstack-swift | 20:43 | |
| *** ChanServ sets mode: +v zaitcev_ | 20:43 | |
| *** zaitcev has quit IRC | 20:46 | |
| *** e0ne has quit IRC | 20:50 | |
| *** diablo_rojo has quit IRC | 21:31 | |
| *** tesseract has quit IRC | 21:32 | |
| *** rcernin has joined #openstack-swift | 22:00 | |
| *** zaitcev_ has quit IRC | 22:19 | |
| *** zaitcev_ has joined #openstack-swift | 22:33 | |
| *** ChanServ sets mode: +v zaitcev_ | 22:33 | |
| *** BjoernT has quit IRC | 22:46 | |
| *** tkajinam has joined #openstack-swift | 22:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!