Monday, 2018-04-30

*** edmondsw has joined #openstack-powervm00:28
*** edmondsw has quit IRC00:33
*** edmondsw has joined #openstack-powervm02:16
*** edmondsw has quit IRC02:21
*** k0da has joined #openstack-powervm04:53
*** edmondsw has joined #openstack-powervm05:53
*** k0da has quit IRC05:57
*** edmondsw has quit IRC05:58
*** Guest1988 has joined #openstack-powervm07:11
*** Guest1988 has quit IRC07:17
*** k0da has joined #openstack-powervm07:35
*** edmondsw has joined #openstack-powervm07:42
*** edmondsw has quit IRC07:47
*** edmondsw has joined #openstack-powervm12:07
*** esberglu has joined #openstack-powervm13:31
esbergluedmondsw: efried: Production CI is down. It's hitting similar jenkins IOErrors that staging was hitting13:36
esbergluIt ended up being an issue with the nic type on the staging mgmt server13:36
esbergluI'm checking with thorst to see if the same issue could be hitting prod13:37
*** tjakobs has joined #openstack-powervm13:37
*** tjakobs has quit IRC13:53
*** tjakobs has joined #openstack-powervm14:20
*** k0da has quit IRC16:02
esbergluefried: edmondsw: Update on prod CI. Was not the nic type issue hit on staging17:24
esbergluLooks like it might be related to running glance under uwsgi17:24
efriedthought we fixed that17:24
edmondswrather, I thought we had a workaround for that, because I don't think the community changes merged17:24
esbergluefried: edmondsw: We ran into issues back when we were upgrading to pike instead of queens17:25
esbergluWe had to manually patch in changes because they never got merged to pike17:25
esbergluBut those fixes are in queens17:25
esbergluHowever, uwsgi is still not technically supported by glance for queens17:25
edmondswwhy didn't we hit this in staging?17:26
edmondswand can you use httpd instead of uwsgi just for glance?17:26
edmondswor what is the recommendation17:26
esbergluThat's the patch to do so. I'm patching into the env and redeploying right now to test17:27
edmondswwe should make a stink with the glance guys that they need to merge these patches...17:28
edmondswthey don't want folks to use uwsgi, but they won't merge the patch to allow folks not to?!?17:28
esbergluedmondsw: Yep17:28
esbergluedmondsw: I'm thinking that since on staging I upgraded to pike, then from pike to queens17:29
esbergluThe issue we're hitting might have been missed17:30
edmondswesberglu I dug up the patches I was thinking of, and I think they should both be there for queens17:40
edmondswignore that second one... meant this:17:41
edmondswesberglu they merged for queens, but not pike17:42
edmondswso maybe there was a problem with those? Or we did something to workaround the issue for pike, and that causes a problem with queens?17:43
esbergluedmondsw: Yep. That's why I was surprised to hit this on production17:44
esbergluBut the error message is a timeout from glance calling uwsgi_response_write_body_do()17:44
esbergluAnd since it's recommended NOT to use uwsgi on queens, I'm giving that a try17:47
*** k0da has joined #openstack-powervm17:50
esbergluMight be some sort of scaling issue as well. I'm not super confident that this switch will solve it, but it's the only idea I have so far17:50
*** k0da has quit IRC18:12
*** k0da has joined #openstack-powervm18:23
*** k0da has quit IRC19:11
*** k0da has joined #openstack-powervm19:56
esbergluedmondsw: Looks like that patch to disable uwsgi for glance is gonna work20:16
esbergluAt least it got past the previous point of failure20:16
edmondswesberglu good20:17
esbergluedmondsw: CI should be up within the hour if no other issues20:18
esbergluI don't think it was blocking anything though20:18
esbergluefried: Prod logserver is now running with
esbergluI linked an example in the comments20:37
efriedesberglu: Cool man.  Did you pick up PS11?20:38
esbergluefried: Yeah20:38
efriedesberglu: Oh, yeah, this isn't working at all the way it oughtta.  Are we stripping the color codes maybe?20:39
efriedesberglu: You completely replaced the os-loganalyze we had patched in there before?20:39
esbergluefried: Yep. This is os_loganalyze master with only the above patch20:39
efriedthanks dude.20:39
esbergluefried: np20:40
esbergluefried: I'm just gonna leave it alone for now. If it takes a while to get this moving I can revert back to what we've been using20:40
efriedesberglu: can you PM me the deets so I can go replace it as Matthieu comes up with more patch sets?  (So we don't have to bug you with it)20:40
efriedesberglu: Hum, this seems also not to make a difference when I remove colorize=true20:42
esbergluefried: Could also be how we are outputting the journalctl logs to files20:44
efriedesberglu: When I download the file, it doesn't have the ANSI codes in it.20:45
efriedesberglu: -a should be preserving the color codes.20:45
efriedesberglu: I suspect os-loganalyze is stripping them.20:45
efriedesberglu: Where are the actual source files on the log server?20:46
esbergluefried: /srv/static/logs/20:46
esbergluThen whatever path is in the URL20:46
esbergluefried: From the commit message "ANSI escape codes are removed for readability"20:47
esbergluI that applies to downloading as well20:47
esberglu*I think that20:47
efriedesberglu: Yeah, the source still has 'em in there.20:47
esbergluefried: I believe your changes also stripped the ANSI codes, but only when displaying20:48
efriedesberglu: That's correct.  But that was for the other colorizer.  We hadn't played with my ANSI one in a while.20:48
efriedI'll leave this alone until Matthieu gets a chance to look at it; then maybe I'll mess with mine again.20:49
esbergluefried: You aware of what patches were applied before I switched it today?20:50
efriedesberglu: Ah, okay.  Thanks.20:51
*** edmondsw has quit IRC21:07
*** edmondsw_ has joined #openstack-powervm21:11
esbergluefried: edmondsw_: CI is up and running on queens21:15
*** k0da has quit IRC21:15
*** edmondsw_ has quit IRC21:15
*** esberglu has quit IRC21:22
*** k0da has joined #openstack-powervm21:28
*** k0da has quit IRC21:30
*** tjakobs has quit IRC22:13

Generated by 2.15.3 by Marius Gedminas - find it at!