Thursday, 2017-08-10

*** thorst has joined #openstack-powervm00:04
*** edmondsw has joined #openstack-powervm00:30
*** thorst has quit IRC00:31
*** edmondsw has quit IRC00:34
*** thorst has joined #openstack-powervm01:08
*** thorst has quit IRC01:08
*** thorst has joined #openstack-powervm01:09
*** thorst has quit IRC01:13
*** svenkat has joined #openstack-powervm01:21
*** thorst has joined #openstack-powervm01:25
*** thorst has quit IRC01:27
*** svenkat has quit IRC02:23
*** thorst has joined #openstack-powervm02:28
*** thorst has quit IRC02:29
*** chhavi has quit IRC02:33
*** esberglu has quit IRC02:40
*** thorst has joined #openstack-powervm04:04
*** thorst has quit IRC04:09
*** chhavi has joined #openstack-powervm04:13
*** thorst has joined #openstack-powervm04:26
*** thorst has quit IRC04:30
*** k0da has joined #openstack-powervm06:11
*** thorst has joined #openstack-powervm06:26
*** thorst has quit IRC06:31
*** k0da has quit IRC07:11
*** thorst has joined #openstack-powervm08:27
*** thorst has quit IRC08:32
*** svenkat has joined #openstack-powervm09:06
*** svenkat has quit IRC09:43
*** apearson has joined #openstack-powervm09:49
*** thorst has joined #openstack-powervm10:28
*** svenkat has joined #openstack-powervm10:30
*** svenkat has quit IRC10:31
*** thorst has quit IRC10:33
*** apearson has quit IRC10:46
*** smatzek has joined #openstack-powervm11:49
*** svenkat has joined #openstack-powervm11:54
*** esberglu has joined #openstack-powervm11:59
*** svenkat_ has joined #openstack-powervm12:00
*** svenkat has quit IRC12:01
*** svenkat_ is now known as svenkat12:01
*** thorst has joined #openstack-powervm12:02
*** esberglu has quit IRC12:04
*** edmondsw has joined #openstack-powervm12:27
*** esberglu has joined #openstack-powervm13:00
efriedesberglu See the dev ML - tonyb is asking for help with our -powervm branches wrt requirements thaw.13:12
*** kylek3h has joined #openstack-powervm13:15
*** k0da has joined #openstack-powervm13:57
esbergluefried: Talking about it in #openstack-release. You know if our repos are under TC governance?13:58
esbergluedmondsw: thorst: ^13:58
efriedThey're not.13:58
thorstadreznec: you want to participate in that?13:58
esbergluefried: edmondsw:
esbergluSeeing some runs like ^14:28
esbergluwhere the compute isn't starting up properly14:28
esbergluWith Internal Server Error 50014:29
efriedesberglu I'll say this again: if we're only failing 56 tests when the compute service doesn't start at all... do we need the other 718 at all?14:31
edmondswmost likely not14:32
efriedHere's another thing that's been on my mind.14:32
efriedOther CIs, when you click a .log.gz, it opens in the browser, and even has convenient color coding and log level filters.14:32
efriedOurs.... download.14:32
edmondswI wonder why that is14:33
efriedesberglu Can you add to the wishlist to figure out why that is and how we can conform?14:33
efriedCause that's way more convenient than save, unzip, page.14:33
esbergluI think it's just txt.gz vs. log.gz14:34
esbergluI can probably just rename them14:34
efriedesberglu That compute log looks truncated.14:34
esbergluefried: Why do you say that? It goes from the service starting to the compute driver being shut down14:36
efriederrumuh... let me download it again.14:37
edmondswefried when I open n-cpu.log with less, it's missing several lines at the top that are there when I open with vi...14:37
edmondswsome kind of formatting issue?14:37
efriedoh, well wtf?14:37
edmondswand moving around the file in less I'm seeing weird behavior, other lines go missing and then come back...14:38
efriedyeahno, I'm not seeing any difference.  less, vi, gvim...14:38
efriedThey all start with -- Logs begin at Thu 2017-08-10 07:07:45 CDT, end at Thu 2017-08-10 07:45:23 CDT. --14:39
efriedAug 10 07:30:39.694738 powervm-ci-powervm-devstacked-33602 systemd[1]: Started Devstack devstack@n-cpu.service.14:39
efriedand end with Aug 10 07:34:58.689495 powervm-ci-powervm-devstacked-33602 nova-compute[29612]:             <ServerAdapter kxe="false" kb="CUR" schemaVersion="V1_6_0">14:39
efriedAug 10 07:34:58.689694 powervm-ci-powervm-devstacked-33602 nova-compute[29612]:                 <Metadata>14:39
edmondswyeah, that's what I see with vi14:39
edmondswwith less I'm seeing it start with this:14:39
edmondswAug 10 07:30:51.054148 powervm-ci-powervm-devstacked-33602 nova-compute[29612]: any related configuration options. Users of neutron, meanwhile, should use the14:39
edmondswmaybe it's something with my mac14:39
efried1551 lines.14:39
esbergluNope same for me I had a different n-cpu log open14:40
efriedokay phew.14:40
efriedWorld got really weird there for a sec.14:40
edmondswstill weird for me :)14:40
efriedI mean, that's still weird, but...14:40
*** tjakobs has joined #openstack-powervm14:40
efriededmondsw Are you using `less -R` ?14:40
edmondswsomeone else's problem :)14:40
efried227495 bytes unzipped14:41
efriedWhat does wc -l give you?14:41
edmondswdon't have the problem with less if I remove the -r14:41
edmondsw1551, but that wouldn't matter14:41
efriededmondsw -r or -R?14:41
efriedThey're different (opposite, almost, I think)14:42
edmondswhmm... -R I don't see the problem...14:42
efriedah, yes, I get the same bizarre behavior with -r14:42
edmondswok, I'll use -R... tx :)14:43
efriedAnd forget what I said about opposite.  -R is -r on roids.14:43
edmondsw'Unlike -r, the screen appearance is maintained correctly in most cases.'14:43
*** tjakobs has quit IRC14:44
esbergluedmondsw: efried: Yeah the log is getting cut off in some runs14:46
efriedIs this a fs space issue again?14:46
efriedor possibly a not-waiting-for-sync issue?14:47
esbergluI miss screen...14:47
esbergluI don't think it's a space issue14:47
edmondswesberglu are you grabbing the logs from disk, or running a journalctl command to generate them?14:49
esberglujournalctl command to output them to files then the files get copied to the logserver14:49
esbergluThe only time I'm seeing this is on these runs where the compute driver is failing to start14:50
edmondswyou wouldn't expect journalctl to have a sync issue14:50
esbergluHowever ALL of the logs could be getting cut off, there isn't a good reference point for the passing runs14:50
efriedesberglu Welll, we could shut down the service(s) before we collect the logs.14:52
efriedThen we'd have a reference point.14:53
esbergluefried: edmondsw: 568015:01
efriedesberglu Why not as part of process_logs?15:02
*** k0da has quit IRC15:03
esbergluIdk it's not really doing any log processing. I guess you could argue it's setting up for log processing?15:03
esbergluActually it can't be part of process logs as we have it right now15:04
esbergluBecause it has to happen before the journalctl output to files15:04
esberglu(now that could probably be moved to process_logs)15:05
esbergluefried: See ps215:09
efriedesberglu Like it.  Anything that reduces shell-code-in-yaml.15:11
esbergluefried: One more change in ps3. In browser logs15:21
efriedesberglu Did you try that out?  Does it work?15:22
efriedHonestly, I kinda doubt it's gonna have the fancy log filtering stuff.15:22
esbergluWell it has to be txt.gz if you want to view in the browser15:23
efriedgood enough for now.15:23
esbergluThats what it will look like15:23
efriedHm, yeah, the filtering stuff is coming through, but not the coloring.  Wonder what's up with that.15:24
esbergluNot the cleanest look, but in browser15:24
edmondswyep, better15:24
*** k0da has joined #openstack-powervm15:24
efriedesberglu It also doesn't actually appear to be gzipped.15:25
efried...when I download it.15:25
edmondswthat probably depends on how you download it, right?15:26
esbergluWhat are you viewing it with? Some tools automatically handle txt.gz15:26
esbergluCheck the file size15:26
esbergluYeah it's definitely zipped. 405K. Unzipped it's 4.4M15:27
efriedI was trying that, but you pulled the rug.15:27
esbergluefried: Should be good now15:27
efriedWell, I guess my browser (?) is unzipping it automatically15:28
efriedBut not renaming it.15:28
esbergluefried: Yeah that's why log.gz didn't work. The browser doesn't know how to handle it, but for txt.gz it does15:29
efriedAnyway, yeah, would like to get that coloring figured out too.  I'd rather suffer the inconvenience of downloading than look at it in b/w with the color code gorp.15:29
efriedBut let's not get distracted.15:29
esbergluefried: This is going to take a while to get into CI, image template rebuild and all that jazz15:30
efriedoh, boo.15:30
efriedthen we might as well get distracted, if it's going to be fairly easy to figure out.15:30
esbergluefried: And we can't make process_logs external to take advantage of the immediate changes like we can with other things15:31
efriedBecause IPs?15:32
esbergluefried: Does the coloring still work when you download?15:33
efriedesberglu Yes15:34
esbergluThis might just be a browser issue that we can't get around then15:34
efriedThat doesn't make sense, because in the same browser for other CIs, it shows up colored.15:35
esbergluOh okay didn't realize that15:35
efriedesberglu How does it show up for you?15:36
esbergluefried: Yeah thats colored15:36
efriedI just picked that one at random, btw.  They're all like that.15:36
efriedHum, if I look at the source, it's truly HTMLified.  I wonder if they're putting it through a postprocessor.15:36
efriedesberglu But you have a way to look at other orgs' CIs, right?15:37
esbergluefried: Yeah15:37
efriedesberglu Nyak, when I look at yours ( view-source: ) it's also HTML - just the classes are empty and the color codes aren't stripped.15:38
edmondswcompare the raw file contents when downloading the gz file and extracting locally (not having the browser do it) and see what's different15:54
efriedHow would I download the GZ file?  curl?15:55
efriedI can't even get curl --raw to stop decompressing it.15:58
*** dwayne has quit IRC16:34
edmondswefried go to the parent directory, right click the link to the gz file and "save link as"16:58
efriededmondsw When I do that, it unzips it for me anyway.16:58
efried(but doesn't rename it)16:58
efriededmondsw Still unzips it.16:59
edmondswthen you're probably stuck16:59
efriedIn fact, it may even be unzipping it on the server.16:59
edmondswit would have to be17:00
efriedwhich is weird.  Like, what's the point of having zippable content?17:00
edmondswsaves disk space, just not network bandwidth17:05
edmondswI'm guessing browsers don't have the builtin support for rendering these files that we assumed they did, so they do it server-side to make it work17:06
efriededmondsw Well, what's weird is, we're not doing any of that rendering on the server side, but *some* of it is happening anyway.17:11
efriedHm, if it's server side... I wonder if we just have an outdated apache or something on our log server.17:12
efriedor missing a plugin17:12
edmondswesberglu that sed command you've got doesn't seem right...17:19
edmondswin 568117:19
efriededmondsw What doesn't work?17:21
edmondswthe command :)17:21
edmondswI don't ever get cleared...17:22
edmondswI would also think we'd need to keep the first part of the hostname so we know what host it is17:22
edmondswesberglu efried figured out a command that would work and commented with that17:30
efriedesberglu See -nova for answer on the log coloring business.17:31
edmondswthat doesn't read well... meant to say "both of you... I figured out..."17:31
esbergluedmondsw: Yep that seems to work17:32
esbergluWaiting for my manual run to finish so I can test and then we can pull the trigger17:33
efriededmondsw How does it help us to keep the first part of the domain name?17:34
edmondswwe were before... I assumed that was how we were figuring out which host was used17:34
edmondswone problem with my proposal is if we ever print just the domain info, not a full hostname, it wouldn't touch that17:35
edmondswdo we do that?17:35
esbergluedmondsw: Not to my knowledge17:35
esbergluI will make sure that everything gets cleared in test17:36
esbergluefried: The first part of the domain name stays17:36
efriedfine fine.17:37
esbergluedmondsw: Is that 1 in getting substituted somehow?17:37
edmondswmeans the first () match17:38
edmondswthat's how it keeps the first part of the hostname17:38
esbergluefried: Catching up on the nova convo17:44
edmondswesberglu sorry, that sed command I recommended needs another tweak... commented17:47
esbergluedmondsw: np. Done17:50
esbergluefried: Sounds like it's time to get this mod_wsgi, uwsgi business figured out17:51
esbergluYou're stacking with that change again?17:51
efriedesberglu Working on it in -nova17:51
efriedesberglu It kinda looked like /tmp ran out of space.17:52
efriedWhile creating the image17:52
efriedSpeculation is that under uwsgi, we copy the whole image through /tmp17:52
efriedwhich is... silly.17:53
*** chhavi has quit IRC18:08
*** dwayne has joined #openstack-powervm18:12
*** smatzek has quit IRC18:32
esbergluedmondsw: efried: Thought I was merging the neo-os-ci change, actually merged the powervm-ci change. Not an issue, just means that one more changeset will be going up18:36
*** smatzek has joined #openstack-powervm18:36
esbergluedmondsw: efried: 568218:40
efriedesberglu Can you copy out /var/log/apache2/* in that same patch?18:45
esbergluefried: Sure18:45
efriedesberglu Should be moot when we get uwsgi figured out... which looks like it's getting close.18:45
esbergluefried: Actually that would have to go in the jenkins job config. I can still do it in a separate patch if you would like18:47
esbergluOr is it not worth spending time on?18:48
efriedHuh?  You can't add [ for f in `ls /var/log/apache2/*.log`; do cp $f /opt/stack/logs/$f.txt; done ] in that same script?18:49
esbergluefried: Yeah brain fart18:50
esbergluAny idea why that second command isn't picking up the log files?18:56
efriedI wonder if sudo refuses to expand wildcards18:57
efriedyou know, for safety.18:57
efriedesberglu Anyway, just copy everything out of there.18:57
*** efried is now known as efried_afk19:02
*** openstackgerrit has quit IRC19:03
*** efried_afk is now known as efried19:27
efriedesberglu uwsgi might be fixed by -- trying now.19:33
esbergluefried: Nice19:44
esbergluefried: edmondsw: thorst: Starting the process to tag RC1 and create the pike branches. For anything to get in after that point it will have to be backported to pike19:56
efriedesberglu Go.19:56
*** smatzek has quit IRC20:07
esbergluPike branches are all up20:13
esbergluefried: Did your stack work?20:17
efriedesberglu No.  But it may be because I had stale settings in my apache confs.  Cleared 'em out, trying again.20:17
efriedesberglu Do we have a devstack bug open for the stupid neutron conf dir bs?20:18
thorstesberglu: cool  :-)20:18
esbergluefried: no20:21
efriedesberglu I can't imagine why we're the only ones seeing this.20:22
esbergluefried: When I first started hitting it I thought it was an issue on our end for that reason20:22
esbergluBut not convinced of that anymore20:22
*** esberglu has quit IRC20:47
edmondswdid we really never open a bug for that? looking...20:52
edmondswefried esberglu I did open a bug:
openstackLaunchpad bug 1699609 in devstack "stack fails unable to find /etc/neutron/neutron.conf" [Undecided,New]20:55
*** k0da has quit IRC20:59
*** esberglu has joined #openstack-powervm21:01
*** esberglu_ has joined #openstack-powervm21:02
*** esberglu has quit IRC21:02
*** thorst has quit IRC21:24
*** thorst has joined #openstack-powervm21:27
efriedesberglu_ edmondsw Workaround seems to be removing the qos service.  Do we need that for anything?21:27
*** thorst has quit IRC21:31
esberglu_I don't think so21:37
efriedesberglu_ Let's kill it and see if we get any additional failures.21:40
*** thorst has joined #openstack-powervm21:49
*** thorst has quit IRC21:54
*** svenkat has quit IRC21:59
*** kylek3h has quit IRC22:07
*** openstackgerrit has joined #openstack-powervm22:12
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: ISCSI volume attachment fixes
*** openstackgerrit has quit IRC22:18
edmondswefried esberglu I wonder if the fact that we're now using a generated tempest conf will help us remove qos...22:20
edmondswI also wonder if that will really fix this, or if we'll just find that another plugin that we do need hits the same issue22:23
edmondswdoesn't look like it, though... qos seems to be the only thing we've got turned on from these:
edmondswso yeah, if we could drop qos, that would probably get rid of the problem for us22:30
efriededmondsw Well, I can confirm that stack goes through with /etc/neutron missing and qos off the enabled list.22:34
edmondswwe should check with thorst to see if he knows of a reason we need it22:35
*** esberglu_ has quit IRC22:47
*** edmondsw has quit IRC22:53
*** thorst has joined #openstack-powervm23:38

Generated by 2.15.3 by Marius Gedminas - find it at!