Friday, 2017-08-04

*** thorst_afk has joined #openstack-powervm00:11
*** svenkat has joined #openstack-powervm00:14
*** svenkat has quit IRC00:49
*** thorst_afk has quit IRC00:52
*** svenkat has joined #openstack-powervm01:05
*** svenkat has quit IRC01:30
*** edmondsw has joined #openstack-powervm01:42
*** thorst_afk has joined #openstack-powervm01:42
*** edmondsw has quit IRC01:46
*** apearson has joined #openstack-powervm02:10
*** thorst_a_ has joined #openstack-powervm02:18
*** thorst_a_ has quit IRC02:18
*** thorst_afk has quit IRC02:20
*** edmondsw has joined #openstack-powervm03:30
*** edmondsw has quit IRC03:34
*** apearson has quit IRC03:34
*** http_GK1wmSU has joined #openstack-powervm03:48
*** http_GK1wmSU has left #openstack-powervm03:49
*** thorst_afk has joined #openstack-powervm04:19
*** edmondsw has joined #openstack-powervm04:31
*** thorst_afk has quit IRC04:33
*** chhavi has joined #openstack-powervm04:35
*** edmondsw has quit IRC04:36
*** thorst_afk has joined #openstack-powervm06:01
*** thorst_afk has quit IRC06:05
*** edmondsw has joined #openstack-powervm06:19
*** edmondsw has quit IRC06:24
*** thorst_afk has joined #openstack-powervm08:02
*** thorst_afk has quit IRC08:06
*** thorst_afk has joined #openstack-powervm09:03
*** thorst_afk has quit IRC09:07
*** edmondsw has joined #openstack-powervm09:55
*** edmondsw has quit IRC10:00
*** wilkm has joined #openstack-powervm10:08
*** openstackgerrit has quit IRC10:18
*** edmondsw has joined #openstack-powervm10:36
*** edmondsw has quit IRC10:40
*** thorst_afk has joined #openstack-powervm11:04
*** thorst_afk has quit IRC11:08
*** smatzek has joined #openstack-powervm11:14
*** svenkat has joined #openstack-powervm11:42
*** svenkat has quit IRC11:46
*** svenkat_ has joined #openstack-powervm11:46
*** edmondsw has joined #openstack-powervm11:47
*** thorst_afk has joined #openstack-powervm11:51
*** thorst_afk has quit IRC11:52
*** edmondsw has quit IRC11:52
*** thorst has joined #openstack-powervm12:06
*** edmondsw has joined #openstack-powervm12:23
*** kylek3h has joined #openstack-powervm12:54
*** apearson has joined #openstack-powervm13:20
*** esberglu has joined #openstack-powervm13:25
esbergluedmondsw: thorst: efried_zzz: Last night I was looking at the prod. controller and there were a bunch of old openstack processes running13:35
esbergluI don't think they were getting cleaned up properly when unstacking during the deploy13:36
esbergluTo avoid this in the future I am just going to run ./ every deployment. It will increase deployment time slightly but we have been doing that less and less13:37
thorstseems reasonable13:49
*** efried_zzz is now known as fried_rice13:49
*** wilkm has quit IRC13:59
*** apearson has quit IRC14:10
*** smatzek has quit IRC14:13
*** smatzek has joined #openstack-powervm14:13
*** apearson has joined #openstack-powervm14:30
esberglufried_rice: Got time to talk through timeout issues today?14:32
fried_riceesberglu Yeah, but gimme 30.14:33
esberglufried_rice: Sure14:33
esberglufried_rice: edmondsw: thorst: 5648 should fix the cell issues in CI15:33
*** apearson has quit IRC15:36
edmondswesberglu hmm... "after the stack". So we remove that once sdague's change goes in?15:39
edmondswor are we still possibly broken after his change goes in?15:39
edmondswbecause "after the stack" is too late to mean anything once his change goes in, so it's one of those 215:40
esbergluedmondsw: We could remove this after his change goes in if his change fixes the issue15:42
esbergluBut it doesn't hurt to leave15:43
esbergluIf the hosts are already discovered while stacking this won't do anything15:43
esbergluIf they don't get discovered while stacking it gives us one more chance to discover them15:43
edmondswesberglu my point is that it no longer gives us one more chance after his change goes in, because the stack would fail, right?15:44
esbergluedmondsw: Let me take another look at the change, but I didn't think that it would fail the stack15:46
edmondswesberglu maybe I misunderstood/misremember... but if not then why did it break anyone?15:47
esbergluedmondsw: Because discover_hosts was being run too soon I believe15:48
esbergluSo they are now waiting until the compute service is actually up to run it15:48
edmondswesberglu oh, right15:50
edmondswyeah, I don't see anything in wait_on_compute that would actually bail on the stack15:51
esbergluSo sdagues change _should_ fix us15:52
esbergluBut no reason to remove the second discover_hosts after that merges because it doesn't hurt anything and could potentially save a run that still fails the stack's discover_hosts15:52
edmondswesberglu so something was blowing up with that `openstack compute service list` call because of the missing EnvironmentFile, but we've fixed that, and should be good now?15:53
esbergluYep. Previously we weren't setting up remote until after the stack which was causing that to fail15:54
esbergluNow we do the remote setup before stacking15:54
edmondswk, left comments15:56
edmondswI agree with fried_rice, we need to figure out why this is even necessary and what we can do about it15:56
fried_riceedmondsw esberglu To make sure everyone understands: sdague's *original* change would *fail* the stack if the compute service didn't come alive within the timeout in mid-stack.15:59
fried_riceWith his *new* patch, it logs a warning, but proceeds anyway.15:59
*** openstackgerrit has joined #openstack-powervm16:00
openstackgerritMerged openstack/nova-powervm master: force python2.7 for tox pep8
fried_riceSo his change won't in fact fix us, other than by giving our compute service a little longer to come up and be discoverable.16:02
fried_riceIn a situation where our compute service doesn't come up in that 60s window, it is pretty unlikely to come up at all, so the new discover_hosts at the end is likewise pretty unlikely to change anything.16:03
fried_riceSo to summarize:16:03
fried_riceUntil sdague's change merges, esberglu's change makes sense.  After sdague's change merges, it's almost certainly a no-op, though it doesn't hurt anything.16:04
fried_riceesberglu I'm ready to look into timeouts now.16:04
edmondswesberglu fried_rice should we merge this, then, or just carry it as a patch until sdague's change goes in?16:04
edmondswI kinda lean toward the latter16:04
fried_riceesberglu Can we "carry it as a patch"?16:05
edmondswoh, maybe we can't...16:05
fried_rice(esberglu In true programmer style, I hope you multiplied my time estimate by three.)16:05
fried_riceedmondsw esberglu I'd be okay with adding a TODO to remove it once (link to sdague's patch) merges.16:05
esberglufried_rice: Was just about to take lunch. Dig in this afternoon or would you rather do it now? No big deal either way for me16:12
fried_riceesberglu Whenever you like.  I got plenty to keep me busy.16:12
esberglufried_rice: I have a quick thing now and then I'll take lunch16:13
esbergluSo initially I thought this was timing out on the rebuild16:13
esbergluBut it looks like it is actually timing out on a show_server rest request16:14
esbergluJust wanted to make sure you were seeing the same. I went through the logs and the server in question goes through the flow as it should based on the test16:15
esbergluAnd actually reaches the correct status. The request to check the status is what seems to be the timeout source16:15
*** esberglu is now known as esberglu_lunch16:25
edmondswAndyWojo you were asking about the OpenStack User Survey the other day... I did get out a blog about that last week if you want to check it out:
*** edmondsw_ has joined #openstack-powervm17:23
edmondsw_fried_rice is there a way to search eavesdrop?17:24
fried_riceedmondsw_ If you figure it out, let me know.17:24
edmondsw_yeah, that's what I figured17:24
fried_riceedmondsw_ Wouldn't be too hard to write a crawler17:25
fried_ricewhich means someone must have done it.17:25
*** edmondsw has quit IRC17:26
*** openstackstatus has quit IRC17:27
*** openstack has joined #openstack-powervm17:29
fried_riceesberglu_lunch I agree with your assessment.  And I can find no indication as to why that request timed out.17:49
fried_riceThe n-api side looks fine.17:49
*** esberglu_lunch is now known as esberglu17:49
fried_riceBut 99% of the time we get timeouts, this test case is one of the culprits, nah?17:50
esberglufried_rice: Yeah17:50
fried_riceesberglu Can we run with tempest patches?17:51
esberglufried_rice: Yeah17:51
esbergluToss a patch up and start dumping some more info?17:52
fried_riceWell, I'm really not sure how more info would help us.17:52
fried_riceBut it might be worth figuring out if the hitch is permanent or temporary.17:52
fried_riceSo what I was thinking is: put an infinite retry loop around that show_server rest request.17:53
fried_riceesberglu So if the condition is permanent, the test itself will eventually time out.18:01
fried_riceBut my guess is it ain't, and this will make that class of problem vanish.18:01
esberglufried_rice: Nice beat me to it18:01
fried_riceesberglu Aaand now I'm gonna go to lunch.18:02
fried_riceI wish.18:02
AndyWojoedmondsw_: thank you. I already completed the survey.18:03
edmondsw_AndyWojo sure. I think you can still update your answers until it closes, if you wanted to.18:04
AndyWojoI answered everything correctly.18:10
edmondsw_AndyWojo ++18:11
AndyWojoI recently got my OpenStack certification, and the SuSE OpenStack cert. Very surprised not to see any IBM certs centered around OpenStack/PowerVC18:16
*** openstackstatus has quit IRC18:26
*** openstack has joined #openstack-powervm18:28
*** esberglu has quit IRC18:43
*** esberglu has joined #openstack-powervm18:43
*** esberglu has quit IRC18:48
*** esberglu has joined #openstack-powervm19:01
*** thorst has quit IRC19:03
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: ci check
*** thorst has joined #openstack-powervm19:06
*** thorst has quit IRC19:10
*** chhavi has quit IRC19:18
*** thorst has joined #openstack-powervm19:18
*** thorst has quit IRC19:20
*** thorst has joined #openstack-powervm19:20
esbergluedmondsw_: I'm dumb and messed up the patching format19:24
edmondsw_esberglu np19:24
esbergluAdded an example for future dumb me's lol19:25
edmondsw_lol... +219:25
edmondsw_could you just put a comment in the conf file itself?19:26
esbergluedmondsw_: Atm no. I was gonna add a TODO for that and to make the list of changes comma separated19:26
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: CI Check2
*** apearson has joined #openstack-powervm19:32
-openstackstatus- NOTICE: Gerrit is being restarted to pick up CSS changes and should be back momentarily20:36
*** smatzek has quit IRC20:50
*** apearson has quit IRC21:00
*** svenkat_ has quit IRC21:08
*** apearson has joined #openstack-powervm21:08
*** thorst has quit IRC21:21
*** thorst has joined #openstack-powervm21:22
*** thorst has quit IRC21:26
*** kylek3h has quit IRC21:38
*** edmondsw_ has quit IRC21:47
*** apearson has quit IRC22:05
*** apearson has joined #openstack-powervm22:06
*** apearson has quit IRC22:06
*** fried_rice is now known as efried_WEEKEND22:16
*** thorst has joined #openstack-powervm22:22
*** thorst has quit IRC22:28
*** esberglu has quit IRC22:28
*** esberglu has joined #openstack-powervm22:29
*** esberglu has quit IRC22:33
*** efried_WEEKEND has quit IRC22:48
*** efried_WEEKEND has joined #openstack-powervm22:58
*** edmondsw has joined #openstack-powervm23:36
*** edmondsw has quit IRC23:40

Generated by 2.15.3 by Marius Gedminas - find it at!