13:59:50 <jokke_> #startmeeting glance
13:59:51 <openstack> Meeting started Thu Jan 31 13:59:50 2019 UTC and is due to finish in 60 minutes.  The chair is jokke_. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:59:55 <openstack> The meeting name has been set to 'glance'
13:59:57 <jokke_> #topic roll-call
13:59:59 <jokke_> o/
14:00:40 <jokke_> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:01:16 <rosmaita> o/
14:01:23 <LiangFang> o/
14:01:47 <abhishekk> o/
14:02:26 <jokke_> Nice we have quorum
14:02:33 <jokke_> #topic updates
14:03:44 <jokke_> just a quick one. We did release b1 and as the rosmaita's test is merged in the master please note that the check and gate will fail until we get the alembic migrations version change in
14:04:08 <rosmaita> (see links in open discussion on agenda)
14:04:32 <jokke_> So please do not spam recheck and load the gate for no reason, it will be solved once the change gets (hopefully) through gate
14:04:40 <smcginnis> .o.
14:04:49 <rosmaita> just got to the gate now, survived the initial check!
14:05:03 <rosmaita> got kicked out last night, post failure and a timeout
14:05:07 <jokke_> yeap
14:05:31 <jokke_> so that's all from for now
14:05:39 <jokke_> #topic release updates
14:05:43 <jokke_> abhishekk: floor is yours
14:05:50 <abhishekk> ack
14:06:05 <abhishekk> subunit.parser error is back
14:06:11 <rosmaita> arrrrgh!
14:06:28 <abhishekk> periodic jobs for functional py3 are failing intermittently
14:06:46 <abhishekk> jokke_, just covered release updates
14:06:53 <jokke_> Was this something we thought we had fixed or is this something that jusappeared for a while?
14:07:08 <jokke_> just disappeared
14:07:13 <abhishekk> more about failure in point 3 added by rosmaita
14:07:24 <abhishekk> jokke_, looks like that
14:07:31 <rosmaita> yeah, it just stopped for a while
14:07:34 <jokke_> kk, lets hop on that
14:07:43 <jokke_> #topic subunit parser
14:08:07 <rosmaita> yeah, so abhishekk answered my question
14:08:12 <rosmaita> which was, is it back?
14:08:16 <rosmaita> and the answer is yes
14:08:36 <rosmaita> abhishekk has a patch up based on the neutron one he mentioned
14:08:44 <rosmaita> abhishek's patch:
14:08:55 <rosmaita> #link https://review.openstack.org/634235
14:09:08 <rosmaita> i was looking at the cinder bug about this that smcginnis worked on
14:09:19 <rosmaita> #link https://bugs.launchpad.net/cinder/+bug/1728640
14:09:20 <openstack> Launchpad bug 1728640 in Cinder "py35 unit test subunit.parser failures" [Critical,Fix released] - Assigned to Sean McGinnis (sean-mcginnis)
14:09:21 <abhishekk> it is very hard to reproduce, I have ran entire test suit by removing 1 by 1 potential failing test case but not able to reproduce
14:09:37 <rosmaita> yeah, i don't know why it only seems to happen in the gate
14:09:46 <smcginnis> It's very random.
14:10:03 <smcginnis> And even though that is marked as "fixed", I just consider that a workaround.
14:10:12 <rosmaita> that's what i was going to ask
14:10:28 <rosmaita> but as long as it decreases the frequency, that would be better, i guess
14:10:32 <smcginnis> There was something with the switch from testr to stestr that introduced this with python 3.
14:11:24 <smcginnis> Treinish did a lot of debugging and tracing around the Sydney Summit for this but were never able to track down the root cause.
14:11:33 <jokke_> smcginnis: yeah I can't recall seeing these while we were still using testr
14:12:21 <rosmaita> smcginnis: you mentioned on the bug that the changing the log level in the setup doesn't always help in the functional tests, something about the tests calling out to other stuff that winds up changing the log level?
14:12:24 <abhishekk> I was banging my head against wall for last two days
14:13:11 <rosmaita> because this thing is so random, i don't know how we can test abhishekk
14:13:13 <rosmaita> 's patch
14:13:29 <rosmaita> maybe i can include it on all of my patches, i seem to hit it a lot!
14:13:46 <abhishekk> rosmaita, no idea
14:13:57 <abhishekk> and I am also not sure it will work or not
14:14:16 <jokke_> I'm wondering if it's resource starvation on gate
14:14:37 <smcginnis> rosmaita: It's fuzzy now, but I did fix one thing where we were setting the log level higher but the code was going back and lowering it again.
14:14:52 <rosmaita> ok
14:14:56 <abhishekk> because I have just checked in zuul and found that py3-functional job is stalled and it will timed out later
14:14:57 <jokke_> which would explain why we see it in gate but locally when we have nuf power we never see it
14:15:00 <smcginnis> The only "fix" seems to be really restricting how much log output happens during test runs.
14:15:55 <rosmaita> abhishekk: that is what i'm seeing a lot, too -- subunit parser error shows up in output, a bunch of tests are listed as '...inprogress', and then eventually failure
14:16:22 <abhishekk> yes, that's the case
14:16:59 <rosmaita> i'm trying to think of a way we can run *only* func-py35 test in gate, but i don't think that's possible
14:17:10 <rosmaita> we only control the jobs in our local zuul
14:17:20 <rosmaita> the other templates are out of our control
14:18:03 <rosmaita> i'd like to run abhishekk's change like 50x to see what happens, but that's going to run a lot of jobs
14:18:10 <jokke_> rosmaita: we should be able to drop the majority on test patch if we just add "*" to the irrelevant files
14:18:51 <rosmaita> actually, maybe change tox.ini
14:18:58 <rosmaita> on abhishekk's patch
14:19:06 <jokke_> rosmaita: on the same patch you want to try out. Just mark it DNM ;)
14:20:19 <abhishekk> I guess we can modify yaml file and keep functional-py35 job only?
14:20:54 <rosmaita> abhishekk: yes
14:21:00 <rosmaita> i think in the project section
14:21:08 <abhishekk> yes
14:21:20 <jokke_> abhishekk: that's what I meant. Just add "*" to all jobs there but the py3 job
14:21:37 <jokke_> that should drop the patch from running any of the other tests
14:22:02 <rosmaita> i think you have to modify what templates are called too though
14:22:14 <rosmaita> we can't define the irrelevant files for them only
14:22:28 <rosmaita> anyway, key thing is if we can not have tempest or grenade run
14:22:36 <rosmaita> those are the ones that are taking forever
14:22:56 <rosmaita> so looks like jokke_ is right
14:23:12 <rosmaita> we can '*'  for tempest-full and neutron-grenade
14:23:30 <LiangFang> do we need to take a look of the code of stestr to identify the root cause?
14:23:54 <rosmaita> LiangFang: if someone has time
14:24:12 <LiangFang> yes
14:24:13 <rosmaita> mtreinish is the author, and he and smcginnis were trying to figure it out with no luck
14:24:44 <jokke_> LiangFang: if you have time to dig into that, I'm sure there would be lots of happy people also outside of glance ;)
14:24:51 <rosmaita> ++
14:24:55 <abhishekk> +1
14:25:01 <smcginnis> If someone can, my hunch was there was some string or stream handling that was getting encoded differently or something under python 3.
14:25:27 <LiangFang> I will take a look tomorrow
14:25:43 <LiangFang> but china new year is coming
14:25:52 <rosmaita> so i think what we do now is abhishekk updates his patch to exclude tempest and grenade
14:25:57 <LiangFang> I will take vacation for one week
14:26:14 <rosmaita> and puts up another with the .zuul.yaml change but without his fixture change
14:26:22 <rosmaita> and we run them both a bunch of times
14:26:33 <rosmaita> should see way more errors in the "unfixed" patch
14:26:43 <abhishekk> rosmaita, ok
14:26:49 <jokke_> rosmaita: sounds like a plan to me
14:26:53 <rosmaita> i can help
14:27:06 <rosmaita> because the problem is that you can't recheck a successful job
14:27:17 <rosmaita> have to make some kind of change
14:27:22 <smcginnis> You can abandon and restore.
14:27:28 <rosmaita> ah, ok
14:27:33 <jokke_> or just poke some comment string
14:27:41 <jokke_> change a character
14:27:56 <rosmaita> i like abandon and restore, just click some buttons!
14:28:01 <jokke_> :D
14:28:03 <rosmaita> ok, sounds good
14:28:07 <jokke_> playing a videogame
14:28:17 <jokke_> Nice, we have a plan!
14:28:27 <abhishekk> so I need to update current patch to remove temepst and grenade jobs
14:28:33 <rosmaita> thanks abhishekk!!
14:28:54 <abhishekk> and another patch without logging change and just functional py35 jobs
14:29:07 <rosmaita> right
14:29:32 <jokke_> abhishekk: yeah, just add the "*" to irrelevant files on basically every job but the py3 job that keeps failing
14:29:50 <jokke_> that way we spawn minimal amount of jobs on the gate trying to find the reson for this
14:30:08 <abhishekk> jokke_, ok
14:31:01 <jokke_> And please remember to remove those before we merge it if it helps :P
14:31:10 <rosmaita> :D
14:31:29 <rosmaita> hopefully, the reviewers will be paying attention!
14:31:39 <abhishekk> only for check jobs right?
14:31:53 <jokke_> otherwise I will go to revert frenzy and pull out everything that was merged after when I find out :P
14:31:57 <rosmaita> yeah, shouldn't hit the gate unless someone approves it
14:32:01 <jokke_> abhishekk: yeap
14:32:04 <abhishekk> I will set WF to -1 to be safer side
14:32:17 <rosmaita> ++
14:32:31 <jokke_> ok, lets move on
14:32:45 <jokke_> #topic configuration test job
14:32:50 <rosmaita> this should be quick
14:33:15 <rosmaita> we have this new test that will break when glance version and the db migration CURRENT_VERSION dont match up
14:33:34 <rosmaita> jokke_ pointed out that we don't want to run that locally, just in the gate
14:33:45 <rosmaita> so my question is about separating it out
14:33:58 <rosmaita> should i move it in the tree to like glance/tests/config/ or somethinig?
14:34:12 <rosmaita> or just blacklist it in the normal tox definitions?
14:34:23 <rosmaita> i'm sort of leaning toward moving it
14:34:35 <rosmaita> but if there aren't any other tests like this, seems kind of dumb
14:34:35 <jokke_> yeah I like the idea of moving it
14:34:41 <rosmaita> ok, that's all then
14:34:52 <rosmaita> i will put up a patch soon-ish
14:34:56 <jokke_> that will make it possible to still run it locally, it just won't run by default
14:35:13 <rosmaita> just needs to be done before stein release
14:35:39 <rosmaita> ok, thanks, next topic
14:36:04 <jokke_> should we call the folder /glance/tests/gate ? that way if we in future want to move something else there as well, we do not need to keep doing new floders under our tests tree?
14:36:18 <rosmaita> that sounds good
14:36:43 <jokke_> gr8, unless there is objections, now we can move on :)
14:36:56 <rosmaita> action me!
14:37:40 <jokke_> #action rosmaita Move the alembic migration test under /glance/tests/gate and make new job definition for it that is not ran locally by default
14:37:44 <jokke_> happy with that?
14:37:45 <rosmaita> ty
14:38:05 <rosmaita> then i won't have to hunt for the name when i forget :)
14:38:09 <jokke_> #topic open discussion
14:38:10 <jokke_> :D
14:38:32 <rosmaita> just wanted to point out that i updated the meeting info like we discussed last week
14:38:57 <jokke_> Quick one, I'll be travelling Fri & Mon (FOSDEM time!) so won't be too reachable before Tuesday
14:39:23 <jokke_> telegram is your best bet if you need me
14:39:33 <rosmaita> also, jokke_ already mentioned holding off on rechecks until https://review.openstack.org/#/c/632185/1 merges
14:39:43 <jokke_> rosmaita: thanks, didn't realize that there was chair section on it as well :D
14:40:09 <rosmaita> yeah, noticed that when i was looking for the eavesdrop link
14:40:27 <rosmaita> abhishekk: you too, i put you as chair of the bug squad
14:40:35 <abhishekk> rosmaita, ack
14:40:40 <jokke_> good good
14:40:52 <rosmaita> you can change the time if you like, just put up a patch to infra like this one
14:41:03 <rosmaita> #link https://review.openstack.org/#/c/633824/
14:41:42 <rosmaita> that's all i have, execpt i will miss next week's meeting
14:41:55 <rosmaita> it will be quiet without me!
14:42:02 <jokke_> haha!
14:42:04 <jokke_> it will
14:42:18 * smcginnis will be buying rosmaita a drink (probably coffee at this time)
14:42:37 * rosmaita will be drinking an appropriate beverage
14:42:45 <jokke_> oh you guys will be meeting up?
14:42:53 <rosmaita> cinder midcycle
14:43:25 <abhishekk> something is wrong with my patch, I have removed a;ll the jobs except functional-py35 but still they are running
14:43:27 <jokke_> nice, so we can expect smcginnis not be around either
14:43:37 <abhishekk> is something wrong I have done?
14:44:01 <smcginnis> I should be online, but hopefully things go well with the midcycle and I am distracted by good discussions there.
14:44:09 <rosmaita> abhishekk: did you remove some of the templates?
14:44:22 <abhishekk> rosmaita, nope
14:44:23 <rosmaita> i think some jobs run whether we want them to or not
14:44:34 <abhishekk> just removed the jobs from check pipeline
14:44:37 <jokke_> abhishekk: yes, don't remove them, add "*" line on to the irrelevant-files section on each of the check jobs in .zuul.yaml
14:44:47 <abhishekk> it has removed non-voting job though
14:44:56 <abhishekk> jokke_, ok will do that
14:45:15 <rosmaita> it may take some experimentation
14:45:29 <rosmaita> you may want to ask in #openstack-infra for the best way to do this
14:45:33 <jokke_> that should bypass the template runs as zuul goes like "OK, nothing relevant has been changed in the repo" :P
14:45:35 <rosmaita> they should be glad to help!
14:46:29 <rosmaita> jokke_: i think because of our project status, there are some jobs that run that we have no control over and cannot override
14:47:00 <jokke_> rosmaita: yes, but they still honour the irrelevant-files override, that's why I said to use that ;)
14:47:37 <rosmaita> i'm not sure about that, otherwise everyone could use '*' to turn off jobs
14:47:42 <jokke_> yup
14:48:02 <rosmaita> anyway, ping AJaeger or fungi or cboylan in infra
14:48:04 <jokke_> I've been seriously tempted few times
14:48:24 <rosmaita> jokke_: i think everyone has! i think that's why you can't do it for all the jobs!
14:48:48 <jokke_> rosmaita: and this might reveal the loophole which might end up to a patch in zuul :P
14:48:59 <jokke_> but for now it should work
14:49:35 <rosmaita> abhishekk: it's late where you are, you can hold off for tomorrow
14:49:56 <rosmaita> just shoot an email when you have the patches up, and i will abandon/restore all day tomorrow
14:49:56 <abhishekk> rosmaita, tomorrow I am on leave
14:50:01 <rosmaita> oops
14:50:14 <rosmaita> i am too, actually
14:50:18 <rosmaita> :)
14:50:18 <jokke_> abhishekk: I think the whole line should look like "- ^*$"
14:50:23 <abhishekk> so I will do it and then take off
14:50:32 <jokke_> haha ... we're all out for tomorrow then :D
14:50:43 <rosmaita> sounds good, i wlll monitor for the rest of today
14:51:16 <jokke_> ok, we can continue this on the #os-glance, lets end the meeting unless someone has something else for us
14:51:16 <rosmaita> i think it must be the red hat pto expires on feb 15 situation
14:51:46 <rosmaita> nothing from me, have a good week everyone!
14:52:08 <jokke_> Thanks all!
14:52:26 <abhishekk> thank you all
14:52:33 <LiangFang> bye
14:52:50 <jokke_> Oh LiangFang Have a very happy new year!
14:53:04 <jokke_> #endmeeting