22:01:44 #startmeeting zuul 22:01:45 Meeting started Mon Feb 19 22:01:44 2018 UTC and is due to finish in 60 minutes. The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:01:46 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:48 The meeting name has been set to 'zuul' 22:01:50 #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul 22:02:05 let's talk about v3 blockers 22:02:13 #topic Release blockers 22:02:21 #link roadmap https://storyboard.openstack.org/#!/board/53 22:02:30 there are 4 items on that list 22:02:39 the first is the zuul_json fix 22:02:49 dmsimard is working on that 22:02:50 hi 22:02:58 awesome 22:03:01 dmsimard: do you need any help on that? 22:03:38 I'm working on that *right now*, I sent a patch that should reproduce the issue and if it does I'll see from there (might toggle keep or autohold) 22:04:09 dmsimard: oh you mean you're trying to reproduce it in infra's zuul? 22:04:37 corvus: I'm unable to reproduce it locally by exercising zuul_stream and zuul_json by themselves (through the test-logs tool script) 22:05:24 https://review.openstack.org/#/c/545994/ should hopefully trigger it 22:06:11 dmsimard: if it does, is the next step to figure out what needs to happen with that same test playbook in a zuul unittest to be able to cause the error there? 22:06:59 mordred: the next step is to see if https://review.openstack.org/#/c/543633/ fixes it somehow 22:07:23 well, we need a test reproduction to see if it fixes it 22:07:37 as I recall, that patch on it's own wasn't enough to fix everything because the feature/zuulv3 version of that patch (same code) wasn't enough to fix it at one point, see the finger URLs here: https://review.openstack.org/#/c/504238/ 22:08:25 This may have changed with ansible dot releases or whatever might have happened since then, I'm just not confident to let that merge without additional validation first 22:08:38 so validation is being done 22:08:44 yah 22:08:51 dmsimard: well, you said the test in the change didn't fail without the fix 22:08:58 so it's not actually a test of the fix 22:09:01 right 22:09:12 * rbergeron sneaks into a seat ever so inconspicuously 22:09:23 corvus: right, so I might need help understanding why that is once we figure out how to properly reproduce it 22:09:46 I didn't need more than that to validate the fix for ara 22:10:14 https://github.com/openstack/ara/commit/6fd7a95ab940d7a78ade0040232a1ffdaf41ccfc 22:10:17 so if there's still a problem (apparently there is, since we saw this, what, 1 or 2 weeks ago?) we need a test case to show the problem, so that we can confirm the fix and verify it doesn't regress in the future 22:10:23 dmsimard: well, let me know if you hit a wall on making it break again 22:10:28 corvus: agree 22:10:29 mordred: ack 22:10:34 corvus: yes, we need some sort of test indeed 22:11:05 i'm also happy to help 22:11:23 okay, so that's thing 1/4 -- hopefully we'll sort it out soon 22:11:24 ok, I'll shout once I get more info 22:11:30 dmsimard: honestly - give it SpamapS - he can get anything to break :) 22:11:49 thing 2/4 is, incidentally, to refactor zuul_stream so we have better testing :) 22:11:50 mordred: nod, I also need to work on that exit code thing we discussed the other day 22:12:06 mordred: i saw patches for that; what's the state? 22:12:20 corvus: yah - I started work on part of that, but have shifted to the javascript stack for now 22:12:41 okay, so we shouldn't focus on that yet.. javascript first 22:12:44 javascript is thing 3/4 22:13:19 https://review.openstack.org/#/q/topic:zuul-stream-rework is the work started down the streaming piece of the puzzle 22:13:38 mordred: is there anything anyone can do to pitch in on that now? or should we just wait? 22:13:40 it needs to be reworked a little bit to have the stream listener be tied to the job and not to the playbook 22:14:10 I'll take a look at those patches and review 22:14:38 corvus: anyone who feels like hacking should be able to take what's there and move forward if they want to - main thing is the lifecycle of the listener needs to be essentially the same as the ssh agent 22:15:10 this is because of the controlpersist 22:15:43 which can only forward something _once_ - not every time - since subsequent connections just re-use the previous forwarded port 22:16:06 so when we forward it the first time, the thign we forward needs to continue to exist for the next times 22:16:20 that means the listener can't start in the callback, right? it needs to be started by the executor 22:16:59 corvus: that's right. and it probably needs to be started as a subprocess rather than a thread, otherwise we've got a bazillion threads running around 22:17:14 corvus: so it may take a few more stabs to find the right thing 22:17:45 ok. so if anyone feels like poking at that, talk to mordred 22:18:01 yah - my feelings will not be hurt if someone wants to dive in :) 22:18:10 we're blocking the v3 release on this basically because there's so little testing of this right now, if someone comes in with a bug, we can't fix it 22:18:23 but also - if nobody else does, I should be able to return to it in a little bit 22:18:58 okay, thing 3/4 is javascript 22:19:19 that's very close - although I've still got two of the patches marked WIP 22:19:22 mordred: we merged 4 pre-req patches today -- does https://review.openstack.org/538099 need any reworking because of that? 22:19:47 corvus: yes - it does - that's the next thing on my list 22:20:14 that reworking shouldn't take long - I just wanted to be able to verify it against the live status page in addition to the unit tests 22:20:28 cause, you know - moving parts 22:20:36 ya 22:21:13 that's the best way to find out what extra unit tests to add :) 22:21:21 EXACTLY :) 22:21:34 http://logs.openstack.org/55/545755/1/check/build-javascript-content/0012b09/npm/html/status.html <-- however, is the end of the stack, and you can see it working 22:21:46 and speaking of tests, that one now includes the test that we can serve the js files in all 3 ways we know zuul is deployed 22:21:59 yup. thanks for that - it's already been helpful :) 22:22:29 i'm looking forward to not breaking openstack-infra's zuul everytime we land a web change :) 22:22:43 ++ 22:22:46 RIGHT? 22:23:04 will be nice 22:23:35 okay, so if mordred and dmsimard ask for help about any of those 3 efforts, please pitch in -- those are our internal release blockers, and we're so close 22:23:52 thing 4/4 on the list is the github3.py release 22:23:52 * mordred is curious on the state of 4 ... 22:23:58 jlk: around? 22:24:12 in fact, I am. 22:24:16 yay! 22:24:26 I'll read up 2/4 tonight 22:24:41 I have no real change in news, other than continued work on teh open issue. https://github.com/sigmavirus24/github3.py/pull/773 22:25:02 A pile of my work was merged into that PR, but there is more to do yet it seems. 22:25:47 jlk: if folks had time to pitch in on that, are there more units of work to hand out? 22:26:00 or it it mostly in the hands of the maintainers at this point? 22:26:50 well... not easily no. It's mostly in the hands of the maintainer. I know part of the work that needs to be done, but a lot of this is kinda beyond me too. It's a fairly big restructuring of how the objects interact, but it's mostly opaque to the API users. 22:27:41 I think it's down to the last few files to go through, but they all intertwine a bit. I don't know what the maintainer's strategy is tor the rest, but I'll ask. 22:27:53 cool 22:28:16 we probably aren't going to be ready to release zuul this week, so we still have a bit of time 22:28:43 but once we clear the first 3 things on that list, and this is all that remains... i think we need to consider a backup plan 22:28:55 which i think is probably to vendor github3.py into zuul temporarily. 22:29:14 hopefully it won't have to live there for long 22:29:19 vendoring code copies is the way of the world these days 22:29:19 I think that's the right thing to do 22:29:25 yah. much as I'm not a fan of vendoring things like that, it seems like an ok choice in this instance 22:29:32 but i think that's the best way we can get zuul out there without making a big fuss. 22:29:52 fungi: so is electing fascists ... but that doesn't mean I want to continue the trend :) 22:30:00 fair! 22:30:01 mordred: ahem 22:30:06 o_O 22:30:09 uhoh. /me politicaled 22:30:14 * rbergeron fans herself 22:30:17 * mordred backs slowly out of the room 22:30:35 oh now I get the context. I was missing it for a moment. 22:30:41 okay, so we'll see how things shake out, and if it comes to that, i'll volunteer to do the vendoring 22:30:59 corvus: kk. I will volunteer to review 22:31:15 and of course, we'll remove it and release as soon as github3.py releases 22:31:21 ++ 22:31:32 sounds like a reasonable approach 22:32:02 okay, that's the whole release-blocker list... thanks everyone whose working on those, and everyone else who's assisting :) 22:32:09 hopefully that doesn't take long enough that we have to start worrying about tracking security bugs in a vendored unreleased depndency 22:32:09 #topic open discussion 22:32:14 corvus: is it worth going ahead and doing that since we have the git url in our requirements now anyway? 22:32:16 fungi: yeah 22:32:35 worth noting a few behavioral changes have merged recently in zuul. We've updated how timeouts work and the status api moved 22:32:57 clarkb: ++ ... .json have been removed from rest api calls 22:32:59 * rbergeron haz a thing for open discussions when the moment presents itself :) 22:33:00 mordred: i don't think it's going to be very impactful for folks until we're nearly ready to release... 22:33:06 corvus: kk 22:33:14 * mordred presents rbergeron a moment 22:33:35 oh. wait. that's an old loaf of bread ... 22:33:37 ooh. neat 22:33:55 oh, not so neat. I guess i should obtain a moment from corvus instead 22:33:58 ;) 22:33:59 anywayyyyyyy 22:34:29 So: Some of you may have seen the lovely message from mr. rich bowen on the dev list re: video things at the PTG - I chatted with him a few weeks ago and... 22:34:36 Over the weekend, we had another issue where zuul.o.o lost access to nodepool.o.o (zookeeper) a few times. This results in zuul unlocking nodes, and nodepool mass deleting all in-use nodes, which deletes them under running jobs. Aside from moving to zookeeper cluster, plan for PTG. Is there other way to help protect running jobs from not losing their nodes under them and resulting in a bunch of 22:34:37 SSH failures 22:34:55 he was super willing to record some interviews (like, group style, 2 or 3 humans, plus anyone who might be using zuul aside from openstack who wants to talk as well) 22:35:05 for us, which might be useful / interesting fodder to add to the website. 22:35:08 http://lists.openstack.org/pipermail/openstack-dev/2018-February/127525.html 22:35:09 rbergeron: \o/ that sounds like good 22:35:16 i would love more website fodder! 22:35:17 So i am here to procure wranglings of a few of you to do such things. 22:35:17 that'd be neat 22:35:37 rbergeron: I'm always happy to sit in front of a camera - especially when I don't have to prepare before-hand 22:35:37 I can vouch for rbowen doing great work with the interviews, had a few in the past 22:35:50 He's on #rdo (rbowen) if you want to chat 22:35:57 (i also said I'd try to wrangle someone into volunteering to also chat about broader infra things, and I should poke dmsimard about talking about ARA, and maybe mordred talking about ansible / shade / modules along with other humans) 22:36:11 (and, you know, wrangling is my thing, so, yeah) 22:36:15 rbergeron: spoiler, I already had an interview about ARA 22:36:17 tobiash: is usually not around for these meetings, but he will be at the ptg and is a non-openstack user; we should see if he'd be willing to chat 22:36:30 in any case: lest we all say "yeah yeah" and then find ourselves exhausted on thursday night: 22:36:48 ya I tend to avoid adding too much for ^ that reason 22:36:51 +1 22:36:59 he's doing things tues-thurs, though he might make an exception for a monday thing for anyone who is only around on monday (AHEM COUGH JLK) 22:37:26 hai. I mean, I could talk about our failed/cancelled CI project, but uh... 22:37:41 * jlk wonders if Clint would be around 22:37:42 so: I can either voluntold a few of you, but i know infra as well as zuul folks have some cross-project things going on / help sessions etc, so if anyone is willing to pinpoint an afternoon or moning where they feel safe :) 22:37:56 oh, I mean IBM is still using it internally with great success as far as I know. 22:37:57 jlk: or you could unveil your secret plans for world domination 22:38:31 jlk: mostly highlighting "yay github works with zuul" is the thing. it's nicer to highlight it with the human who has done a bunch of that work :) but... after, post-thing-via-video, whatever is cool. 22:38:32 I think clint said he won't be making it 22:38:35 or i can dress up as you 22:38:46 oh yeah I can talk about that too. 22:39:27 and also: it'd be neat if like, you know, not just people employed by my gracious employer overlords 22:39:30 yah. also - since jlk works at github now - an interview on how awesome zuul github integration is from him would be neat 22:39:31 pretty please 22:40:01 yup, I think I can toe that line and make it clear I did the work before joining GitHub, and that it's not an official GitHub effort :D 22:40:06 then I can take words out of context and say "github endorses using zuul for all CI" 22:40:12 jlk: but he can do things monday if you have things to say, or if we can pair you up with a human, and maybe we can do another pair of humans in a few days :) 22:40:15 jlk: oh - darn - you anticipated my evil 22:40:34 mordred: you're always evil, is anticipaaaaation really a thing there? 22:40:36 rbergeron: cool, with jlk, and maybe asking tobiash tomorrow, we might get 2 non-openstack folks 22:40:40 rbergeron: good point 22:40:44 mordred: careful editing of his interview in post production can get around it 22:40:52 are there any other non-openstack folks going to the ptg? 22:40:56 fungi: the stephen colbert treatment? 22:41:00 I should be back in Dublin again on Thursday and could slip away for an hour or so if it doesn't workout otherwise. 22:41:06 non-openstack or not-just-openstack? :) 22:41:27 rbergeron: not-just-openstack -- i guess folks that can give a non-openstack-infra perspective on zuul 22:41:37 rbergeron: also - melvin is signed up for an interview about openlab, so that should be another non-redhat-zuul talk 22:41:47 jlk: ack, I will see what we can do for monday and also keep that in mind. 22:42:01 thanks 22:42:03 mordred: oh good, that was a thing i brained for a moment before the squirrels came earlier today 22:42:14 there's the software factory perspective 22:42:28 there'll be some folks involved in software factory at the PTG 22:42:33 and maybe anyone using zuulv3 for openstack third-party testing 22:42:47 so: should I pick some times on the sheet and propose them on the zuul list -- or just sign up for things and also eyeball the infra plan and hope that if we screw up that it will still be okay? :) 22:42:49 rbergeron: have I mentioned that squrrels have literally been trying to eat through our roof above our bedroom? 22:42:59 dmsimard: ahh, do you know who exactly? 22:43:02 mordred: TMI 22:43:04 rbergeron: did you see my rough schedule for infra rooms? 22:43:05 texas-sized squirrels 22:43:12 rbergeron: I'll double check and get back to you 22:43:16 very loud dubstep scares them away, fwiw 22:43:26 rbergeron: tldr is monday tuesday is more infra office hours for random projects. Expect to have a bit of zuulv3 related stuff there 22:43:39 clarkb: I did eyeball it, or maybe i only eyeballed the etherpad and not an actual schedule of activities 22:43:42 dmsimard: rbergeron I know jpena and apevec are also attending from SF 22:43:45 then wednesday and first half of thursday just focus on zuul and transition into more general infra topics thursday afternoon and friday 22:43:47 clarkb: yep, caught that 22:44:01 *nod* 22:44:12 clarkb: fwiw, i'm expecting monday/tuesday to be a _lot_ of zuul v3 job config q&a 22:44:26 fungi: ya 22:44:28 * fungi predicts a line out the door 22:44:41 everything good around here has a line out the door, so ++ 22:44:44 (okay, hopefully not *that* bad) 22:44:45 * rbergeron will be the gatekeeper with the marshmallows 22:45:06 ...and bad jokes and puns 22:45:52 what, no calamari? 22:45:53 okay, so: i'll find slots and do my best to make them work with times and will just blather at y'all on the channel. i think that's probably best (and probably in the morning because wow, i have been typing for like 11 hours) 22:46:09 rbergeron: why don't you send something to the zuul list (whether it's "i think you should do these times", or "i signed you up for these times") since it may get better exposure there, especially for the not-just-openstack folks 22:46:28 Shrews: nicely played 22:46:35 corvus: sounds good, and thank you for the "yes, more humans will see that" reminder :) 22:46:49 (even i forget this here and there despite reminding others of it all the time!) 22:47:20 shrews: only if you want me to vanish 22:48:03 #action rbergeron to mail list with times she signed people up for after thorough research and hope for non-conflict to do zuul interviews for yayyyy, content 22:48:34 unrelated: corvus, can we mail the meeting logs to the zuul list? (I thought this happened automagically but i may be thinking of fedora instead) 22:48:42 (or maybe they go to infra) 22:48:49 (i guess it is slightly related as it made me think of this) 22:49:03 rbergeron: they go to web 22:49:07 rbergeron: the logs are hosted at http://eavesdrop.openstack.org but aren't emailed otu 22:49:32 (and with that: that is all the squirrels that i have) 22:49:39 ah, okay. i may be confuzzled 22:49:40 we could (manually) email logs (and agenda reminders) if folks would find that useful... 22:49:51 rbergeron: http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-02-13-19.01.html 22:50:00 (i can see how that would be miserable on openstack-dev, but maybe not on the zuul list) 22:50:01 if we do that, we should start with the next meeting cause i totally haven't been #ing 22:50:11 corvus: it's possibly not a terrible idea to email log summary and links 22:50:16 corvus: i have mostly found it useful in that sometimes folks are like "OH HEY NEAT I HAD NO IDEA THERE WERE MEETINGS" 22:50:25 we used to have someone doing some of that, but determined that people used their calendars and knew where to find the logs/minutes anyway 22:50:31 also no meeting next week? It will be middle of the night almost right? 22:50:33 oh, speaking of next meeting, let's not have one next week? 22:50:38 "and i can see i might have things to yap about" 22:50:50 yeah, it'll be like 10-11pm local 22:50:54 corvus: ack :) and yes, no meets next week plz 22:50:57 also, i won't be around for the meeting after that -- do folks want to have it, or call it a PTG recovery day? 22:51:31 wfm 22:51:35 i'll probably be drinking whiskey to warm up after tramping through a bog or something 22:51:44 I will be traveling so will be ptg recovery day for me 22:51:52 I'll be on a plane 22:51:53 my friend is in dublin and he saw an actual tanker truck of guinness 22:51:56 convenient bit of trivia, while we're in dublin they'll be on utc basically, so i won't need to spend the week converting times to add them to my calendar (for once) 22:52:04 rbergeron: OMG 22:52:21 pabelanger: right. like, super cliche yet also awesome 22:52:32 +1 22:52:39 pabelanger: to your question earlier -- no we don't have a way of avoiding that other than "fix whatever communication problem is between zuul and zookeeper" 22:53:07 so hopefully this gets better with the cluster 22:53:21 https://scontent.fphx1-1.fna.fbcdn.net/v/t31.0-8/28061721_10213872342754761_459075879379154238_o.jpg?oh=7b377a0a69bb3df528ad5cac28837917&oe=5B11639B 22:53:30 corvus: yah, that is what I figured. Wanted to ask 22:54:15 ++ cluster 22:54:59 okay, i saw no objections to canceling next 2, and some support 22:55:22 #info no meeting on feb 26 or mar 5 22:55:35 anything else? 22:55:53 safe travels 22:55:56 plz to have safe metal tubing 22:56:25 and respond to pabelanger's infra dinner things! 22:56:41 ++. thanks all, see some of you soon :) 22:56:44 thanks! 22:56:45 #endmeeting