22:01:07 #startmeeting zuul 22:01:08 Meeting started Mon Jan 8 22:01:07 2018 UTC and is due to finish in 60 minutes. The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:01:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:09 \o 22:01:11 The meeting name has been set to 'zuul' 22:01:27 hey 22:01:31 #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul 22:02:01 * mordred waves to humans 22:02:15 #link release roadmap https://storyboard.openstack.org/#!/board/53 22:02:51 let's go through the 3.0 things real quick -- 22:03:15 did someone pick up the github ingestion via zuul-web yet? 22:03:34 i guess that's the last thing that's still in the old webapp, and once we get rid of that we can drop it entirely? 22:04:04 I picked this up before christmas 22:04:20 tobiash: ah great -- it's https://review.openstack.org/504267 right? 22:04:30 What's missing is an end to end test yet 22:04:39 corvus: I thought was assigned jlk, but seen he's moved onto new job recently. 22:05:03 pabelanger: jlk handed it off to tobiash 22:05:07 corvus: yes 22:05:08 o/ 22:05:14 ack, I missed that 22:05:21 tobiash: i agree with your last comment there -- we probably only need one or a few explicit tests for this 22:05:32 tobiash: are you planning on finishing it up like that? 22:06:12 (i updated the assignment in storyboard) 22:06:24 Probably not this week as we're going productive this week 22:06:34 But I hope next week 22:06:39 tobiash: congrats and good luck :) 22:06:44 tobiash: that sounds great, thanks 22:06:47 dmsimard: are you still planning on fixing the zuul_json issue? 22:07:10 thanks 22:07:23 oh, we lost jlk as a contributor? :( 22:07:24 :-) 22:07:30 yes, in fact I rebased the patch last week in order to reproduce the issue (which would have been easier to troubleshoot, now that I'm root) but for some reason it did not trigger 22:07:49 Shrews: i haven't heard that from jlk 22:08:00 I've looked around and saw some fixes in upstream ansible for that issue but it is confusing since we should not have those fixes yet in our version of Ansible. 22:08:11 * dmsimard gets link 22:08:16 Shrews: he just posted on twitter he is working as SRE at github.com :)So, hope not. 22:08:22 I think jlk is just busy getting inserted into his new job matrix. 22:08:32 Last we talked, he hoped to continue Zuul'ing in some capacity. 22:08:48 that is also what I have heard from jlk 22:08:54 https://github.com/ansible/ansible/commit/c30ee42fe1f0a9666a90f4d63121780f2a186c54 should fix our issue 22:09:05 hopefully he's busy inserting zuul into his new job matrix ;) 22:09:06 i see no reason why working at github would preclude contributing to zuul -- in fact, it's a position uniquely suited to doing so. i hope he will continue. :) 22:09:13 it also claims performance and memory improvements but I dunno. 22:09:21 we have a human on the inside now to bug with API questions :) 22:09:41 mordred: ++ 22:09:49 didn't know he was at github now, grats him 22:09:51 yes, let us all infiltrate all the places 22:10:06 also, the big thing jlk is working on right now is getting github3.py released so we can release. that seems very likely to meet with github.com supervisory approval, i'd think :) 22:10:08 dmsimard: well, today is his first day, so not knowing is totally fair :) 22:10:16 corvus: I certainly hope so! 22:10:36 corvus: in summary, I don't know whether or not there is something to fix yet since I am now unable to reproduce the issue and there has been an (unreleased) fix upstream. 22:10:41 I'll add my findings in the story. 22:11:14 dmsimard: do you mean you've been unable to repro using our production zuul, or locally? 22:11:33 doing a recheck/rebase on https://review.openstack.org/#/c/504238/ typically reproduced the issue 22:11:35 it no longer does 22:11:43 * fungi has to disappear now, but will catch up on the rest of the meeting from the log later tonight 22:12:03 dmsimard: oh interesting. okay. so maybe we can drop this from the 3.0 blockers? 22:12:11 it's possible a change occurred in zuul-stream fixed it 22:12:34 corvus: I don't know yet -- I could try reproducing the issue without the patch (that was supposed to address the issue..) and see what happens. 22:12:42 should be fairly easy 22:13:08 dmsimard: okay. if you decide it's no longer an issue, go ahead and drop the zuulv3.0 tag from the story when you update it, please 22:13:13 ack 22:13:32 mordred: did i see you post something about a zuul-stream refactor? 22:13:50 https://review.openstack.org/531171 looks like 22:14:12 mordred: is that a start to addressing the 'refactor zuul-stream and add testing' story? 22:14:52 the story for that is "It's currently largely untested and difficult to make changes to." 22:15:57 mordred: please let me know when you are back 22:16:06 corvus: it's sort of a patch related to that - but that's a little bit more about changing how the log stream data gets schelpped around to be potentially more something we could upstream 22:16:11 oh hai! 22:16:44 I also wrote up a proposal here: https://github.com/ansible/proposals/issues/92 22:17:13 and I guess it could help testing because we could have it log to a file/buffer in tests and check the contents rather than needing to build up a full on daemon process and listener and all that 22:18:08 clarkb: well, spinning things up and tearing them down isn't really a problem for the test runner 22:18:52 i think it's more that there's a structural issue with the fact that most of this behaves differently if you are sshing into a remote node vs locally 22:19:13 yah - and also things like rebooting a test node will kill the zuul_console process 22:19:30 mordred: well, i'm talking about unit tests 22:19:40 so the aim here is that we won't need a zuul_console process, and also that the surface area for things breaking will reduce 22:19:50 corvus: ah- yes, well, that as well :) 22:20:46 amongst the issues with the current system are surprise, fear and an almost fanatical devotion to the pope ... 22:20:57 the reason this story was created was because there are, even now, still a lot of times we put something in the console log that we shouldn't, or don't put something in that we should. and we have zero testing for any of that, and any time we try to change it, we break production, so it's effectively frozen.. 22:21:11 so that's the thing that we need to get unblocked before the v3.0 release 22:21:29 we need to be able to accept a bug report from someone saying "this should have been in the console log" and act on it 22:21:31 yup. and that refactor is still on the todo-list - and is largerly orthogonal to the above patch 22:21:46 mmm, largerly 22:21:47 okay that's helpful 22:21:50 or, I *believe* it's orthogonal 22:22:21 it's possible we'll discover that the other patch enables the refactor in important ways 22:22:25 mordred: yeah, i could see it interseting if testing somehow becomes more feasible with the above, otherwise, i'd be terrified to land the above anyway 22:22:33 ++ 22:22:52 +1 for not needing zuul_console process! I've been having reboot issues actually. :) 22:23:06 Do we no longer have a zuulv3-dev node that we could test sensitive things on ? 22:23:24 no, we deleted it 22:23:37 We could hook it up to openstack-dev/sandbox or something 22:23:57 pabelanger, leifmadsen_: i see you're working on the docs -- anything blocking you there? 22:24:14 dmsimard: for stuff like this I think the effort is likely better spent adding tests to the test suite 22:24:42 yeah, we're not landing anything else without tests :) 22:24:52 corvus: just discussing where we want the example config project repo to live, that was about it. We should have some rst docs up this week I believe 22:25:04 pabelanger: example config project repo? 22:26:05 corvus: some of the discussions were just about having something existing online, so we can use the git driver connection for it. But, possible we want to docs to also include more details how to build that out 22:26:12 pabelanger: since we don't expect anyone to actually reuse that, how about we just inline the content into the docs and walk people through creating their own? 22:26:35 corvus: yah, that is an option for sure. 22:26:59 i mean, we should document using zuul-base-jobs (which will need to be a config repo -- but it only holds jobs). and zuul-jobs, of course. 22:27:21 but i don't think we have plans for a reusable repo with, say, pipelines at the moment. 22:27:45 that's still something everyone will need to create locally. for now. :) 22:28:18 okay, I'll bring it up with leifmadsen_ tomorrow and we can discuss it more in #zuul 22:28:26 Making that more generic would ne cool 22:28:42 yep, but a very long-term task :) 22:28:51 pabelanger, mordred: what's the status of reporting on github from openstack-infra? 22:29:56 I last tested it before holiday break, but haven't progressed more due to upcoming refactor of things into zuul-web. But it was working, except for 1 thing I cannot remember ATM 22:30:20 pabelanger: were there any errors in the logs? 22:30:25 corvus: nope 22:30:46 corvus: I was waiting for the injest patches to land before moving forward 22:30:47 pabelanger: ok. mordred: maybe you want to turn on your shade test? 22:30:55 corvus: it's possible I didnt communicate that to anyone ... 22:30:59 why is that a blocker? 22:31:34 corvus: it's not - I just figured that scalability was one of the larger concerns and the injest patch changes the structure of how that works... 22:31:47 corvus: BUT- can *totally* turn on the shade patch very easily and we can see how it goes 22:32:19 yah, I think getting more data from shade patch works for me 22:32:21 mordred: i'm not going to say 'no' to more data :) i say go for it whenever you're ready :) 22:32:39 and hopefully we'll have the injest patch in next week too 22:33:11 i'm far enough with the cross-source deps work (which is the next item on the list) to say i think we can have it landed in the next few days / end of week at latest. 22:33:22 corvus: kk. I'll get that going 22:33:39 so once that lands, we'll have some *really fun* things to test 22:33:57 Yay 22:34:25 mordred: istr you said your shade job could be used to verify the cross-source work as well? 22:36:38 mordred: also, i think you were 90% of the way through js tooling patches before the holidays -- i've added a patch to your series which we can use to validate that all the URLs that we expect to work do work 22:36:39 would that be the whole "changes to ansible openstack modules triggers shade tests and reporting" thing? b/c that would be exciting 22:36:46 Shrews: yep 22:37:02 * Shrews is giddy with excitement 22:37:26 mordred: so when you pick that up, we can test the three different ways of serving the webapp that we know about 22:37:45 corvus: yup! 22:37:54 (to all of the things) 22:37:58 cool 22:38:06 the nodepool static driver from tristanC is in review 22:38:26 Shrews: are we running the finger gw in openstack prod now? 22:38:46 corvus: yes 22:38:51 Shrews: \o/ 22:38:53 Oh, nice! 22:39:06 finger UUID@zuulv3.openstack.org works 22:39:08 I think we are still running the executors on a low port right? 22:39:09 are we at a point to remove root user from zuul-executor now? 22:39:14 pabelanger: yes 22:39:15 Shrews: do you want to delete all the special user handling code from the executors, and have them serve finger on port XX79 by default? 22:39:22 Shrews: awesome 22:39:23 corvus: sure 22:39:34 i love deleting code 22:39:38 Shrews: cool, i think that's probably the last thing there and we can clear that from the list 22:39:54 pabelanger, leifmadsen_: ^ fyi quickstart docs change :) 22:40:06 corvus: ++ 22:40:25 as mentioned, jlk is working on getting github3.py released... 22:40:36 and finally, clarkb put 2 new things on the list about secrets and branches 22:41:23 these came out of the zuulv3-issues etherpad 22:41:42 my understanding of this is that without fixing this you cannot use the same secret name on different branches? 22:41:48 which seems like something we should fix 22:42:25 I want to say it was kolla that ran into this and they worked around it by using different names 22:42:25 clarkb: yeah, i agree these seem like release-blocking bugs 22:43:38 okay. that's the list. i won't go through it again next meeting, but i thought it would be good to reset after the holiday break. 22:44:08 #topic RAM governor for the executors 22:44:43 dmsimard: i think this is your topic? 22:46:32 dmsimard: lemme know when you're back 22:46:37 #topic mailing lists 22:47:12 i'll send out an email about this, but while (most of us) are here... we should now have the infrastructure in place to host mailing lists at lists.zuul-ci.org 22:47:46 i would like to have a zuul-announce list, where we make release announcements and notifications about changes to job syntax/deprecation/etc (which will be important for things like the shared zuul-jobs repo) 22:48:03 +1 22:48:22 and obviously, we need at least one other list for discussion... should we create a zuul-discuss list? or a zuul-dev list? or both? 22:48:37 I kind of think both 22:48:57 (ie, do we need a dev/other split, or can we start without that and just have a combined list?) 22:48:58 zuul-discuss for end user support? 22:49:26 corvus: but I could see starting with zuul-discuss and splitting a zuul-dev later if it gets too much for people 22:49:29 * fungi returns early and catches up 22:50:05 what about zuul-users 22:50:21 I really don't like the dev/general split we have for openstack 22:50:28 and also ops. 22:50:44 It creates a ton of cross posting confusion and also easy ways to ignore important threads 22:50:45 there's a lot of overlap -- any dev work we do will affect users and could benefit from general discussion, so for that, i like having one list. i don't want people to feel like they're "bothering the devs" though if they have questions like "how should i write a job that does ..." 22:51:30 corvus: yes - those are both sides of my thoughts :) 22:52:05 I think there are at *least* 3 personas "I am running a zuul" "I am writing jobs for a zuul" and "I am developing zuul" - many of us here are all three 22:52:23 i feel like maybe it's safer to start with one, and create more if needed -- so maybe we should have "zuul-announce" and "zuul-discuss" for now? 22:52:31 wfm 22:52:34 sounds good 22:52:47 sure 22:53:07 ok, i'll email the infra list with that as a proposal, get feedback from folks not here ("we decided in irc what the mailing lists should be!" is especially ironic), then make lists in a day or 2. 22:53:46 #topic open discussion 22:54:31 I had one question come into my mind this last week 22:54:42 corvus: mentioned in #openstack-infra, but as a followup from earlier, https://review.openstack.org/#/q/topic:turn-on-ansible has the patches to start running stuff on ansible patches 22:54:47 https://etherpad.openstack.org/p/zuulv3-issues is almost cleared out of active items. I've marked things as fixed that were fixed, gotten a few things fixed that weren't, and filed bugs for items that need longer term tracking 22:54:54 mordred: thx 22:55:03 IIRC we had some kind of plan to merge feature/zuulv3 into master before the 3.0 release. I don't recall what the prereqs for that were. 22:55:05 it would be great if everyone could take a look over that and make sure their items are accurate and possibly open bugs for them if they need longer term tracking 22:55:16 then I can unpin the tab in my browser :) 22:55:41 SpamapS: we need to update the puppet-openstackci deployment tooling to handle master as v3 22:55:48 SpamapS: yes! it's still imminent -- we want to land something to puppet-openstackci like https://review.openstack.org/523951 first 22:56:12 so that we don't introduce zuulv3 to the world of openstack third-party ci ops by automatically upgrading them 22:56:13 Ok, just looking for targets of opportunity. :) 22:56:35 but a puppeteer.. I am not 22:56:45 afaik that's the only thing blocking that, as soon as it lands we'll merge 22:57:17 and then i will need to unlearn 2 years of "git reset --hard origin/feature/zuulv3" 22:57:40 thats ok I always check out master first then have to switch to feature/zuuvl3 22:57:50 that'll unlearn itself pretty quickly when we delete the branch 22:58:12 fatal: ambiguous argument 'origin/feature/zuulv3': unknown revision or path not in the working tree. 22:59:53 the stuff of dreams 23:00:11 :-D 23:00:25 time's up, thanks everyone! 23:00:28 #endmeeting