22:03:49 #startmeeting zuul 22:03:50 Meeting started Mon Mar 19 22:03:49 2018 UTC and is due to finish in 60 minutes. The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:03:51 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:03:53 The meeting name has been set to 'zuul' 22:03:58 #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul 22:04:09 that's not really true -- i didn't clear out the agenda from last time 22:04:16 i don't think there are any new items there 22:04:36 #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-03-12-22.01.html 22:04:41 #topic Action items from last meeting 22:04:47 tobiash send a proposal to the mailing list to discuss moving meeting to 20:00 or thereabouts 22:04:52 that happened 22:04:59 21:00 has been proposed as a compromise 22:05:11 i don't know that there's been any consensus yet 22:05:20 maybe we need a poll. or maybe we need an executive decision. 22:06:03 corvus: I support any new time that works for people -unless it's a time I don't like 22:06:19 o/ 22:06:26 mordred: your candor is a model for all 22:07:23 i'll try to push on this more this week. 22:07:30 I'm happy for it to be 20:00, don't compromise just for me :-) 22:08:12 I have no problems with any of the times around what wehave now 22:08:36 jhesketh: thanks. i'll try to find out what 21:00 would mean for the eu folks to see if that's worthwhile, or if we need to do 20:00. 22:09:03 in the mean time, we'll try to make the meeting logs useful, and have useful mailing list conversations too. 22:09:37 i think we're already pretty good about not doing big decisions just based on meetings -- there more of an opportunity for folks to sync up on current issues. 22:09:42 2100UTC will be 2300 central european time I think 22:09:49 at least during the daylight savings hours? 22:10:15 (don't trust my dst and timezone math though) 22:10:29 clarkb: i don't trust anyone's dst math :) 22:10:42 * jhesketh nods 22:10:45 next action item: corvus create zuul-security team in storyboard 22:10:48 i did that 22:11:12 i added a bunch of folks from zuul-core which have been active reviewing things recently... 22:11:44 we discovered you don't get email while a bug is private 22:11:52 but I think storyboard has decided that is a bug? 22:11:59 lets just switch to Mars standard time 22:12:07 clarkb: i think that's the case 22:12:29 i'm not sure it's going to immediately be fixed though 22:12:45 should we just make the membership match zuul-core... or should we have people opt-in to it if they're interested? 22:12:59 * fungi sneaks in late and sits in the back 22:13:07 at least currently it is somewhat opt in as far as needing to poll for bugs 22:13:31 tobiash set up a dashboard 22:13:36 er, worklist 22:13:59 oh, but i think it depends on having the zuul-security tag. so that could be lossy. 22:14:18 we could always make a periodic zuul job that polls for us. 22:14:46 #info zuul-security team created in storyboard 22:14:48 as long as it uses an account with access to those bugs, i guess 22:14:58 s/bugs/stories/ 22:15:20 #info ping corvus if you are want access and are on zuul-core and don't have it already 22:15:34 also, i have no idea if storyboard shows you what teams you are on 22:16:08 (i'm an admin, and team administration is admin-only, so i can't reliably investigate that myself) 22:16:13 i've also discovered that storuboard.o.o is lagging in sb-wc updates owing to a (until recently incomplete) transition in the way it was being deployed, so it's still missing the custom reporting urls feature 22:16:22 corvus: I don't see it under my profile so I'm guessing not? 22:16:58 clarkb: the only thing i can think is maybe if you visit the teams page not as an admin you can see your own. but i'm totally just guessing. 22:17:12 like, *if* it's possible, that's where i'd guess it might show up. 22:17:28 i think we're close to the webclient deployment updating regularly again, but spotted some possible ui regressions in draft rendering worth digging into before merging anything else there 22:17:53 fungi: cool, that will be really helpful for this as well as general bug reporting 22:18:05 fungi: let me know if I can be of use there - it's all probably my fault anyway 22:18:16 I don't even see where I can find the team so ya not sure there is a straightforward way in the current ui 22:18:27 (someone reported issues reporting a zuul bug the other day because the new-bug form was behaving in a way that would be improved by the custom url) 22:18:34 mordred: see recent scrollback in #storyboard about fonts 22:19:18 #info zuul-security members will need to search for private stories since notification is not working 22:19:23 oh, right, instructions for reporting suspected vulnerabilities 22:19:31 i think i had an action item there 22:20:05 #link https://review.openstack.org/554352 Add instructions for reporting vulnerabilities 22:20:13 draft in need of reviewing 22:20:32 fungi: thanks, yes that was the next and final action item from last week 22:20:44 i'm still working on the process documentation for people handling those 22:20:57 hopefully that'll be up for review in a couple days 22:21:10 honestly, all this taken together, i'm worried we'll miss vulnerability reports. i think we need to do *something* about that. 22:21:23 i agree 22:21:46 maybe fix storyboard. or do a periodic job to poll. or ask folks to send an email (to a private list) after opening a story. or use a private list instead of storyboard. 22:22:48 fixing sb notifications seems like the most useful step. i'm just hesitant to volunteer for something else and not get to it 22:23:06 i was going to give the first one a shot, but i ran into issues with the test framework, and the storyboard folks haven't had time to give me any hints on how to proceed. 22:23:23 https://review.openstack.org/553102 is as far as i got 22:24:02 i can try to push it some more. but i need moral support. :) 22:24:15 aha, so trying to get testing in place for the fix 22:24:34 looks like trying to reproduce it first then work on a fix? 22:24:38 yep 22:25:05 i mean, we're in this position because there is zero testing for this feature :( 22:25:26 i'll point diablo_rojo to this meeting log too, since she's been keep to track issues preventing sb from being used by projects who need vulnerability management 22:25:59 er, keen 22:26:40 okay, i'll give that one more shot, and if i can't make any progress, let's discuss options 2-4 next week. 22:27:03 (note, i'm still looking for encouragement :) 22:27:41 #action corvus attempt to help with storyboard private story notification fix 22:27:50 i find encouragement is best found in its liquid form :) 22:28:24 Shrews encourages corvus to drink then hack on storyboard 22:28:26 liquid couragement 22:28:46 #topic Release blockers 22:29:05 okay, so the latest on the release is: 22:29:24 [drumroll] 22:29:35 1) mordred's going to check on whether there's any more JS stuff we have to land now because we won't be able to rolling-fix it later 22:29:59 2) refactor and testing for log-streaming 22:30:15 3) at least one security issue which is still marked private 22:30:27 4) github3.py release/fix 22:30:29 [eol] 22:30:47 re github3 we've pinned to an older commit for now right? 22:30:56 (so we can't go as is) 22:31:04 (the hopefully-soon-not-having part of #3 i suppose is what you mean there) 22:31:15 I'm 75% of the way through implementing the last thing I think will verify we're good on #1 (it's looking good) 22:31:38 clarkb: yep. in order to actually release, we either need a newer github3.py release which fixes the problem with ghe. or wait long enough for ghe to catch up. or vendor github3.py. 22:31:47 On #4, I can speak to that 22:31:52 jlk: great! 22:31:54 \o/ 22:32:01 well, I was going to say something like what you just did... 22:32:13 but, I think I can convince upstream that we should do what we can to support older versions of GHE 22:32:23 that it's not realistic to just draw an line and say "sorry". 22:32:37 jlk: oh that sounds promising. last i looked at the issue it seemed to be trending toward wontfix 22:32:39 There is desire upstream that we always return the exact data 22:33:00 but given API versions, I don't think we can do that and we'll have to have some flexibility for GHE users 22:33:09 I meant to spend some time on it last week and I ran out of spoons 22:33:18 so hopefully this week. 22:33:39 jlk: the exact data meaning the same data always? 22:33:41 * corvus marvels at spoon-driven-development 22:33:51 but I guess the apis can't always accomodate that so you need to do best effort (see also shade) 22:33:57 jlk: libraries for a public service and libraries that consume N potentially copies of that same service running locally at various verions are a bit different aren't they? 22:34:04 clarkb: as in the same keys populated with data in the response 22:34:26 mordred: yes. that's the point I'm going to make 22:34:43 and that I think it's okay if when dealing with GHE you may get a key that is populated with None 22:34:52 jlk: ++ 22:35:05 there is ambiguity in "did I get a full response" vs "the API cannot give me that data" 22:35:18 ghe is github enterprise? 22:35:20 but again, price one pays for having to deal with N copies of the service w/ older API contracts 22:35:21 fungi: yep 22:35:22 fungi: yeah 22:35:27 jlk: yup 22:35:37 * fungi is terrible at acronym expansion 22:35:40 thanks 22:35:56 corvus: https://en.wikipedia.org/wiki/Spoon_theory 22:35:59 jlk: javascript supports a variable being marked at undefined as well as null ... I wish we had the same in python 22:36:36 jlk: would you be able to convince github to donate some of those older GHE's to CI for this sort of thing? :) 22:36:51 corvus: I could have said I ran out of tomatoes too... https://en.wikipedia.org/wiki/Pomodoro_Technique 22:37:09 mordred: yeah, we could fake it with string data, but that's awful. 22:37:23 you could say python has a separate concept of undefined 22:37:35 just manifests through raised exceptions ;) 22:37:40 mordred: few good choices in python land. Drop the key, populate the key with a None LIKE object (but not None), use None. 22:37:56 drop the key would probably be most accurate 22:38:02 but violate the return same data each time 22:38:16 SpamapS: maybe! I actually haven't talked to the people at GitHub yet that deal with 3rd party libraries. I don't even know if there is anybody at GitHub that does such a thing. 22:38:19 (I should do that) 22:38:34 "A pomodoro is indivisible" - I disagree - I divide pomodoros all the time - you just need a (reasonable) knife 22:38:43 step 1. post job req 22:38:47 clarkb: yeah, it makes it difficult to consume the library if you talk to both GHE and public. 22:39:14 client consumers have to reason about with has_attr vs checking for None value 22:39:19 mordred: if you take a knife to your pomodoro timer it's very unlikely to tick anymore. 22:39:40 SpamapS: I was just talking about tasty pomodoros 22:39:56 * ianw has no idea what a pomodoro is 22:40:02 mordred: your taste in objects confuses me. I suggest eating something other than timers. 22:40:09 ianw: (italian word for tomato) 22:40:20 I see I've now totally derailed this meeting :D 22:40:23 squirrel! 22:40:37 corvus: those are divisible too 22:40:42 wait we're doing squirrel method now? How do we get the squirrel back from mordred over IRC? 22:41:01 * mordred hands everyone a fluffy squirrel 22:41:01 you throw a tomato at it 22:41:12 gee, thanks? 22:41:13 anyway, back on the topic 22:41:14 haha i was guessing spanish word for pomegranate 22:41:43 I'll make a case that github3.py needs to handle old versions of GitHub Enterprise, pick the least crappy way of doing that, and push for a bugfix update 22:41:50 jlk: thanks! 22:41:55 jlk: ^5 22:42:11 He's said he's willing to see quick bugfix releases go out, and in theory I have the credentials to make that happen, so hopefully that part of it will be low friction. 22:42:34 i think the zuul vulnerability is going to cause us to release no sooner than next week anyway... 22:43:18 so we can probably check back in next week and see how things look on the gihthub3.py front 22:43:25 openstack has been a good tester of those patches :) 22:43:50 we didn't talk about #2: log streaming, did we? 22:44:08 not yet 22:44:09 what's the streaming problem now? 22:44:19 fungi: it's not tested 22:44:23 got ity 22:44:35 so as the saying goes, broken 22:44:53 (every time we've landed a bugfix to it, we've broken it. i'm uncomfortable releasing something where we can't land a bugfix) 22:44:54 also it doesn't survive node restarts - but the "not tested" part is the most important thing 22:45:02 corvus: ++ 22:45:14 the refactor in progress makes it more testable by design 22:45:32 there still may be some things we should use the new 'remote' tests for as well. 22:45:52 corvus: ya may be a good idea to test the ssh socket forwarding stuff 22:46:07 just beacuse that seems like a potentially fragile thing that may break if no test explicitly covers it 22:46:19 clarkb: yep, i think that will need to be covered by 'remote' 22:46:45 i believe Shrews has started pitching in on this effort 22:47:04 ya I reviewed some Shrews patches today to catch up on what was going on there 22:47:11 just trying to push it along by fixing tests until mordred can get back to it 22:47:22 it needs some more work before it's functional, and since the 'remote' tests post-date it, it will need some new tests written 22:48:37 i think our choices here are: 1) to continue to plug away in the direction we're going; 2) write some 'remote' tests for the old system and call it testable; 3) release with the understanding that if there are bugs, we won't be able to land fixes until someone writes a test from scratch. 22:49:23 i think those are ordered in decreasing amount of time required. 22:50:12 thoughts? especially from Shrews, mordred, clarkb: ^ 22:50:15 i don't know if this counts for anything, but i think the push toward the new way is making log streaming super complex (more so than it is now, imo). that might introduce bugs and whatnot 22:50:43 but, of course, there are the benefits that may outweigh that 22:50:57 Shrews: you'd know better than me. The old system seems complex too :) I think 2) maybe viable as a temporary stand in if we need it 22:51:05 I don't think its too hard to add in remote tests? 22:51:25 Shrews: i think we'll need generous testing for it. that may end up being the biggest time cost. 22:52:27 so ... I think I agree with Shrews ... the new thing there is a bit more complex ... 22:53:06 it might be worthwhile to maybe shift focus to 2 for now, then circle back and finish the new thing when it's not a blocker and we can spend our time on getting it right 22:53:06 clarkb: yeah, i think the weirdest thing about the old system wrt testing is that once we launch zuul-stream with the first remote test, it stays running for the remainder. that's... probably okay? (maybe we could do a cleanup fixture to try to kill it or something if we thought that was important) 22:54:53 I'm dropping my slack patch this week, so I can catch up to master, and I'll do whatever testing I can. 22:55:08 or change how it's being started to die once the client closes its socket 22:55:10 okay, maybe i should push up a change to add, say, two tests for the current system using 'remote', and we can see how that looks, then maybe other folks can pitch in if we want more coverage. 22:55:17 and honestly, i'm not entirely clear on how the new way helps improve our testing, so maybe mordred is right by suggesting #2 for now? 22:55:39 fungi: i don't think that's possible due to timing considerations 22:56:36 corvus: ++ 22:57:18 ahh 22:57:31 Shrews: well, not having to worry about the streamer running as a remote process makes things a bit more isolated. i guess we're about to find out how important it is. 22:58:15 i do like the new system though for several reasons. chief among them is that it requires nothing special running on the test node and survives reboots. :) 22:58:27 corvus, Shrews: yah, exactly. pre-remote test framework, trying to figure out how to deal with the zuul_stream process was ... unhappy thoughts 22:58:57 #action corvus create some remote tests for zuul-stream 22:59:21 maybe i can do that in such a way that they'll work for the new system too 23:00:07 i mean, fundamentally, i care most about things like "did we report that a playbook started into the console log" 23:00:25 that should be the same in both systems 23:00:35 we're at time... 23:00:39 thanks everyone! 23:00:56 #endmeeting