22:01:54 #startmeeting zuul 22:01:56 Meeting started Mon Mar 26 22:01:54 2018 UTC and is due to finish in 60 minutes. The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:01:57 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:59 The meeting name has been set to 'zuul' 22:02:01 #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul 22:02:11 #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-03-19-22.03.html 22:02:21 #topic Action items 22:02:25 corvus attempt to help with storyboard private story notification fix 22:02:30 so i haven't gotten very far with that 22:02:39 we didn't encourage you enough 22:02:48 er, i should clarify 22:02:52 i tried again really hard 22:02:56 and still didn't get anywhere 22:03:32 o/ 22:03:36 the test infrastructure around this in storyboard is not functioning the way i expect, and i'm at a loss for how to fix it 22:03:55 (only half paying attention, please ping me if I'm needed) 22:04:05 so we need to decide if we're okay living with the fact that we won't get email about critical security bugs 22:04:14 or if we want to do something else 22:04:18 that's depressing. i'd like it to not be the case, but am dealing with my own "why did tests not catch this?" moment in sb now as well 22:04:44 yeah, i'd also love to have the time to work on a systemic fix to the issue i'm running into, but it would be a *large* change 22:05:01 i know how to do that, but don't have time. i do have time to make a small change, but don't know how. 22:06:01 what's our other option? or is there none? 22:06:01 we could try to peridocally poll storyboard with a cron. however, in pure coincidence, the cron i have running for the zuulv3 dashboard just started timing out. i don't know if the same would be true for a query for private stories. 22:06:07 i could try it and see. 22:06:30 that's one option. another might be to create a mailing list for folks to send security bug reports to. either the entire report, or just a notification after they create a bug. 22:06:51 honestly, if we can work around this with a cron job, that's *probably* the slightly safer/friendler path. 22:07:00 (for reporters) 22:07:09 it's probably a bit more work on our part to make sure the cronjob doesn't bitrot 22:07:22 (also, could be a periodic job in zuul, doesn't need to be cron ;) 22:07:54 i'm wondering if the recent api slowness is limited to searches by story tag since that's where i've mainly noticed it so far. more of a topic for #storyboard but it may mean that searching on other criteria is not currently slow 22:08:07 fungi: good point 22:08:19 so maybe i should try writing an automated query for this and report back? 22:08:32 however, i also haven't looked to determine whether you can query by private-ness 22:08:52 fungi: hopefully 'list all zuul stories' doesn't become prohibitively expensive :) 22:09:16 crontab seems like a good next step 22:09:27 #action corvus write automated query for private zuul stories (to find security issues) 22:09:42 the other action item from last week is: corvus create some remote tests for zuul-stream 22:09:54 i did that. i think they merged! 22:10:08 #topic Why not tagging the current master, e.g. release early, release often? (tristanC) 22:10:09 fair, if zuul has _that_ many stories before sb performance improves, then maybe we're doing something wrong ;) 22:10:30 tristanC is not here 22:10:48 but i guess the topic is self-explanatory :) 22:11:47 a beta release wouldn't be the worst thing imo (especially if github thing is fixable at this point) 22:11:51 my straightforward answer to that question is that as soon as we are able to release, we will. 22:11:56 but ya I realize ^ 22:12:03 could do zuul 3.0.0.0rc1 or something i suppose 22:12:07 i know of two issues: 22:12:34 1) javascript -- mordred has some changes that he thinks we should merge now because otherwise, migrating folks later will be painful/impossible 22:13:04 (as in, we've got at least one more case where we must merge a change then go do manual things to make it work) 22:13:26 we don't want to be landing those kinds of changes in the future 22:13:29 * mordred hopes to have that in shape for folks by tomorrow morning 22:13:48 * mordred currently on very bad in0-flight wifi 22:14:15 2) github3.py -- we're still in the position where if we get a working release of this before item #1 clears, great! otherwise, we need to vendor it before release. 22:14:32 either way, i think those both preclude releasing master at this point 22:14:57 wfm 22:15:11 corvus: it was mentioned by tobiash that the github3.py issue for license api thing is fixed? I guess it just needs a release now? 22:15:16 however, once we do release, we need to do one more thing, then we can return to our pattern of releasing frequently. like, maybe every few weeks. 22:15:25 clarkb: oh i missed that! 22:15:29 maybe it's time to ping jlk :) 22:15:30 jlk can probably confirm 22:15:44 So. 22:15:54 I think we fixed at least some of the issue tobiash found, but maybe not all 22:16:05 #info release is blocked by some pending javascript patches (mordred to prepare ASAP) 22:16:14 and I've been really bad about being able to set up something to validate, so it'd be helpful if we could get a fresh attempt by tobiash 22:16:32 #action mordred propose release-blocking javascript patches 22:16:33 There hasn't been a release, so he'd have to test from develop branch. If we can confirm, I can do the work to make a release happen 22:16:46 #action tobiash test most recent github3.py to see if it's ready to release 22:17:20 sweet! hopefully, with the magic of timezones, he can do that while i'm asleep :) 22:18:14 There have been other GitHub Enterprise folks testing and they've found at least one other issue that has an open PR, so I'm not sure if that effects tobiash or not 22:18:22 #info ping jlk with github3.py test results to make release 22:18:38 jlk: have a link handy? 22:19:40 digging itup 22:19:56 https://github.com/sigmavirus24/github3.py/pull/813 is the pr 22:20:10 https://github.com/sigmavirus24/github3.py/issues/794 is the (closed perhaps too early) issue 22:20:17 good ol' sigmavirus24 22:20:32 * prometheanfire misses sigmavirus 22:20:34 #link https://github.com/sigmavirus24/github3.py/issues/794 closed github enterprise issue 22:20:50 #link https://github.com/sigmavirus24/github3.py/pull/813 potential github enterprise issue? 22:20:55 electrofelix reported that issue? 22:21:00 small world 22:21:10 both in fact :) 22:22:28 nice 22:23:33 i think that's it for this subject... 22:23:43 #topic Open Discussion 22:23:45 Do we know electrofelix? 22:24:12 jlk: oh, yes, he's a contributor to jenkins-job-builder, and some other openstack-infra-related projects 22:24:50 that's his nick in irc; can often be found in #openstack-infra or #zuul 22:25:26 a.k.a. pelix 22:25:45 oh that name is new to me 22:25:50 jjb core reviewer 22:26:11 he was pelix for a while beforee electrofelix 22:26:48 in other news, i think i'm pretty close to getting git.zuul-ci.org up and running 22:26:57 I have an awkward patch up for ipv6 support in zuul: https://review.openstack.org/556327/ would love to discuss that this week. 22:27:23 it will be especially nice to be able to tell people to use "https://git.zuul-ci.org/zuul-jobs" 22:27:36 yah, that would be good 22:27:39 and cool 22:27:42 pabelanger: is there an equivalent patch for nodepool? 22:28:11 corvus: oh! I wonder then if he's another zuul + github user 22:28:16 Shrews: yah, that merged this morning https://review.openstack.org/556209/ 22:28:30 ah, ok. missed that 22:29:15 though biggest impact for zuul-{base-,}jobs since it goes in everyone's config if they want to cd our stdlib 22:29:47 jlk: if not currently, i'm given to understand, he'd like to be :) 22:29:58 i missed that too 22:30:04 i'd really like it if we stopped merging changes without docs 22:30:08 in fact, i'm going to propose a revert of that 22:30:52 we have past the point where we should merge a change without tests and docs 22:31:04 thats awkward that statsd has to have an explicit ipv6 flag 22:31:17 yeah, i assume it's because it's udp? 22:31:29 maybe instead of a flag we just detect if it is an ipv4 addr or ipv6 and do the right thing? 22:31:48 clarkb++ 22:31:48 we can do that, but no way to stop statsd from then detecting it again 22:32:10 clarkb: we'd need to know if the ipv6 address is routable from us, right? 22:32:39 corvus: yes, I think assuming it is if you have global ipv6 address is mostly safe 22:32:46 https://github.com/jsocol/pystatsd/blob/master/statsd/client.py#L152 is how it works today in statsd. If we want to propose a change, I'm okay witht hat 22:32:48 (this seems to be behavior of most applications) 22:33:05 clarkb: yeah, i'd prefer that approach too 22:33:09 yeah, if both are global then fine, otherwise that's not entirely trivial to determine 22:33:36 so basically, do a lookup of the host, see if it has a global ipv6 address, if so, see if we do, if so, set the ipv6 flag... 22:33:43 (why doesn't py-statsd do that?) 22:34:04 and "happy eyeballs" approaches don't work so well for connectionless datagrams 22:35:39 if someone had an internal (not global) statsd server only on ipv6, this scheme wouldn't work for them. they'd need a flag. 22:36:25 oh, there's one more thing related to releases i should mention -- 22:36:33 its unfortunate that is even a thing considering ipv6's available address space being there to avoid that problem entirely 22:36:43 but yeah, if you're hoping to take a non-global route to a v6 destination they can't reach via v4, then maybe the onus is on them to make dns only resolve the thing they can reach or use an address literal? 22:37:00 fungi: yeah, that might be enough of a workaround 22:37:43 after we do release 3.0.0, and before we release 3.0.1 -- we should land support for reno. i don't want to do it before 3.0.0 (because i don't want to even pretend we can write those release notes). but going forward, i'd like zuul and nodepool to have real release notes. 22:37:55 so that basically means that's the first post 3.0.0 thing we need to land 22:38:22 so, use socket.getaddrinfo() again to detect if ipv6, then setup flag on statsd client. wfm 22:38:50 pabelanger: yeah, let's give that a shot. because i think we all agree this option is weird. :) 22:39:05 indeed 22:39:49 pabelanger: not quite as simple as all that, need to check that you have a global route i think? but basically yes 22:40:36 fungi: check the route, or check to see if we have a global ipv6 address? 22:41:11 fungi: okay, I was going to use the same logic for fingergw we did for ipv6, I don't think we checked global route there 22:41:54 well one or the other should work, just pointing out it's not sufficient to check that socket.getaddrinfo() works via v6 22:43:03 fungi: okay, maybe after meeting we can look at https://review.openstack.org/551015/ and see if we'd have the same issue 22:43:34 yep, we can hash it out outside the meeting for sure 22:44:03 i had an action item a while back to follow up on the meeting time change. i did not do that last week, but have started doing it this week. 22:44:36 i don't think there's a way we can make everyone happy, so i'll probably end up making an executive decision one way or the other. however, there's another alternative i'd like us to consider: 22:45:24 end the practice of regular meetings and focus more on mailing list discussions for wide dissemination/collection of information, and ad-hoc irc discussions for when that's not practical 22:45:39 as the folks who randomly showed up to today's meeting, what do you think about that? :) 22:46:07 wfm 22:46:20 I admit, i like the structure IRC meetings, but happy to try something new if people would like 22:46:34 as a completely non-representative sample, i think it would be nice to venture headlong into this new governance unencumbered by the baggage of weekly meetings 22:47:26 I know monty struggles with mailing lists based on some recent conversations. But I imagine ad hoc conversations can get him pointed to relevant threads as needed 22:47:52 culturally the openstack community has used scheduled meetings as a crutch to avoid discussing over mailing lists 22:48:20 would be nice to see if not having scheduled meetings reduced that tendency 22:48:32 I also have a really hard time with threads that get outlooked, but maybe I just need a better tactic for dealing with those 22:49:09 it's really hard to get to a decision on a mailing list. so it will require we develop some different skills. 22:49:32 go from "i should add that to the agenda for some upcoming meeting which may be a week away and might already be too overbooked to get to my topic" to "i'll post to the mailing list about this now" 22:49:38 and yeah, meetings provide a structure, timeboxed by their nature, which we will have to be cognizant of. (bi-?)weekly summaries may help with that. 22:49:50 I'm pretty bad about email for this project, but I can respond to random channel pings for data 22:50:19 corvus: re making decisions and lack of time boxes maybe once a thread has had reasonable conversation we set a N day timer on completing the decision making 22:50:24 yah, summaries might help with that 22:50:24 also, we have a dedicated mailing list, which may make it a bit easier for folks to deal with than, for instance, openstack-dev. 22:50:24 similar to specs I uess 22:51:39 +1 to trying MLs 22:51:58 okay, maybe i'll reply to the ml thread about the meeting time with the suggestion we consider terminating it. 22:52:20 i'll indicate in this meeting, at least, we saw some value to the meeting, but there wasn't widespread revolt 22:53:01 i don't find the meetings useless, but concede they may be an antipattern 22:53:42 probably the biggest thing is that it's an hour where we all actually plan on talking about zuul. for those of us with multiple hats, that can be surprisingly important. 22:53:51 ^ (for me) 22:53:55 fair! 22:54:20 ++ 22:54:38 but it's an hour that several other people plan on being asleep during, so... there you have it :) 22:55:03 anything else, or should we wrap it up? 22:55:10 i'm good 22:56:11 fwiw, ML discussions have been extremely valuable (e.g., the zk retry stuff), but was surprisingly lacking on much participation. not sure if an IRC meeting would be much better (or worse) 22:57:07 just more food (mmmm, food) for thought 22:57:43 personally i like the meetings, as someone quite remote it's often very vaulable to get decisions made in them 22:58:55 ianw: if we do keep it, maybe we could be more strict about the agenda (so people can more reliably anticipate if they should attend), and cancel it if there's nothing on it. 22:59:26 ianw: otoh, we could ping the folks active on an email thread and say "hey, so how about we decide on that thing we were emailing about" 23:00:11 thanks all! 23:00:12 #endmeeting