22:02:36 <jeblair> #startmeeting zuul
22:02:37 <openstack> Meeting started Mon Apr 24 22:02:36 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:02:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:40 <openstack> The meeting name has been set to 'zuul'
22:02:49 <jlk> mordred: Shut the front door!
22:03:03 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:03:24 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-04-17-22.03.html
22:03:51 <jesusaur> o/
22:03:54 <clarkb> hello
22:04:11 <jeblair> jlk has a time constraint and has requested that we talk about the github topic which i put on the agenda first
22:04:20 <jeblair> #topic Github patches
22:04:39 <jlk> Hi, thanks.
22:04:53 <jlk> So, github patches.
22:05:15 <jlk> There is a series of them, the topic is 'github-v3'
22:05:28 <SpamapS> o/
22:05:31 * SpamapS made it back
22:05:42 <jlk> These are mostly cherry-picked from the zuul 2.5 patch series to add github support. I've massaged them to fit into v3 as best I could.
22:06:17 <jlk> The first 10 or so pass tests after being rebased on some recent v3 changes, the rest are in progress of being rebased
22:06:22 * SpamapS notes that jlk is excellent at massaging the pythons
22:06:34 <SpamapS> careful though, they bite. :)
22:06:36 <jlk> and I think there's another 5 or so from BonnyCI's fork of zuul that have yet to be brought into gerrit.
22:06:47 <jeblair> we've been deferring review of them so that we could focus on getting the bulk of the fundamental zuulv3 changes (many of which affect how these should operate) in place first
22:07:31 <jeblair> jlk and SpamapS asked me if maybe we could consider whether it's time to take a serious look at merging these soonish
22:08:00 <jeblair> and i think now is a good time
22:08:30 * mordred agrees
22:08:40 <SpamapS> \o/
22:08:42 <jlk> I agree as well :D
22:08:50 <jeblair> i think the major things that would impact it are out of the way now
22:08:53 <jlk> I look forward to the feedback, and will be quick to respond.
22:09:23 <jeblair> (we're certainly not done with v3, but i don't think these are blocked on anything major anymore)
22:09:51 <jlk> These may also shed light on places where model.py is too gerrit specific, so some things may need to move out into driver space, which I'm also happy to do work on.
22:10:12 <jlk> ditto scheduler.
22:10:17 <jeblair> i took a high-level look at them, and they generally look like they fit in pretty well; i think that merging them won't be a major distraction at this point.
22:10:39 <jlk> (and now I have to skeedaddle, so my son doesn't have to wait in the school office. My client will keep logging)
22:11:22 <jeblair> i'm signing up to review them; if anyone else is interested, let me know and i'll make sure we know to watch for your reviews
22:11:32 <jeblair> (or, you know, just start reviewing if you want :)
22:12:53 <jeblair> i'll start reviewing the early ones while jlk continues the large rebase underway
22:12:53 <SpamapS> There are some (not)fun github-isms that don't mesh well to the way OpenStack uses Gerrit+Zuul.. but most of them are handled well in Zuul's model AFAICT.
22:12:53 <SpamapS> One just has to use Github differently than one uses Gerrit.
22:12:54 <SpamapS> which is not surprising. :)
22:12:54 <jeblair> SpamapS: yeah, though a lot maps over with sufficient abstraction.
22:13:10 <mordred> SpamapS: as a person who has yet to find great ways to use github, it might benefit me to have a really high-level paragraph or somethign which describes the workflow/differences ?
22:13:58 <SpamapS> mordred: the biggest thing is that "reviews" are a new beta feature of github
22:14:06 <SpamapS> and they've got some very strong opinions
22:14:10 <SpamapS> one of which we strongly disagree with
22:14:16 <mordred> yah. the "votes are sticky" one?
22:14:24 <SpamapS> but basically.. when you push -f over your source branch for a PR, they do not dismiss positive reviews
22:14:33 <SpamapS> They're committed to that.
22:14:36 <mordred> wow relaly?
22:14:41 <mordred> they think that's not a bug?
22:14:45 <SpamapS> They've noted that we're not alone in reporting it as a bug.
22:14:48 <jeblair> many people will enjoy abusing that.
22:14:51 <SpamapS> No they think it's a feature.
22:14:54 <mordred> wow
22:15:03 <jeblair> i'll note that gerrit has that feature.  it's just (sensibly) off by default.
22:15:33 <SpamapS> Zuul can enforce the review<->sha bond.. but that feels like zuul imposing things on github.
22:15:56 <SpamapS> also you cannot review your own PR
22:16:14 <SpamapS> so there's no self-approve without something like a magic label.
22:16:31 <jeblair> SpamapS: generally speaking, if we can find a way to let zuul operators/users *choose* to enforce something, i think that's in scope.
22:16:53 <SpamapS> jeblair: Agree. We've put that one down as "let's warn users and see how they feel about adding it as a feature"
22:17:24 <jeblair> SpamapS: (to be more concrete, if it could be implemented as, say, a pipeline requirement of "review==sha" or whatever would seem sensible)
22:17:37 <SpamapS> But the events from github match well enough with events from gerrit... it's still just repos.. and there are refs for things.. there's not too much to change in the core now.
22:17:41 <jeblair> then you can choose to have that req or not
22:17:59 <SpamapS> jeblair: yeah that's how I'd want it.
22:18:08 <jeblair> yeah, i got about 75% of the way through the stack before i started having thoughts like "let's rethink this..."
22:20:22 <jeblair> anyone else want to say anything on this topic?
22:22:00 <jeblair> well, i'm excited.  thanks jlk for putting up with all the rebasing and doing such excellent massaging.  i know mordred will be thrilled when we can start reporting on ansible PRs.  i will be too.  :)
22:22:16 <mordred> \o/
22:22:21 <jeblair> i'm going to continue to run this meeting backwards...
22:22:25 <jeblair> #topic Bubblewrap
22:22:42 <jeblair> i mostly just wanted to check in and see what the (infra?) blockers are for bubblewrap
22:22:49 <jeblair> something about packaging...
22:22:51 <mordred> fwiw - I put a topic on tomorrow's infra meeting related to "where do we stick debian packages"
22:22:57 <SpamapS> ubuntu backports, I think, is effectively stalled
22:23:11 <jeblair> mordred: oh cool, so that (can be?) in service of bubblewrap
22:23:14 <mordred> and ppa's seem hosed, at least they did not build anything for vhd-util
22:23:32 <SpamapS> Simplest way to get bwrap into Xenial would be a PPA.
22:23:35 <mordred> jeblair: yah - we have 2 things we need in infra potentially that are not in our upstream repos at the moment - vhd-util on xenial and now bubblewrap
22:23:42 <SpamapS> PPA's what?
22:23:44 <mordred> SpamapS: it would - if ppas were building things
22:23:50 <SpamapS> #ohmy
22:23:57 <SpamapS> That's a very bad sign.
22:23:59 <mordred> SpamapS: I uploaded a new source pacakge for xenial to the vhd-util ppa
22:24:03 <mordred> and nothing happened
22:24:07 <mordred> I mean, it accepted it
22:24:17 <jeblair> but in a mongodb sort of way?
22:24:20 <SpamapS> PPA builders take a back seat to the OS builders
22:24:23 <SpamapS> so it may have been queued
22:24:31 <pabelanger> Ya, shouldn't be too hard to get a bublewrap repo setup for repreprp in openstack-infra
22:24:36 <pabelanger> if we want to do that path
22:24:55 <SpamapS> since they just opened artful
22:24:57 <SpamapS> lots of rebuilds
22:24:58 <mordred> SpamapS: it doesn't even show as queued
22:25:03 <SpamapS> mordred: link?
22:25:06 <mordred> SpamapS: there are literally no records in launchpad
22:25:17 <mordred> SpamapS: https://launchpad.net/~openstack-ci-core/+archive/ubuntu/vhd-util
22:25:34 <SpamapS> yeah wow
22:25:36 <clarkb> I was thinking about doing our own repo and wondering if maybe it made more sense to use nix?
22:25:53 <mordred> well, I mean, that would require retooling literally everytihng we do
22:25:55 <mordred> but sure?
22:25:58 <clarkb> then in theory we'd have packages that would work on centos fedora, ubuntu ,debian gentoo etc
22:26:03 <SpamapS> nix?
22:26:06 <clarkb> mordred: no it wouldn't nix is different than nixos
22:26:09 <SpamapS> Oh
22:26:10 <mordred> oh
22:26:18 <mordred> I thought you were suggesting moving to nixos
22:26:24 <SpamapS> well he is
22:26:25 <mordred> which, I mean, I'm game for crazy things
22:26:28 <clarkb> no I'm talking abuot nix
22:26:34 <clarkb> basically just the package manager
22:27:07 <clarkb> which apparently works quite well alongside other package managers in distros?
22:27:12 <clarkb> since its all self contained and magical
22:27:16 <mordred> well - I think this is honestly a grea topic for tomorrow's infra meeting
22:27:19 <SpamapS> it's a bit aggressive.
22:27:37 <mordred> I think today's zuul meeting status is "bubblewrap is stalled pending resolution of where to stick bubblewrap"
22:27:42 <mordred> yah?
22:28:02 <jeblair> yeah, that is a good takeaway
22:28:04 <SpamapS> I could also push harder and try to get xenial-backports upload rights
22:28:37 <jeblair> #info starting to review/merge github patches
22:28:46 <SpamapS> and we could also ./configure ; make ; sudo make install bwrap ;-)
22:28:51 <pabelanger> what about standing up a fedora zuul-executor for now? until we figure out ubuntu packages?
22:28:54 <jeblair> #info bubblewrap is stalled pending resolution of where to stick bubblewrap package
22:29:08 <SpamapS> pabelanger: need to test brwap
22:29:09 <jeblair> pabelanger: unit tests are still a concern
22:29:10 <SpamapS> bwrap
22:29:18 <SpamapS> so we'd need a fedora test job
22:29:20 <SpamapS> which isn't a bad idea btw
22:29:26 <SpamapS> but.. then do we have ZK?
22:29:29 <pabelanger> ya, we have fedora-25 in the gate
22:29:34 <pabelanger> yes, zookeeper for fedora
22:29:43 <pabelanger> I've been using it to test zuulv3 roles
22:30:36 <jeblair> that's good to keep in mind as an option.  let's see where we go in the infra meeting, and if the answer is "it will be some time before we have it available in infra", then we can add a fedora job and push forward on implementation.  that would only block infra rollout.
22:30:52 <pabelanger> k
22:30:53 <mordred> ++
22:31:32 <jeblair> #info a fedora25 test job is an option for testing bubblewrap
22:32:00 <jeblair> anything else we should discuss here?
22:32:09 <SpamapS> the implementation I already did has probably bitrotten
22:32:18 <SpamapS> but feel free to review it
22:32:34 <mordred> SpamapS: I believe in your implementation
22:32:44 <SpamapS> https://review.openstack.org/453851
22:32:59 <SpamapS> https://review.openstack.org/453852
22:33:06 <jeblair> #link https://review.openstack.org/453851 is bwrap implementation; does not pass tests but is ready to review
22:33:49 <jeblair> #topic Status updates: Zuul test enablement
22:34:12 <jeblair> i don't think we enabled many tests recently, though we did go and make them faster
22:34:30 <SpamapS> yeah we're yak shaving around the last few tedious/complex tests
22:34:33 <jeblair> clarkb is continuing looking into some issues as well
22:34:59 <SpamapS> I truly believe the weird optimizatoin problem with python in xenial is the issue.
22:35:11 <SpamapS> but there may be some interactions too
22:35:23 <clarkb> ya I'm working on cleaning up the thread leaks
22:35:47 <jeblair> are there any test enablement patches that are in need of review?
22:35:50 <clarkb> I've got apsched leaks fixed I think but that causes configuration verification to afil? is voluptuous introspecting Driver and Connections?
22:36:11 <jeblair> clarkb: yes
22:36:22 <clarkb> I'm also working to narrow down kazoo thread leaks (looks like they start in test_openstack)
22:36:45 <SpamapS> I haven't seen any test re-enablements come in lately
22:36:47 <clarkb> separately I've got broken yappi profiling and use cStringIO stuff. I'll try to make sense of it all and psuh a stack of changes
22:37:02 <SpamapS> 15 @skip's left
22:37:03 <jeblair> that reminds me we can probably drop test_openstack now; it was mostly a demonstration.  but obviously after clarkb uses it to track down what he's seeing.
22:38:16 <SpamapS> ah yeah, well covered in the others most likely
22:38:30 <clarkb> oh there are also gearman leaks somewhere too, the code I've got to debug thread leaks should help me track them all down
22:38:46 <clarkb> basically modified the if >1 thread then log error code to also log all the threads and their stacktraces
22:38:59 <jeblair> clarkb: ah, thanks.  wondered what you were doing there :)
22:39:05 <clarkb> then set .testr.conf OS_LOG_CAPTURE to 0
22:39:10 <clarkb> so that logs are captured unconditionally
22:39:56 <jeblair> anything else on this?
22:40:58 <jeblair> #topic Status updates: Zuul sample jobs
22:41:21 <jeblair> pabelanger: have you started up the executors again?
22:41:30 <jeblair> er, executor.  single.  :)
22:41:45 <pabelanger> I wanted to confirm still good to do that, if so, was planning on today
22:42:11 <jeblair> pabelanger: yeah, i think after installing branch tip (remember that's still manual), should be fine.
22:42:33 <pabelanger> okay, I'll do that after our meeting. Have some jobs I'd like to start testing again.
22:43:45 <jeblair> cool
22:43:57 <jeblair> #topic Progress summary
22:45:21 <jeblair> i don't think much is changing here atm
22:45:57 <jeblair> there's lots of tasks available if folks are interested.  i think it's generally correct.
22:46:09 <jeblair> #topic Open discussion
22:46:47 <pabelanger> zuul package for fedora landing hopefully this week :)
22:46:53 <jeblair> pabelanger: neat!
22:47:16 <pabelanger> next up will be nodepool
22:47:38 <jeblair> we've had lively discussions in #zuul today related to log streaming (which Shrews is working on implementing)
22:48:11 <jeblair> i should probably copy some of that into a story
22:50:00 <jeblair> thanks everyone!
22:50:03 <jeblair> #endmeeting