22:02:54 <jeblair> #startmeeting zuul
22:02:55 <openstack> Meeting started Mon Jun 26 22:02:54 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:02:56 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:58 <openstack> The meeting name has been set to 'zuul'
22:03:06 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:03:21 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-06-12-22.04.html
22:03:41 <jeblair> #topic Status updates: Standard job library
22:04:24 <jlk> o/
22:04:30 <jeblair> i don't think we've made substantial progress on this since last meeting, though pabelanger did do a lot of work to get gearman running with ssl which we will need for secrets support
22:05:08 <jeblair> mordred and i are planning on pushing pretty hard on this in the next week or two, i believe
22:06:03 <jeblair> once we get going, hopefully we can get the bonnyci folks, and software factory too, looking at the shared jobs repo
22:06:11 <Shrews> ++
22:06:58 <jeblair> anything else to add on this?
22:08:18 <jlk> Ready to review! :)
22:08:57 <jeblair> #topic Status updates: Documentation
22:09:21 <jeblair> i did a lot of work on documentation over the past week
22:09:25 <jeblair> #link https://review.openstack.org/463328
22:09:31 <jeblair> #link https://review.openstack.org/475928
22:09:42 <mordred> yah. them's a lot of words
22:09:45 <jeblair> those are the biggest changes, but they are at the top of a stack.
22:10:08 <jlk> I just spent a significant chunk of today (re)reading the first one.
22:10:12 <SpamapS> very excited to see docs starting to form
22:10:17 <mordred> ++
22:10:18 <mordred> me too
22:10:19 <Shrews> i LOVE when others write much needed documentation
22:10:24 <jeblair> the whole stack is ready for review, and it was green earlier today, though i merged tristanC's config default change which conflicts with some of them.  they will be easy to fix, so expect new revs tomorrow.
22:10:51 <jeblair> i know there's still a lot more to do (especially we need more examples, some of which should be in-line, some of which should be appendices)
22:11:03 <jeblair> and i think there's still some polish that could happen on higher-level overview stuff
22:11:31 * Shrews read that as "Polish"
22:11:32 <jeblair> but i'm hoping that's enough of a skeleton for us to build on  (i mean, it's more than a skeleton, it is pretty substantial, but you get the idea)
22:11:41 <jeblair> Shrews: reverse polish?
22:11:56 <jlk> jeblair: hopefully you can refrain from rebasing the changes, right? So that sub-patchset level comparisons can still be used if/when you make changes?
22:12:40 <jeblair> jlk: i will need to rebase most of the stack in order to fix the conflicts, but i will do a two-step rebase to make it easier
22:12:54 <jeblair> jlk: i'll rebase without changes first, and annotate that so you can ignore that inter-patchset diff
22:12:57 <jlk> kk
22:13:04 <jeblair> jlk: then i'll fix the conflicts in its own chaneg
22:13:23 <jeblair> maybe.  if possible.
22:13:46 <jeblair> if i can't, i'll just make sure that i only change the minimum to fix conflicts in the rebase.
22:14:11 <jlk> appreciated
22:14:54 <jeblair> i've tried to build a narrative structure in each of the user/admin guides so we're sort of taking users or admins through a learning process.
22:15:10 <jeblair> i'm sure it's not fully there yet, but that's what i was aiming for
22:15:59 <jlk> it's a good read
22:16:04 <jlk> They're good words, Jom.
22:16:06 <jeblair> i'm very eager to land these asap, and then resume our practice of changing docs with code.  :)
22:16:27 <mordred> yes. this will be quite nice
22:17:30 * Shrews will review Jom's words tomorrow
22:18:10 <jeblair> the second change is all kinds of reorg, much of which changes again later.  i don't know how to make that better, sorry.
22:18:54 <jeblair> also, keep in mind that the docs job renders everything, so folks may enjoy reading the final rendered form in later changes.  :)
22:19:10 <jeblair> i've consulted it several times today
22:19:27 <jeblair> anything else?
22:19:54 <clarkb> if you pull the second change and look at it locally I think git itself might be able to better show you the file moves?
22:19:57 <clarkb> maybe not
22:21:57 <jeblair> clarkb: yeah, i'm not sure.  i took notes while writing it.  :|
22:22:24 <jeblair> #topic Status updates: Github parity
22:23:07 <jeblair> jlk: do you have any outstanding changes related to this?
22:23:28 <jlk> There are a couple open.
22:23:45 <jlk> Depends-On for github, and a fix to support push events better (non pull-request)
22:24:02 <jlk> I also noticed while reading through docs that we could do a better job of supporting statsd metrics in the github driver
22:24:20 <jeblair> good point; now's probably a good time to add that
22:24:37 <jlk> Oh, and Depends-On is partial, it does not do the helpful thing of back-searching to discover needed-by changes to re-enqueue.
22:25:01 <jlk> the back searching is going to be limited by what github supports in search, as we discussed at AnsibleFest.
22:25:37 <jlk> We might be able to get partially there just by checking our own cache of change information, but that could lead to non-deterministic behavior.
22:26:16 <jeblair> yeah, we still need to decide how to handle that.  we brainstormed some ideas about that; i think our choices are roughly: (a) don't support it; (b) move depends-on into pull request messages rather than commits; (c) perform some kind of local memoization so we just ask ourselves for the info
22:26:29 <jeblair> (c) is growing on me
22:26:46 <jeblair> the amount of data we would need to keep is tiny, i think.
22:26:54 <mordred> I believe b and c are better outcomes than a
22:26:57 <jlk> C feels fairly easy to implement, if we ignore lost cache during restarts.
22:26:57 <jamielennox> c is useful as we should know, but aren't we already doing (b)
22:26:59 <clarkb> bigger problem is probably making sure that you've collected all the data right?
22:27:03 <clarkb> for c
22:27:26 <jeblair> jlk: i think we can have the scheduler go ahead and store the data on disk to persist between restarts.
22:27:27 <jamielennox> i don't think it makes sense to have depends-on in commit messages for github
22:27:29 <mordred> clarkb: well - the only thing that could be blocked on a depends-on is a previously-gotten event
22:27:32 <jlk> clarkb: so far, we are. Every change we runtime cache we grab every commit message of every commit on the change.
22:27:47 <mordred> ah - different question. jlk's answer is better than mine
22:28:07 <jlk> jamielennox: you prefer that they be read from pull request comments instead?
22:28:38 <mordred> I thought we did not get an event when a PR summary changes?
22:28:55 <jamielennox> i thought we were doing from the pull request body, i mean you have to merge the full PR
22:28:58 <jlk> I'll have to test.
22:29:09 <jamielennox> but not comments on a PR
22:29:10 <clarkb> jlk: right but hat happens when you take a 4 hours downtime and github changes a bunch?
22:29:21 <mordred> btw - I think we should do c even if we do b - because gh rate-limiting means if we can avoid the extra query it would be valuable
22:29:42 <jlk> well.... lets take a step back here
22:29:42 <clarkb> or are you saying that its ok because next eent you'd just repopulate complete cache?
22:29:49 <jlk> clarkb: (yes that)
22:29:57 <mordred> ++ to that
22:30:07 <mordred> if there's a zuul downtime people are going to need to recheck anyway
22:30:21 <jlk> If we listen to jamielennox , and shift the data source to pull comments
22:30:22 <jeblair> (or admin enqueue events or something)
22:30:30 <jlk> we can avoid some of this problem
22:30:51 <mordred> well- I think pull comments are the wrong place - pull summary would be the more similar place, no?
22:30:56 <jlk> (it'll introduce some new problems, but not insurmountable)
22:31:03 <jamielennox> mordred: right, not arbitrary comments added to the PR
22:31:04 <mordred> summary goes into merge commit message
22:31:06 <jamielennox> just the summary
22:31:06 <mordred> jamielennox: ++
22:31:07 <jlk> hah, well, okay
22:31:19 <mordred> I thought we did not get events when that updates?
22:31:24 <jlk> yes, we'd just have to be aware when that changs.
22:31:26 <jlk> es
22:31:28 <jlk> one sec
22:31:34 <jamielennox> not sure
22:31:57 <jamielennox> also, it seems right to me on the PR, but i'm used to gerrit so all my PRs are one commit long
22:32:17 <jamielennox> but there are some great/shocking examples of 200 commits to a PR that i think mean PR is about the only place they make sense
22:32:22 <jlk> Changing the PR summary generates a comment event with an action of "edited"
22:32:40 <mordred> cool. so we can react to that and notice a depends-on has been added or removed
22:33:07 <jlk> sure, we'd just have to ensure we're only looking at the "special" comment
22:33:10 <jlk> which is the PR summary
22:33:34 <jeblair> jamielennox: i think the behavior is that if any of those 200 events have a depends-on footer, then the pr depends on that.  but putting it in the commit lets you say "this commit depends on this functionality", and that's something that will transit through multiple git repos, etc.  it's a property of the commit, rather than the (somewhat more transient) pull request.
22:34:36 <mordred> jeblair: yah - I think I'm coming to the other pov on that ... for github users, the PR is the unit of work, not the commit - removing a depends-on footer if it stops being true would also become potentially hard
22:34:54 <mordred> for folks who do not rebase their branches but instead push new patches on top of a branch
22:34:55 <clarkb> especially since old commits and comments go away when you rebase right?
22:34:56 <jamielennox> yea, ok, i see it's useful to reference it in the commit that uses it. but everything else we do is via PR, like you can't address a commit in a PR like BonnyCI/zuul#32@1 so your PR is really your unit in GH
22:34:59 <jlk> The PR Summary is implemented as the first comment of the PR, we'd have to identify this and make it "special"
22:35:29 <jamielennox> jlk: there's nothing that identifies the PR body as special at all in an event?
22:35:41 <mordred> clarkb: old commits do - old comments stay around
22:35:41 <jlk> not that I've seen :(
22:36:04 <clarkb> mordred: only top level comments though right?
22:36:23 <jlk> ugh
22:36:27 <mordred> clarkb: hrm. I thought in-line comments also stuck around and were marked as "for a previous revision" with a red x
22:36:35 <jlk> haha
22:36:38 <jlk> guys you're on a tangent
22:36:43 <mordred> we are indeed
22:36:56 <jlk> those are "different" as they're "review comments"
22:36:59 <jlk> not to be confused with Pull Request Review comments
22:37:03 <jlk> There's comments to the PR which you see on the front page
22:37:07 <mordred> ah
22:37:13 <jlk> there's review comments which are made in-line with the source code
22:37:23 <jlk> and then there's Pull Request Review comments
22:37:33 <jamielennox> which GH implemented as a special type of issue, which confuses this more - but digression
22:38:09 <jlk> From the webhook data I'm looking at, I don't even see an identifier for _which_ comment was edited.
22:38:29 <mordred> jlk: you can edit comments other than the summary?
22:38:35 <jlk> yes
22:38:44 <jamielennox> jlk: so you could still say a comment was editted on this PR and fetch it again? not greatest but it would catch it
22:39:01 <jlk> http://paste.openstack.org/show/613760/ is hte payload
22:39:03 <mordred> oh wow. you really can edit comments. that's so weird to me
22:39:12 <jlk> jamielennox: yeah, we could fetch and then re-examine the "first" comment.
22:40:05 <jeblair> okay, so there's some support for moving this to PR summary.  how about we dive deeper into this and make sure that doing so is both workable and a solution to the needed-by problem, and regroup?
22:40:09 <jamielennox> hmm, crazy that the PR body is only considered a comment
22:40:15 <mordred> ++
22:40:20 <mordred> to both jeblair and jamielennox
22:40:21 <jamielennox> ++
22:40:30 <jlk> ++
22:41:06 <jeblair> jlk: okay if i action you with that?
22:41:27 <jlk> worksforme
22:41:32 <jeblair> #action jlk look into moving depends-on to pull-request summary to see if it is both workable and a solution to the needed-by problem
22:42:06 <jeblair> cool, thanks.. let's move on to make sure we don't run out of time
22:42:17 <jeblair> #topic Status updates: (web) console streaming
22:42:48 <Shrews> oh, hey. that's an agenda item now
22:44:01 <Shrews> not sure what to say except that i've hit a roadblock and had to swallow my pride and ask for help from the original author and Dear Leader
22:44:29 * mordred is also going to help swear at things
22:45:08 <clarkb> have the problems been described anywhere yet?
22:45:32 <Shrews> clarkb: not publicly, no
22:46:05 <jeblair> hopefully we can figure out in the next couple days if the problems are "merely technical" or something more fundamental that would cause us to re-evaluate our approach
22:46:48 <jeblair> i think we're at the "this doesn't seem to work; not sure what's going on" stage
22:46:56 <Shrews> yeah. if it's a stupid programmer (aka, me) error, we can move forward. if it's an architectural thing, then public input may be needed
22:48:11 <jeblair> i'm going to skip some topics and move on to...
22:48:18 <jeblair> #topic Removing py2 support from v3 nodepool (Shrews)
22:48:31 <SpamapS> One thing I've never liked about async "futures" style coding is that it is often just hard to see wtf is going on inside the machine.
22:48:43 <jeblair> #link https://review.openstack.org/476683
22:48:43 <SpamapS> (that was regarding the web streaming)
22:48:58 <Shrews> mordred and I were discussing https://review.openstack.org/476683 this morning, and it came up that maybe we should stop supporting py2 in feature/zuulv3 nodepool
22:49:35 <jeblair> SpamapS: [me too; part of why i want to keep that kind of code in its own sandbox]
22:49:40 <mordred> yah - if we don't support py2 in zuulv3 of zuul, I'm not sure why we should support it in nodepool - of course, we need to start running infra's nodepool in v3 before we drop it
22:49:58 <clarkb> Shrews: mordred my concern with that right now is python35 integration test does not work on nodepool
22:50:07 <mordred> clarkb: that's a great concern!
22:50:08 <clarkb> I think we need to get the testing solid on python3 first then make that assertion
22:50:11 <mordred> clarkb: we sohuld, you know fix that :)
22:50:14 <mordred> ++
22:50:17 <clarkb> because right now python2 is what works based on testing
22:50:19 <Shrews> clarkb: that's a very valid point
22:50:27 <jeblair> clarkb: the devstack one, or the zuul one?
22:50:32 <clarkb> (I think it may work on feature branch but not master)
22:50:36 <clarkb> jeblair: the devstack one
22:50:45 <mordred> oh. yah - I only mean on feature/
22:51:03 <clarkb> mordred: so I think you have to do both at the same time actually
22:51:10 <mordred> but we should _definitely_ make sure testing is solid before doing anything
22:51:14 <clarkb> and the reason for that is we have a habit of fixing things on feature first then maybe backporting to master
22:51:22 <clarkb> and you can't backport anymore if different pythons are supported
22:51:52 <Shrews> devstack, from what i've seen. weird errors, like: http://logs.openstack.org/23/476223/2/check/gate-dsvm-nodepool-py35-src-nv/af51324/logs/devstacklog.txt.gz#_2017-06-26_17_03_17_402
22:52:09 <clarkb> Shrews: ya it has to do with setuptools and permissions and stuff I think
22:52:12 <clarkb> Shrews: its good python fun
22:52:16 <mordred> so - let's maybe reframe this to "before we release nodepool v3, we should drop python2 support" - largely because I dont' think we want to grow new python2 based users that makes it hard to drop support
22:52:47 <Shrews> mordred: yes, that's a good way to put it. A goal to move towards
22:52:52 <mordred> and between now and then, we should definitely make sure that all the python3 testing is solid and that infra is running nodepool v3 on python3
22:53:09 <mordred> since those are both just good ideas for sanity and consistency
22:53:23 <jeblair> mordred: i'm okay with that, though i note that it doesn't address the immediate need, which was driven by "we want to land some code in v3 that is much simpler (like, a hundred lines plus an external dependency) with py3
22:53:36 <SpamapS> does this maybe up the priority of shimming?
22:53:41 <mordred> jeblair: indeed
22:54:21 <Shrews> jeblair: to be fair, i *really* like harlowja's change, but it isn't something that's needed RIGHT NOW.
22:54:29 <clarkb> I'm ok with doing it I just think it has to be done on both branches together with working tests
22:54:39 <clarkb> Shrews: also that, it may save like a second on shutdown
22:54:48 <harlowja> i save all the seconds, lol
22:54:48 <clarkb> not super urgent but a nice code cleanliness thing
22:54:49 <Shrews> well, could be several seconds, but yeah
22:55:13 <jeblair> SpamapS: i think the main gains of shimming were load testing plus ease of transition for openstack-infra; so i don't see this as driving a further need for that.
22:55:44 <SpamapS> Oh I thought maybe shimming would let us get onto feature/ sooner.
22:55:54 <SpamapS> so we wouldn't be backporting so much
22:56:23 <jeblair> SpamapS: oh, hrm.  i guess it would?
22:56:35 <jeblair> like, we could go ahead and merge feature back into master
22:57:20 <jeblair> but it sounds like our immediate needs aren't important enough to push that though
22:57:56 <jeblair> we can shelve harlowja's change until we're ready to do that anyway, and we'll survive
22:58:05 <harlowja> wfm
22:58:14 <jeblair> i will buy harlowja a cookie
22:58:37 <Shrews> so, is it a goal that we remove py2 support from nodepool before release then?
22:58:55 <jeblair> +1
22:59:14 <jeblair> #topic Surprise bonus topic: should we meet next week?
22:59:36 <jeblair> i just realized i'm going to be sitting around watching a smoker all day next monday
23:00:00 <jeblair> (it's the day before a holiday, so i will probably make it a 4 day weekend)
23:00:09 <Shrews> i'm considering taking all of next week off, so +1 from me
23:00:14 <jeblair> would folks like to cancel the meeting, or someone else want to chair?
23:00:37 <jlk> I'm okay w/out a meeting
23:00:57 <jlk> I'll report my investigations by way of a gerrit change
23:01:35 <jeblair> okay, i'll wait for a volunteer chair until tomorrow and without one, i will send a cancellation notice
23:01:39 <jeblair> thanks all!
23:01:40 * SpamapS is +0
23:01:46 <jeblair> #endmeeting