22:03:36 <jeblair> #startmeeting zuul
22:03:38 <openstack> Meeting started Mon Feb 27 22:03:36 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:03:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:03:42 <openstack> The meeting name has been set to 'zuul'
22:04:18 <jeblair> #link last meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-02-13-22.02.html
22:04:27 <jeblair> #link inaccurate agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:04:39 <fungi> accuracy is for nerds
22:05:14 <jeblair> i propose we skip the status updates this week and have a more free-form ptg recap and sync up
22:05:26 <fungi> seconded!
22:05:36 <SpamapS> indeed, much has changed :)
22:05:38 <adam_g> yes plz!
22:05:41 <jeblair> motion carries
22:05:49 <jeblair> #topic PTG recap and sync up
22:05:55 <jeblair> we ran jobz!
22:06:04 * fungi prepares to crib this for his infra ptg summary
22:06:20 <SpamapS> And we ansibled safely.
22:06:30 <fungi> for... definitions of "safely"
22:06:31 <pabelanger> there was much rejoicing
22:06:32 <rbergeron> "we ran jobz." *mic drop* should suffice for a summary, right?
22:06:38 <jeblair> rbergeron: ++
22:06:50 <clarkb> not sure what safe means here :)
22:07:18 <jeblair> i think we should keep running the zuulv3-dev server for a while so we can continue to poke at a running system
22:07:26 <SpamapS> clarkb: caution tape was not broken.
22:07:53 <SpamapS> jeblair: how long? Like, until we decide to just rename it to zuul? ;-)
22:07:54 <fungi> jeblair: sounds like a fine idea
22:08:07 <jeblair> we have some proposed solutions to keep it from interacting poorly with the production nodepool
22:08:39 <jeblair> SpamapS: hehe :)  probably we will re-install, due to all the terrible things we have manually done (and may continue to do) in order to expedite changes
22:08:55 <jeblair> (before we go into production for real)
22:09:12 <SpamapS> what, you don't login to production systems and live-edit normally?
22:09:18 <fungi> and so as to be able to test that we properly rendered such terrible things accurately in configuration management as this moves forward
22:09:21 <pabelanger> I would like to bring infracloud-chocolate online for nodepool.o.o, which means we'll break zuulv3-dev.o.o. But, I have a patch up to fix it! See 438490 and 438677
22:09:30 <jeblair> SpamapS: nothing normal about us :)
22:09:38 * rbergeron passes pabelanger a piece of chocolate for this
22:09:41 <pabelanger> otherwise, we'll run out of HDD space on nb01.o.o and nb02.o.o in a few days
22:09:54 <jeblair> pabelanger: then i guess we should consider that a pretty high priority
22:10:05 <pabelanger> yes please
22:10:16 <jeblair> #agreed keep zuulv3-dev online for continued testing
22:10:23 <fungi> no argument here, though does this uncover a verbosity or frequency issue we should eventually tackle?
22:10:37 <fungi> long post-v3 even
22:10:40 <jeblair> #agreed bring infra-chocolate back online
22:11:15 <jeblair> fungi: i don't think the hdd space issue is something that needs to be fixed in nodepool, aside from handling the situation we're in (which pabelanger is working on)
22:11:46 <jeblair> fungi: essentially, we're in a half-configured state because of the strange situation we're trying to accomodate.
22:11:52 <SpamapS> If you get desperate for a nodepool target ... I can probably carve a small quota out of BonnyCI's cloud. We're pretty short on public IPs though.
22:12:01 <SpamapS> I can probably request a v6 range.
22:12:12 <fungi> jeblair: okay, so it's not merely a "the provider you want to use is offline" scenario?
22:12:21 <fungi> in that case, makes sense
22:13:00 <fungi> SpamapS: we can get by with v4 for the mirror server and cloud api endpoints, v6 for everything else
22:13:37 <fungi> oh, and probably still depending on rfc-1918 v4 for the test nodes with an overload nat to at least one v4 address
22:14:14 <jeblair> fungi: i think so; i think there are thing we aren't doing (like manually deleting images) that we would do in a provider unexpectedly offline scenario.  i could be wrong about that, but at the very least, this situation doesn't map to that in a straightforward way in my mind.  :)
22:14:20 <clarkb> fungi: yes enough things hit github and the like
22:15:04 * fungi just found http://bonnyci.org/
22:15:51 <SpamapS> fungi: ohai ;)
22:17:42 * rbergeron taps on the channel to make sure everyone did not just get eaten by a pirate
22:17:44 <SpamapS> So.. given what we had at PTG, at some point I'd like re-evaluate our board of "Production ready Zuul" and make sure we have the right things on there and given the right level of priority.
22:17:53 <fungi> so the details as i have them are: 1. zuul v3 poc ran a simple hello world job as of tuesday afternoon, 2. people stuck around after tuesday to continue hacking and got it running general tox-based jobs, 3. what else?
22:18:07 * SpamapS realizes he's the only one who didn't have to read bonnyci.org's about page since he wrote it. ;)
22:18:21 <jeblair> related to that -- SpamapS and jlk are going to start work on forward-porting the github patches.  we are not ready to merge them (finalizing the config syntax and finishing some more fundamental v3 work is still the review priority), but doing so will help their team start running a tentative zuulv3 which helps them pitch in on other work.
22:18:36 <jeblair> rbergeron: looks like the answer is yes and no :)
22:18:53 <rbergeron> fungi: we went from single node to multi-node in a matter of minutes? :)
22:19:14 <SpamapS> we may want to slow down so we don't go to plaid
22:19:38 <jeblair> yeah, let's iterate through some topics
22:19:41 <jeblair> SpamapS: first
22:19:49 <jeblair> SpamapS: you suggested we re-eval the board
22:19:52 <fungi> rbergeron: thanks! yes, multi-node worked
22:19:56 <jeblair> SpamapS: want to do that in the latter half of the meeting?
22:20:06 <jeblair> or over the course of this week and reconnect next meeting?
22:20:21 <pabelanger> fungi: collect / publish logs on Friday
22:20:33 <rbergeron> jeblair: given this is my first meeting with y'all where i'm actively paying attention... can i add hashtag-info's and stuff for minutes? (i am more engrained in fedora culture and don't want to over-step my info boundaries or whatever) :)
22:20:40 <jeblair> rbergeron: do it
22:20:42 <SpamapS> jeblair: sounds good
22:20:47 <fungi> pabelanger: that mostly worked? we're still missing safe publishing of public logs right?
22:20:53 <jeblair> ahem
22:21:03 <jeblair> #topic Re-evaluate zuulv3 priority board
22:21:07 <SpamapS> jeblair: sorry lag.. later sounds good, we can just do a high level overview of process here, and reeval the board through the week
22:21:25 <pabelanger> fungi: they work, published only to zuulv3-dev.o.o. But yes, we need to dive deeper into ansible filtering on tasks
22:21:26 <rbergeron> #info things accomplished at PTG: 1: zuul v3 poc ran a simple hello world job tuesday afternoon; 2: ppl continued hacking post-tuesday and got it running general tox-based jobs
22:21:36 <jeblair> meeting-chair fail
22:21:51 <jeblair> we just vomited a bunch of stuff in channel
22:21:58 <jeblair> which is expeceted for an agenda-less meeting
22:22:01 <SpamapS> I got a huge burst of lag
22:22:05 <SpamapS> dunno if maybe we all did
22:22:10 <rbergeron> #info Further accomplishments: single-node to multi-node in a matter of minutes! (3), (4) collecting / publishing logs on friday
22:22:17 <jeblair> but now i think we have a bunch of stuff to talk about, so let's try to go through them one at a time
22:22:24 <pabelanger> ++
22:22:26 <phschwartz> SpamapS: I did. seems to be a freenode issue
22:22:43 <jeblair> the current topic is the zuulv3 priority board
22:23:25 <rbergeron> yay lag.
22:23:48 <phschwartz> jeblair: we can note that what I am working on is a priorty as it affects the layout directly. don't have the issue number at the moment.
22:23:50 <SpamapS> #link https://storyboard.openstack.org/#!/board/41
22:24:08 <jeblair> SpamapS: okay, so over the course of this week, we will update that and check in again next meeting
22:24:19 <jeblair> SpamapS: what else on that topic should we cover at this meeting?
22:24:40 <SpamapS> ^ See the above board. With a successful push toward a working zuulv3 completed at PTG, it's a good time to just peruse the lists, and make sure everything there is still relevant, and fill in what we've learned.
22:25:07 <SpamapS> My guess is a few things have just gotten bumped up in priority and a few may have been bumped down.
22:25:32 <jeblair> SpamapS: it may be good to restate our priorities to, as part of updating that
22:25:53 <jeblair> i suspect they haven't changed significantly, but keeping them in mind will help us filter that appropriately
22:26:00 <SpamapS> I'm most concerned with making sure we widen the available tasks as I personally have some new engineers coming online and they'll need a nice fat queue of low hanging fruit to get ramped up.
22:26:24 <SpamapS> (everybody say hi to eggshell and eventingmonkey when you see them in #zuul ;)
22:26:29 <jeblair> SpamapS: good to know
22:26:31 <rbergeron> #action please review lists in storyboard; make sure things are relevant, add in what we're learned. some items may have been reprioritized, though likely not significantly
22:26:41 <SpamapS> ^5 rbergeron
22:26:43 <SpamapS> that's all
22:26:51 <jeblair> #topic PTG summary
22:27:01 <jeblair> fungi wanted to know what to say about the ptg
22:27:13 <jeblair> i think maybe that got covered while i was banging the gavel
22:28:07 <rbergeron> jeblair: i think i may have covered that in #info's above, though i can press my up arrow a bunch and redo them?
22:28:09 <jeblair> also, i think mordred was working on a blog post that covered not only some of what we did, but will lay out some info for our community
22:28:48 <fungi> yeah, i'll gladly take what people brainstormed earlier, thanks!
22:29:39 <rbergeron> #action mordred we think is working on blog post covering things that happened at PTG, also laying out info / vague roadmap for community
22:29:53 <jeblair> i think that captures that pretty well :)
22:30:02 <jeblair> #topic github patches
22:30:27 <jeblair> SpamapS, jlk, and i were talking about this a bit earlier in #zuul
22:30:46 <rbergeron> fungi: awesome. hopefully whatever mordred has cooking will cover the rest but *those were big hooray accomplishments*
22:31:12 <jeblair> i wanted to bring it up here just to keep folks apprised
22:31:57 <jeblair> like i said, it's not a review priority yet, but we're far enough along that now is a good time to start working on them so they'll be ready when we are able to review and merge them; and in the interim, it helps get more folks involved
22:32:34 <clarkb> is plan to test those interactions to use something like requests mock like shade is using?
22:32:45 <SpamapS> In case any of you didn't read BonnyCI.org, we're trying to run a public zuul-aaS with github as a trigger source. The patches need porting to v3 before we can run anything meaningful on v3, so we're pushing on that now. :)
22:33:02 <clarkb> Gerrit is easy in that it hasn't changed much and we "test" it thousands of times per day, but I don't think we will have that with github at least not to start
22:33:03 <SpamapS> clarkb: Yeah something like that is definitely going to be necessary.
22:33:23 <jeblair> though i think we all know that at this point zuulv3 is neither secure or stable; so folks should not run it in production yet, and we should keep authoring of zuulv3 content (jobs, etc) to a minimum since we know the syntax will change.
22:33:39 <rbergeron> #info github patches aren't a review priority yet, but it's a good time to begin work on them so they'll be ready when we are able to review/merge; this will also benefit our bonnyci friends :)
22:34:03 <SpamapS> Heh right, we mostly just want to get good at running the new pieces. :)
22:34:09 <rbergeron> #info in case there was any question: zuul v3 is neither secure or stable, plz don't run it in production :)
22:34:21 <rbergeron> #info (but it will be :D)
22:34:35 <SpamapS> if you check out http://bonnyci.org/beta/ .. we _are_ looking for intrepid projects to take the first few chaotic steps with us.
22:34:56 <SpamapS> but we're not making any timeline promises and don't want to do more than play with it until v3 is production ready.
22:36:42 <jeblair> oh, i have another topic; then i will ask other people for more topics
22:37:00 <rbergeron> dmsimard: i say that and we seem to be having weird lag so... yeah :) (when "more topics" comes up it seems, real soon now)
22:37:07 <jeblair> #topic upcoming priorities
22:37:08 * dmsimard nods
22:38:08 <jeblair> my feeling is that with a dev zuulv3 running, some of the things we should focus on immediately are related to config syntax --
22:38:48 <jeblair> including reporting configuration failures  (so that as we propose changes to zuulv3 jobs, we can see errors)
22:38:50 <phschwartz> jeblair: agreed
22:39:12 <jeblair> and all of the outstanding tasks that are going to affect the config language (like the one phschwartz mentioned)
22:39:42 <jeblair> once those things are in place, we should be able to somewhat (still not greatly) expand what the zuulv3 dev server is doing
22:40:02 <pabelanger> Personally, I like to see us increase our test coverage on ansible plugins too
22:40:18 <jeblair> we put a bunch of things in the ptg etherpad (some of which is related to this) which we should tackle as well.  we may need to translate those to stories.
22:40:32 <jeblair> phschwartz: yes, i think there's a story and someone assigned to that now
22:40:35 <jeblair> w00t
22:40:50 <rbergeron> #info config syntax, including reporting configuration failures and any outstanding tasks that will affect the config language, are the things to focus on atm
22:41:00 <jeblair> pabelanger: ^ sorry, meant to address you
22:41:45 <pabelanger> jeblair: great, will look for it
22:41:56 <jeblair> #topic unagenda topic collection
22:42:14 <jeblair> rbergeron: should we talk about update emails?
22:42:33 <jeblair> dmsimard: you have an ara thing to discuss?
22:42:38 <jeblair> anything else?
22:43:04 <rbergeron> #info other items in the ptg etherpad
22:43:06 <dmsimard> oh hai, rbergeron suggested I ask here
22:43:07 <rbergeron> #link https://etherpad.openstack.org/p/pike-ptg-zuul
22:43:16 <rbergeron> jeblair: i woul dlike to, but let dmsimard go ahead real quick :)
22:43:23 <jeblair> #topic ara
22:43:24 <rbergeron> figured he might like to have the wider air time
22:43:29 <jeblair> dmsimard: go for it!
22:43:58 <dmsimard> I figured you might be interested in providing some comments/feedback regarding some UI work I've been looking to land in ARA real soon
22:44:18 <dmsimard> For a good use case, here's a before and after of an OpenStack-Ansible run with 1700ish tasks
22:44:24 <dmsimard> Before: http://logs.openstack.org/98/435698/1/check/gate-openstack-ansible-os_magnum-ansible-func-ubuntu-xenial/329f931/logs/ara/
22:44:33 <dmsimard> After: http://46.231.133.111/index.html
22:45:14 <dmsimard> Beyond the UI/UX work, one of the focus was to streamline the static generation which is very relevant in OpenStack gate scenarios (such as devstack runs et al)
22:45:56 <rbergeron> #info dmsimard seeking comments / feedback for UI work in ARA
22:46:04 <dmsimard> If we take for example this same OpenStack-Ansible run, it took 6m21s second to generate before (with 3710 files @ 31MB gzipped)
22:46:20 <jeblair> #link Before: http://logs.openstack.org/98/435698/1/check/gate-openstack-ansible-os_magnum-ansible-func-ubuntu-xenial/329f931/logs/ara/
22:46:22 <clarkb> I kinda like that things aren't paginated in the old thing. Eeasier to ^F
22:46:24 <dmsimard> Now we're looking at 18 seconds, 8.1MB on 1700 files
22:46:28 <jeblair> #link After: http://46.231.133.111/index.html
22:47:04 <pabelanger> clarkb: maybe a config flag to disable
22:47:10 <pabelanger> but looks nice, the UI that is
22:47:27 <jeblair> i think i have some rendering bugs, but i'll followup with dmsimard after the meeting
22:47:44 <rbergeron> dmsimard: so when can we get you to hook up to zuul v3? :) (even though the jobs are like, "woot hello world pep8 wheee" :)
22:47:52 <rbergeron> (though perhaps we're not ready, i have no actual idea)
22:47:56 <mordred> o/ zomg so sorry computer device did not bing
22:48:20 * Shrews waves from atop a mountain
22:48:26 <SpamapS> )I think ARA will be more helpful in the devstack-gate case, but can't hurt for others
22:48:48 <dmsimard> clarkb: thanks, I'll chat with you after the meeting see if we can work something out
22:48:49 <rbergeron> SpamapS: or maybe for bonny. but, yes
22:49:05 <dmsimard> rbergeron: I think we need to draft a spec around a zuul reporting UI in general
22:49:10 <dmsimard> this UI could be ARA or it could be something else
22:49:13 <jeblair> yeah, i'd like to have it so that we can hook ara into zuulv3.  probably in some manner that lets operators install callback plugins.  i definitely want that early in v3, but still think we need to defer it for a little bit longer while we get our internal house in order.
22:49:13 <rbergeron> dmsimard: thanks for sharing and coming :D
22:49:25 <SpamapS> rbergeron: well not so much for bonny, but more in the "when our tests involve lots of ansible tasks" vs. "command: foo" :)
22:49:46 <dmsimard> for example, it could be a soft coupling (i.e, just a pre-run task that sets up ara and post-run task that generates the report) or a hard coupling (actual centralized ara implementation in Zuul for example)
22:50:16 <rbergeron> dmsimard: https://storyboard.openstack.org/#!/board/41 is da place, i think :) (or someone will correct me, in 3, 2,... )
22:50:39 <dmsimard> Should we draft a broad spec for some sort of reporting interface ?
22:50:39 <rbergeron> at least to keep notes on things as they progress if nothing else, i'd guess
22:50:52 <dmsimard> otherwise I'm done, just a general request for feedback :)
22:51:00 <rbergeron> :D
22:51:02 <jeblair> SpamapS: we'll see how it shakes out, but even "command: foo" may involve more than one ansible task (to set up git repos, install packages, archive logs and artifacts, etc).
22:51:06 <SpamapS> the board/41 URL is for minimum requirements to put zuul in production
22:51:26 <jeblair> SpamapS: then again, maybe it's 3 ansible tasks and that's still not interesting.  :)
22:51:53 <rbergeron> spamaps: true dat
22:52:05 <rbergeron> dmsimard: we can probably chat it out back in #zuul to figure out the right spot
22:52:09 * fungi got sucked into the visiting family black hole, as predicted, but will catch up from the meeting logs later
22:52:12 <SpamapS> for things that aren't blockers to that, I'd say maybe we need a zuulv3.1 ? :-P
22:52:16 <jeblair> fungi: o/
22:52:28 * rbergeron has a short meeting item to squeeze in also
22:52:34 <jeblair> SpamapS: ++
22:52:43 <jeblair> #topic status update emails
22:53:15 <rbergeron> so: it's been suggested that maybe having a general status update mail to keep folks apprised and reminded of the fact that things are moving would be useful.
22:53:50 <rbergeron> not sure if there's enough for a weekly thing, or maybe every-other week. i am happy to ... make a story and draft up approximately what it would look like in storyboard
22:54:02 <rbergeron> or ... just send something out after this meeting and let people be like wtf :)
22:54:24 <rbergeron> (i kinda would mneed to know where to send it to. btw, i'm volunteering since this is stuff i can do without driving people insane)
22:54:32 <rbergeron> (if people think it's a good idea)
22:54:40 <jeblair> at least, i answer enough questions like that that i was about to start doing it when rbergeron volunteered :)
22:54:46 <jeblair> so i'm in favor :)
22:54:58 * mordred feels like it's all he does - so also in favor
22:55:34 <SpamapS> yesplease
22:55:38 <rbergeron> partially related to that: at some point i htink it would be useful to figure out the "pretend we had a date that we might go into production, what are the milestones walking backwards from that date that people actually need to pay attention to or where we need to seriously do loud messaging about wtf is going on"
22:55:42 <jeblair> i think the -infra and -dev mailing lists would both be good?
22:56:04 <rbergeron> kk. i will stick a drafty-type thing in an etherpad for this first one just to let y'all eyeball it, and then i'll ship it out. and then i'll just ship.
22:56:05 <SpamapS> we're better than we should be at pretending ;)
22:56:13 <rbergeron> weekly or biweekly? anyone have a preference?
22:56:45 <jeblair> maybe biweekly for now, escalating as needed when important milestone approach?
22:56:51 <rbergeron> #action rbergeron to draft up zuul v3 status mail for ppl to poke at, then send. and then just be shippin' in weeks afterwards.
22:57:11 <rbergeron> jeblair: i think that's appropriate. though i think this one can go out now since woot! all the magic updates
22:57:35 * rbergeron notes it would be useful if mordred is working on a blog post to maybe have that blog post to point to as well but probably not a blocker (?)
22:57:44 <jeblair> rbergeron: ya
22:57:49 <mordred> blog post is going well
22:58:11 <mordred> _someone_ gave some good feedback that needs incorportating
22:58:13 <jeblair> rbergeron: also, that will probably take care of fungi's need to report on the ptg.  :)
22:58:14 <rbergeron> #info mail to be bi-weeklyish, escalating as Zuul v3 Arrrrrrives.
22:58:20 <rbergeron> jeblair: exxxxxxactly
22:58:32 <rbergeron> hooray, i can do something helpful! woot
22:58:38 <rbergeron> okay, that's all from me :D
22:58:41 * mordred hands rbergeron a pie
22:58:46 <jeblair> #topic pie
22:58:55 <SpamapS> Speaking of pie
22:59:03 <pabelanger> most welcome
22:59:04 <rbergeron> it's delicious.
22:59:20 <SpamapS> Starting to smell like Zuul's baking a pie of its own.. I wonder how long we'll stay under infra's umbrella.
22:59:38 <SpamapS> certainly through v3
22:59:55 <mordred> yah. that was a topic that came up a few times around the ptg it seems - all I know is that I like both pie and not getting rained upon
23:00:04 <SpamapS> same
23:00:09 <SpamapS> TIME? ;)
23:00:17 <jeblair> #endmeeting