22:07:51 #startmeeting zuul 22:07:52 Meeting started Mon Nov 20 22:07:51 2017 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:07:53 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:07:56 The meeting name has been set to 'zuul' 22:08:12 The meeting should be stored in UTC, and then display is adjusted by local display zone info 22:08:15 #link agenda: https://wiki.openstack.org/wiki/Meetings/Zuul 22:08:22 but people fail at thinking like that, so some events are stored actually in the local timezone, so that the "time" doesn't change when the offset does 22:08:24 #link last meeting: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-11-13-22.10.log.html 22:08:39 Thanks to Shrews for chairing the last meeting! 22:08:46 it was nice and short, so i just linked to the transcript 22:08:57 i found it helpful, even if short 22:09:36 i will only add that the infra+zuul team did not have rotten vegetables thrown at us so i consider it a success (in fact, many nice things were said about v3) 22:09:51 it=summit 22:10:04 yay 22:10:11 yes, none of the vegetable they threw were rotten 22:10:19 a lot of excitement around zuulv3 at summit 22:10:20 woo 22:10:41 #topic Add support for shared ansible_host in inventory (pabelanger) 22:11:18 so, this is something I found when trying to convert a zuulv2 job to native zuulv3 22:11:32 it is more an optimization I think on CI resources and hopefully something we want to consider 22:11:38 #link https://review.openstack.org/521324/ 22:12:10 right now, if I use our current nodeset stanza, I'd have to request 6 nodes from nodepool, when in fact, the way the playbooks are written, I really only need 1. 22:12:20 pabelanger: i take it the zuulv2 job was running ansible within the job, and you're trying to move that ansible up to top-level zuulv3 internal ansible? 22:12:29 also, host groups doesn't work in this case, because of the way variable scoping is handled 22:12:36 jeblair: correct 22:13:12 I'm not sure I'm following, have a bad case of mondays 22:13:28 pabelanger: cool. so this isn't a regression from v2, more of an impedence mismatch with zuulv3 internal ansible and native ansible. which is cool -- we want to make it as transparant as possible. 22:13:30 * dmsimard reads review 22:13:39 if there are differences in how variable scoping works, I could see that being something other folks would run in to should they attempt to do what pabelanger is trying to do too 22:13:49 yah - what jeblair said 22:13:53 pabelanger: oh, different hosts which lead back to the same host 22:14:07 jeblair: Yup, in fact, requesting the 5 nodes from nodepool work fine. I just didn't want to land it and eat up a bunch of nodes for each patch 22:14:31 dmsimard: right 22:14:32 "I want 3 different ansible hosts, but I only need one node" 22:14:34 wouldn't you put the single node in different groups with zuulv3 22:14:39 then have your playbooks operate on the various groups? 22:14:46 but one node could be in say 6 groups 22:14:52 clarkb: yah - that apparently behaves differently in some ways 22:14:57 clarkb: yes and no, you can do both 22:15:11 clarkb: (that was my first suggestion as well) 22:15:15 pabelanger: will it run ~5x as fast if scheduled across 5 nodes? if so, the larger node size doesn't sound too terrible 22:15:16 clarkb: yes, that is possible but it would require a rewrite in my case 22:15:28 clarkb: like, technically, there's nothing that prevents "keystone.domain.tld" "nova.domain.tld" to ultimately resolve to the same IP address while also being different "hosts" in ansible 22:15:56 the problem here is that we use IP addresses, not hostnames 22:16:10 pabelanger: what's the deficiency with using groups? 22:16:30 i know "something about var scoping" but is there something more specific i can reference? 22:16:30 pabelanger: I think we might break some assumptions in roles if we do this, especially multinode roles 22:16:46 jeblair: play host filtering is an example 22:17:03 so if we want to support the story, as much as possible, of "run your existing ansible as part of your testing" - then if there is a semantic distinction between 2 hosts with the same IP and a single host in two groups, then I think we need to allow a user to express which they want 22:17:09 jeblair: it is likely better is I work up simple playbook example, because I'm likley not going to explain it very well. 22:17:27 jeblair: if you want to do an "all in one" but your playbooks/roles are built to target different hosts 22:17:27 dmsimard: isn't that an anti-pattern? (ie, shouldn't you filter plays based on groups or roles anyway?) 22:17:30 (also, yah, having a little clarity on the things that are different between the two would likely be helpful for all of our deeper understanding) 22:17:43 I have a (bad) example, one sec 22:18:06 https://github.com/rdo-infra/rdo-container-registry/blob/master/hosts 22:18:32 it so happens that I'm installing everything on the same host, but the playbooks are made to target specific groups to install specific things 22:18:45 pabelanger: ^ does that make sense ? 22:19:35 dmsimard: iiuc, that case should be handled currently with our group support (aside from the openshift_node_labels, but that's only because we don't do any host-specific inventory vars right now. we could, but that's a different change) 22:19:40 dmsimard: right but we have different groups ability so your think shoudl work right? 22:19:42 and then there's an example of var scoping in that example, if you look at the nodes group 22:19:54 right, but more specifically ansible_host seems to create a new SSH connection (which resets variable scope) where using groups doesn't. It will just run everything from start to finish using 1 connection. Based on my local testing 22:20:19 pabelanger: yah - that's, I think, the most important distiction 22:20:21 pabelanger: that strangely rings me a bell 22:20:29 pabelanger: multiple ssh connections make sense, how that's connected to variable scoping i don't understand 22:20:32 but, I'll setup a simple playbook / inventory demonstrate the issue 22:20:47 pabelanger: ++ 22:21:06 jeblair: hostvars can arguably be workaround by supplying a host_vars directory so it's not a big deal I think 22:21:15 (or group_vars) 22:21:28 I don't think we need to support providing them in the nodesets (unless we really want to) 22:21:53 dmsimard: either way, based on my current understanding, i think it's orthogonal to this question, so we can set it aside for now 22:22:00 yah 22:22:01 * dmsimard nods 22:22:18 a reproducer would indeed help 22:22:35 okay, will get that done for tomorrow and we can discuss more 22:23:14 okay, my personal summary here is: i'm not opposed to this on principle, but before we proceed, i'd like to understand it a bit more; pabelanger will help by suppling more examples and details. and if we do need to proceed, we should evaluate dmsimard's concern about assumptions in multinode jobs. 22:23:31 that jive for folks? 22:23:33 ++ 22:23:35 I agree 22:23:38 from a philosophical point of view, I'd prefer to minimize the number of times we have to say to someone "to use your existing ansible in zuul, you need to rewrite it" 22:23:39 yeah. 22:23:47 there will be some cases in which that is unavoidable, of course 22:24:15 sgtm 22:24:19 mordred: ++ 22:24:23 yah, this was the closes way I could reproduce an existing inventory file I was testing with in v2 22:24:26 #agreed pabelanger will help by suppling more examples and details before we proceed with this. if we do need to proceed, we should evaluate dmsimard's concern about assumptions in multinode jobs. 22:25:31 i -1d the change with a quick summary too, so we don't lose it 22:25:40 ack 22:25:50 #topic Allow run to be list of playbooks (pabelanger) 22:26:27 So, this was actually the first way I solved above, but kept it alive because it might be usefully. Having a job run multiple playbooks 22:26:31 this seems to touch on similar issues... 22:26:33 ah :) 22:26:36 yup 22:26:38 that was my question 22:26:41 #link https://review.openstack.org/519596 22:26:53 so this is a semi-alternate to the other change 22:27:25 Yah, gave the option to run back to back playbooks with specific hosts 22:27:34 kinda like we do on puppetmaster today with ansible 22:27:51 so, not sure if we want to consider supporting it or leave until later 22:27:55 considering it is something we already do elsewhere it seems to make sense as a feature 22:28:51 actually, why do we do that on puppetmaster? 22:29:35 I know we wrap each ansible-playbook with timeout, did we break it out due to memory issues? 22:30:12 is it something about parallelism? or exiting on failure? 22:30:26 jeblair: I think it's acutlaly just historical 22:30:53 we had a run_puppet.sh - and we started using ansible to run it by modifying that script one bit at a time 22:31:39 jeblair: mordred the big reason for it today is decoupling infracloud from everything else 22:31:42 pabelanger: how would this have solved your problem? even if zuul ran multiple playbooks in sequence, it would still have the same vars? 22:31:54 because infracloud is more likely to fail and adds a significant time to the round robin 22:31:59 jeblair: vars set by tasks in the plays get reset across playbook invocations 22:32:03 clarkb: right, but those just operate completely in parallel, right? 22:32:12 jeblair: yes 22:32:14 mordred: oh i see 22:32:20 so ya I guess in the context of a job you'd just have two jobs for that 22:32:23 yah 22:32:31 jeblair: it would be same vars, but multiple ssh connection attempts. That seems to be the key to resetting them to how I expect them in the playbooks 22:32:53 again, I think a more detailed example playbook might help here 22:33:01 and happy to write up 22:33:05 i'm still questioning the connection between ssh connections and variables 22:33:16 i'm pretty sure those are two independent concepts 22:33:57 these are group_vars if that helps 22:34:21 does zuul set group vars? 22:34:50 I don't think so 22:34:54 it doesn't, ansible is loading them based on the playbooks/group_vars folder 22:35:11 there's either inventory-wide vars or extra-vars which both apply to everything 22:35:29 pabelanger: so you're getting different variables because you're running playbooks in different paths? 22:36:22 jeblair: I get different variables if I switch to groups in my inventory file 22:36:32 well 22:36:38 groups of groups 22:36:56 I haven't tested group_vars and host_vars, it'd be interesting to test actually.. typically you'd have host_vars and group_vars inside {{ playbook_dir }}, but in our case those paths aren't "merged". However, I believe you can set group_vars and host_vars inside roles, and that would be more likely to work. 22:37:11 pabelanger: er, i'm trying to understand how this change solves your variable problem from earlier 22:37:13 for exmaple: http://git.openstack.org/cgit/openstack/windmill/tree/playbooks/inventory is my working v2 inventory file 22:38:00 jeblair: bascially, it allows me to stop doing http://git.openstack.org/cgit/openstack/windmill/tree/playbooks/site.yaml 22:38:17 and create a run: statement for each playbook 22:38:28 pabelanger: why is that preferable? 22:38:41 pabelanger: don't you feel like you're moving too much logic *into* zuul? 22:38:49 seems equivalent to me as well 22:39:25 FWIW that's exactly what we're doing with the base and multinode integration jobs, we're running "one" playbook that includes multiple playbooks 22:39:33 I don't currently see that as a hindrance 22:39:35 jeblair: yes, this is a work around, because I created 521324, which I'd much rather have 22:39:42 before* 22:41:14 I'll have to run here in about 5 minutes, but don't want to leave people hanging. I'm happy if we want to contiune this topic in #zuul too 22:41:39 pabelanger: to be clear, i'm, again, not permanently opposed to 519596, but before we merge changes like that, i'd like to have a really clear idea of why they are necessary, or what problem they solve, or what situation they improve. so far we've got the "include list" as one thing, but that seems like an anti-pattern and a mild argument against merging 519596 22:41:47 if there's a variable aspect to this, i still don't understand it 22:42:35 sure, I'll get some working examples that better show the issues I ran into converting v2 job to native v3. These were both my attempts to address some issues I was having 22:42:44 pabelanger: okay, thanks 22:43:06 #agreed pabelanger will provide more examples and explanation for this 22:43:25 I have to run now, will catch up on minutes when I return 22:43:31 #topic open discussion 22:44:11 I have a topic... 22:44:18 For open discussion, I just wanted to point out that we formally started looking at what it means to run a Zuul v3 that is not the one in OpenStack 22:44:18 I too have one, but go for it jlk 22:44:35 jlk won first :P 22:44:38 dmsimard: who's "we" ? 22:44:45 IIRC there is one at BMW is there not? 22:45:06 We is RDO Infra (analogous to openstack-infra) and Software Factory developers 22:45:08 there are a few in fact 22:45:10 (and for a hot minute, there was Bonny. Sigh.) 22:45:23 dmsimard: neat! 22:45:43 Software Factory had arguably been running Zuul v3 for a while 22:46:25 But there's some interesting questions and design challenges in thinking how we want to share configuration between zuuls (zuul-jobs, openstack-zuul-jobs, project-config, and specific project) 22:47:09 i think SpamapS was also trying out zuul-jobs sharing 22:47:28 jamielennox had some thoughts in this space as well 22:47:43 I started a thread about it in the context of TripleO http://lists.openstack.org/pipermail/openstack-dev/2017-November/124733.html and we also started hunting down issues we come across in zuul-jobs here: https://etherpad.openstack.org/p/downstream-zuul-jobs-issues 22:47:46 we probably want to focus on sharing zuul-jobs first and not the others right (they aren't generally supposed to be reconsumable so figuring it out for zuul-jobs where it is is a good start) 22:47:48 sharing between instances is definitely a design goal for zuul-jobs. 22:48:34 clarkb: it's funny that you mention that, because one of the ideas that has been floating around is to centralize the playbooks/roles/jobs/etc for TripleO in tripleo-ci and then use that across all Zuuls 22:48:36 i think openstack-zuul-jobs and individual openstack projects may be useful for openstack third-party-ci, but that's less of an explicit goal, and i think large amounts of 'shadow' and 'include'/'exclude' may be needed. 22:48:41 yes. I could also imagine that in-repo jobs and possibly openstack-zuul-jobs might be things that OpenStack Third Party Zuuls will want to consume 22:48:50 jeblair: yes, in the context of third party CI and such. 22:49:02 jeblair: yup 22:49:37 project-config definitely isn't meant to be shareable -- *however* -- we do want to have at least a stub/example base job. that should end up either in zuul-jobs or zuul-base-jobs at some point soon. 22:49:44 ++ 22:49:51 starting by figuring out sharing of zuul-jobs and getting it right will go a long way 22:50:17 ++ 22:50:32 mordred: ya thats what I'm thinking. Those are the bits that should be reconsumbale so lets start there and learn what we learn 22:50:46 dmsimard: so thanks for diving in and thanks in advance for your patience :) 22:51:19 (cause we just *might* have gotten some things wrong in the first pass) 22:51:25 no stress 22:51:26 last week we merged a couple changes that broke Zuuls config and zuul didn't catch them upfront. The one I remember off the top of my head is parenting to a final job. I know that pabelanger ran into something else when he had to restart zuul due to OOM as well. 22:51:31 ^^ is my item 22:51:53 it would be good if we could address those config issues pre merge 22:51:53 a couple? 22:51:56 clarkb: by broke zuul's config... what do you mean? 22:52:01 how did more than one breaking change merge ? 22:52:08 jlk: was thatyour topic? or did we talk over you? 22:52:10 how did *one* breaking change merge? 22:52:11 jeblair: in pabelanger's restart zuul case zuul would not start up again 22:52:18 mordred: I can wait through clarkb's :) 22:52:37 is there a bug report? 22:52:41 jeblair: in the job parented to final job zuul kept running but the new jobs that merged would not run 22:52:50 jeblair: I believe both were added to the zuulv3 issues etherpad 22:53:14 clarkb: the second sounds like an expected run-time error; the first sounds like a bug. 22:53:30 ok we can followup on them after meeting 22:53:35 * clarkb gets out of jlk's way 22:53:35 ack 22:53:51 * dmsimard has an easy item after jlk 22:54:04 * jeblair has a quick item after dmsimard 22:54:21 hi there! So tristanC landed a nodepool driver for k8s, and I'm asking if the group collective has the appetite to discuss/debate k8s (and container in general) driver approaches for Zuul/Nodepool. 22:54:30 "is now the time" or should we table this for later? 22:54:43 and by now, I don't mean in this channel now, but on list and in #zuul and whatnot 22:55:07 s/landed/proposed/ 22:55:14 sorry that's what I meant :( 22:55:16 silly fingers 22:55:37 jlk: yah - just wanted to make sure nobody else was confused :) 22:55:38 it's worth discussing not just because it's important but because it'll be a precedent from which the design of other "drivers" will be built from IMO 22:55:57 jlk: I was actually just writing an email to the list on this meta topic actually 22:55:59 i'd really like to get a v3 release out before we dive into this, because i think it will help get and retain other folks into our community. also, it's just embarrasing not to release. 22:56:25 +1 22:56:30 considering the lack of such features isn't a regression I'm on board with that too 22:56:45 * mordred agrees with jeblair - but does think there is at least one facet of the discussion that is worth having pre-release to make sure we don't back ourselves into a corner when we release 22:56:49 same, a release would be good, so long as we don't have to wait for zuul4 to add container support 22:56:51 my preferred approach would be to get the release out the door quickly, then start on the next dev cycle. there are things on the roadmap for release that others can pitch in on (some of them have my name next to them, but they don't have to be me and that would speed things up) 22:57:38 ok mordred you will start a topic on the ML ? 22:57:44 i would be really surprised if we are unable to release by februrary, and think earlier is likely (though i caution, there really aren't that many work weeks left in this year) 22:57:49 the thing I think is worth sanity-checking ourselves on pre-release is making sure we aren't doing anything thatwould fundamentally block the addition of a nodepool driver that produces build resources that do not use ssh 22:57:53 dmsimard: yes, I shall 22:58:02 let's pick up the discussion there 22:58:16 works for me, thanks! 22:58:56 this dovetails into my topic, which is in next week's meeting, i'd like to check in on the release roadmap items 22:58:56 There's only 1:30 left.. but Ansible 2.4.1 is out and addresses most/all of the regressions/issues introduced in 2.4. How do we upgrade without breaking people that have written <2.4 things ? 22:59:25 i'll add that to the angenda for next, and i expect we'll have the roadmap items in storyboard soon 22:59:33 Alternatively, how do we potentially allow different version of Ansible ? Because some things works in one, some in others, etc. 22:59:35 Ack 22:59:44 jeblair: ++ 23:00:11 I really hope that we don't get too deep in "what version of ansible" land for Zuul users. 23:00:13 at least not "choose your own" 23:00:19 dmsimard: most of the 2.3 to 2.4 breakages were in the python layer for us, right? 23:00:32 mordred: for ara and zuul themselves, perhaps 23:00:40 mordred: there's still some "bug fixes" that broke things 23:00:58 some behavior changes in includes/imports 23:01:05 also variable scopes 23:01:16 they don't really follow semver :( 23:01:27 jlk: I actually have some thoughts on how we might consider doing that without death - but as of right now I agree with that sentiment 23:01:31 jlk, dmsimard: some folks have expressed a use case for multiple ansible support, and it may be in keeping with our philosohpy of trying to be as transparent as possible with ansible. for instance, kolla tests all their stuff with all current supported versions of ansible. 23:02:11 sure, but I thought at that point, you write a first job that installs the ansible you want to test with, then.... 23:02:12 this is a cool topic which I'll continue in #zuul :) 23:02:13 dmsimard: I'm not sure there is a great story yet today for how to upgrade zuul's ansible from 2.3 to 2.4 and verify that people's job ansible doesn't break 23:02:18 so it may not be out of the question for zuul to maintain compatibility with current supported versions. but i think that needs a design proposal. :) 23:02:26 i think the quick answer to dmsimard's question is -- 23:02:36 we try using 2.4.1 and fix what's broken :) 23:02:39 yah 23:02:43 yuck 23:02:53 anyway, we're over time :) 23:03:01 Shrews has done much work on that already, we should be in pretty good shape 23:03:10 thanks all! 23:03:13 #endmeeting