19:00:10 #startmeeting infra 19:00:11 Meeting started Tue Oct 24 19:00:10 2017 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:14 The meeting name has been set to 'infra' 19:00:20 o/ 19:00:21 o/ 19:00:28 hello infraers 19:00:35 helo 19:00:36 hi! 19:00:51 hi 19:01:02 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:11 #topic Announcements 19:01:27 #info Summit in two weeks 19:01:48 seems like we just had one 19:02:06 This is a general fyi that many of us will be traveling to the summit in Sydney soon. Expect things to slow down starting mid next week 19:02:14 #info You probably need a Visa if you don't have one yet 19:02:25 yeah, i have to start my journey a week from tomorrow 19:02:48 or an "electronic travel authorization" if you don't need a "visa" 19:02:53 right that 19:03:17 so make sure you've got one 19:03:36 #info No meeting November 7 19:04:04 I'm just going to go ahead and call it on ^ 19:04:10 ++ 19:04:34 but I will be around to host the meeting next week 19:04:55 #topic Actions from last meeting 19:05:02 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-17-19.00.txt Minutes from last meeting 19:05:16 fungi did the backup policy on secrets get a doc patch yet? 19:05:31 something i was probably supposed to do there, so... no not yet 19:05:35 #action fungi Document secrets backup policy 19:05:43 * Shrews arrives late 19:05:53 ianw: I don't think I've seen backups for the actual secret keys yet either? 19:06:00 bah no, will do 19:06:01 oh, right, something about making sure it says not to treat us as a backup of your secrets 19:06:15 #action ianw Work to backup zuulv3 secret's keys 19:06:17 fungi: ya that 19:06:34 (fwiw I didn't expect those things had been done as so much effort is still in sorting out bugs/jobfixes/etc) 19:06:37 we can't provide them to you if you lose them, but we'll back up our private keys for our own disaster recover of the ci system 19:06:43 will do 19:07:00 #topic Priority Efforts 19:07:06 #topic Zuul v3 19:07:31 jeblair had an item about the PTI and job variants here 19:07:40 Are we ready to remove zuul v2 config from project-config? Then we need to merge first https://review.openstack.org/#/c/513910 and then recheck and merge https://review.openstack.org/#/c/507180/ 19:07:46 oh hi 19:07:52 We first want to tag, don't we? 19:07:58 AJaeger: yeah, i think we're ready 19:08:13 we can always retroactively tag 19:08:27 * AJaeger removes the WIP from 513910... 19:08:56 (anyone want to suggest a commit to tag? anyone want to actually perform the tag?) 19:08:59 jeblair: go ahead, sorry for sneaking in front... 19:09:04 I can make the tag 19:09:06 (i don't care enough to suggest a commit :) 19:09:27 jeblair: probably commit before 513910? 19:09:38 clarkb: wfm 19:09:44 or we could even tag 513910 since it doesn't remove the jobs yet 19:10:11 so the pti thing -- i think the migration doc is pretty clear on "don't put python-jobs template in your repo, put it in project-config" 19:10:13 after the meeting I'll pull up a git log and propose a commit to tag but I'm guessing something like master or master~1 19:10:24 i'm happy to `git tag -s farewell-jenkins ...` the immediate merge commit parent of whatever change removes our v2 configs 19:10:40 as noted, doesn't have to happen pre-merge 19:10:59 but projects may need/want to create project-pipeline variants of the jobs in those templates 19:11:48 (like, use a trusty on this old branch, or adding files or irrelevant files) 19:12:00 or adding required projects like with neutron jobs 19:12:03 ya 19:12:10 generally speaking, those could go either place -- project-config or in-repo. 19:12:17 with one notable exception 19:12:35 jeblair: I'd like to have a clear policy on what is ok: Do we reject variants in project-config? Don't care? Ask to have them in project-config? 19:13:04 or don't run any py27 jobs (the template adding that also adds pep8, which makes that more complex) 19:13:07 once the project template is applied in project-config, you can't elect not to run a job on a branch, you have to drop the template and add the job specifically for the branch 19:13:13 yah, that :) 19:13:24 it's probably worth treating these as two separate cases 19:13:44 since not running a pti job on a branch is, strictly speaking, a policy violation 19:14:02 even though, in the case which brought it to our attention (nodepool), it's a very forward-looking one :) 19:14:44 anyway, that case requires a project-config change anyway, so it's not going to slip by us. i think we can set it aside for the moment and focus on the other case -- variants that add projects or file matchers, etc 19:15:04 oh you know what, file matchers may not work either for the same reason 19:15:19 branch matchers do not 19:15:19 jeblair: I think that may be what the release job is running into 19:15:21 (i bet that's what's going on with the thing i'm supposed to look at in the etherpad) 19:15:23 to reduce our work, we could say: templates have to stay in project-config, any variants go in-repo ;) 19:15:56 AJaeger: i'm inclined to favor that idea too 19:16:35 if we would go down this road, would we -2 any variant and ask to add to in-repo? 19:16:46 is this something that is considered a bug or just behavior we have to live with? 19:17:02 asking beacuse if it is just something that needs to be fixed do we want to move back to project-config later? 19:17:10 clarkb: this is largely a policy question 19:17:28 Similar, do we allow new legacy jobs like https://review.openstack.org/514768? I'm inclined to -2 this change that adds a legacy tempest troveclient job 19:18:10 clarkb: ultimately, i'm inclined to say that we should trust projects to manage their own variants 19:18:35 reviewing those isn't really worth our time, and it's desirable that they be in-repo. 19:18:38 jeblair: would we still force the variant to be run via project-config? 19:18:46 jeblair: then I would love to have a check that we do not run non-voting jobs in gate or voting jobs only in check ;) 19:18:54 there are rules about defining a job in one repo and attaching it to another repo right? 19:19:00 slightly concerned we'll end up with teh variant of "run no jobs" if not 19:19:21 dhellmann: yes. one of the rules is you can totally do that. 19:19:40 so I can define a job in releases and attach it to nova without their consent? 19:19:45 clarkb: yeah, having the project-template attached to the project in project-config will cause the job to always run. 19:20:00 jeblair: gotcha so even if the variant is defined elsewhere we'll force it to run 19:20:10 dhellmann: nope. another rule is you can only attach it to your own project. only project-config is exempt from that rule. 19:20:14 in that case ya I agree that allowing projects to define their own variants seems to be simplest 19:20:34 jeblair : ok, that's what I thought, thanks 19:20:42 dhellmann: you could define a template in release and if nova uses it, any changes in release repo, would be used... 19:21:01 dhellmann: but then nova could remove the template usage 19:21:17 AJaeger : yeah, that'll work fine. I was hoping to put the list of which projects use which release jobs in one place. it sounds like that place is still project-config, even if we move the job definitions elsewhere. 19:21:32 dhellmann: yes 19:21:45 so i think where this is most likely to break down is in the competing interest between having the template in project-config saying "always run pep8" and a project wanting to say "don't run pep8 on these files". 19:22:02 basically, as soon as any variant matches a change, the job is going to run. you can't undo that 19:22:31 jeblair: would the main job with no exclusions match too? 19:22:35 this is both in the case of the 'don't run on this branch' example from earlier, but now that i think about it, the more mundane 'don't run on this file'. 19:22:48 so the job settings aren't applied in "layers"? 19:23:05 dhellmann: they are, but once a job is selected to run, it can't be unselected 19:23:14 jeblair: so, to not run a job, I just add a "files: this-file-does-not-exist" ?;) 19:23:22 dhellmann: cause basically, that's just not applying extra layers 19:23:37 let me rephrase 19:23:56 jeblair : I guess you're saying the settings in the project tree don't override the settings in project-config 19:24:07 using a first-match algorithm on the decision to run a job 19:24:12 they may extend them to add jobs, but not change the rules for selecting jobs to run 19:25:01 the openstack-python-jobs template in project-config is attached to all projects. if a change matches any of the matchers are on *that template*, the jobs will run, regardless of whether a project adds their own variant to their own pipeline. 19:25:14 * dhellmann nods 19:25:26 dhellmann: yeah. they can even alter the way the job is run (ie, increase timeout, add projects, change nodes). the only thing they can't do is cause it not to run. 19:25:42 ya that is why releases py35 job runs even though there is a variant that specifies different irrelevant files 19:25:53 I guess that's not surprising and is probably a feature. 19:25:55 jeblair: so, irrelevant-files/files cannot be added in a variant? 19:26:05 AJaeger: only in an additive capacity 19:26:16 so adding files would work fine 19:26:21 files would work but irrelevant-files wouldn't? 19:26:31 you can use them to say "also run this job on this file". or, "extend the timeout if the change touches this file" 19:26:40 but not "don't run this, even though another rule says to" 19:26:48 dhellmann: right 19:26:53 got it 19:26:57 if you wanted to note additional files which should also cause the job to run (assuming the job was previously set to run only under more limited circumstances)? 19:27:08 fungi: yep 19:27:14 jeblair: thanks for this discussion, this is helpful! 19:27:35 so, tl;dr, there's no way to make the py35 job not run for data files that don't change the test results? 19:27:41 so i think if our policy is "you have to have the pti template attached to your project in project-config" there's some implicit policy around when those jobs run. 19:27:51 dhellmann: you could modify the base py35 job 19:27:56 dhellmann: i think there are 2 things we can do (while continuing this policy) 19:27:56 dhellmann: but otherwise I dn't think so 19:28:04 * clarkb lets jeblair explain better 19:28:09 we can try to have a good set of matchers on that project-template 19:28:14 and i think we actuall do that now? 19:28:16 I wonder why the releases repo falls under the pti we use for product repos. 19:28:24 jeblair: ya there are already some irrelevant files on the project-template 19:28:29 ie, i think the project template does have irrelevant-files 19:28:39 copied from the nova/neutron copypasta from v2 i think 19:28:55 so we can continue to manage additions to that and try to find a happy medium for everyone 19:28:55 dhellmann: currently the pti is getting applied to every deliverable of an official team. we may want to decide at the tc level that the pti is only applicable to some subset? 19:29:21 or we could adopt a policy that lets folks remove the project-template from their project in project-config, perhaps only if they also add back the individual jobs but with the matchers they want. 19:29:30 fungi : Yeah, I think that's a conversation worth having. Not just for this case, which it seems we can solve regardless. 19:29:45 or, rather, we at the governance level have done a poor job of defining the extent to which deliverables have to conform to the pti 19:29:58 jeblair: that is what nodepool has done 19:30:07 and so project-config-core reviewers are taking the safest and most literal interpretation 19:30:12 fungi: yeah, I've noticed several times that different people have a different definition of "all repos" :-) 19:30:14 though i expect, once we merge feature back to master, we can re-apply the template 19:30:51 also, to be fair, the pti doesn't say 'you have to have a zuul project-template'. it just says 'you have to run these jobs'. it's probably fair for releases to still follow the pti, and for us to adapt our thinking on how that happens. 19:31:21 (and more importantly, if you run these jobs, they run this way) 19:31:36 does the infra team *want* to enforce the PTI for all projects? or would you prefer that happens some other way? 19:31:59 jeblair: in fact this came up for swift not too far back. their unit tests include tests requiring some special local filesystems provided which results in them needing to run an alternate job for them. the decision at the governance level was that the sort of deviation they required was acceptable 19:32:09 i'm not sure i'd characterize our desire as that to 'enforce' as much as 'facilitate'. 19:32:15 sure, ok 19:32:35 We're not enforcing all PTI templates on jobs - I'm not -2 a change that removes python-jobs or a repo creation without python-jobs... 19:32:51 ya we've also given leeway in the past with eg noop jobs to start 19:32:59 dhellmann: basically we'll do our best to review what's tested based on guidance provided in project governance 19:33:27 but, definitely not the police 19:33:54 ok. so it sounds like we should clarify that, and independently the infra team wants to consider the approach to apply the results 19:34:13 so maybe the happy medium is: 1) use the template if you can. 2) if you want to change the matchers on the template to improve them and it would work for everyone, do that and we'll merge the change. 3) if that doesn't work for you, it's okay to drop the template in project-config, but then (a) add the jobs back in project-config (b) add the jobs back in your own repo. 19:34:23 (i don't know which of 3a or 3b we should suggest) 19:34:47 jeblair: does adding the jobs back in project-config outside of a template work? eg this behavior is specific to templates? 19:35:00 probably 3a vs 3b can depend on whether the project has (yet) started to do any in-repo config 19:35:04 if part of the point of being able to set the jobs up outside of project-config is to lower the infra team's review burden, it seems like (b) is the approach to go with 19:35:09 clarkb: yeah, and when you do that, you can add whatever (project-specific) matchers you want. 19:35:15 got it 19:35:38 fungi: ya that seems like a reasonable place to start 19:36:00 i agree 3b is preferred but if a project doesn't have or doesn't want in-repo configuration at that point then 3a is fine they just agree to put up with our review bottleneck 19:36:11 that seems fair 19:36:42 its not like that bottleneck is a regression and generally should be improved if >0 projects do it in repo 19:36:50 in most cases i expect teams to prefer 3b as a solution over 3a anyway 19:37:09 once they get comfortable with the notion 19:37:42 okay, i think i have a feeling for this, thanks. this has been really useful to discuss in a meeting setting :) i'll write it up and send it to the list 19:37:58 awesome, thanks jeblair! 19:38:00 thanks, jeblair ! 19:38:08 i agree, very helpful 19:38:27 One related question: Do we allow new legacy jobs like https://review.openstack.org/514768? I'm inclined to -2 this change that adds a legacy tempest troveclient job. 19:38:50 AJaeger: I'm inclined to allow it if only because it is easy to copy what saharaclient did and this appeas to be a bug in the migration itself 19:38:53 Or do we handle this like 3a vs 3b - if in-repo exists... 19:39:02 i'm maybe inclined to let that slide for a few weeks. maybe a month or so. 19:39:45 So, encourage them to go in-repo but let them make their own decision for some time - ok, let's try... 19:39:46 just because the v3 native replacements for complicated jobs are only now becoming available, and we're impinging on peoples' schedules enough as it is :) 19:40:15 maybe stop doing that after sydney 19:40:27 and also that allows teams comfortable with writing v3-native jobs to lead the way, and provide a larger body of examples and experience for other teams to learn from 19:40:29 wfm 19:40:46 wfm 19:40:58 ok anything else zuul? we are running out of time and have a few other items on the list 19:41:18 onward 19:41:19 nothing from me 19:41:22 #topic General topics 19:41:32 #topic IRC highlight string for project-config cores 19:41:52 config-core has been proposed as a replacement for project-config-core to reduce the amount of typing necessary 19:42:11 i'm happy to match on the alternative there 19:42:29 whatever has some loose consensus is fine by me at any rate 19:42:35 ya probbly best thing is to add both sets of highlights and that way users can get away with teh short version or use the long one if already learned 19:42:47 that's what I did earlier today... 19:43:05 lets do that then. It is easy 19:43:49 #agreed Add both config-core and project-config-core highlights to irc clients 19:44:05 please disagree if there is a reason not to 19:44:17 config-core: i agree 19:44:28 perfect 19:44:32 hopefully that highlighted for you :) 19:44:33 thanks for the test, ianw ;) 19:44:39 #topic Remove old/EOL branches 19:44:44 tonyb: this one is yours 19:45:00 tonyb: did you make it? 19:45:09 Otherwise I can step in... 19:45:35 tonyb has proposed two topics for branch removal: stable/newton and old numeric ones 19:45:52 the agenda lists the two emails. 19:46:04 The numeric one got no feedback at all AFAIU. 19:46:26 AJaeger: the projects using the numberic branches are who you were looking for feedback from right? 19:46:27 I'd like to see the removal done to make our jobs easier - less to migrate 19:46:49 clarkb: yes, those where what tonyb looked for feedback. 19:47:15 No feedback can mean those projects are dead, haven't read it, whatever - but he only asked for disagreement and nothing happened 19:48:16 worst case we'll have tags for the branches and can restore them 19:48:23 I think it is probably fine to ask for forgiveness in this case 19:48:35 if they were important they'd be in a branch-override in project-config or something, which they're not afaics? 19:49:53 So, two questions basically: Shall we do it? And who volunteers? jhesketh_ did it the last few times AFAIR but he seems not be around 19:50:00 ++ to doing it 19:50:40 AJaeger: is this time sensitive as far as running the script? 19:50:41 i'm happy to work with tonyb on it, since we're timezone buddies 19:51:20 ianw: that would be great. I'm happy to help too so feel free to ping if I can help 19:51:25 clarkb: not really time sensitive - but we normally clean up jobs only after removal. And I would prefer to simplify jobs earlier instead of seeing them migrated in-repo... 19:51:37 AJaeger: good point so should be done soon 19:52:12 AJaeger: does that cover it? anything else you and tonyb want to go over on this topic? 19:52:27 this covers everything from my side 19:52:38 #topic Sydney team evening ideas 19:52:41 (but I expected tonyb to be here, so don't know whether it covers everything from his) 19:52:56 AJaeger: ok we can follow up with tonyb later in the day (when tonyb is awake) 19:53:05 #link https://ethercalc.openstack.org/lx7zv5denrb9 19:53:10 ianw has put together a thing ^ 19:53:23 if you are going to be in sydney and are interested in walking around and finding food and drink you should take a look 19:53:27 i realised the tuesday is melbourne cup day 19:53:43 ianw: does that mean everyone is going to be in the bars watching a sporting event? 19:54:12 the event is at like 3pm, but many will continue. i'm not sure if the conference has organised anything 19:54:19 it is "the race that stops a nation", as they say 19:54:37 sounds fun :) 19:54:58 but ya just wanted to amke sure people saw that ethercalc, there is an infra list thread. Please take a look if interested 19:55:08 #topic Project Renames 19:55:09 does "X" mean i am available? 19:55:14 #undo 19:55:15 Removing item from minutes: #topic Project Renames 19:55:21 jeblair: that is how I was treating it 19:55:24 jeblair: yes 19:55:38 #info X on ethercalc means you are available that night for team thing 19:55:41 cool 19:55:46 #topic Project Renames 19:56:09 These didn't happen beacuse we were focused on helping the release team. I think it is probably a good idea to get more settled into zuulv3 before we try to schedule this 19:56:24 ++ agree 19:56:29 #link https://etherpad.openstack.org/p/rename-2017-10-20 19:56:30 it also came up that we may have to modify our "playbook" for performing these due to zuulv3 19:56:36 a preliminary plan 19:56:48 Probably post sydney we can resync and go from there 19:56:54 but i think we decided the early zuul steps were wrong? 19:57:10 ianw: ya I think there were concerns about renaming things in a way that kept zuul happy 19:57:27 lets pick it back up after the summit 19:57:31 #topic Open Discussion 19:57:36 quick you have 1.5 minutes 19:58:12 I did something to one of my molars last Friday eating a brick that we called toast and have a dentist visit tomorrow morning (I was just there too...) 19:58:37 right, renaming projects without breaking zuul's configuraiton loading is the tricky bit we don't have answers to yet, i think (short of forcing the change in while zuul is offline) 19:59:01 changeS 19:59:01 ok, thank you everyone 19:59:06 see you next week then in sydney 19:59:08 #endmeeting