19:00:32 #startmeeting infra 19:00:32 Meeting started Tue Nov 28 19:00:32 2017 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:33 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:35 The meeting name has been set to 'infra' 19:00:44 hello, who is here for the infra meeting? 19:00:50 o/ 19:00:54 o/ 19:01:01 o/ 19:01:25 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:21 There are a couple items on the agenda. Jeblair will be joining us late so I think we will do zuulv3 after the general topics list 19:02:31 #topic Announcements 19:03:03 its been a quiet week with the US thanksgiving holiday, I'm not aware of anything that needs announcing 19:03:07 is there something I've missed? 19:03:28 i guess the venue for the ptg hasn't been officially announced yet 19:03:51 so maybe flag that to announce in next week's meeting if it has been by then 19:04:05 * clarkb scribbles a note 19:04:13 It should be announced by this afternoon. 19:04:35 in that case look to read the openstack-dev mailing list later today for an announcement on the ptg location :) 19:04:43 o/ 19:04:55 o/ 19:04:58 #topic Actions from last meeting 19:05:08 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-11-21-19.01.txt Minutes from last meeting 19:05:22 Fungi's action to write the secrets backups doc is complete \o/ 19:05:43 oh, that was complete last week 19:05:51 i think i linked it during the meeting then 19:06:06 ya it ended up in the actions list but now is done so we can keep it out 19:06:09 maybe it hadn't merged yet at that point 19:06:53 I'm not aware of any specs that need review or otherwise need to be brought up and will skip zuulv3 for now so that jeblair can join us which means straight ot general topics 19:07:00 #topic General topics 19:07:11 #topic Zanata 4 upgrade 19:07:22 #link https://review.openstack.org/#/c/506795/ initial change needed for zanata upgrade work 19:07:52 we are seeing some of the initial changes come in that will allow us to upgrade from zanata 3 to zanata 4. I think the i18n team would like to see this get done before the string freeze which is late january. 19:08:18 it would be great if we can help make that possible (code reviews, actually merging code/running upgrades) 19:08:35 o/ 19:08:58 I did the last one so am fairly familiar with the service, is anyone else interested in learning about the java/wildfly/zanata things? If so let me know and we can work to help get this going with the i18n team 19:09:12 i have a passing familiarity from the last upgrade, and i think we're in similar tz's, so put me down to help 19:09:27 ianw: awesome thanks 19:09:45 I expect it will be mostly straightfroward after talking to aeng, no need for a java update or distro upgrade 19:10:18 unlike, say, the next gerrit upgrade 19:10:26 i'll start by reviewing ^^ :) 19:10:43 fungi: +1 19:11:35 #topic Priority Efforts 19:11:44 jeblair: is here, on to zuul 19:11:50 #topic Zuul v3 19:12:17 I wanted to do a quick recap of the zuul cloner venv removal because there were some hiccups with it and want to amke sure we don't forget to do the last cleanups 19:12:32 pabelanger: ^ I think the fixes for the pyyaml install have gone in job side, have we remvoed it from the image again? 19:12:45 clarkb: we have! 19:13:11 I think we are ready to actually move to https://review.openstack.org/513506/ now, which removes zuul-cloner from base jobs 19:13:55 #link https://review.openstack.org/513506/ remove z-c shim from base job is ready for review now 19:14:30 for 513506, we'll need to be ready to fix jobs that are broken by it, and re-parent them to legacy-base 19:14:42 fungi: I think we basically agreed to remove jenkins from the ci group in gerrit last week as well. Did that happen yet? 19:14:51 as we expect native zuulv3 jobs 1) not to use zuul-cloner, 2) parent to base 19:14:52 it has not, but i can do it now 19:15:00 fungi: +1 19:15:08 10 seconds ;) 19:15:12 works for me and is an easy revert if we need to 19:16:00 okay, that was more than 10 seconds, but done now 19:16:11 frickler: tristanC have a third party CI of zuul-jobs agenda item 19:16:16 i had to remember the group name because i forgot i'd linked it from the agenda 19:16:25 I don't think tristanC is here, but frickler is. Want to fill us in? 19:16:41 I'm not directly related to the CI 19:16:47 somehow i missed that email 19:16:50 #info The retired "Jenkins" account has been removed from the Continuous Integration Tools group in Gerrit now 19:16:56 but i've read it now. and i support the concept in general 19:17:04 but I wanted to make sure that tristan gets more feedback on his mail 19:17:13 though i was about to send back a reply suggesting that we just use 'recheck' for the recheck command 19:17:25 #action fungi send an announcement about the removal of Jenkins from Continuous Integration Tools 19:17:32 i'll do that after the meeting 19:17:35 fungi: thanks 19:17:50 #link http://lists.openstack.org/pipermail/openstack-infra/2017-November/005688.html for details on third party CI for zuul-jobs 19:18:01 i've long advocated that third-party cis should not have their own recheck command language 19:18:48 i don't think it should be a problem for this repo 19:19:26 agreed. I think the other considering to make is whether or not we want it to +1, +/-1, -1 or just +0 with logs 19:19:31 *other consideration 19:20:01 i'm happy to try out voting if other folks are 19:20:51 I think voting helps get more eyeballs on the problem and not allowing voting makes it easier to ignore the failures. So I am happy to try voting as well 19:21:13 ++voting 19:21:20 hi 19:21:29 sorry, forgot to fix calendar event timezone.. 19:21:38 i have no objection to voting third-party ci systems as long as their feedback is reliable 19:21:40 yah, I think we can give voting a try 19:22:24 I don't have backlog to get context, but yes, we (RDO/Software Factory) would like to be third party CI against zuul-jobs. I don't know what shape this will take yet. 19:22:59 the same applies for me 19:23:05 dmsimard: current context is software-factory third-party ci: http://lists.openstack.org/pipermail/openstack-infra/2017-November/005688.html 19:23:10 dmsimard: basically jeblair has asked that we not have any special recheck syntax, just support 'recheck' like upstream zuul. And we seem to be comfortable to try it out in a voting manner (so we'll need to update gerrit ACLs) 19:23:13 which i somehow missed over thankgiving 19:23:43 Ultimately, one of the objectives is to leverage TripleO(-ci) jobs, roles, and playbooks from within review.rdoproject.org which is anologous to review.openstack.org, so it will be very important for us to be able to re-use zuul-jobs (and potentially other things, but that's another topic) outside of OpenStack 19:24:39 I'd start with non-voting first to get some confidence that we're doing the right thing 19:24:58 wfm 19:25:09 that's the usual approach i believe 19:25:11 just to pile on, i agree wrt standardizing on "recheck" across ci systems. rechecking individual ci systems is moderately dangerous for the same reasons we've resisted requests to recheck individual jobs 19:25:11 dmsimard: thats fine, just wnted to bring up possible voting early as a lot of projects don't allow it at all and wasn't sure were we stood on that 19:25:29 so as to avoid surprises later if there were major disagreements :) 19:25:53 clarkb: well, it's not clear to me yet what these third party jobs will look like yet 19:26:12 clarkb: i.e, will it be running base-integration/multinode-integration but just from another zuul for example ? 19:26:41 also, to be clear on the third-party ci situation, we've also said in the past that we'll disable accounts for any ci systems which start reporting on infra team repo changes without prior discussion 19:26:42 I feel like there's stuff we'll realize once we get started 19:26:49 dmsimard: it could be running your most important jobs 19:27:24 I would object to not allowing third-party CIs have their own recheck syntax, sometimes we want to only nudge our third-party CI when there is an obvious problem with it without wasting upstream CI's resources 19:27:33 my thought is not to be too prescriptive about what they're doing. we should have ongoing conversations with third-party ci operators to make sure we're making the most of things, but in general, third-party operators are probably best placed to decide what's important to them and what unique things they can bring to the table. 19:27:39 * rcarrillocruz waves 19:27:46 mmedvede: "zuul enqueue" via the rpc in that case 19:28:05 tobiash: it depends, what's the purpose or the use case ? I don't think running a tripleo-based job against zuul-jobs is necessarily worthwhile -- we're interested in testing the roles individually 19:28:13 fungi: I thought we are not encouraged to comment on the same patch twice without an explicit recheck 19:28:35 dmsimard: but ya sounds like you can go ahead and start trying things out in a non voting capacity, may even want to start in a non reporting manner first. See how that goes then tweak from there 19:28:40 mmedvede: that might be a policy some team has put into place, but it's not our policy afaik 19:28:49 mmedvede: there is nothing about that in https://docs.openstack.org/infra/system-config/third_party.html#requirements 19:28:55 dmsimard: yes, that probably depends on what's important to you as zuul-jobs user 19:29:01 mmedvede, fungi, jeblair: unless mistaken, the recheck keyword is controlled by the third party CI so there's nothing preventing operators from responding to "recheck" and "check myzuulname" 19:29:04 mmedvede: there is a note in there saying that "recheck" should retrigger all systems. 19:29:18 mmedvede: we're specifically talking about third-party ci systems which want to vote on changes to the openstack-infra/zuul-jobs repo (and potentially other deliverables of the infra team in the future) 19:29:29 dmsimard: nothing except their willingness to abide by the guidelines we've established 19:29:46 wait, I thought we said recheck foo was good a while back. I feel like we go back and forth on this every few months 19:29:47 fungi: indeed, let's not get too far derailed on this :) 19:29:56 jeblair: the important part is that they answer to "recheck", right ? if they answer to "check foo" is that an issue ? 19:29:57 pabelanger: i have never said that. 19:30:13 jeblair: other infro-root have, IIRC 19:30:36 dmsimard: we don't have an established policy on that. i would like to, in the context of zuul-jobs only, establish a policy that we don't do that and all systems honor recheck. 19:30:38 only. 19:31:12 jeblair: that's fine, on our end that means setting up a separate pipeline (because we already have a pipeline meant for third party) but that's not expensive 19:31:50 dmsimard: (or, if it happens to honor something else, just don't mention it) 19:31:56 sure 19:32:01 dmsimard: "our" in this context being rdo ci? 19:32:22 fungi: yeah, RDO's zuul answers to things like "check rdo experimental" (so we don't trigger "check experimental") 19:32:32 and possibly other things 19:33:06 jeblair dmsimard : agree recheck should absolutely retrigger all the CI systems. But this does not exclude ability for third-party CIs to also be triggered separately. I would like there to be an official blessed syntax for this. Right now each CI comes up with their own 19:33:22 i'm curious why someone would want upstream experimental pipeline results but not rdo's experimental pipeline results. still, that's not crucial to this topic 19:33:34 ya, I think we may be starting to get into another topic entirely 19:33:41 we can come back to that if there are no other zuulv3 items or finish them 19:34:05 I think we may want to talk about the merging of branches though that wasn't on the agenda. Any other zuulv3 items? 19:34:25 I have something 19:34:29 mmedvede: part of the resistance, historically, is that we feel leaving comments in the code review system is a bad api anyway, and would like to eventually have some other interface fro such tasks 19:35:00 and not tie ourselves to a standard involving arbitrary code review comment strings 19:35:51 dmsimard: what was your zuulv3 item? 19:36:06 I'd love at least a first round of reviews on the 'sqlite over http' ara middleware series to 1) always have ara reports regardless of failure/success 2) reduce even further the impact of storage/inode on the log server 19:36:17 The reviews are tagged here: https://review.openstack.org/#/q/topic:ara-sqlite-middleware 19:36:48 And you can see a practical implementation here -- 19:37:00 dmsimard: does that also depend on getting the ara install updated independent of the zuul install on the zuul executors? 19:37:04 Without the middleware: https://logs.rdoproject.org/33/10433/1/check/rdo-registry-integration/Ze74352b77e17444cace463fc9c994213/ara-database/ 19:37:06 With the middleware: http://logs-dev.rdoproject.org/33/10433/1/check/rdo-registry-integration/Ze74352b77e17444cace463fc9c994213/ara-database/ 19:37:33 SSL cert is bad ^ 19:37:42 pabelanger: yeah, logs-dev :( 19:37:45 (i replied to the ml thread with a summary of our discussion on the third-party ci issue) 19:37:55 pabelanger: I spun it up without getting proper certs (yet) 19:38:26 clarkb: it doesn't depend on the version of ara on the executors, no 19:38:48 dmsimard: cool, so we can work this independently. I'll make a note to review it 19:39:04 clarkb: it doesn't even depend on the version of ara on the logserver (where it would sit like htmlify/os-log_analyzer), it's just a wsgi script that happens to be bundled in ara at the latest version but otherwise can be carried in tree 19:39:25 Didn't we have a set of patchs to install it onto our own dev server? 19:39:32 logs-dev.o.o for example 19:39:38 that's the topic I linked earlier, yes: https://review.openstack.org/#/q/topic:ara-sqlite-middleware 19:39:46 okay cool 19:39:49 #link https://review.openstack.org/#/q/topic:ara-sqlite-middleware changes to run ara out of sqlite db using middleware. Will cut down on inode use on the logs server hopefully allowing us to add ara to successful jobs again 19:40:00 will look over here today 19:40:07 I have a todo to resolve a conflict between htmlify and ara rewrite rules but it's otherwise at least ready for reviewing 19:40:39 I -W one of the patches but it's still worth reviewing :)( 19:41:23 I'll probably go ahead and rebase the stack since it's been a while 19:41:26 that was it for my topic :) 19:41:45 #link http://lists.openstack.org/pipermail/openstack-infra/2017-November/005695.html ml thread on merging feature branches back into master on nodepool and zuul repos and shifting dev work to those branches 19:41:56 If you haven't seen it yet and are interested in zuul ^ is probably worth a read 19:42:01 jeblair: anything you want to add to ^ here? 19:42:34 ya 19:43:01 i'd love for someone from the third-party-ci community to jump in on the puppet-openstackci work 19:43:36 that is something that should be straightforward to accomplish and doesn't require any zuulv3 knowledge -- the opposite in fact -- it's work to keep zuulv2 working with puppet-openstackci 19:43:36 there's an irc channel where they hang out, worth a try to get their attention 19:43:58 true, though there's a problem if they aren't paying attention here. 19:43:59 mmedvede may also know individuals that might be interested? 19:44:00 they might not be subscribed to the MLs 19:44:32 we also have project-config-example repo - what should we do with that one? It uses Zuul v2/Jenkins right now 19:45:17 again, if they aren't subscribed to openstack-infra it's a problem. i will send an announcement to third-party-announce to draw attention to my post, however. 19:45:24 clarkb: it has been relatively quiet, lennyb fyi ^^ 19:45:41 jeblair: thanks 19:47:45 AJaeger: good question... i wonder whether it needs branching or can have v2 and v3 content side-by-side 19:47:49 ok any other zuulv3 items before we move on to open discussion? 19:48:01 that's in my opinion part of eth puppet-openstackci work to determine 19:48:17 fungi: and somebody would need to update it. Question is whether anybody is using it at all... 19:48:36 AJaeger, fungi: can likely support side-by-side as we did during our transition. 19:48:48 convenient 19:48:52 though should probably just switch to v3 soon. 19:48:52 clarkb: I'll take a look at puppet-openstackci for zuulv3 branch merge workarounds 19:49:02 mmedvede: thanks 19:49:10 jeblair: ^ sounds like you may have a volunteer? 19:49:14 \o/ 19:49:37 mmedvede: feel free to delegate/distribute the load to any other interested 3rd-party ci ops who show interest too 19:49:58 though hopefully the work involved is relatively minimal 19:50:06 this is my hope :) 19:50:12 ++ 19:50:20 but getting some of them to help test it out may make sense 19:50:34 ya I think having third party ci involved just for ^ is worthwhile 19:50:42 even if they aren't able to actively review the changes or write them 19:50:55 most of the work is going to be moving our zuulv3 stuff from system-config back into puppet-openstackci 19:51:38 pabelanger: ++ 19:53:16 #topic open discussion 19:54:03 As a general heads up with the firefighting largely behind us I'd like to start organizing the infra TODO list. Basicalyl something that shows new and old infra folk what work is happening and where they can help out if they have spare cycles 19:54:20 You'll probably see me ask for eyeballs on a storyboard board in the near future 19:54:52 * mordred is back to not being able to login to storyboard, fwiw 19:54:57 fun 19:54:59 I still need to send out ML post about xenial upgrades, I'll try to get that out later today 19:55:00 the Zuul v3 migration issue etherpad has still some items, could we all review it over the next days, please? 19:55:08 AJaeger: ++ 19:56:37 ++ 19:58:08 on an openfloor note, unbound reviews are up to try and see if this helps with our ongoing DNS resolution failures: https://review.openstack.org/#/q/topic:unbound-ttl 19:58:30 should be good to go in, they're set to not change anything and effectively be no-op so that we can try it selectively in some jobs. 19:58:48 dmsimard: is that something we might want to try in a limited fashion on the jobs affected by the problem? 19:59:25 clarkb: that's exactly the purpose, yes, we're actually not changing the defaults from unbound, but jobs can specify vars for cache-min-ttl and it'll be configured accordingly 19:59:35 gotcah sounds good 20:00:09 alright that is all the time we have, find us in #openstack-infra or on the infra mailing list. Thanks everyone 20:00:12 #endmeeting