19:01:10 #startmeeting infra 19:01:11 Meeting started Tue Sep 5 19:01:10 2017 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:14 The meeting name has been set to 'infra' 19:01:20 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:27 o/ 19:01:30 o/ 19:01:44 i guess it's that time again 19:01:55 #topic Announcements 19:02:09 #info The PTG is next week 19:02:20 #link https://etherpad.openstack.org/p/infra-ptg-queens 19:02:31 #link https://ttx.re/queens-ptg.html for general PTG info on what to expect 19:02:40 clarkb: I don't believe in PTGs 19:03:02 mordred: good news! it is happening either way :) 19:03:22 the tooth fairy will be there 19:03:26 Also, I compeltely failed at signing our new Queens release key so will remind everyone about that agin now 19:03:29 #info Queens Cycle signing key ready for attestation 19:03:36 #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0x4c8b8b5a694f612544b3b4bac52f01a3fbdb9949&fingerprint=on Queens Cycle signing key 19:03:41 #link http://docs.openstack.org/infra/system-config/signing.html#attestation attestation process 19:03:47 thanks! 19:03:57 i mentioned it to the release team in their meeting last week as well 19:04:01 if you are an infra-root and haven't done that yet, please do that soon as fungi wants to swap keys out after the trailing cycle releases happen 19:04:16 yeah, basically immediately post-ptg 19:04:22 or as soon as i sober up ;) 19:04:39 (just kidding, we have a gerrit maintenance as soon as i get home!) 19:04:47 #topic Actions from last meeting 19:05:18 fungi: I am expecting we still don't have switchport counts? not suprising considering Houston has bigger problems to worry about now 19:05:36 i think we can drop that from the perpetual action items and discuss at the ptg how we want to go forward 19:05:43 fungi: ok that wfm 19:06:07 #action ianw upgrade mirror-update server and bandersnatch 19:06:07 it don't suppose it's really doing much good to bring it up every meeting 19:06:27 still pending, didn't want to touch before release 19:06:28 ianw: ^ I expect that can actually happen nowish since most of the release stuff is done tomorrow/thursday 19:06:46 ianw: yup np. I just don't want to forget it and I think we are finally to a point where it is mostly safe to do 19:06:46 yep, will look into 19:07:02 #action clarkb update infracloud docs to include ssl setup info 19:07:18 I still need to write ^ that change but am hoping to do that today before the details escape me 19:07:35 #topic Specs approval 19:07:50 #link https://review.openstack.org/#/c/492287/ 19:08:49 This is a cleanup change and all of the dependencies are compelte for it now. Can we open that to voting for those interested in Gerrit and teh Gerrit upgrade? I'll look at getting it in on Thurdsay assuming there aren't items that need addressing 19:09:26 That was really the only spec I saw that was ready. Please let me know if I missed any 19:09:48 #topic Priority Efforts 19:09:52 i'm good with opening that for council vote (thought you had done that last week actually) 19:10:05 #undo 19:10:06 Removing item from minutes: #topic Priority Efforts 19:10:37 fungi: I don't think the dependent changes had merged by the time I looked at the specs again on thursday? I did approve the ssh keys for zuulv3 spec though 19:10:53 #topic Priority Efforts 19:11:23 ok really quickly before getting into zuulv3. Just wanted to remind everyone that Gerrit upgrade to 2.13 is happening Monday after the PTG 19:11:35 #link https://etherpad.openstack.org/p/gerrit-2.13.-upgrade-steps 19:11:56 reviewing ^ would be great if you have time (ha) 19:12:08 #topic Zuulv3 19:12:43 hi! it's less than 1 week before our planned cutover to zuulv3 19:12:46 * clarkb hands meeting baton to jeblair 19:12:55 here's the list of outstanding items we've been working from: https://etherpad.openstack.org/p/zuulv3-pre-ptg 19:13:04 going down that list... 19:13:07 #link https://etherpad.openstack.org/p/zuulv3-pre-ptg 19:13:29 i've been working on the devstack jobs, and i think the devstack-legacy job, which is the job that we'll base the automatically converted devstack jobs on is about ready 19:13:34 well, it is ready 19:13:44 lgtm 19:14:00 it's probably going to need slight revision in order to plug into what the migration script outputs 19:14:14 but it demonstrates all the mechanical things it needs to do 19:14:32 it's also actually really close to a framework we can use for *most* automatically converted jobs 19:15:00 at any rate, i think the next step for that is now waiting on the migration script 19:15:09 so i'll move part of that to the done section 19:15:21 the other part of the devstack work is the zuulv3 native devstack job 19:16:07 we're making progress on that -- it's actually really cool to look at -- you can see how things are shaping up for reuse in zuulv3 and how devstack jobs can be understood by mere humans 19:16:13 speaking of migration script, I'm shifting my attention to migration script today 19:16:24 that's here: https://review.openstack.org/500202 19:16:32 jeblair: ++ it's super cool 19:16:47 mordred: ya, let's talk about migration script in just a min 19:17:16 #link https://review.openstack.org/#/c/500202/ devstack zuulv3 native job 19:17:19 re the devstack job -- unfortunately, we've run into an issue possibly related to zuul_console or zuul_stream where we stop getting output after devstack runs wget 19:17:43 weird 19:17:48 yah. it's ... 19:17:51 digging into that is now at the top of my list (except that some emergency things jumped ahead of it this morning) 19:17:56 one of the weirdest things ever 19:18:01 so hopefully i'll start looking at it in earnest this afternoon 19:18:09 does sound very bizarre 19:18:32 i'm considering it a blocker for 2 reasons -- i still think we need the native devstack job in place before the migration so we actually have something to show and something for folks to build off of 19:18:39 (otherwise, chaos and anger ensues) 19:18:43 yup 19:18:57 and also, it may represent a general problem that may appear in other contexts 19:19:05 well, and people continue to cargo-cult old cruft 19:19:26 (though i rank that a little unlikely because devstack surpasses everything else with what it does with file handle redirection :) 19:19:28 does it cause the jobs to fail too? or is it just breaking the UI? 19:19:29 yah - the second thing is the most troubling, because it's a very confusing thing to occur that makes no sense 19:19:39 clarkb: it causes the job to hang 19:19:54 gotcha so not something we could attempt to live with if we wanted to 19:20:00 although it does get FURTHER than the wget before it hangs 19:20:02 yah 19:20:05 (ansible proceeds somewhat (unsure how far, for obvious reasons) past the point it stops logging) 19:20:22 it's worth noting that it's not just the remote log streaming that hangs ... 19:20:23 could we switch to curl --silent? 19:20:27 wonder if wget security mechanisms are getting overzealous closing extra file descriptors or something 19:20:31 the local writing of content to the files in /tmp also stops 19:20:45 pabelanger: if our ci system can't run wget we must hide our faces in paper bags 19:21:02 i have some paper bags 19:21:11 jeblair: agree 19:21:19 yah. I would consider not being able to run wget a showstopper - largely becaue whatever breaks wget is gonna break somethign else too 19:21:30 completely agree 19:21:37 anyway, i'll start in on that soon and keep folks updated on it 19:21:42 next thing on the list is jobs that use special slaves 19:21:52 #link jobs that use special slaves https://etherpad.openstack.org/p/zuulv3-special-jobs 19:21:56 mordred: ^ 19:21:59 so, they don't work (for obvious reasons) but if folks want to see POC jobs using the new devsack base job: https://review.openstack.org/#/c/500365/ is a rewrite of all of shade's dsvm jobs using the new base job 19:22:19 mordred: (cool thanks! good to exercise the new api) 19:22:33 I have the majority of these done, including a spectacular stack related to wheel mirrors that you shoudl all enjoy and also run away from screaming 19:22:59 #link https://etherpad.openstack.org/p/zuulv3-special-jobs subset of zuulv3 cutover prep work, specifically for jobs that run on special nodes 19:23:03 mordred: that last one means we drop the special builders, right? 19:23:07 yup! 19:23:12 sweet 19:23:35 yay 19:23:40 mordred: what's still tbd? 19:23:48 there's still a few hangers on - pabelanger was just asking about next things to work on, so I think I may hand-off the last of that to him while I shit to migration script 19:23:56 * jeblair giggles 19:24:42 jeblair: I'll update the etherpad after the meeting with a short-list of review links / todo 19:24:54 oh - also - I have two other changes related to not-migration it's worth mentioning 19:24:55 mordred, pabelanger: thanks! 19:25:08 that I need some folks to review 19:25:09 exciting 19:25:28 https://review.openstack.org/#/c/500320 and https://review.openstack.org/#/c/489719 19:25:50 oh yeah, if we find time for those, that will be really nice 19:25:58 https://review.openstack.org/#/c/500320 is a must-have - or a different impl is - it's a thing we missed in our original tox job - which is upper-constraints handling 19:26:06 it may actually end up being somewhat migration related, once we start thinking about things like neutron plugin unit tests 19:26:12 like, we cannot go live without eitherhttps://review.openstack.org/#/c/500320 or an alternate 19:26:15 #link https://review.openstack.org/#/c/500320 constraints handling in tox jobs 19:26:25 the thing I want to discuss about it explicitly 19:26:42 is the approach taken of adding openstack/requirements to the required-projects list of the base job 19:27:12 hrm 19:27:24 why can't it be in the base unittest job + base devstack job? 19:27:27 currently every job in v2 that runs tox is cloning openstack/requirements itself in the job, so I don't think doing that is any MORE burdensome than the current thing, but it feels weird 19:27:36 jeblair: because unittest is in zuul-jobs 19:27:55 mordred: it feels like maybe the right thing to do is make openstack-unittest then 19:27:58 and I'll consider it a crying shame if openstack cannot use the tox-py27 job 19:28:05 tox-py35-constraints has it I think 19:28:12 i'd like to explore every possibility available before we do that 19:28:20 mordred: one thing to consider is we don't need the entire repo, we just need the single file (not sure if that makes anything easier though) 19:28:40 clarkb: well - we also honestly need to be able to do depends-on with upper-constraints values too 19:29:07 jeblair: can we make a unittest job in project-config that shadows the one in zuul-jobs? 19:29:10 pabelanger: are you saying tox-py35-constraints is a child job of tox-py35, and tox-py35-constraints has openstack/requirements? 19:29:18 mordred: ya so some repo manipulation needs to happen 19:29:28 can the base tox job archetype grow a mechanism to add -c /some/path/constranits.txt if a file exists at that location, and then all we have to do is add some task to drop the right content there? 19:29:36 yes. tox-py35-constraints is what https://review.openstack.org/#/c/500320 is aiming to remove the need for 19:29:47 fungi: right -so that's what's in that patch 19:29:48 jeblair: yes to both 19:29:49 though i guess the set problem means that adding that task still leaves us with a subclass of the tox jobs 19:29:49 mordred: oh, why do we want to remove it? 19:30:02 let me back up real quick 19:30:38 mordred: (or we could take this as something to hash out in #zuul after meeting) 19:30:38 first - what the patch does is adds logic to the tox role to look for an upper-constraints file, if given, and if so it will set the environment variable UPPER_CONSTRAINTS_FILE 19:31:08 all of the openstack tox jobs that need/understand UC files respond to that env var 19:31:35 so that part of the logic allows the tox job itself to be used by people who care or don't care about upper-constraints files and for both groups it'll DTRT 19:31:52 the second part of the equation is getting a file on disk and telling the tox job to look for it 19:32:14 mordred: i think that part is fine; the thing that rubs me the wrong way is cloning openstack/requirements everywhere; there are *a lot* of jobs that don't need that 19:32:14 that is the part where the current patch sets a variable and a required-projects 19:32:20 jeblair: indeed 19:32:33 jeblair: we could perhaps justmake it a project-template 19:32:47 jeblair: that adds the repo to required-projects and also sets the variable 19:33:06 so people can use python-jobs-with-constraints perhaps? 19:33:16 I think I must have missed something, I thought tox jobs that needed requirements, we'd just create tox-py35-contraints and tox-py27-constraints in openstack-zuul-jobs, everything else could use tox-py27 from zuul-jobs 19:33:19 i'd love to see if we can make it a proper job 19:33:24 and then for individual job consumption, people can always set the variable and add the repo if they want 19:33:34 can we pause for a minute though? 19:33:37 sure 19:33:57 i'd like to ask whether we want to continue this conversation here, or just note that this is something we need to work out after the meeting? 19:34:04 sure. we can do that 19:34:09 we have 5 items after this in the zuul list alone 19:34:17 ya I think we can sort that out after meeting (this is going to be a full meeting) 19:34:18 I just need people to engage on this topic 19:34:31 let's resume this immediately after the meeting in #zuul 19:34:34 kk 19:34:41 next up: migration script 19:34:44 we can also talk about the other patch I mentioned then too 19:35:05 mordred: you mentioned earlier you're going to start working on this 19:35:29 mordred: is this still at one-person stage, or do you need/want anything from other folks? 19:35:36 yes. I'm shifting attention to that today - I think the other special jobs are far enough along, we can grab stragglers at the end if we need to 19:35:38 s/start/resume/ 19:35:57 I think it's one person for the rest of today 19:36:13 and maybe part of tomorrow 19:36:14 cool, i reckon you'll let us know when there's more stuff to jump on 19:36:34 next up: migration docs 19:36:36 but at that point I'm expecting it'll be to the point where it can be run locally, someone can find a problem with a migration and add a workaround 19:36:49 #link infra-manual zuulv3 migration docs https://review.openstack.org/500218 19:37:03 that change and its parents remove all of the TODO items from the zuulv3 infra manual page 19:37:17 it would be great if folks can review it, and also read the whole thing and identify gaps 19:37:42 in some form or other, it at least roughly covers most of the things i would want to communicate to folks making the jjb -> zuul transition 19:37:54 it's not complete documentation, but it should get us on the same page 19:38:10 i think once we have the migration script, we may want to add things related to that 19:38:13 jeblair: i'll review those today 19:38:29 like "the migration script output some stuff that looks like this, here's what you should do" 19:38:34 can't write that yet 19:39:07 we already talked about zuul_console breakage... so next is docs job incorrectly publishing 19:39:27 we published the zuulv3 docs over top of the zuulv2 docs 19:39:34 i think pabelanger just checked in on this before the meeting, and there's a patch that should fix that 19:39:46 so hopefully this is mostly taken care of already, just needs a review and some republishing 19:39:59 yes 19:40:15 pabelanger: can you take care of making sure that patch gets merged and appropriate jobs re-run to get the right content back in place (and the new version exercised)? 19:41:05 I'll review this immediately after the meeting too 19:41:14 (since I started looking at it just before) 19:41:18 jeblair: yes, I'll do that after the meeting 19:41:38 cool thx 19:41:41 finally -- zuul-cloner shim; i sent an email out about this and Shrews started working on it 19:42:28 idea here is just that if you run zuul-cloner it doesn't break on you to easy migration? 19:42:35 *ease 19:42:55 clarkb: also it translates the golang-style paths to zuul-cloner style paths 19:43:05 src/git.o.o/openstack/foo -> openstack/foo 19:43:25 maybe even supports clonemap files so jobs that rely on that work as-is 19:43:37 working on the clonemap login now 19:43:50 logic* 19:43:56 Shrews: w00t thanks! 19:44:04 nice 19:44:07 that's all the things on the list -- anything missing? 19:44:16 (blockers for migration/cutover) 19:44:30 oh - I noticed a thing, I should add it to the list ... 19:44:41 new servers we need for saturday? 19:44:53 pabelanger: yes! 19:45:01 eg: nl02.o.o is online, and we should cut over to it asap 19:45:07 pabelanger: did you spin down those logstash servers? 19:45:08 nl01.o.o is trusty 19:45:21 I noticed this weekend while working on some base job stuff that our configure-mirrors role does not have parity with the current configure-mirror script 19:45:32 mordred: nice thanks :) 19:45:34 jeblair: I just stopped logstash services first, so if we are okay to stop them, we can do that now 19:46:03 logstash-worker016.o. to logstash-worker20.o.o 19:46:06 I think we can likely make a legacy-openstack-configure-mirrors thing in project-config that does a dumb version of a direct translation 19:46:08 pabelanger: cool, sounds like you, fungi, and clarkb can probably handle that 19:46:08 mordred: jeblair fwiw the existing script is written such that you execute the same script hat nodepool exectures to load that data up 19:46:14 for mirrors 19:46:16 and make it nicer moving forward 19:46:16 yup 19:46:37 kk 19:46:48 clarkb: yup. I think we need to have legacy-openstack-configure-mirrors do that for now 19:47:14 pabelanger: ya logstash job queue count looks ok so should be fine to turn those servers off/delete them 19:47:16 clarkb: and then we can circle back and talk about how/if to refactor it to be more 'native' later 19:47:50 clarkb: ack 19:48:51 okay, the next event(s) in our schedule are to do trial cutovers this weekend -- saturday and/or sunday evenings. obviously that will only happen if these blockers are sufficiently resolved by then. 19:49:25 maybe let's discuss details on that friday? 19:49:48 sounds great 19:49:54 wfm 19:49:58 ++ 19:50:10 clarkb: eot from me 19:50:33 ok 19:50:47 there is also a new thread on the mailing list about Zuul UI things 19:51:13 I've sent a quick reply but definitely don't have enough background on everything going on in Zuul to say much. Would be nice if someone could respodn to that properly 19:51:30 #topic PTG team dinner 19:51:38 clarkb: will do 19:52:06 Tuesday evening looks like the best night for us because it is when we have the most overlap without conflicts. I have't heard objections yet but basically said lets do Tuesday evening at beer garden 19:52:15 ++ 19:52:35 Mmm, garden beer 19:52:45 If this doesnt' work please propose alternatives but as of today that is the plan :) 19:53:00 Once in Denver I'll send mail/ping people with details on how we are transporting 19:53:16 #topic Project renames 19:53:27 #link https://review.openstack.org/#/c/500768 19:53:47 we have one project rename. I don't expect we'll get to taht this week for obvious reasons, or next 19:54:16 Assuming gerrit upgrade goes well after PTG maybe we can plan a rename for that week or week after? This is mostly a heads up, but lets not worry about it until later 19:54:18 i expected it would happen after the gerrit upgrade 19:54:32 sounds good 19:54:53 In theory we'll be knocking out a couple long standing todo items for infra over the next couple weeks so lets focus on doing that first 19:55:16 #topic Open Discussion 19:55:24 woo managed a couple minutes leftover for this 19:55:29 #link https://etherpad.openstack.org/p/GoqkPTucMK infra addition to tc's top 5 help wanted 19:55:29 \o/ 19:55:37 the tc has suggested we propose something along those lines to help increase visibility with companies looking for effective ways to get more involved 19:55:54 please hack that up with whatever edits you think it deserves 19:56:09 but i'd like to get it pushed up to gerrit in the next day or two 19:56:33 * clarkb makes note to review that 19:56:35 fungi: nice, was just talking to some downstream people about how to get move involved upstream 19:56:56 i borrowed a little wording from the summary in system-config doc, though with tweaks and most of that is wholly new prose 19:59:13 doesn't look like there is much else. Thank you everyone 19:59:17 #endmeeting