19:01:05 #startmeeting infra 19:01:05 Meeting started Tue Jul 9 19:01:05 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:08 The meeting name has been set to 'infra' 19:01:10 o/ 19:01:23 #link http://lists.openstack.org/pipermail/openstack-infra/2019-July/006410.html Meeting Agenda 19:01:38 #topic Actions from last meeting 19:01:44 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-07-02-19.01.txt minutes from last meeting 19:02:05 mordred: any progress with the github account creation and -infra org cleanup? 19:02:16 nope 19:02:40 jroll sent a followup agreeing with ttx's suggestion about repository descriptions 19:02:50 (on the openstack-discuss ml thread) 19:02:57 * mordred missed that - goes to look 19:03:09 apologies for that dragging, am just looking for volunteers to help at this point 19:03:27 #action mordred create opendevadmin github account 19:03:34 #action mordred clean up openstack-infra github org 19:03:42 #link http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007607.html Assuming control of GitHub organizations 19:04:28 jroll: yes, volunteers! (and if there are none, maybe that opens the door for alternative choices about whether mirroring there is even important?) 19:04:50 indeed 19:05:32 seeing how often people link to github on #openstack-infra: Many are used to it - but seems it might not be important enough to take action 19:05:57 they can always get used to linking something else ;) 19:06:20 exactly 19:07:49 seems like we usually get volunteers when we put it in those terms 19:08:02 I guess if no one shows up in the near future we can always bring that up as the alternative 19:08:11 (as a non github user I'm not terribly interested myself :) ) 19:08:36 #topic Priority Efforts 19:08:42 #topic OpenDev 19:09:14 I'm not aware of anything new on the opendev side of things since last week. Anyone else have something to share here? 19:10:05 the mirror stuff but that's got its own topic 19:11:14 #topic Update Config Management 19:11:49 corvus: has been improving our job setups and has a change into zuul which will allow us to avoid testing everythign in system-config when zuul.yaml updates 19:12:04 #link https://review.opendev.org/669752 Test job updates without explicit file matcher 19:12:34 This should help us move more quickly on our ansible changes as we won't end up testing the world (I think it will help with some of the other efforst we have to add jobs as well) 19:12:51 yeah, with that we can really scale up the number of system-config jobs 19:13:56 ++ 19:15:01 will reduce the number of test nodes we need for system-config changes too which should make everyone happy 19:16:50 Any other config management improvement updates? 19:17:26 mordred: let's you and i sync up after the meeting and figure out next steps for the gitea project creation change? 19:17:34 i think we said we'd revisit that this week 19:17:48 one of us should start on that soon :) 19:18:26 corvus: kk 19:19:06 #topic Storyboard 19:19:13 fungi: diablo_rojo any udpates to share? 19:19:37 clarkb, got some of our migration tracking stuff up to day 19:19:39 date 19:19:52 poked at a few more teams to see if they want to migrate, havent gotten responses yet 19:19:57 i took a vacation ;) 19:20:17 Would be great if we can get some help from mordred on the database indexing and whatnot to speed search up 19:20:31 Since our intern got a job and left us :'( 19:21:45 diablo_rojo: might want to followup on https://review.opendev.org/651942 as well 19:22:14 AJaeger, thanks for the heads up. Will do! 19:22:29 clarkb, thats all I have really 19:22:44 diablo_rojo: oh no! (although awesome for the intern) 19:23:56 mordred, yes, happy for her also, selfishly, very sad for StoryBoard and the amount of effort I put in to get an intern 19:24:13 yeah 19:24:30 #topic General Topics 19:24:37 First up checking in on our trusty life support situation 19:24:49 fungi: were you able to get your change to pass tests after we switch to the legacy base jobs? 19:25:01 and if so has that change been merge yet? 19:25:38 yeah, it worked and has merged 19:26:02 and just a bit ago i cleaned out the weirdly miscloned files where vcsrepo wants to put its repos 19:26:10 cool I guess its rerun and see what the next errors are? 19:26:12 so waiting to see if puppet will dtrt now 19:27:03 Ok next on the agenda was an update on the afs and mirrors situation. Sounds like dhowells is one vacation for a bit as is ianw so we've stopped using the kafs mirror. 19:27:07 looks like the redirect was not the problem for those though 19:27:14 :( 19:27:19 so still hunting for an explanation 19:27:27 could be submodule related? 19:28:03 I did want to ask if we think we should go ahead and replace all the mirrors with bionic + 1.8.3 at this point. That setup seems stable and will get us LE certs and tls 19:28:27 nit: ianw mentioned 1.8.4 AFAIR in his email 19:28:58 ah well whatever is in the infra PPA :) 19:29:14 seems fine to me 19:29:19 I'm not sure this is urgent but does potentially open the door to more interesting job activities 19:29:23 1.8.3 - link is http://lists.openstack.org/pipermail/openstack-infra/2019-July/006411.html 19:29:27 (they can more strongly trust our mirrors if so) 19:29:47 so, I misremembered 19:30:06 it might be worth figuring out how to build these packages institutionally, rather than the ianw pushes to ppa approach? i get the feeling we may be dependening on these for a while, and we may need to update them. 19:30:16 most of my memories are misrememberies 19:31:09 i think the goal was to convince ubuntu package maintainers to either update the prerelease they shipped in bionic, or get it added to bionic-backports? 19:31:35 we never have come up with a good way to do that though. maybe we could have a zuul post job push the config to the ppa? i've forgotten how ppas work though. 19:31:49 afs folks seem a bit unnerved by ubuntu deciding to distribute a prerelease in their lts distro 19:32:09 fungi: ya I think the two ideas ianw had were to either use kafs or have ubuntu update their clearly broken package(s) 19:32:10 it's not ideal 19:32:39 we've found kafs needs more work (which we are helping with but still needs time) and I'm never sure how possible updates to ubuntu packaging is 19:33:09 i think the only real benefit to the ppa is that we can have it build arm64 packages 19:33:56 working on a way to maintain the packages as a team rather than an individual effort gives us a backup plan in case ubuntu doesn't jump when we say, and gives us options if we need to update versions. 19:34:04 corvus: ++ 19:34:26 maybe a topic for next week's meeting, assuming ianw is back? 19:34:28 running debuild on a test node, running apt-ftparchive over that, generating a detached release file on the executor and pushing that somewhere wouldn't be complicated, and we do have arm64 nodes too 19:34:42 thinking out loud here we should probably do both things then? ubuntu should ideally have working openafs and we can also manage the ppa more programatically 19:35:02 corvus: and ya maybe we let ianw chime in when back and then go from there 19:35:06 yeah, ubuntu having working afs is definitely the ideal 19:35:13 i just don't want to be caught without a backup plan 19:36:23 makes sense, for sure 19:37:03 sounds like a plan we'll bring it up with ianw and work towards sustainable bionic openafs 19:37:16 The last thing I had on the agenda was an update of the fortnebula cloud addition 19:37:38 I had hoped we'd be "done" at this point but have run into glean + centos/fedora + network manager + ipv6 only cloud problems 19:37:52 gentoo images also don't work but for different reasons 19:38:21 (what's the gentoo problem in 6 words or less?) 19:38:36 Where I am at in the debugging is I have nodepool's test centos image with new glean unit file and old unit file ready to upload into fortnebula to boot test nodes there. And have also held a nodepool test job so I can figure out why our supposed fix doesn't work in testing 19:38:51 the gentoo problem is similar to the centos problem. basically no ipv6 addr configured on the interface 19:39:00 okay, just different tooling 19:39:05 but gentoo uses systemd-networkd and not networkmanager so the mechanism of failure should be different 19:39:07 yup 19:39:23 that's more than six words ;) 19:39:26 :P 19:39:30 fungi: you're more than six words 19:39:41 yup yup yup yup yup yup yup 19:39:45 what I'd like to do with the images I'm uploading to fn is confirm the fix works there so I can turn my focus to the test jobs 19:39:54 and I guess I'll find what I find 19:40:14 I did test all of the other images one by one and they all seemed to work (ubuntu, opensuse, debian) 19:41:21 and the fix which isn't working in testing did work when booting the nodes there directly (outside of our nodepool test job), right? 19:41:58 fungi: yes, but that was on modified images in place (i updated the unit file, reloaded the systemd daemon config, deleted the /etc/sysconfig/network-scripts/ifcfg-eth0 file then rebooted) 19:42:32 its possible that something else related to booting a second time is causing it to work which is why I'm now going to double check with fresh images directly from the test job (I copied the image out of the test job) 19:43:02 sure 19:43:27 so ya I'll be poking at that today and into the near future probably 19:43:35 And that takes us to the end of our agenda 19:43:40 #topic Open Discussion 19:43:48 Anything else? 19:45:29 FYI: fungi has a change up to switch to ansible-lint 4 in zuul-jobs, see https://review.opendev.org/667695. I plan to +2A by end of week if there are no objections. 19:45:58 thank you for the heads up 19:46:55 pay attention to the child changes of that one, since they do give us some indication of where ansible-lint devs are headed and whether it will be an increasingly steep treadmill over time 19:47:13 fungi: maybe we are in a position to give feedback to those devs? 19:47:35 "hey here is a bunch of real world ansible and it doesn't beneift from x or y but could really use a check for z" type of feedback maybe 19:47:45 one of those child changes needs a fix in ansible-lint, others might have a -1 19:48:18 sure, or to decide whether ansible-lint is (or even wants to be) an effective tool for this particular use case 19:49:28 yeah, i'm unconvinced that linting for code style is a useful thing. we've wasted hundreds of person-hours on it. but i've tried to give those changes a fair evaluation in the spirit of compromise. 19:49:31 i split the "fixes" for each new rule into separate changes so each can be evaluated on their own merits and included or abandoned as we choose 19:49:45 if someone wanted to remove it entirely, i would happily +2 those patches. 19:49:58 and i think we should all be aware that it may come to that depending on the direction that ansible-lint takes. 19:50:08 ya what I'd really like are linter checks for deprecated behavior in ansible 19:50:19 but we've got a plan for finding and addressing those by running ansible instead 19:50:23 yeah. and things that are actually catching gotcha behavior 19:50:29 yes. love that. 19:50:55 "I don't think that means what you think it means" 19:51:02 helpful 19:51:07 "I don't like the color of your shirt" 19:51:08 not helpful 19:51:11 ++ 19:51:22 inconceivable 19:51:32 * corvus is wearing a *black* shirt, so i assume that's okay ;) 19:51:51 * mordred is not wearing a shirt - given it's the carribean and there is no A/C where he's sitting 19:52:00 this is why we irc 19:52:05 corvus: ++ 19:53:02 its actually cool and rainy here 19:53:02 clarkb: you said something about a short meeting? :) 19:53:11 corvus: ya we better stop now to keep to that :) 19:53:14 thanks everyone! 19:53:18 * fungi wore shorts to the meeting 19:53:24 * AJaeger too 19:53:24 haha 19:53:29 find us in #openstack-infra or at openstack-infra@lists.openstack.org 19:53:32 #endmeeting