19:03:42 #startmeeting infra 19:03:43 Meeting started Tue Aug 22 19:03:42 2017 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:03:44 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:03:46 The meeting name has been set to 'infra' 19:03:51 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:04:03 #topic Announcements 19:04:15 PTG is quickly approaching. 19:05:08 Pike is right in the middle of release fun as release candidates come out and all that (so we should be careful to be slushy and avoid breaking things) 19:05:25 That is all I had. Anyone have anything to add? 19:06:05 clarkb: just that mirror nodes should be Xenial now, so if anyone has issues we're not aware of, let us know 19:06:42 ya I've added that to general discussion so that we can talk about it a bit more 19:06:46 #topic Actions from last meeting 19:06:59 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-08-15-19.03.txt Minutes from last meeting 19:07:17 o/ 19:07:29 ianw: I'm guessing that mirror update hasn't been updated to xenial yet. Have you heard from the release team on whether or not we can move forward on that? 19:08:52 I completely failed to send the gerrit upgrade email last week so shoudl get that done nowish. And I haven't seen word of infracloud switchport counts 19:08:58 #action fungi get switchport counts for infra-cloud 19:09:08 #action clarkb send reminder of Gerrit upgrade after the PTG to the dev mailing list 19:09:30 i also completely failed on that, sorry 19:09:41 #action ianw upgrade mirror-update server and bandersnatch 19:10:09 that is all we had from last meeting. So moving on 19:10:11 #topic Specs approval 19:10:41 #link https://review.openstack.org/#/c/462207/ is an update to the zuulv3 spec around zuul's handling of ssh keys 19:10:53 jeblair: SpamapS ^ is that generally ready for approval? 19:11:39 clarkb: yes, sorry! 19:11:52 I think we may also want to add in the way we switch out the ifnra key for the job key? 19:12:34 aiui pre playbook removes infra key from agent and adds in a per job key? Not sure if you want that to be a separate spec update though. In any case I think getting that updated as necessary would be good 19:12:43 clarkb: we could -- though that's already implemented so we don't really need to drive consensus on it. may be better to find a good place to document that for posterity. 19:12:55 maybe expand our base job documentation 19:13:00 wfm 19:13:19 The other spec I wanted to bring up was mentioned last week. 19:13:40 #link https://review.openstack.org/#/c/492287/ mark gerrit contact store spec as completed 19:13:47 o/ 19:13:49 It has a couple depends on that could use reviews from infra-roots 19:14:14 basically all cleanup at this point, pretty straightforward. 19:14:38 Are there any other specs that we should bring up today? 19:16:28 sounds like not. Would be great to get 462207 reviewed and I expect that fungi will merge that when he gets back (I don't have approval bits in that repo yet). And reviews to finish up the contact store priority effort much appreciated as well 19:16:33 #topic Priority Efforts 19:16:52 I don't think there is much else to add on the gerrit upgrade or contact store front. 19:17:00 jeblair: anything from zuulv3? 19:17:18 nope, still chugging along 19:18:09 #topic General topics 19:18:40 One of the big things that happened last week was we upgraded our region local mirror instances to xenial from trusty in an effort to get more reliable http servers and afs 19:19:03 We weren't sure if it would fix all the problems but ianw pointed out debugging a current lts is far more beneficial than the last one 19:19:12 Ya, switching to xenial has been an improvement from what I see 19:19:47 also the patch for htcacheclean 4096 directories is also working great 19:19:59 clarkb: only one in the list still is OVS, is that ready? 19:20:13 yup I agree. Though I did notice that yesterday we had 4 cases of pip hash mismatches in rax-dfw all within a minute of each other so I think we may still have some minor issues around afs (assuming it was an afs problem in the first place) 19:20:13 "the list" being https://etherpad.openstack.org/p/mirror-xenial for reference 19:20:38 #link http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22THESE%20PACKAGES%20DO%20NOT%20MATCH%20THE%20HASHES%20FROM%20THE%20REQUIREMENTS%20FILE.%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s for query on pip hash mismatch 19:20:50 #link https://etherpad.openstack.org/p/mirror-xenial for finishing up mirror upgrades 19:21:09 ianw: I didn't really want to touch them until we got the account situation sorted out there just to avoid potentially causing more problems 19:21:11 hmm, i'll look into that, because a cluster is consistent with the segfaults we saw 19:21:22 ianw: clarkb: the ttl for proxypass also seems to be an improvement for 502 proxy errors. I haven't see any issues from buildlogs.centos.org today 19:21:28 hmm i noticed that pycrypto doesn't build properly when using the mirrors, possibly related to the mirror update? https://review.openstack.org/#/c/496027/1 19:22:09 cmurphy: hmm, that shouldn't be different ...looking 19:22:25 pabelanger: that is good to know thanks. 19:23:05 jeblair: you haven't heard anything back from jean daniel have you? Checking my email I don't have anything 19:23:19 ianw: those tests try and fail to install gerritbot which fails building pycrypto, but it seems to work fine on the same vm without the pip.conf changes 19:24:17 clarkb: negative 19:24:43 whoops. /me waves 19:24:47 cmurphy: that seems to be a problem with the host side, rather than the mirror side? 19:25:21 #action clarkb follow up on ovh account situation and make sure mirrors get rebuilt there as xenial 19:25:33 cmurphy: ill comment 19:25:34 I'll go ahead and try to track that down now that we expect people to be back from vacation aiui 19:26:18 The other bigish thing that happened recently was we no longer allow glean to update dns resolvers based on dhcp content in our images 19:26:42 we were having dns resolution problems in clouds like rax and it turned out we weren't using our local unbound install, we were hitting the cloud resolvers instead 19:26:45 Yes! That has helped bigtime for centos 19:26:52 pabelanger did a bunch of work in glean to fix that. 19:26:59 now that we are using unbound, DNS failures are basically gone 19:27:10 except for on suse where we now have the problem of using unbound :) 19:27:24 #link https://review.openstack.org/#/c/496310/ should fix unbound for suse if people can review that today 19:27:36 git branch 19:27:38 #link https://review.openstack.org/#/c/496341/1 is alternate suse fix 19:27:43 oops. sorry 19:27:59 there ended up being two changes pushed for that, if we prefer one more than the other I will be sure to abandoned the second one 19:28:08 k, I'll look at suse fixes after meeting 19:28:17 thanks 19:28:53 Overall I think these changes are doing quite a bit to make test jobs more reliable so thank you to everyone that helped get them in place 19:29:15 Ya, tripleo gate pipeline hasn't reset in 24 hours now 19:30:09 The last item I had for general discussion was something that happened this morning. nose-html-output is still in use by a couple projects (horizon and swift I think?) Its basically unsupported by us due nose being highly discouraged as a test runner and lack of subunit. As a result I have added amotoki to the core list as they have an interest in keeping the tool running at least for the short 19:30:11 term 19:30:27 if there are other individuals interested in supporting that project I think we should just go ahead and add them to the core group 19:30:52 (good news is sounds like there will be new investigation of getting horizon to emit subunit so yay) 19:31:13 nice 19:31:54 #topic Open discussion 19:32:21 Anything else? 19:32:31 anyone mind if i get to a bindep release today? 19:32:42 there's some rpm changes that will be helpful for testing python3.6 19:32:46 can we remove fedora-25 DIB? 19:33:19 re bindep any concern about that affecting the release? IIRC the unittest are pretty robust since its basically just a language parser so probably fine 19:33:24 also, somebody asked if we could remove debian-jessie too. Since openstack-deb team is getting disbanded 19:33:28 pabelanger: not quite, but i can do some cleanup today to get us there. fedora-26 is ok with devstack now 19:33:43 pabelanger: ya I had asked about debian but fungi pointed out projects like dib are using the image so I think we can keep it around 19:34:00 For open discussion, just wanted to mention I officially got the green light from Red Hat to spend time on upstream/infra and help you guys out. My first work item is to improve devstack-gate bash/ansible things. 19:34:04 clarkb: i think, looking at the changes, low risk, mostly just the changes to add "--whatprovides" to rpm calls 19:34:10 i have a mass approval request if anyone wants some internet points https://review.openstack.org/#/q/status:open+topic:refactor-infra-beaker the idea is to hopefully reduce the need for mass updates 19:34:12 k, the reason I ask, is we only have 100Mbps link on nb03.o.o, so it does take some time for new images to upload. 19:34:40 rackspace is crazy fast, like 30mins to upload 80GB 19:34:41 i don't think dib is testing on debian 19:34:42 dmsimard: yay! thanks! 19:34:56 dmsimard: nice! and thank you 19:35:18 dmsimard: ^5 19:35:20 ianw: did that change to add the output of all deps get merged? I +2'd it but I think the submitter would like it if it was in the next release 19:35:48 it did merge so thats good 19:36:13 #link https://review.openstack.org/#/q/status:open+topic:refactor-infra-beaker mass approval revuest to reduce the need for mass updates 19:36:47 clarkb: https://review.openstack.org/#/c/492693/ ? yep 19:36:58 ianw: ya 19:37:21 the only outstanding changes are refactor ones ... bit iffy on code motion without other reasons for it 19:37:26 pabelanger: probably the next step is do a job listing for the debian image and see if its just our bindep fallback test and the debian packaging jobs 19:38:35 clarkb: agree 19:39:00 cmurphy: I'll try to review that after lunch 19:39:10 ty clarkb 19:41:27 for those of us going to the ptg is there interest in trying to have a team dinner again? anyone interested in lining that up? 19:42:02 that might be better off as a ml thread 19:42:27 +1 19:43:36 I'll give it to 19:45UTC before ending the meeting but beginning to sound like that is it 19:45:24 ok thank you everyone. You can find us in #openstack-infra if we missed anything or you remember something important 19:45:32 #endmeeting