19:01:27 #startmeeting infra 19:01:28 Meeting started Tue Oct 30 19:01:27 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:32 The meeting name has been set to 'infra' 19:01:40 #topic Announcements 19:01:55 Hopefully this doesn't surprise anyone but the Berlin summit is less than two weeks away 19:02:22 This is your warning to do any necesary prep work :) I need to put together a project update and onboarding session 19:02:41 #link https://civs.cs.cornell.edu/cgi-bin/vote.pl?id=E_aac97f1cbb6c61df&akey=b9e448b340787f0e Vote for OpenStack T release name 19:02:54 tonyb put together a poll for us to pikc a name. You should really vote for train 19:03:06 note that if you share an ip address you'll need to find an alternative from which to place your vote 19:03:28 say, an unpopular starbucks 19:03:39 or via your phone? 19:03:39 or maybe the public library (they still have those, right?) 19:03:44 o/ 19:04:12 Anything else worth announcing? 19:04:38 not to my knowledge 19:04:44 #topic Actions from last meeting 19:04:51 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-10-23-19.01.txt minutes from last meeting 19:05:15 Not in that ^ meeting but in the opendev messaging meeting I took an action to start drafting an email to talk about opendev more broadly 19:05:37 I've got time set aside for that on this meeting agenda but if you want a sneak preview https://etherpad.openstack.org/p/WUNOTv8MuP 19:05:56 #topic Priority Efforts 19:05:56 * fungi sneaks a peek 19:06:07 #topic Update Config Management 19:06:29 May as well jump right in. I don't think much happened on this last week as we were distracted by zookeeper, openstacksdk, and nodepool related things (at least I was) 19:07:04 gerrit query for reviews: topic:puppet-4 and topic:update-cfg-mgmt 19:07:10 * mordred aplogizes for his hand in the openstacksdk things 19:08:13 i, for one, am glad you choose to do openstacksdk things 19:08:14 don't apologize 19:08:16 have I missed related work on ^ worth talking about? I guess we may still need a decision on where to store the ara database? I think the conversation we had a week ago mostly ended up with maybe we should rebuild bridge server to be appropriately sized for ansible and mysql 19:08:34 then we reduce data leakage concerns and have server that performs as expected across the board 19:09:15 well, there was already a trove instance created, so i thought we said in the interim we can use that in parallel with any work to replace the bridge server 19:09:24 ah 19:09:39 I support all of the choices 19:09:47 but that in sizing the replacement bridge.o.o we should take running a mysql db for ara into account 19:09:53 I was distracted by zookeeper move at the time so fungi's memory likely better than mine 19:10:08 ++ 19:10:39 yeah, the information leak is theoretical, and so short-term concerns while we're getting this up and running aren't as significant as the longer-term risks of continuing to use a separate db system 19:10:54 sounds good 19:11:04 mordred: hae a storyboard containerization update? 19:11:27 oh my 19:11:30 * fungi hopes it's not an old margarine tub 19:11:56 so - I have decided to step back from that just a little bit- a bit more to wrap my head around 19:12:11 and gonna just do a normal update - so I can properly think about what the containerized version looks like 19:13:09 I'll hopefully get some patches up for review later today or early tomorrow 19:13:11 now thinking that "margarine-tub" would have been a great alternative name for pbrx 19:13:18 fungi: still might be 19:13:31 seems reasonable. I do think we shouldn't be afraid of doing the quicker get off trusty moves without blocking too hard on containers if it isn't straightforward upfront 19:13:37 yup 19:13:43 that's kinda where I ended up 19:13:53 also - to be fair - I'm *also* overthinking it 19:14:21 anything else related to updating config management before we go to storyboard? 19:14:25 (this is the segway) 19:14:29 so basically upgrading storyboard-dev.o.o and then storyboard.o.o from trusty to xenial for now? 19:14:34 fungi: yah 19:14:47 er segue? english weird 19:14:48 wfm 19:14:59 sanguine 19:15:01 I'd like to eventually end up with storyboard running in python3 in containers on bionic 19:15:07 mordred: shiny 19:15:10 but let's just make progress 19:15:29 #topic Storyboard 19:15:38 I did review the storyboard spec 19:15:45 me too! (finally) 19:16:03 i also need to apologize to diablo_rojo for the pile of comments i left on it 19:16:11 and frickler too. Thank you all for doing that 19:16:18 Looks like we are waiting on an update on that now? 19:16:45 if any of my comments are unclear or need added detail, please let me know and i'm happy to expand on them 19:17:33 are tehre other storyboard updates? I had to miss the meeting last week due to dentist visit 19:18:46 some talk of forum planning 19:19:17 there's a remaining blockers to migration session on the forum schedule 19:19:21 fungi: project moves have slowed down? 19:19:22 #link https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22839/storyboard-migration-the-remaining-blockers Summit Session 19:19:27 I'm guessing that is due to the remaining blockers? 19:19:30 for the moment, yes 19:19:37 well, and also some roadblocks in testing them 19:20:02 diablo_rojo has discovered that test imports of neutron abruptly stop partway into what we would expect to have loaded 19:20:27 suspecting at this point that we're missing support for lp api result pagination or something along those lines 19:20:34 I guess it is good that we test them then :) 19:20:55 but need to add some more debugging output to figure it out 19:21:09 indeed! 19:21:13 oh joy 19:21:41 neutron is by far the largest single lp project we've attempted to import, so no surprises it's where we're finding new problems 19:22:09 mordred: ^ probably want to coordinate the redeployment of -dev with neutron debugging 19:22:26 (just to avoid deleting old server in the middle of a test migration) 19:22:52 clarkb: yes - I agree. 19:22:53 #topic General Topics 19:23:10 Opendev. As mentioned earlier I've started drafting an email to communicate the things 19:23:32 It depends a bit on the work lsell will be doing foundation side, but I'm trying to set a tone of we aren't abandoning you and this is something people have wanted for a while 19:23:48 I think it is also helpful that we now have data saying 98.5% of all the test resources we use are openstack related 19:23:57 point of order. i move that any specs related to the opendev transition be deemed priority efforts 19:23:58 ++ 19:24:15 * mordred seconds fungi's point of order 19:24:17 That said I am not sure I'm entirely happy with that draft so please give me feedback or suggest wys to make it better 19:24:21 mordred: fungi I'll third that 19:24:22 * mordred also knows that's not how robert's rules work 19:24:38 yep 19:24:48 i was abusing terms for fun 19:24:53 me too 19:24:59 #link https://etherpad.openstack.org/p/WUNOTv8MuP draft email announcement 19:25:08 just wanted to be clear in case any robert's rules superfans were lurking 19:25:08 now the link shows up in the correct spot in the meeting logs 19:25:41 I'll try to sync up with lsell real soon now to see how the other bits are going 19:26:37 Doesn't look like the dns configurations are fully in place for opendev.org yet, but progress was amde on that lst week too 19:26:48 I expect corvus will be motivated to finish that when back at work 19:27:40 Trusty server upgrades. As hinted to earlier with storyboard this continue to happen. The zookeeper cluster work last week was basically a trusty to xenial upgrade for our zookeeper server. The trusty zookeeper is running on nodepool.openstack.org. It is on my list of todos today to delete that server 19:28:04 this server has been around for a long time and was a fairly important piece of the puzzle for us so if you know of anything that needs saving or cleanup before I delete it please let me know 19:28:18 (possbily useful openstack api related scripts) 19:28:41 also, no obvious problems after switching from single-node zk to a cluster 19:28:47 so that was nice 19:29:06 correct. I chceked the zk stat output on all three servers for days after and it has continued to look happy 19:29:10 ++, now we just need to get it ssl :) 19:29:11 cacti also makes it looks happy 19:29:39 Once nodepool.o.o is gone I'm going to grab another server off the list and go look at what that server needs. Maybe pbx? 19:30:01 I feel like I've got some momentum with this and am hoping to keep doing a server ish a week if I can 19:30:08 i guess that did eventually get rebuilt on ubuntu, huh? 19:30:12 fungi: yup 19:30:14 i'm hoping to get to some upgrades too, if nothing else pops up 19:30:49 are the git servers and load balancer our only centos servers now? 19:30:58 ianw: great. Please do let me know if I can help (that goes for anyone doing this work I am highly motivated to get this type of thing done and happy to help where I can) 19:31:01 fungi: yes 19:31:35 #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup list of servers that need upgrading 19:32:42 Zuul queue backlogs. Since the PTG we've been experiencing long zuul queue backlogs. There have been a varity or reasons for this over the last month and a half many of which we've fixed. But the general situation is that we are still suffering from this 19:32:58 #link http://lists.openstack.org/pipermail/openstack-dev/2018-October/136141.html Most recent update from clarkb on the Zuul backlog situation 19:33:14 The good news is we are getting better data from zuul around where the cost is 19:33:25 Using that we can measure if our changes improve things 19:33:32 and also prioritize which work would be most effective 19:33:38 ++ this is great 19:34:50 Another item worth pointing out is that I think we are in part running into lack of logging in jobs that help us understand where the time spend is within a job. I've been asking various projects to improve the logging so that the statement isn't just "infra is slow or unreliable" and instead is "on infra $task takes this long and is $resource bound" 19:35:19 For example tripleo points to docker image download costs. But as far as we know we don't have a measurement of that so step 0 is lets measure that then we'll know if changes improve it 19:35:24 (or if it is even a major cost at all) 19:35:42 OSH has performance issues but we don't knwo if they are related to cpu/disk/network/etc so to improve that we need that information 19:35:56 yup. logging here is also helpful, because while we do have docker mirrors, it may not always be clear if we're actually using them or not 19:36:01 There is zuul work to build in some performance data into base jobs too which should help 19:36:17 I think we've taken for granted that devstack does all of this for you 19:36:21 yup 19:36:29 and now that we are doing more and more non devstack testing the lack of that data is apparent 19:37:06 If random projects drop by asking for help implementing that stuff any help you can give will likely go a long way 19:38:21 And that was it for the agenda I had written down 19:38:30 #topic Open Discussion 19:38:40 Anything else? we've got time so feel free 19:39:14 i've got nothin' 19:40:25 my kids have a halloween parade thing tomorrow at school. I am not yet sure if I am expected to go view that 19:40:31 but I may be afk for an hour or two tomorrow for that 19:40:51 fungi: ^ did the storm take precedence over halloween planning? 19:41:50 I'll give this until 19:45 UTC and if we don't have anything to talk about by then I'll end the meeting 19:42:16 yeah, we sort of didn't decorate much 19:42:36 gonna chill with some friends and frighten/rot the teeth of children in their neighborhood this year 19:44:57 oh! we get off DST this weekend in this country at least 19:45:14 oh - is that this weekend? 19:45:18 friendly reminder that meeting times are UTC so if your local government observes DST you may have the meeting shift by an hour 19:45:26 yeah, i need to skim my recurring schedule for any meetings which shift relative to dst 19:45:37 or rather which follow dst and so shift in utc 19:46:02 the meeting will be an hour earlier in my local timezone next week 19:46:24 and with that I think I'll call the meeting. Thank you everyone 19:46:28 #endmeeting