19:01:09 <ianw> #startmeeting infra
19:01:09 <openstack> Meeting started Tue Mar 26 19:01:09 2019 UTC and is due to finish in 60 minutes.  The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:10 <jroll> \o
19:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:13 <openstack> The meeting name has been set to 'infra'
19:01:20 <dtroyer> o/
19:01:33 <ianw> #link http://lists.openstack.org/pipermail/openstack-infra/2019-March/006298.html
19:01:50 <ianw> #topic Announcements
19:02:21 <ianw> clarkb remains AFK, hence me # tagging this one
19:02:29 <cmurphy> o/
19:02:44 <ianw> no other announcements
19:02:56 <ianw> #topic Actions from last meeting
19:03:06 <ianw> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-03-19-19.01.html
19:03:23 <ianw> clarkb was to ask about infra ptg days being thursday friday
19:03:33 <ianw> anyone know if that happened?
19:03:54 <corvus> i think i remember an irc message suggesting that it did
19:04:03 <frickler> I seem to remember that that was confirmed, yes
19:04:08 <fungi> i believe an updated ptg schedule has been published for feedback which has our time moved
19:04:12 * fungi looks
19:04:44 <corvus> the published published one doesn't appear to have been updated: https://www.openstack.org/ptg/#tab_schedule
19:05:00 <fungi> #link http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190321/3a3989bb/attachment-0001.png
19:05:22 <fungi> from
19:05:28 <fungi> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004122.html Strawman Schedule
19:05:47 <fungi> that seems to show infra/qa sharing a room thursday and friday
19:06:34 <ianw> ok, i think let's leave an action item to just make sure everything published comes into sync?
19:06:39 <fungi> diablo_rojo_phon probably has to let jamesmcarthur know when it's settled so the official one can be updated
19:07:12 <ianw> #action clarkb double check infra ptg days published as thu/fri
19:07:14 <jamesmcarthur> correct :)
19:08:08 <ianw> ok, all action items accounted for
19:08:22 <ianw> #topic Specs Approval
19:08:34 <ianw> I don't think there are any specs for approval at this point?
19:09:22 <ianw> #topic Priority Efforts
19:09:54 <ianw> cmurphy: i saw a few more puppet4 changes going out, just a matter of review at this point?
19:10:14 <cmurphy> ianw: pretty much
19:10:31 <cmurphy> clarkb caught up with me already so just proposed a few more
19:10:38 <fungi> there was some fiddling over the weekend with the pip provider to work out getting gear upgraded on logstash01
19:10:51 <fungi> but cmurphy got that sorted out
19:10:56 <cmurphy> yay
19:11:18 <ianw> oh again, more things related to the change in output of pip?
19:11:32 <cmurphy> the same issue as before
19:11:36 <cmurphy> we just hadn't merged that fix
19:11:41 <cmurphy> till last weekend
19:12:14 <ianw> oh right ... heh plenty of time for pip to have changed itself again in between :)
19:12:21 <cmurphy> heh
19:12:32 <ianw> #topic Opendev
19:12:53 <ianw> i did send out the git -> https changes as discussed in last few meetings.  no real issues
19:13:24 <ianw> a few weird things with some repos not having .gitreview, and some missing zuul branch configs, both of which i didn't code for, but otherwise ok
19:14:00 <ianw> jroll: want to update on where the renaming thread is at?
19:14:17 <fungi> (for openstack)
19:14:27 <jroll> ianw: it seems we have consensus on the plan, I need to spin a new version of the governance patch
19:14:48 <jroll> it should be straightforward to make a list of the renames from there
19:15:10 <ianw> what's that review # for a link?
19:15:15 <jroll> (spoiler: we're going with option #2, all unofficial projects in one namespace, all official projects in the openstack namespace)
19:15:23 <jroll> #link https://review.openstack.org/#/c/645601/
19:16:17 <corvus> sounds good (though unofficial projects can still self-select their own org at https://ethercalc.openstack.org/opendev-transition )
19:16:31 <fungi> well, all unofficial projects who don't request their own namespace from opendev before the maintenance in one namespace anyway
19:16:38 <fungi> er, or what corvus said
19:16:41 <jroll> sorry, yes, that
19:16:45 <corvus> we have some takers there now (3 people!)
19:16:49 <jroll> we'll make an attempt to tell them about it :)
19:16:59 <corvus> jroll: much obliged! :)
19:17:10 <ianw> #link https://storyboard.openstack.org/#!/story/2004627
19:17:24 <corvus> but yeah, happy to stick anyone who doesn't respond into "unknown/"
19:17:33 <corvus> or whatever (or whatever/) :)
19:17:59 <ianw> so it's fungi signed up to do the modrewrite & gitreview update implementation of the renames once this is finalised?
19:18:41 <fungi> yes, an initial stab at incorporating corvus's test redirects is already in place on files.o.o and covers git.openstack.org urls
19:18:52 <fungi> i'm still working on the variant for the other git sites
19:19:16 <fungi> but all the vhosts are configured there now with their ssl certs/keys installed
19:19:45 <ianw> cool, well that all seems like it's moving along
19:19:49 <corvus> i'm planning on testing whether zuul itself follows those redirects for the git driver later today/tomorrow
19:19:57 <fungi> for the git-review and zuul configuration patching, i had a quick question...
19:20:03 <corvus> ianw: re task 29707 -- i asked fungi for a cert for docs.opendev.org earlier; he went ahead and got review.opendev.org too.  if the LE stuff is ready, great, and if it's not, we can use the comodo.
19:20:13 <fungi> particularly on the zuul configuration itself
19:20:55 <ianw> corvus: it's very close to reviewable; i just finished yesterday all the implementation bits i can think of.  will beat into shape with docs etc and request reviews in next day or so
19:21:04 <fungi> is it sufficient to just patch the contents of {.,}zuul{,.d/*}.yaml or do we also need to try to update any references to checked-out paths in playbooks and roles?
19:21:06 <corvus> ianw: w00t
19:22:16 <frickler> fungi: well, there are things like this https://opendev.org/openstack/openstack-chef/src/branch/master/playbooks/integration.yaml#L6
19:22:17 <corvus> fungi: that's a good question ... if we don't update the playbooks/roles, i feel certain that some things will break; however it's understandable we may want to reduce the complexity there and leave that for humans...
19:22:44 <frickler> can we make zuul create symlinks for the old path?
19:22:59 <frickler> similar to the redirects?
19:23:27 <ianw> fungi: you mean things like "upper_constraints: "{{ ansible_user_dir }}/{{ zuul.projects['git.openstack.org/openstack/requirements'].src_dir }}/upper-constraints.txt"
19:23:29 <corvus> frickler: well, there's also the more 'zuulish' way of doing it which is using {{ zuul.projects['git.openstack.org/openstack/openstack-chef'].src_dir }}
19:23:41 <fungi> ianw: frickler: yeah, those examples
19:24:03 <frickler> corvus: will that work better when we s/git.o.o/opendev.org/ ?
19:24:21 <corvus> frickler: no, but it won't be affected by symlinks, so jobs that use that would need a different workaround
19:24:46 <frickler> corvus: ah, right
19:24:51 <ianw> fungi: i feel like that will have to be updated, it will just break otherwise?  if that's zuul.projects keeping an alias or a direct code change ...
19:25:02 <fungi> so i have a feeling there are enough of these cases that my script should also recurse top-level playbooks and roles directories
19:26:11 <fungi> i know zuul itself just expects its configs to have .yaml extensions, but ansible is likely to be more flexible in that regard. does also looking for *.yml in (any child of) those directories cover it?
19:26:12 <corvus> fungi: that's probably not a bad idea.  though playbooks can be anywhere (zuul.yaml will tell you where they are, but of course they could include_tasks in other directories)
19:26:24 <corvus> fungi: generally yes.
19:26:39 <fungi> oh, interesting, so the zuul configuration can set explicit paths to playbooks and roles?
19:26:48 <corvus> fungi: playbooks, yes, roles no
19:27:03 <corvus> roles are either at the top level of the repo or in roles/
19:27:23 <fungi> oh, so they can also be in the top level directory. okay
19:27:44 <corvus> i bet if you just did roles/* and each of the directories holding a playbook, you'd get 99.99%
19:28:25 <fungi> what's the configuration directive i should look for which sets the path to a playbook tree?
19:29:01 <ianw> codesearch for "zuul.projects\['git.openstack.org" actually showed up way less than i thought it would be ... only around 60 matches
19:29:04 <corvus> fungi: the values under 'pre-run', 'run', and 'post-run' are (lists of) paths to playbook files relative to repo root
19:29:41 <fungi> thanks, that ought to be manageable
19:29:44 <corvus> fungi: eg "pre-run: playbooks/my-playbook.yaml" is typical, but could be "pre-run: some/weird/dir/whatever.yaml"
19:30:08 <fungi> and those are relative to the top-level directory of the repository?
19:30:11 <corvus> yep
19:30:17 <fungi> perfect
19:30:34 <fungi> and they have to end in .yaml or .yml?
19:30:41 <corvus> fungi: feel free to ping me with questions or even requests to help write code chunks
19:30:45 <corvus> fungi: .yaml for zuul
19:30:53 <corvus> er
19:30:54 <corvus> 1 sec
19:31:03 <fungi> ansible will let them be anything though, i guess
19:31:13 <corvus> i think zuul.yaml has to be .yaml i think playbooks (even those run by zuul) can be anything
19:31:22 <fungi> yeah, didn't want to monopolize the meeting. i've got enough for a first stab at it
19:31:27 <fungi> thanks!
19:32:07 <ianw> any other updates?  i think dtroyer was all good with your bits last week?
19:32:26 <dtroyer> yup
19:32:57 <ianw> #topic Storyboard
19:33:22 <ianw> any updates?
19:33:44 <fungi> the missing support for swift form-post signatures in openstacksdk has been proposed
19:34:14 <fungi> #link https://review.openstack.org/639760 (openstack/openstacksdk) Add support for generating form-post signatures
19:34:30 <fungi> this was a blocker for secure implementation of the story attachments feature
19:35:45 <fungi> other than that, lots of pending changes to review
19:36:19 <fungi> yesterday was the proposal deadline for outreachy candidates, so there was quite the flurry of last-minute activity
19:36:32 <fungi> (or maybe the deadline is today)
19:37:19 <ianw> cool, do the reviews require a lot of web-architecture pre-knowledge?  or is it pretty standalone?
19:38:05 <fungi> there's a bit of a split. the storyboard repo holds the api implementation in python
19:38:39 <fungi> the storyboard-webclient repo holds the web front-end and is mostly javascript, using an oldish angular framework
19:39:28 <fungi> luckily we also have draft builds of the webclient rendered in ci jobs, so you can see whether or not they work and/or if they introduce unexpected behavior
19:40:33 <fungi> i don't know whether diablo_rojo_phon or SotK are around and have anything to add
19:41:01 <diablo_rojo_phon> Nothing from me I don't think.
19:41:09 <diablo_rojo_phon> Oh.
19:41:19 <diablo_rojo_phon> Outreachy application deadline was today
19:41:32 <fungi> though we also have a storyboard irc meeting most weeks this time on wednesdays, if anyone wants to continue more in-depth discussions about such things (as well as any time in #storyboard)
19:41:34 <diablo_rojo_phon> So the flow of potential new contributors will be ebbing now
19:42:10 <fungi> yeah, the channel was full of folks trying to figure out how to install storyboard and run local tests for the past month or more
19:42:32 <fungi> but a big plus is that helped to fine-tune some of our developer docs and workflow
19:43:24 <fungi> anyway, that's all i've got for now
19:43:52 <ianw> #topic General Topics
19:44:12 <ianw> #topic PTG planning
19:44:34 <ianw> just wanted to stamp the link in again
19:44:38 <ianw> #link https://etherpad.openstack.org/2019-denver-ptg-infra-planning
19:45:13 <ianw> i'm sure ideas, questions and comments are welcome
19:45:58 <ianw> we seem to be free of other general topics, which must mean that everything is perfect and we can all just go to the beach
19:46:20 <frickler> we had nb nodes run full again
19:46:31 <frickler> possibly related to suse breaking their repos
19:46:31 <ianw> #topic Open Discussion
19:47:05 <ianw> frickler: oh right, is the short version of the unbound issues that it was a super old package without the latest dnssec keys due to the mirror going away?
19:47:42 <frickler> ianw: yes, except that it is still broken, mirror updates hopefully happening soon
19:48:32 <frickler> also it wasn't mirror going down, but repo actively being published in an empty state iiuc
19:49:17 <ianw> cool ... that explains why it looked so similar to the actual brokenness we had due to dnssec being stripped by opendns ... red herrings everywhere!
19:49:38 * fungi wants a pickled herring sandwich now
19:49:53 <frickler> yes, the effect was the same, unbound failing to validate any dnssec response it receives
19:50:03 <frickler> returning servfail to the clients
19:50:08 <ianw> i think that a cron job on the nb* to clear out +x day files on /opt/dib_tmp might help slightly
19:50:38 <ianw> but also, i had to clear out a few builds that were on disk, but not in "nodepool image-list"
19:50:55 <frickler> how much quota do we have for volumes? maybe adding like 1tb to each would also help
19:51:06 <frickler> so we don't fill it up within a few days
19:51:29 <fungi> nb01 and nb02 have a 1tb volume each
19:51:52 <fungi> nb03 is a harder one to solve since there's no cinder there, and instead we need to ask for a flavor with a larger rootfs
19:52:20 <fungi> though cleanup got utilization on those volumes down to something like 50% anyway
19:52:29 <fungi> not sure how much headroom we really want to provide
19:53:25 <frickler> I think if we had space for like 1-2 weeks of builds, that would allow us to easier react on issues
19:53:25 <corvus> aiui Shrews is looking into nodepool leaks
19:54:49 <ianw> frickler: i think that a background cleanup job might provide a little more breathing space, for a small error
19:54:49 <fungi> yeah, there was something about making sure nodepool removes the on-disk files corresponding to failed builds?
19:55:13 <fungi> rather than just leaving them lying around
19:56:45 <ianw> ok, we're just about at time, as usual discussion can continue in #openstack-infra
19:56:49 <frickler> does anyone have a simple recipe to monitor /opt space in cacti? otherwise I can look into that
19:57:33 <ianw> frickler: as in something to alert us?
19:58:07 <frickler> ianw: as in being able to track usage. currently we only have / unless I looked wrong
19:58:10 <fungi> as in right now /opt isn't tracked in cacti for the nodepool builders
19:58:35 <ianw> oh right, yeah that would be a start
19:58:39 <corvus> i thought the script did that.  but if it's not working, it's just some clicking
19:59:25 <fungi> i think it will add any filesystems which are present when the server is first added
19:59:44 <fungi> but if new filesystems are added after the config has already been generated, that doesn't get updated
19:59:45 <corvus> i guess we stopped running the script all the time?
20:00:02 <corvus> or, rather, we have it ignore hosts that are already there?
20:00:16 <fungi> sounds likely, but i haven't looked deeply
20:00:20 <ianw> what's "the script"?
20:00:35 <corvus> maybe there's an option to run it once for that host manually
20:01:01 <ianw> i think that tracking it explicitly would be great, and we can then correlate when it starts to climb over the long term
20:01:14 <ianw> frickler, want an action item to look into this?
20:01:16 <corvus> i have no idea where the script ended up :)
20:01:30 <frickler> ianw: yeah
20:01:49 <corvus> https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/cacti.pp#n64
20:01:51 <fungi> #link https://opendev.org/openstack-infra/system-config/src/branch/master/modules/openstack_project/manifests/cacti.pp#L72-L76 Exec[cacti_import_xml]
20:02:04 <ianw> #action frickler look into disk usage tracking, particularly of /opt on nodepool builders
20:02:12 <corvus> it's the create_graphs.sh script
20:02:34 <fungi> oh, nevermind, yeah it's what corvus linked
20:02:38 <corvus> https://opendev.org/openstack-infra/system-config/src/branch/master/modules/openstack_project/files/cacti/create_graphs.sh
20:02:40 <corvus> there it is
20:02:46 <corvus> #link https://opendev.org/openstack-infra/system-config/src/branch/master/modules/openstack_project/files/cacti/create_graphs.sh
20:03:18 <fungi> and the cron at line 87 in the manifest runs it
20:03:28 <frickler> o.k., I'll take a closer look tomorrow
20:03:37 <corvus> hrm.  that sort of looks like it should be running and idempotent
20:03:41 <corvus> so there maybe a bug there
20:03:53 <corvus> apparently, there's a log file :)
20:04:00 <ianw> ok, i gotta dash.  i'll end meeting here and we can pick this up later
20:04:02 <ianw> #endmeeting