19:01:23 #startmeeting infra 19:01:23 Meeting started Tue Mar 31 19:01:23 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:24 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:26 The meeting name has been set to 'infra' 19:01:28 #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006616.html Our Agenda 19:01:59 o/ 19:02:07 o/ 19:02:15 #topic Announcements 19:02:22 no new announcements this week 19:02:51 #topic Actions from last meeting 19:02:57 osf "community meeting" conference call tomorrow 19:03:05 er, thursday 19:03:13 #undo 19:03:14 Removing item from minutes: #topic Actions from last meeting 19:03:27 * fungi can't remember what day of the week it is at the best of times 19:03:39 fungi: thats the high level overview one right? then the one where meetpad is particularly relevant is april 2, 4 6? /me looks for that email 19:03:51 yeah 19:04:12 #link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013699.html 19:04:44 2, 6, 7 for time zone coverage on PTG brainstorming 19:04:53 details in that email (which was sent broadly 19:05:08 #topic Actions from last meeting 19:05:16 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-24-19.01.txt minutes from last meeting 19:05:33 fungi took actions to create the new opendev meeting channel and a service-discuss mailing list 19:05:44 those changes are up for review at gerrit topic:opendev-comms 19:05:44 thanks, fungi! 19:06:24 the bikeshed decisions there were to call the new ml "service-discuss" and the new irc channel "opendev-meeting" 19:06:39 and also whether to include statusbot in the new irc channel 19:06:44 and we've got a topic a bit later to discuss what that means for the future of this meeting (as well as other things) 19:07:04 it might fight with meetbot over channel topics, but i expect our uses of the two won't overlap sufficiently in there for it to become an issue 19:07:49 ya I expect it will be fine and if the topic gets out of sync many channel lurkers will be able to update it 19:07:53 so, opendev-meeting is only us - not a public place like #openstack-meeting? 19:08:31 o/ 19:08:34 AJaeger: its public in that anyone can join but the topic/theme of the channel will be focused on opendev meetings and incident management 19:08:45 AJaeger: it won't host arbitrary meetings 19:09:12 do we think there might be a desire for a general meeting channel? 19:09:28 it seems that more and more projects are choosing to host meetings in their own channels 19:09:39 maybe we just wait until someone asks if they can use it, and otherwise don't worry about it? :) 19:09:39 it seems like even in openstack the desire for a central meeting channel has faded 19:09:41 clarkb: ok, we're on the same page - then, another name might be better since it's a different policy than #openstack-meeting 19:09:45 corvus: ya thats what I am thinking 19:10:27 * fungi is happy to update those changes if someone has a better name, but maybe we can talk about it during the corresponding meeting topic 19:10:34 ya lets continue 19:10:41 #topic Specs approval 19:10:49 we landed the meetpad spec and work there has progressed! 19:11:14 I wanted to call out the xwiki spec had a merge conflict as a result and could use some trivial reacks (though I think I'm comfortable approving it without those if people don't want to double check the erbase) 19:11:19 #link https://review.opendev.org/#/c/710057/ xwiki for wikis 19:12:19 mostly just a heads up there that a new ps has arrived and votes weren't preserved, but I'd like to not hold that back for too long 19:12:25 #topic Priority Efforts 19:12:32 #topic Update Config Management 19:12:43 There are exciting updates to our gerrit with docker changes 19:12:51 heck yes 19:12:53 mordred: ^ would you like to fill us in? 19:13:08 * diablo_rojo sneaks in late 19:13:12 manage-projects is now mostly working (there's one recent glitch- but we think that was related to a weird manual run) 19:13:14 but ... 19:13:16 drumroll 19:13:27 we're now triggering manage-projects with zuul and not from cron! 19:13:39 \o/ 19:13:45 which means project-config changes for project creation will get applied immediately 19:14:06 ya we've seen ~2 repos be weird, but possibly they are just side effects from when things were broken 19:14:07 and - it's showing that our structure for doing infra-root playbooks works - so we shoudl be able to peel off the rest of run_all in pretty short order 19:14:23 we should fix them (probably manually after reading jeepyb) then continue to monitor subsequent things 19:14:27 yeah- we're also going to have web-accessible logs - so people should be able to see when something is being weird 19:14:38 I agree - I think they are manual pushes 19:15:27 but we should start actually seeing logs from project creation on the promote job on the p-c patch - so we should know when stuff goes poorly 19:15:35 ++ 19:15:43 once that patch also lands, right? 19:15:45 in fact- once this is solid ...we might want to have jeepyb start exiting 1 19:15:51 another thing we should probably call out is our use of allowed-projects and semaphores? 19:15:59 https://review.opendev.org/#/c/716376/ <-- collecting logs 19:16:19 mordred: why have it exit 1? 19:16:31 exiting 1 on a creation failure 19:16:32 corvus: because that way we'll see red on the job to create the projects 19:16:37 yeah - when something fails 19:16:52 ack 19:16:54 since it'll be running in response to file changes rather than just on a cron 19:17:10 yeah - so this: https://review.opendev.org/#/c/716359/ is adding a semaphore since we're also triggering this from project-config 19:17:31 though that raises the question, when a change creates multiple repos and one breaks do we want it to keep going or just short-circuit and fail 19:17:32 we don't need to do it on all of our infra-prod jobs - but we do when it's a job that could get triggered by more than just system-config 19:17:38 with goaccess we run periodically and have a job timeout that is less than the day period for periodic jobs. that means we largely avoid any overlapping jobs. In general though I think these zuul applied ansible playbooks will want semaphores as production isn't throw away test isntances with overlap :) 19:17:43 so - a thing for opendev folks to keep in mind 19:17:56 fungi: I think keep going - but report that there was an error 19:18:01 mordred: ya thats the key is if it can be enqueued more than once (supercedent promote pipeline avoids the issue with single repo case) 19:18:11 yup 19:18:12 so stash error conditions and then summarize at the end 19:18:19 yeah. well - it'll alreayd log them 19:18:37 so the behavior is currently right - we just want to flag that there was an issue somewhere and the exit code should be non-zero 19:18:42 everything else should work great 19:19:04 right, that's what my poorly-constructed sentence was attempting to convey 19:19:14 thanks for confirming 19:19:14 fungi: sentences are english hard being 19:19:52 is there anything else to call out on this subject? are we running gerrit in a container yet? 19:20:55 no - we still need to do a restart 19:21:01 gerrit2 95054 186 38.0 65169020 23520072 ? Sl Mar20 30043:21 GerritCodeReview -Xmx48g -jar /home/gerrit2/review_site/bin/gerrit.war daemon -d /home/gerrit2/review_site --run-id=1584714516.94871 19:21:04 nope 19:21:22 and there's still a cut list on the etherpad to followup on 19:21:26 gerritbot is next on my list 19:21:35 then there's some utility cron jobs and stuff 19:21:47 do we have a link to that etherpad? 19:21:58 the plan for gerritbot is to move it to eavesdrop as part of this, right? 19:22:08 https://etherpad.openstack.org/p/gerrit-2020-03-20 19:22:10 or was that going to be a later step? 19:22:16 fungi: yeah- well - first step I'm containerizing it 19:22:21 got it 19:22:30 #link https://etherpad.openstack.org/p/gerrit-2020-03-20 gerrit on docker todo list 19:22:31 then yeah- since it's new ansible - might as well do it on eavesdrop 19:22:55 i think i suggested that eavesdrop might be a better long-term home for it, but it's not critical, so whatever ends up being easier :) 19:23:11 yeah - I thnik it'll be the exact same amount of work since it's completely net-new ansible 19:23:44 eavesdrop or eavesdrop replacement if we want to start migrating the other things there to bionic, i don't suppose it matters 19:24:34 ya we can probably figure out what makes the most sense as it gets ansible dockered 19:25:00 ++ 19:25:23 ianw: you've also been pushing on the nodepool builder on docker and fedora-31 images. Does it make sense to bring that up here? 19:27:01 * mordred just +A'd the nodepool change to bump dib 19:27:41 probably the biggest thing to be aware of there is that fedora31 images will be joined fedora30 images on the new nb04.opendev.org builder running on docker 19:27:41 umm, if that bump is in, then we can merge the configs and i can try things out today 19:27:43 also - if people haven't seen the new python-stow element they should look at it - it's super cool and is, I think, a great pttern 19:28:18 if that's all going well, we can think about migrating some of the other builds to the container builder, and eventaully replace the other builders 19:28:24 ++ \o/ 19:28:37 then we can think about containerising the launchers 19:28:39 exciting 19:29:38 #topic OpenDev 19:30:26 I've had a think on what the formalization of the opendev leadership and advisory board next steps are and I think we should get the new comms channels up (particualrly the mailing list for this) then use that as a venue to solicit volunteers 19:30:56 that way the history of the thing will be preserved in a place where people are likely to go looking for it in the future 19:31:31 in addition to that I think it would be great if we can have our next meeting in the new channel 19:31:31 i concur 19:32:00 ++ 19:32:24 so if you've got a moment it would be great to brainstorm the channel name (here for a few minutes or on the change(s)) 19:32:47 however, I expect the mailing list addition is less contentious and maybe we can land that one more quickly 19:33:27 what was the other suggestion, #opendev-annex? 19:33:45 fungi: or -auxilliary (except its hard to spell and I'm sure I got it wrong there) 19:34:02 can you post the links to the reviews for reference? 19:34:07 annex is easy to spell and short I'm on baord with that too fwiw 19:34:10 i'm sure we'd just shorten that to #opendev-aux anyway 19:34:31 #link https://review.opendev.org/#/q/topic:opendev-comms Opendev comms changes 19:34:42 #link https://review.opendev.org/715972 Add a service discussion mailing list for OpenDev 19:34:43 fungi: ^ 19:35:14 ahh, thought he wanted them individually linked, but sure 19:35:23 #undo 19:35:25 er that was for frickler but tab complete failed :) 19:35:31 individual links are fine too :) 19:37:02 then for the actual leadership position we can make a call for volunteers on the new mailing list and if we have more than 1 run an election among opendev contributors. I'm personally happy to continue in this role, but would also be happy to see others take it on as well :) 19:37:59 if we're worried about folks mistaking the #opendev-meeting channel for a general meeting space, then we can likely just clarify its focus in theh /topic 19:38:18 this all sounds good to me; i don't have a preference on the secondary channel name 19:39:01 yeah, i'm not going to bikeshed over the name but if folks want to propose alternatives i'm fine with us entertaining our options 19:39:34 i gave it some thought before i pushed the patch and couldn't come up with anything better either, fwiw 19:39:35 fungi: that sounds like a plan. AJaeger ^ perhaps you can suggest an alternative and we take it from there? 19:40:16 you should -w the patches if you want further discussion on them, I almost approved them this morning already 19:40:19 * AJaeger is bad at naming 19:41:15 I think the mailing list can land as is, I've not heard any concerns about that one, then maybe -W the meeting channel change for now in order to continue discussion on name options in gerrit? 19:41:17 fungi: ^ 19:41:36 clarkb, fungi, give it two days, if no names come up, we merge... 19:41:52 AJaeger: wfm 19:42:29 That takes us to the last opendev item I had on the agenda. A heads up about improving documentation to give peopel ending up at https://opendev.org a bit more context on where to go next 19:42:31 sure, i can wip the system-config change for the irc channel 19:42:57 er, i guess the project-config change actually as that's the earlier onw 19:43:01 so far we've added a getting started link to the links on the nav bar in gitea. This takes you to the opendev infra manual getting started doc which is a simplified version of the dev docs 19:43:45 I expect there is more we can do in this space, whether that is improving themeing of the docs to feel more "cohesive" or updating content to better address the neds of people arriving from gitea 19:44:23 that sounds great :) 19:44:43 oh also we removed the openstack theme from the infra manual 19:44:57 it uses the alabaster sphinx theme now with an opendev logo 19:45:33 definitely interested to hear feedback on this as I think one issue I suffer from is familiarity with the system means I gloss over the docs :) 19:45:48 ideas are good! 19:46:39 before we move on, I'm not sure if it was clear before but next week I'd like to have the opendev meeting in the new channel but will continue to keep the existing time slot 19:46:50 I'll be sure to send out specifics when I send an agenda out 19:46:58 #topic General Topics 19:47:19 fungi: anything to report on wiki? I know you indicated you would try to keep updating the puppet as a guide for potential replacement tooling 19:48:23 nope 19:48:31 not yet anyway 19:48:59 #topic Open Discussion 19:49:05 Anything else? 19:50:26 I'm excited about the progress we've made over the beginning of the yaer. Feels like a lot of the plans and ideas we've hard are coming together 19:50:32 ++ 19:50:45 also I can't type 19:50:52 typing is hard 19:51:35 * fungi types harder 19:51:42 what are the next steps on meetpad? any help needed there? 19:53:08 yeah, it's up and running, but we found in testing that the etherpad function wasn't working in some (maybe even most) cases 19:53:30 corvus: mark tried ti and discovered there isn' 19:53:32 it was returning errors, so it's pretty obvious when it doesn't work 19:53:32 er 19:53:39 isn't an http -> https redirect? we may want to add that too 19:53:43 clarkb: correct 19:53:47 and http doesn't work at all 19:53:51 so we need to add that 19:53:57 that should be easy 19:54:14 the hard part is figuring out what's (sometimes) broken about etherpad 19:54:47 this is presumably also newer etherpad than on our existing etherpad.o.o 19:54:51 so if anyone wants to visit it (via https) and see if they get an error from etherpad, and see if they can figure out what the problem is, that would be great 19:54:59 fungi: no, it is our existing etherpad.o.o 19:55:08 that's the main attraction of the service 19:55:14 oh, okay 19:55:21 one idea that came up on friday was maybe we need to update etherpad though 19:55:22 so not a separate containerized etherpad deployment 19:55:26 correct 19:55:37 jitsi is passing flags that our etherpad doesn't seem to honor so it may have other integration points that we need to accomodate 19:55:41 we probably do need to update etherpad anyway 19:55:57 clarkb: true, but this is only a sometimes failure 19:55:58 however, worth keeping in mind that they've also changed the theme in etherpad recently 19:56:03 so it's going to look a lot different 19:56:08 corvus: ya I doubt it will fix the underlying problems 19:56:58 o.k., I'll try to take a look tomorrow 19:56:59 etherpad-dev.o.o is offline too, i can take a look at that after the meeting 19:57:06 it's possible, but since it sometimes works and sometimes fails, it means that if the problem were to be solved by an etherpad upgrade, it'd have to be some kind of stabilization or race thing, not a straightforward incompatibility. 19:57:42 the good news is that our deployment mechanism worked flawlessly the first time. including LE certs. :) 19:58:03 we're getting good at this 19:58:06 though due to a bug i introduced in the config, it's currently in the emergency file until https://review.opendev.org/715572 lands 19:58:51 that change has now been approved 19:59:22 and we are just about at time. 19:59:33 Thanks everyone! feel free to continue conversation on irc or the mailing list 19:59:43 and we'll see you next week hopefully in the new location! 19:59:52 #endmeeting