19:01:23 #startmeeting infra 19:01:24 Meeting started Tue Apr 3 19:01:23 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:27 The meeting name has been set to 'infra' 19:01:52 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:15 #topic Announcements 19:02:31 \o 19:02:33 Zuul v3 released last week after much hard work by many in this group and others 19:02:53 so raise a glass of your favorite beverage if you haven't already done so (or do it again!) 19:02:53 thanks everyone! 19:02:58 [and there was much rejoicing] 19:03:06 congratulations 19:03:57 i will be happy to raise a drink again if it be the wish of this group 19:03:58 The OpenStack Summit and the colocated OpenDev (with focus on CI/CD) events are coming up in about 6 weeks or so. Apparently laws/regulations that affect airbnb just chagned in vancouver, BC so double check if you planned to stay with them 19:04:33 (I don't actually know how big an impact that will have but it was mentioned by someone that we should warn conference attendees to double check) 19:05:24 anything else we want to announce before digging into the agenda proper? 19:05:26 corvus, yes, do 19:06:24 it's been suggested that as opendev is a more grassroots sort of event, encouraging people to pass the word along to their respective professional circles of non-summitgoers about it would be good 19:07:12 right, we hope to see you and your friends there, I expect it will be a good time 19:07:25 #topic Actions from last meeting 19:07:26 lots of collaborative space/time scheduled 19:07:37 oh should I undo to keep talking opendev? 19:07:55 nah 19:08:04 let's keep this bus in drive 19:08:04 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-27-19.03.txt 19:08:26 I wasn't able to follow lsat week's meeting due to needing to have lunch on tight conf schedule but I don't see any actions after a quick grep 19:08:42 #topic Specs Approval 19:09:00 There are a couple updates to existing specs that I'd like to get in soon. 19:09:01 I agreed to post a patch, which I did: https://review.openstack.org/#/c/557979 19:09:16 and anteaya ^ has patch for survey spec 19:09:26 * clarkb linsk them all via meetbot 19:09:35 thanks 19:09:41 #link https://review.openstack.org/#/c/557772/ mark zuul v3 spec as done 19:09:51 oh, yeah i should have mentioned to you that i put the "Amend top-level project hosting spec" change up for council vote 19:10:03 #link https://review.openstack.org/#/c/555104/ Amend minor detail on git repo hosting for top level project hosting 19:10:16 #link https://review.openstack.org/#/c/557979 implementation of survey spec 19:10:22 (by being a poor stand-in meeting chair i help ensure i won't have to do it often!) 19:10:56 fungi: heh its fine. I was already planning on moving ahead with that. Do we feel like the zuul v3 spec being marked done needs to go through formal council vote? if so are we good with giving that until thursday afternoon pacific time? 19:11:24 I'll likely go ahead and get the minor top level project hosting amendment in today since it was put up for council vote last week 19:11:40 i'm ambivalent on whether shuffling entries around in the index file needs formal voting 19:11:55 I have no opinion 19:11:59 I'm hopign that the presence of the 3.0.0 tag on zuul makes that one non controverial 19:12:09 yup 19:12:11 corvus: ^ do you think it needs any more review than it has received? 19:12:14 it has my vote, at any rate 19:12:22 nah 19:13:02 alright then I'll work toget those approved after the meeting. And anyone interested in survey tooling should review that third change from anteaya 19:13:06 once it merges, we're down to one long-standing priority spec 19:13:22 which takes us to the next agenda item 19:13:30 #topic Priority Efforts 19:13:56 I've put on the agenda that now might be a good time to reevaluate current efforts and decide if any of them should be bumped up to priority effort status. 19:14:03 for our one remaining priority effort, i finally got the 4-byte utf-8 support into production over the weekend and successfully imported stories for tripleo-ui 19:14:10 fungi: nice 19:14:24 fungi, congratulations 19:14:35 my next big task there is figuring out why gerrit commenting on stories broke around the time we upgraded to gerrit 2.13 19:14:59 gerrit paid attention to my new patch 19:15:03 fungi: as the person that helped drive the 2.13 upgrade my failure to catch that means I'm happy to help debug it now. Let me know if I can be of assistance 19:15:13 it changed the status when I uploaded 19:15:26 sorry storyboard paid attention* 19:15:28 yeah, i've noticed status tracking working 19:15:31 it probably involves me making sure pabelanger is done testing things with review-dev01 so i can try some hunches 19:15:42 what specifically is broken? 19:16:11 fungi: I have not yet, just gerrit testing 19:16:11 when you add a story footer in your commit message, the its-storyboard plugin for gerrit is supposed to leave a comment on the story with a url to the change 19:16:29 oh, so it's changing task status but not leaving words 19:16:34 yep 19:16:34 ah 19:16:41 yes, there were no words 19:16:47 fascinating failure mode 19:17:00 indeed. and i'm hoping it won't involve a foray into java 19:17:07 i'm accustomed to things breaking much more completely than that :) 19:17:33 there's a possibility it's just a side effect of changes in how we're doing commentlinks in the gerrit configuration since its plugins seem to rely on that to infer some configured behaviors 19:17:46 and those did change with 2.13 19:18:02 zaro__ gave me a few suggestions to dig into there 19:18:26 anyway, we've had some projects say they want to hold off migrating to sb until that's fixed 19:19:14 and i really didn't want to monopolize the priority efforts discussion, so let's get on with discussing adding some new ones ;) 19:19:55 since we are about to remove one of the long standing efforts that has taken a bunch of our time I thought it may be worthwhile to decide if any other in progress efforts (or possibly new ones) should be bumped up to priority status 19:20:26 on the wiki agenda I threw out some ideas. I don't expect we'll decide this today, but wanted people to start thinking about what is going to be important to the infra team for the next few months or so we can take it from there 19:20:49 not to keep bringing up storyboard, but if people do have spare cycles and want to help hack on it, they're very welcome to do so ;) 19:21:04 I don't think the survey spec is a priority or will take that much effort, I care about but I don't think it should be elevated 19:21:10 presumably it's staying on the priority list at least 19:21:14 (sb i mean) 19:21:17 fungi: ya I don't expect we'll remove it 19:21:31 agree, we still have some trusty systems we need to move to xenial that we didn't get to last virtual sprint 19:21:42 would be open to having another sprint to finish that off 19:21:47 I just wanted to spark the conversation and idea here but will probably move discussion to the infra ml so that we can get broader reach and input and decide together there 19:21:51 i do agree with the things on the suggested list 19:22:01 +1 19:22:13 i'd _love_ for someone who isn't me to be interested in picking up the wiki work i dropped on the floor 19:22:27 because clearly i'm not doing well at getting around to it 19:22:46 so start thinking about it and I'll get a thread started shortly where we can hear all the great ideas 19:23:00 but i'm very happy to provide a high-bandwidth sync up of where it's gotten to for anyone who's interested 19:23:22 fungi: thanks! 19:24:10 #topic General Topics 19:24:29 I'm going to flip the order on these two because one is time sensitive 19:24:50 pabelanger has spun up a new review01.openstack.org running xenial which means we now have IP addresses we can warn people about changing to in the future 19:25:00 I think the last remaining question is the scheduling of that server migration 19:25:10 indeed 19:25:13 pabelanger: have you asked the release team if R-17 works for them? 19:25:38 clarkb: yah, looks like kubecon EU is that week and all of the release team is going to it 19:25:47 pabelanger: cool I guess that makes it a perfect time? 19:25:58 yah, I think so 19:26:06 infra-root does May 02, 2018 work for migrating the gerrit server? 19:26:11 I can eb around for that myself 19:26:19 sounds fine. i have high hopes it'll be a brief outage. mostly just mechanical stuff like moving the cinder volume, changing dns and stopping/starting services 19:26:26 fungi: yup 19:26:34 yah, https://etherpad.openstack.org/p/review01-xenial-upgrade is the steps I am expecting 19:26:36 It's sufficiently early that I can suppose I'll be around 19:26:43 which was done on review01-dev 19:26:47 pabelanger: assuming it works for you and you've got fungi and I on board and now dmsimard I think we can commit to that date 19:26:58 yeah, i'm happy to help with it 19:27:08 clarkb: ++ 19:27:24 I'll start spamming the ML then with info 19:27:44 2000UTC is the current proposed time on the therpad for the ml email 19:27:50 which wfm 19:27:53 #link https://etherpad.openstack.org/p/review01-xenial-upgrade Ubuntu Xenial move for production Gerrit 19:28:10 wfm as someone who can jump in later if required 19:28:19 pabelanger: sounds good, thanks for putting this together. One fewer gerrit todo on the list marked off 19:28:46 np! 19:28:46 just to confirm, may 2 is a wednesday 19:28:54 yes 19:29:09 clarkb: may 2 wfm 19:29:31 2000z is right after the end of the storyboard meeting, so looks fine as far as my availability is concerned 19:29:38 #agreed Upgrade review.o.o's control plane server to Xenial at 2000UTC May 2, 2018 19:30:07 p.s. it might be worth looking at https://review.openstack.org/#/c/555633 before this, so that the gerrit jobs start working again, in case a new .war or something is required 19:30:16 ianw: ++ thanks 19:30:33 ack 19:30:35 #link https://review.openstack.org/#/c/555633 Get the gerrit jobs working again 19:31:12 The next topic is dmsimard's. Our meetbot can be abused during times of internet trolling 19:31:23 yeah 19:31:44 in particular On April 1st which happened to also be Easter there was at least one webclient nick starting meetings and calling them maintenance then telling people in channel they all had to leave 19:31:44 It's probably the same person that has done the few occurrences I've seen 19:31:59 to be fair we've known about this for years, and yesterday was the first time i've ever seen it happen in the wild (and consider the date) 19:32:23 fungi: I think it has happened at least once before with this startmeeting and tell people to leave story 19:32:25 I didn't realize the date -- I was mostly concerned about someone potentially putting up offensive topics and stuff like that. 19:32:27 we could revert https://review.openstack.org/505669 and ask people to opt-in allowing meetbot to set topics 19:32:39 they hit a couple dozen of our channels, and i managed to get the vast majority of those cleaned up a little over an hour later when the meetings expired for them 19:33:07 Can meetbot ensure they are logged in? 19:33:26 er, i guess not yesterday it was sunday. anyway, april 1 19:33:36 * fungi has lost all sense of time any more 19:34:04 corvus: I think one of the problems with that is meetbot is also our channel logger 19:34:14 corvus: so even if people don't want meetings it is desireable to have logs? 19:34:15 jlvillal: i'm not sure if it supports that, but could probably be added easily 19:34:21 clarkb: that's not what that change does 19:34:28 clarkb: that only allows meetbot to set /topic 19:34:32 ah 19:34:33 And also if the person who starts the meeting leaves then stop meeting. 19:34:46 jlvillal: yeah i don't know if anyone's looked into checking whether their nick has identified with nickserv. we've had suggestions in the past of doing that, or relying on +v in channels, or even whitelisting specific chair nicks in advance for specific meetings. all of these come with certain drawbacks 19:35:13 my personal feeling on it is regardless of the bots in use people can troll public channels and generally be annoying. Addressing this in the bot itself won't change that 19:35:18 jlvillal, I've seen many times where people who chair lose connection 19:35:28 but they want the meeting to continue 19:35:42 revoking +t for the meetbot except in channels where people want to hold meetings (and in fact they can still do that even without having it change the channel topic) seems like a low-cost workaround 19:35:52 the project working group is one group I have seen that happen with 19:36:05 which means maybe educating users and having channel local ops may be a good approach 19:36:14 (then ops can kick the offender and they can stop the meeting) 19:36:50 if meetbot supports allowing anyone who is a chanop to #endmeeting at any time, that also might be a nice alternative 19:36:51 I do wonder how much this behavior is tied to being able to set the topic 19:37:01 fungi: I'm not sure if does but it would make sense as a feature 19:37:07 And you said this meetbot abuse has happened before? 19:37:23 when? 19:37:27 at least once, I think it coincided with another holiday like US thanksgiving or something 19:37:32 sounds like dmsimard and clarkb have seen prior occurrences. those didn't come to my attention 19:37:41 ok 19:37:43 I put up links in the agenda 19:37:56 #link http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-04-01.log.html#t2018-04-01T16:39:49 19:38:02 #link http://eavesdrop.openstack.org/irclogs/%23openstack-operators/%23openstack-operators.2018-04-01.log.html#t2018-04-01T16:59:48 19:38:15 Those are the two I've seen, there may be others. Just happened to walk on those. 19:38:19 yeah, irc vandals seem to like to pop up on weekends and popular holidays when they think channel operators (and probably server operators for the network) will be otherwise occupied 19:38:28 Yeah I saw those logs from Sunday 19:38:32 I've only recently started seeing it more, it is a side effect of 505669 corvus linked above? 19:38:39 Need to step away to pick up kids, be back in a bit 19:38:52 pabelanger: on what other dates have you seen it? 19:38:58 dmsimard: oh, those were both part of the same incident. they hit somewhere between 20-30 of our channels (i stopped counting and just cleaned them up) 19:39:48 corvus: I'd have to go back into logs and see, but thought I remember doing endmeeting in some channels to clear the topic. 19:39:58 may have just been openstakc-meeting too 19:40:06 pabelanger: because someone vandalized it? 19:40:21 anyway, if it's an infrequent nuisance, i'm inclined to prefer keeping our tools flexible and just dealing with the result (i mean, we made it so anyone can #endmeeting after an hour anyway) 19:40:26 corvus: yah, some random user did startmeeting test IIRC 19:40:30 then left 19:40:39 pabelanger: i'd hardly consider that vandalism :) 19:40:47 that sounds less malicious and more like someone didn't know what they were doing 19:40:51 i mean, *i've* done "#startmeeting test" :) 19:40:58 is it possible it's some sort of markov chain thing that has happened across something that works? it's ... odd 19:41:12 and what do you think the topic of maintenance has to do with it 19:41:21 since that is the name of those meetings 19:41:24 there was a similar incident last where where someone did "#startmeeting kolla" in #openstack-dev (i think) because they thought that was how they were supposed to summon people for discussion 19:41:31 Regget: I think that was the pretense used to say people should leave teh channel bceause it is under maintenance now 19:41:32 er, similar incident last year 19:41:42 oh 19:41:57 fungi: i agree 19:42:00 I agree with fungi though. It is currently infrequent and relatively easy to address when it happens 19:42:17 So how do we know it was not a maintenance person doing his work on weekends so he doesn’t interfere with meetings 19:42:23 if we need to educate our users a bit more on why/how it happens and what they can do about it (local hcannel ops etc) then we can do that 19:42:40 Regget, because the maintenance people are in this meeting 19:42:42 Regget: because it was an unauth'd nick from a webclient 19:42:47 ok 19:42:50 Regget: and not a chanop 19:42:54 Regget, and that is not how we do mantainance 19:42:59 ok 19:43:18 Regget: i'm probably overestimating what constitutes common sense with respect to irc networks, but the only people doing maintenance on the irc servers should be freenode staff and they wouldn't use people's random bots to change channel topics to signal that 19:43:28 ok 19:43:32 I see now 19:43:46 fungi: thee is a separte question of maintenance on the bot itself but no one should have to leave the channel for that 19:43:56 indeed 19:45:24 In general we can educate people that yes public communications channels do get trolled occasionally. If you have questions or concerns you can ask an chanop of the channel(s) you lurk in 19:45:29 we _do_ use bots to change channel topics so as to let people know about maintenance work we're doing on _other_ systems, but those maintenance activities wouldn't require people to leave any irc channels (to the contrary, it's where we're communicating about them) 19:46:49 hope he comes back, he was asking good questions 19:46:50 if anyone is interested in improving our ability to respond to trolls in general, this spec needs volunteers: http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html 19:46:58 he/she 19:46:59 fungi: perhaps we should communicate that if we use a bot to ask people to take an action, that action will be documented somewhere on an openstack.org url so someone can verify whether or not a bot message with an action is actually from us? 19:47:31 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html This spec needs volunteers which will help us be in a better position to update our IRC bots to address these issues when they happen 19:48:06 mordred: using the status page on the wiki I guess? 19:48:36 mordred: yeah, the first place which springs to mind is the infrastructure status wiki, but referring people to a wiki any random person on the 'net can edit may not be any better 19:48:59 fungi: ya that was going to be my next concern and not wanting to suggest twitter as an official of record location either 19:49:07 yah. I was more thinking some non-wiki location - but I haven't thought through the whole thing deeply yet 19:49:09 perhaps a link to the status page should be included in maintenance annoucments 19:49:17 as in verify here: link 19:49:31 anteaya: I think the problem is then a troll could just send a verify here link to anywhere 19:49:39 if its advertised out of band its a bit more controlled 19:50:25 in any case I think the two steps forward here are 1) improving our irc bot implementation as per corvus' spec. Volunteers needed. and 2) Better docuemntation/education for users of IRC to avoid confusion when trolls are crafty enough to maybe look legit 19:50:29 for the specific "leave the channel" scenario, the only situations we've had where we wanted people to exit a channel are when it's being renamed/forwarded. and for that we just give them a heads up (with a link to an ml thread about it generally) and then kick the stragglers eventually 19:51:11 I can work on drafting an email to the dev ml with info on what infra and freenode would do as far as maintenance goes and who to contact if you have questions 19:51:26 clarkb, true 19:51:39 does that seem like a reasonable place to start? 19:51:42 we also used to try and do that to reclaim channel founder privs, but it's usually close to impossible unless there are just a few people in there so tends to be easier to reach out to freenode staff and get them to override 19:51:45 can point to the irc bot infra spec there too 19:52:00 yeah, that sounds like a great idea 19:52:47 corvus, FYI: I might be interested in assisting with the IRC bot. 19:53:00 clarkb, yes and also perhaps add the text to the IRC services area in the docs 19:53:11 anteaya: ya we can work it into the docs too 19:53:29 ok we've a few minutes left for the what did we miss open discussion topic 19:53:34 #topic Open Discussion 19:53:40 David did a thing 19:53:48 but I think he said he is picking up kids 19:53:56 gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 :) 19:54:00 jlvillal: awesome! i'm happy to work with you on that and pitch in 19:54:10 #link https://www.youtube.com/watch?v=6gTsL7E7U7Q&t=1697 dmsimard did a talk on how to help infra 19:54:12 corvus, cool :) 19:54:22 feel free to share the video if you know people who are interested 19:54:34 * dmsimard back from picking up kids 19:54:41 https://review.openstack.org/#/q/topic:emit-job-header-nodes is a way we could start printing nodepool info in jobs again, if people are interested 19:55:00 dmsimard, how did the talk go? 19:55:11 good-sized crowd? 19:55:40 It went great, less people than expected showed up but since it was recorded it was still worth it. Still got a few good questions throughout the talk. 19:55:55 https://review.openstack.org/#/q/topic:fedora-27 is also needed to migrate jobs from fedora-26 to fedora-27, would like some help landing that too 19:56:01 sometimes small groups can be more effective 19:56:06 yay for questions 19:56:22 and we're getting a bunch of new infra contributors from montreal as a result, right? ;) 19:56:35 The format of the talk was very informal on purpose and was more or less a ~1hr live demo with recursive and meta references. It was fun. 19:56:35 oh this was in Montreal? 19:56:53 Yeah, we have regular OpenStack meetups in Montreal. 19:57:08 nice, I think I was at one in the past 19:57:15 when I went to a pycon 19:57:25 ah yeah there was a pycon about 2-3 years back :) 19:57:29 bit of a drive to go every month 19:57:31 fungi: we'll see :D 19:57:40 * fungi crosses his fingers 19:58:21 My hope is that the talk remains relevant and a form of informal documentation since it was recorded 19:59:48 pabelanger: I'll help with those reviews 20:00:07 alright thanks everyone 20:00:11 thank you 20:00:13 #endmeeting