19:03:39 #startmeeting infra 19:03:40 Meeting started Tue Mar 27 19:03:39 2018 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:03:41 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:03:44 The meeting name has been set to 'infra' 19:03:51 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:57 #topic Announcements 19:04:20 #info clarkb is travelling this week, which is why I am returning as a guest lecturer 19:04:51 he thankfully didn't leave me any new announcements, and i wasn't creative enough to come up with more than that one 19:05:15 #topic Actions from last meeting 19:05:46 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-20-19.01.html Minutes from last meeting 19:05:52 "1. (none)" 19:05:57 and done 19:06:40 #topic Specs approval: Amend top-level project hosting spec (corvus) 19:06:54 #link https://review.openstack.org/555104 Amend top-level project hosting spec 19:06:59 lecturer? Ok, I'll move a few rows up and listen... 19:07:18 i went around and got informal support for this change from almost everyone who reviewed the original 19:07:25 enough so that i went ahead and actually implemented the change 19:07:37 yeah, this one is really just a minor adjustment to the git hosting part of the hosting spec anyway 19:07:52 but now (or soon) would be a good time to raise objections or questions on it :) 19:08:25 i think we may be only minutes away from declaring that entire spec implemented 19:08:25 we're just about to the point where that spec can be moved to implemented if this change is approved, right? 19:08:34 heh 19:08:40 yeah, i mean, we might already be there if i weren't typing with a sandwich in my hand 19:08:45 fastest enter key in the west 19:09:26 i'll risk incurring the wrath of our illustrious ptl and declare open season for roll call on this, since he's already put a roll call vote on it himself 19:09:54 i think that's appropriate 19:10:16 #info Infra Council voting is open on the "Amend top-level project hosting spec" change until 19:00 UTC on Thursday, March 29 19:10:44 any other comments before we continue with the agenda? 19:11:20 #topic Priority Efforts: A Task Tracker for OpenStack - utf8mb4 transition (fungi) 19:11:59 just a quick note that diablo_rojo discovered a while back we couldn't import lp bugs containing 4-byte characters into storyboard 19:12:35 i've been hacking on solving it, and there are a couple remaining outstanding changes i've tested manually on storyboard-dev so that we can move forward there 19:12:55 i bet i can guess which emoji caused the problem 19:12:57 \o 19:13:04 #link https://review.openstack.org/555787 Use utf8mb4 for MySQL database charset (puppet-storyboard) 19:13:04 Looks like py27 & py35 failed again on your latest patch :/ 19:13:13 (╯°□°)╯︵ ┻━┻ < corvus 19:13:31 there was a very specific emoji which took out etherpad.o.o during a summit... 19:13:31 https://review.openstack.org/556626 For utf8mb4 shorten teams.name and users.email (storyboard) 19:13:40 interesting 19:14:02 actually, the problem emoji was something like "person with both hands raised in celebration" which appears in babel install stdout these days 19:14:04 corvus, out of curiousity, what emoji? 19:14:23 But also didn't show up in the lp bug actually. 19:14:40 the one which notoriously took down etherpad at the summit was teh snowman 19:15:06 fungi: oh was it? i thought it was... nevermind. 19:15:11 anyway, barring anything beyond a minor typo failing the unit tests on 556626 those could both use quick reviews 19:15:39 * mordred has already reviewed 19:15:39 i thought it was, pardon my language, U+1f4a9. 19:15:59 * dmsimard googled that 19:16:01 corvus: it's possible the first etherpad outage was a different codepoint, yes, and then the subsequent deaths were all snowman tests 19:16:03 * mordred did too 19:16:41 anyway, i didn't have much else to add here other than it would be nice to be able to keep up our momentum on project imports, so figured i'd highlight these priority changes 19:16:54 +1, we need to ditch launchpad one day :) 19:17:10 (and ubuntu one SSO) 19:17:38 #topic Priority Efforts: Zuul v3 - git.zuul-ci.org (corvus) 19:17:47 i guess this is now a thing? 19:17:52 er, as mentioned, this is nearly done 19:17:57 it may in fact be done 19:18:04 i just hit a missing css file in one of my tests 19:18:06 it renders for me 19:18:14 and me! 19:18:30 so i need to figure out whether all the servers are identical, or if i left git08 different somehow, or if i just caught it mid-rollout 19:18:31 done-ish! 19:18:50 i may ask for help testing in a bit, but am not ready now 19:18:53 $ git clone https://git.zuul-ci.org/zuul 19:18:56 just worked fine for me 19:19:01 anyway, hopefully it's the latter, and it's all done :) 19:19:24 i suppose the next big chunk will be updating zuul repos to switch git references 19:19:40 yeah, that's mostly documentation 19:19:49 the zuul and gitreview files won't change 19:19:55 sure 19:20:28 more critical that people who want to continuously consume the zuul stdlib will be able to use git.zuul-ci.org urls now 19:21:01 though i suppose things like puppet-zuul can also be adjusted in time 19:21:25 anything else you wanted to say about this, or does anyone still have any questions about it? 19:21:33 Are we planning to move things like puppet-zuul and puppet-nodepool there ? 19:21:35 obviously closely related to the spec change discussed earlier 19:21:53 dmsimard: i'm not aware of any immediate plans to do that, but it would likely be possible 19:22:01 dmsimard: not atm; i don't think the zuul project has taken on packaging/distribution tasks 19:22:12 okay -- and another question 19:22:18 (and if we did, i can think of two choices which would probably be ahead of puppet on our list) 19:22:24 heh 19:22:35 zuul and zuul-jobs are currently mirrored on both git.o.o and git.zuul-ci.org by gerrit automatically ? or by other means 19:22:51 (and I guess the other repos) 19:23:03 dmsimard: automatically. git.zuul-ci.org are symlinks on the git farm to the git.o.o repos 19:23:11 dmsimard: it's all the same server, just some magic linking to make it appear in different sites 19:23:24 ah, nice. thanks. 19:23:24 the magic is configured in projects.yaml in project-config 19:23:32 I'll have a look 19:23:35 dmsimard: also worth reading that spec/change 19:23:42 which roughly outlines how it works 19:24:11 I don't have anything else, we can move on :) 19:25:11 #topic Clouds update (dmsimard) 19:25:22 ohai 19:25:30 clarkb suggested maybe you could give us a brief summary of the limestone situation 19:25:57 well, you or somebody, but i picked you because you seemed to have the most background on it 19:26:06 feel free to delegate ;) 19:26:22 The networking is still giving us a bit of trouble. My understanding is that there's two main problems still in the process of being resolved 19:27:00 #1 In some roles we assumed that nodes always had an ipv4 which was the case until Limestone came around with ipv6 only nodes 19:27:25 well, most recently the case, it was last the case with osic before the great ansibling 19:27:41 #link https://review.openstack.org/#/c/556747/ 19:27:51 see https://review.openstack.org/556784 too 19:27:55 #2 We've witnessed some DNS lookup failures (most notably with pypi? maybe with other things) so we need to fix DNS things too. 19:28:39 What we've done for the time being is that we pulled limestone out by assigning custom nodepool labels (prefixed with limestone-) and these are being tested under https://review.openstack.org/#/c/556849/ 19:28:39 current theory on this from me is that we're using the v4 nat gateway, and we know that to be a little unreliable, because of a bad ipv6 detection in unbound 19:28:45 #link https://review.openstack.org/556740 19:29:02 so in both cases it's basically been a regression of things we had working with osic under a similar network model, but lost in the zuulv3 transition because we didn't have anywhere to keep us honest about continuing to support that model 19:29:04 These custom labels will allow us to make sure that the patches we want to land are indeed fixing the issues we're seeing without impacting our regular customers :) 19:29:11 Yah, I feel we had this same issue on osic with DNS 19:29:20 and added the logic to do ipv6 there too 19:29:35 fungi: I wasn't root back when OSIC was a thing so I'll have to defer to the others for that 19:29:37 pabelanger: oh, we did. we had a whole dns-over-ipv6 chooser in the old ready scripts 19:29:52 Yup 19:30:13 which, to be fair, we still have, though as 556740 shows, it may not be working 19:30:56 i'm uncertain if it was working, and something changed, or it never worked 19:31:15 dmsimard, mordred: if you have a sec to look at https://review.openstack.org/556740 -- several of us are curious as to (whether or why) that's necessary 19:31:48 corvus: I've seen that and I am dubious as well, I intend on testing it but haven't got around to it yet 19:32:00 corvus: these variables should be equivalent 19:32:10 that's what i thought, there's a couple of links in there to the logs 19:32:29 should we start by throwing some debug lines in that role? 19:33:09 corvus: i can ... 19:33:33 since it triggers on rax, there's a fair probability of hitting it for comparison purposes 19:34:36 makes sense as that's also got global ipv6 by default 19:35:42 ya as does ovh except it doesnt work :( 19:36:36 i have looked at the supplied log, and the logs of a result including 556740 and agree that the old one lacks unbound_use_ipv6, and the new one has it 19:37:03 so that seems to indicate the change actually changes behavior. why is still a mystery. 19:37:21 I wonder if we are missing facts on that playbook 19:37:36 but hostvars is setup 19:38:03 it's the first pre-playbook, from base 19:39:13 *weird* 19:40:44 i also verified the change correctly sets the value to false on a host w/o ipv6 19:41:20 so we have evidence it does what we want, but no documented explanation for why it works that way? 19:41:29 fungi: that's where i am on the subject 19:41:34 i left a bewildered +2 19:41:42 that's about where i got to :) 19:42:03 something we'd like to bring to upstream ansible attention? 19:43:42 I'm a happy to add some debug statements and see if I can find out why it is failing, but agree, weird 19:43:57 i can try a few debug statements and file an issues 19:44:21 ianw: happy to debug along side 19:44:39 sounds good 19:45:05 i like when we figure out how to fix thnigs, but i'm left feeling uneasy when we can't explain why the fix works (or is even necessary) 19:45:36 if it's fixed via unintended side-effect then there's an increased risk it will re-break 19:45:49 i reckon we can continue to debug this post-meeting. this is the only limestone-blocking change that hasn't merged? 19:45:50 when assumptions spontaneously change out from under us 19:46:28 wait a sec 19:46:41 this role runs in base/pre, right? 19:46:49 corvus: i think so 19:46:54 that means it's trusted, so it's not self-testing 19:47:14 which means my analysis of the results are wrong, and it may not be safe to land without going through the base-test dance. 19:47:28 * fungi hadn't looked, assumed this was being tested via support already landed to base-test 19:47:50 don't we run it from base-minimal? 19:47:59 for ozj 19:48:00 we've got nice big warnings around the base job itself, but not the roles it uses 19:48:52 it may be covered by the integration jobs; if so, perhaps it's safe to land? i'm not sure what that means for evaluating whether the change works though. 19:49:58 anyway, perhaps dmsimard, ianw, i, others can work through that some more post-meeting 19:50:33 wfm 19:50:48 cool, we'll move on with the agenda in that case 19:50:50 thanks! 19:50:58 #topic LimeSurvey spec (anteaya) 19:51:02 #link https://review.openstack.org/349831 Add survey spec 19:51:06 i gather anteaya is interested in breathing some new life into this proposed spec pleia2 put together a couple years ago 19:51:13 yes 19:51:39 we don't have an open source survey tool and I think that it would be useful to have one 19:51:42 I'd use it 19:52:03 since the last patchset a puppet-limesurvey module has appeared 19:52:04 we've heard from various corners of the community that they'd be interested in having ways to put together ad hoc surveys 19:52:29 we could at least use it to verify interest in the tool. ;) 19:52:41 and I have learned via testing with fungi that users just need a token to take the survey, the don't need to register 19:52:50 corvus, :) 19:52:56 heh 19:53:06 so perhaps we can launch it with the current server auth 19:53:13 so i guess this is as much to refresh visibility into the spec and see if anyone has any recent input on it before we see about putting it up for a council vote? 19:53:22 whilst those that are dueling in the auth wars figure out a winner 19:53:36 could the civs software from cornell also be used for surveys? Just thinking about the discussions around elections 19:54:00 civs can be used for certain kinds of surveys 19:54:01 pabelanger, well the only thing I've ever seen it used for is elections 19:54:22 hard to use it for select one answer out of these ones, or fill in some text 19:54:40 also, correct me if I'm wrong civs sometimes has load issues on its servers 19:54:47 well, what civs can't really do well is multi-question sorts of surveys, nor surveys with free-form answers 19:54:57 indeed yes 19:54:58 we could host our own instance of civs and may want to for other purposes 19:55:10 civs is the tool for the method it advertizes 19:55:17 I think civs supports ' Allow voters to write in new choices', but I have never tested it 19:55:17 but the two tools aren't really filling the same needs 19:55:30 unsure about multi options 19:55:31 but the surveys I want to offer need more flexibility in how questions are answered 19:55:54 ack 19:56:24 any objection to moving ahead with server auth for the time being? 19:56:41 pabelanger: less about multi-option, more about your typicak feedback-style surveys where you're setting x-out-of-y values for some things, setting multi-select for some, entering your opinions on certain questions 19:56:46 I'm on my own dime here so I will have to stand this up on a server for myself if this gets bogged down 19:57:00 but thought I would try here first 19:57:39 what's the server auth for? 19:57:46 for the admins 19:58:01 to set up the surveys and create the tokens for inviting users 19:58:03 is it like, we make a secret password and give it out to people who ask to be able to create surveys? 19:58:09 or takers of surveys 19:58:52 well if that is the way infra wants to do it 19:59:11 out of the box is is register with username email and password 19:59:29 it's not a preference, i'm trying to understand the question "any objection to moving ahead with server auth for the time being?" 19:59:35 and, I believe, click on emailed verification 19:59:45 * anteaya nods 20:00:01 i don't see any reason we couldn't turn up a trial deployment, especially if there's already a puppet module for it so it's just a patch to system-config and someone to enter hiera keys and run the launch script, which may allow us to suss out what the missing bits are for a more self-service experience for people who want to set up their own surveys (maybe we could set up surveys through 20:00:02 so I don't know how infra wants to deal with who can register 20:00:03 structured metadata and code review instead of through service-specific accounts for example? i haven't looked at how it stores them or what sort of api it might have, just guessing) 20:00:51 oh, anyway, we're at time for today 20:00:55 also, there's an openid module for apache, which may be able to be used as the "webserver-based" authentication. 20:01:15 corvus, could you add a link to that in the spec? 20:01:23 thanks for the time 20:01:35 yeah, that's something clarkb has successfully coupled with some stuff 20:01:53 its a bit clunky and has bugs but mostly works 20:02:22 noted 20:02:44 anyway, we're two minutes over, but doesn't sound like there are major concerns if somebody's volunteering to do the patch to system-config (and i can do the hiera/launch bits if nobody else wants to volunteer for those) 20:03:06 my patch won't work but I can stand one up 20:03:11 corrections welcome 20:03:18 that's why we have code review and tests! 20:03:23 thanks everyone! 20:03:24 :) 20:03:30 #endmeeting