19:03:38 #startmeeting infra 19:03:38 Meeting started Tue Dec 15 19:03:38 2015 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:03:39 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:03:41 The meeting name has been set to 'infra' 19:03:42 o/ 19:03:47 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:58 unlike last week, we have a very _light_ agenda, so we can take our time 19:04:10 #topic Announcements 19:04:25 #info Reminder: Gerrit 2.11 upgrade is tomorrow (Wednesday, December 16), starting at 17:00 UTC. 19:04:26 o/ 19:04:41 #link http://lists.openstack.org/pipermail/openstack-dev/2015-December/081037.html 19:05:02 #topic Actions from last meeting 19:05:12 #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-12-08-19.00.html 19:05:20 #info DONE: jhesketh finalize infra-cloud sprint planning details on the infra ml 19:05:24 ohai 19:05:27 #link http://lists.openstack.org/pipermail/openstack-infra/2015-December/003554.html 19:05:34 thanks jhesketh 19:05:40 #info DONE: fungi get gerrit maintenance included in thingee's dev digest 19:05:48 #link http://lists.openstack.org/pipermail/openstack-dev/2015-December/082077.html 19:05:50 pleia2: thanks to you too 19:05:54 yes nicely communicataed jhesketh 19:06:00 #info DONE: nibalizer send follow-up gerrit maintenance reminder 19:06:07 #link http://lists.openstack.org/pipermail/openstack-dev/2015-December/082247.html 19:06:21 #info DONE: anteaya plan an infra virtual sprint for knowledge transfer and holiday festivity, friday, december 18th 19:06:28 #link https://wiki.openstack.org/wiki/VirtualSprints#Infra_.22holiday_party.22_knowledge_transfer_virtual_sprint 19:06:35 o/ 19:06:36 #link https://etherpad.openstack.org/p/infra-holiday-party-2015 19:06:57 all action items completed! 19:07:00 #topic Specs approval 19:07:09 #info APPROVED: Complete the reviewable release automation work 19:07:17 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/complete-reviewable-release-automation.html 19:07:32 there are no new specs on the proposed list for this week 19:07:45 at least not according to our agenda 19:07:48 #topic Priority Efforts: Gerrit 2.11 Upgrade 19:07:57 zaro: are we still in good shape for tomorrow? 19:08:04 it sounds liket here will be a spec omming in the near term to host limesurvey 19:08:06 #link https://etherpad.openstack.org/p/gerrit-2.11-upgrade 19:08:21 yes, i believe so. nothing really new to report. 19:08:36 any remaining blocking reviews we need to hurry and finish? 19:08:46 i just pushed up a change to make cleanup a little easier, https://review.openstack.org/#/c/258088/ 19:08:54 and everything we identified to test on review-dev is in place? 19:08:55 I think we flushed out most of the review queue over the last few days 19:08:59 nothing blockin the upgrade though. 19:09:02 that was my impression as well 19:09:12 #link https://review.openstack.org/258088 19:09:14 awesome 19:09:33 anybody hanging around with questions about the upgrade maintenance coming tomorrow? 19:09:47 for the actual process do we have volunteers? 19:09:50 the javamelody plugin doesn't get automatically updated, but that's minor thing we can fix later 19:09:55 I have to pop out early afternoon but can be around in the morning 19:10:02 * ruagair is happy to help however he can. 19:10:07 DavidO says he'll attend the party 19:10:09 i will be here 19:10:22 I'll be here, not sure what I can do besides answer questions in channel 19:10:29 i'll be around the entire time and am happy to drive root-requiring tasks especially 19:10:31 I'll be around as well 19:10:32 I'm expected a flood of is gerrit down? 19:10:40 anteaya: yeah 19:10:48 anteaya: yes, running interference in channel is probably helpful 19:10:49 so I can work on those 19:10:59 * anteaya will run interference 19:11:08 do we want to use our incident channel to do actual maintenance coordination, so as to keep clear of the noise in our usual channel? 19:11:16 anteaya: just say every five minutes "Yes, gerrit is down, see #topic" in a friendly way ;) 19:11:16 zaro: yay DavidO will be here! 19:11:20 sounds like we are well staffed 19:11:35 fungi: I think that makes sense 19:11:40 fungi: maybe a good idea? i sort of suspect we'll end up there anyway eventually :) 19:11:43 AJaeger: only stays friendly the first few times 19:11:51 jeblair: oh, so pessimistic! 19:11:51 AJaeger: you want to tag team? :) 19:12:04 yeah -incident is fine with me 19:12:20 I'll stay in -infra and monitor 19:12:21 fungi: i have no idea how you typo'd "real" that way 19:12:36 anteaya: I might come an hour later, 17:00 UTC is dinner time here... 19:12:36 bwahaha 19:12:51 AJaeger: happy to have your support whenever the time is right for you 19:13:36 anteaya: ok, I try to be there... 19:13:46 #agreed Gerrit upgrade maintenance coordination will happen in #openstack-infra-incident to keep the channel clear and improve efficacy 19:13:48 AJaeger: thanks, I'm grateful 19:13:50 * olaph can help run interference as well 19:14:04 olaph: wonderful, thank you olaph 19:14:06 a reminder, for those who want to follow along but forget to /join the channel initially, it _is_ logged 19:14:11 #link http://eavesdrop.openstack.org/irclogs/%23openstack-infra-incident/ 19:14:17 thanks, fungi! 19:14:51 anything else we need to decide or questions we need to answer about the gerrit upgrade before we move on to the next topic? 19:15:18 I'm good 19:15:20 o/ 19:15:27 fungi: did you get your question answered about the project rename fixes? 19:15:29 hey mordred 19:15:42 jeblair: yep, zaro worked the commands needed into the maintenance plan 19:15:45 fungi: I think they are in zaro db script 19:15:49 woot 19:15:50 's 19:15:51 and tested them as well 19:15:57 The wifi, it is bad in far southern Quebec 19:16:01 ah. fix_project_rename() 19:16:06 with an updated sanitized dump of our production database 19:16:09 mordred: ah but the wine is good 19:16:46 #topic Priority Efforts: maniphest migration 19:16:47 * jeblair watches morderd attempt to use wine instead of wifi to participate in the meeting 19:16:53 ruagair said something in the agenda about a dev instance 19:17:05 yah 19:17:14 jeblair: i feel like i've seen that happen before too 19:17:17 ruagair: I saw a chnange go by last night that implied mod auth openid mostly just works (tm) ? 19:17:21 Yep, so I've sorted openID: People can start playing with a dev instance here: http://os03.mcwhirter.io/ 19:17:22 ruagair: if so neat! 19:17:49 I'm currently generating unique usernames with a random string. 19:17:53 #link http://os03.mcwhirter.io/ 19:17:57 i have logged in! 19:17:59 nice 19:18:00 We'll need to consider if this is th ebes approach. 19:18:08 ruagair, cool 19:18:26 ruagair: for openstackid I think we can use the email addr and treat it as unique 19:18:31 it's certainly _a_ way forward, so you're not blocked on the missing usernames question 19:18:33 woo, I'm 39oywejia82h1zzspxr8bsva 19:18:34 I want to say that is true of lp too 19:18:34 I can sign in 19:18:38 There's still more migration issues to be sorted but we need to start playing with it and considering what changes we need to make. 19:18:46 but I can log in :) 19:18:50 what's the usernames question? 19:18:52 Phab does not like that clarkb :-) 19:18:59 ruagair: huh interesting 19:19:00 That's where I started but it barfs. 19:19:11 (i am uludwh7i7wh5lqi5ed6epeq.) 19:19:13 jeblair: storyboard did not have usernames, phabricator wants usernames, lp does not by default provide usernames 19:19:31 so you get your ip id token 19:19:31 openstackid also does not have usernames 19:19:32 but you can change you uisername, so we could leave that up to people or scrape launchpad. 19:19:34 s/ip/lp 19:19:48 ruagair: is that something maybe we can patch? 19:19:48 where can you change your username? 19:19:57 liek is it just mad that there is an @ inthe username? 19:20:07 i'd love to see openstackid grow a preferred username 19:20:08 jeblair: "change username" :) 19:20:27 jeblair: on the right hand side of the profile screen 19:20:27 pleia2: still not seeing it :( 19:20:34 i should say, an option to fill in a preferred username 19:20:36 fungi: I just want to avoid adding on features to that service until it is standard compliant 19:20:42 Perhaps clarkb. I've not looked that deeply yet. 19:20:45 jeblair: click on the head at the top right of the screen 19:20:45 oh, it's a profile setting, not a setting setting. 19:20:46 fungi: since right now it doesn't really do openid :( 19:20:50 jeblair: yep! 19:20:52 pleia2: i clicked on the wrench. 19:20:57 ah :) 19:20:57 Yes, the '@' is th eissue clarkb 19:20:59 clarkb: true, though we'll likely want usernames for wiki, gerrit, et cetera 19:21:21 fungi: good point 19:21:41 do users have to be logged in to see tasks etc? (or is that just because this instance demos the auth?) 19:21:43 For purposes of unique, could one just s/@/a/ to create an identifying string (assuming we don't need to convert the other way)? 19:21:44 yup and email addrs change so are a bad unique identifier 19:21:51 is change username an admin priv? 19:21:54 and using LP because I can't use openstackID at present. 19:21:57 because the dialog is sort of written that way. 19:21:57 ruagair: we can request a username from launchpad in the openid request, no? 19:22:01 persia: I don't see change username 19:22:03 It may be jeblair 19:22:05 i mean, phab is the first thing we've tried to use it with that really wants a username, but other things we want to switch to openstackid will also need that 19:22:06 pleia2: ^^ 19:22:16 persia: sorry wrong number, meant pleia2 19:22:23 I hope so mordred. I will be looking into that. 19:22:32 mordred: I would have to go and reread the spec but yes you can request differnet attributes 19:22:44 The random number was merely just to unbock usage. 19:22:48 anteaya: if you click on the head graphic on the top right it'll take you to a profile screen, profile screen has change username option in the menu on the right 19:22:49 ruagair: ++ 19:22:53 and mod auth openid lets you map an arbitrary attribute onto the username it passes to the app 19:23:02 jenkins requests username from lp 19:23:05 pleia2: not for me it doesn't 19:23:28 :\ 19:23:35 jeblair: should be possible then 19:23:42 So far everyone who has been able to change names are admins, so that's clearly going to be an issue. 19:23:55 pleia2, ruagair: anteaya is not an administrator 19:23:56 pleia2: I have "Edit Profile", "Edit Profile Picture", "Flag For Later" 19:23:58 ah 19:24:06 Thanks clarkb. I'll follow up on that. 19:24:08 so i think that confirms the earlier supposition change username is only for admins 19:24:09 * AJaeger is neither an admin 19:24:12 anteaya is http://os03.mcwhirter.io/p/2ctl83esnfk2k41dmkf2vb7/ 19:24:15 ruagair: well, at any rate, it's a start, and has helped us to define one need without blocking on it 19:24:22 I am indeed 19:24:26 ruagair: nice work 19:24:51 Yep. That instance will remain up and stable so people can prod it and I'll be doing dev elsewhere. 19:25:02 thanks ruagair! 19:25:04 do we need someone to ask for a username field in openstackid? 19:25:06 ruagair: thanks 19:25:07 I'm also going to need a spirit guide make getting this work into puppet./ 19:25:24 and an English coach. 19:25:27 ruagair: opening a random bug that should have comments I don't see comments just the title and header stuff 19:25:36 not sure if that is known problem with migration 19:25:54 ruagair, i can give you a hand with puppet 19:25:58 I've not used Manufest yet clarkb, so I'll look into. 19:26:07 Thanksd yolanda. 19:26:17 jeblair: yeah, and i'd prefer it not be me to ask since i've been dragging my heels on working through the openstackid platform upgrade so already probably on their list of not so favorite people 19:26:32 * fungi makes a note to up the priority on that 19:26:51 fungi: in fairness we semi stalled out on npm related things 19:26:56 because npm 19:27:37 maybe if i hurry up and get openstackid.org onto ubuntu trusty then i can turn that around into a feature request for usernames ;) 19:27:54 I would rather use openstackid too. 19:28:13 fungi: what is needed to do that? 19:28:44 I see review migrating comments clarkb. thanks for the heads up. 19:28:45 pabelanger: the puppet-openstackid module needs to become smarter about installing the right things depending on whether it's precise or trusty 19:29:00 fungi: I might be able to help with that 19:29:07 so that we can confirm an openstackid-dev.openstack.org upgrade to trusty while leaving openstackid.org on precise temporarily 19:29:20 pabelanger: great! get up with me after the meeting and i can point you at the current reviews 19:29:34 fungi: ack 19:29:36 i bet the request 'add a username to openstackid' is going to explode in complexity -- because what's the point of at authn system storing a username and _not_ using it as the login identifier... 19:30:01 (probably needs to accept both username and email as identifiers at least) 19:30:12 yeah, i can see it being nontrivial 19:30:24 actually hrm 19:30:26 any other field is just data, but this touches a lot 19:30:28 I think it may already do first.lastname 19:30:31 as your id 19:30:45 yah - but that's going to be wonky when we migrate gerrit 19:30:52 because my user id on gerrit is "mordred" 19:30:54 not "monty.taylor" 19:31:01 right 19:31:07 (actually, my id is 2 - my name is mordred) 19:31:09 but, for example, mediawiki uses it to decide what to display for your edit citations and where your userpage lives, gerrit uses it to bootstrap non-openid authentication (rest, ssh) 19:31:33 so they're not really using it as part of the openid authentication, just requesting it as additional details about the account they're creating for you 19:31:43 I'd love to be able to claim a username in openstackid, as well as tell it what my IRC nick is 19:31:52 mordred: ++ 19:31:59 however it does need to be unique, or else you introduce new vulnerabilities in the consuming services 19:32:08 so that we could conceivably write bots that query irc nick from openstaciid so that they can ping people by them 19:32:10 but now I'm getting greedy 19:32:10 but the existing thing may work for ruagir once it does acutal openid 19:32:15 one of the benefits of launchpad is the irc nick field, I use it a lot when looking for people 19:32:16 clarkb: ++ 19:32:17 since it doesn't have an @ and is unique 19:32:17 I see many troubles if we don't allow for that, irc nick as username that is 19:32:20 maybe wikimedia can help with migration? since they've probably already done it. 19:32:35 pleia2: and also ssh keys and pgp keys 19:32:40 fungi: yeah 19:32:41 though I bet individuals could have a fullname of clark@ boylan 19:32:44 because lol 19:34:29 okay, seems we've burned down this topic 19:34:37 Do you have an manifest task id for th etask that should have comments, clarkb? That would be useful to track. 19:34:45 Thanks everyone :-) 19:34:47 ruagair: http://os03.mcwhirter.io/T1010621 19:35:56 zaro: mediawiki didn't migrate from storyboard to maniphest, they migrated from gerrit to phabricator 19:36:04 Got clarkb. Thanks. 19:36:11 zaro: assuming that's the migration you're talking about 19:36:11 * persia is reminded of a discussion last year about a daemon that used nickserv to get an authentication countertoken from the data fields 19:36:20 I suspect the LP -> Maniphest will be much more graceful. 19:37:01 #topic Holiday meeting schedule: Are we meeting on December 22nd and 29th? (pleia2) 19:37:06 fungi: they did neither thing - they're still using gerrit - but iirc, they use LDAP auth on the backend or something 19:37:17 mordred: oh, even more interesting 19:37:19 I'll be around both days, but a lot of teams have already rolled up the carpets for meetings in 2015 19:37:44 i'm around both dates as well, and happy to chair a meeting if people want/see benefit in having one 19:37:44 * ruagair is around for the 22nd 19:37:52 I expect I'll be around those days, fine chatting in here if we have items, fine not chatting in here if we don't 19:38:13 * zaro is around both days 19:38:21 how about we plan for informal meetings that may be entirely open discussion and/or end pretty quickly 19:38:30 wfm 19:38:30 Likely around too 19:38:31 ++ 19:38:36 fungi: should I still send out meeting announcements? 19:38:39 i probably won't be around for the 29 but will for 22. 19:39:05 pleia2: entirely optional, but if you feel compelled to send announcements then that would still be perfectly accurate 19:39:13 ok, great 19:39:41 if someone throws something on the agenda we can discuss, but we should also plan for the possibility of not having many people around to reach consensus on important opics at short notice 19:39:47 topics too 19:40:02 fair 19:40:20 I'm just going to say things to the empty channel and assume silence means assent 19:40:30 mordred: don't you do that anyway 19:40:31 ? 19:40:43 yup 19:40:49 he is having trouble finding an empty channel these days 19:40:51 that's why I'm certain that I'll do it then too 19:40:56 heh 19:41:13 both days wfm 19:42:02 i think i can make the 22nd and I can make the 29th 19:42:33 cool, sounds like we'll have pretty well populated meetings then 19:42:48 ++ 19:43:13 well then, without further ado, i give you... 19:43:15 #topic Open discussion 19:43:44 so much time today 19:43:48 don't all talk at once 19:43:57 i can give a quick update on f23 19:44:02 please! 19:44:08 I have a bit on builders I'd like to mention, too 19:44:11 but I will wait for ianw 19:44:28 * fungi was kidding, feel free to all talk at once if you want 19:44:29 glean is almost working, i think all reviews there are processed 19:44:51 there is one more dib workaround for a systemd feature -> https://review.openstack.org/#/c/257173/ 19:45:10 \o/ 19:45:14 but after that, i'm fairly sure they'll come up with networking ... 19:45:24 that's in. basically, in progress and close 19:45:28 s/in/it 19:45:32 fungi: I'm on bad network - but I can try an update on puppet apply 19:45:34 The builders are cleaned up after the last round of reviews and I think they are ready to go. I would like to get some plan started on when/how we can do that... 19:45:37 ianw: have you been able to test the networking config in json thing? 19:45:44 ianw: if not thats fine, just curious 19:45:58 fungi: there are two issues blocking moving fowrad - one is getting the inventory to use hostnames where possible instead of uuids so that our logs are readable 19:46:02 clarkb: not live on rax, but faking it 19:46:05 greghaynes: meaning the nodepool image build workers? 19:46:09 fungi: yes 19:46:18 fungi: I've gotten that change landed upstream ansible, now working on getting occ and shade support patches landed 19:46:27 https://review.openstack.org/#/q/status:open+project:openstack-infra/nodepool+branch:master+topic:nodepool-workers,n,z 19:46:30 for context 19:46:32 so that should be done soon 19:46:36 So, apache::vhost::custom landed in puppetlabs-apache today: https://github.com/puppetlabs/puppetlabs-apache/commit/b9f630a60811694916ecf8758103d4bca2ac6038 I've already gone ahead an update puppet-nodepool to use it: https://review.openstack.org/#/c/205596/ 19:46:49 mordred: oh cool--so it was outside bugs/missing features we were stuck on? 19:46:49 so, we can start the process of porting to puppetlabs-apache 19:46:56 the second is a mismatch in ansible and puppet groups - and for that I'm going to write a simple thing to exapand a yaml group expressions into generated static inventory files 19:47:09 fungi: yah - well, 'outside' except I'm the author of that too :) 19:47:11 fungi: but yes 19:47:19 you're on the outside looking out 19:47:29 yah 19:47:29 greghaynes: I may be able to do it tuesday ( a week from today) 19:47:39 with reviews happening again between now and then 19:47:46 anywho - we're CLOSE - byt I still need to get the group expanded written and tested before we can finish throwing the switch 19:47:51 s/to do it/to help deploy it/ 19:48:01 clarkb: That works for me if it is earlier in the PST day 19:48:16 Doesn't have to be crazy early, I just am occupied that evening 19:48:55 i've picked the bindep stuff back up off the floor. working through getting a functional test of our fallback list which project-config and bindep can both gate on, implementation of that necessitated moving forward with solidifying the bindep use in our jobs a bit so some of that has already landed 19:49:03 I've split log_processor into its own project, and the new python project boilerplate could use some review: https://review.openstack.org/248868 19:49:48 after fallback tests are running on relevant platforms (possibly voting for our required platforms and non-voting for optional platforms) i've got a couple alterations to bindep i want to finish so i can tag a 1.0.0 release 19:50:14 from infra cloud, not much progress on our side this week. We've been fixing several problems with bifrost and dhcp, and some problem on glean and hostnames. 19:50:56 yolanda: that sounds like progress to me. saw at least one related change fly by in channel earlier and some discussion 19:51:41 fungi: do you have any updates about the trystack.org SSL certs? or what needs to be done to help move that along? 19:51:52 have some people ping me about that internally 19:52:02 fungi, well yes, slow but solid progress :) 19:53:10 i'm making progress on the fairly radical config changes for zuulv3 19:53:20 jeblair: yay 19:53:43 * mordred hands jeblair a cookie 19:53:50 pabelanger: i meant to look into replacing our certs with an account at startssl.com which seems like the least expensive way out, though increasing complexity of the openstack_project::static class is another option 19:54:22 pabelanger: the current maintainers of the trystack.org server aren't keen on maintaining a redirect for a little while? 19:54:22 yolanda: oh - I might need your help to track down an issue with an occ patch and bifrost 19:54:23 i think if someone wanted to start on the nodepool side of things, that can probably proceed in parallel, but it's not super urgent 19:54:33 mordred sure 19:54:40 what's going on? 19:54:46 jeblair: I'd like to hack on the nodepool side of things 19:54:49 What about Let's Encrypt certs fungi? 19:54:59 * ruagair has been using those for a few weeks, quite happily. 19:55:07 ruagair: that discussion would take a lot more than 5 minutes 19:55:14 Fair enough :-) 19:55:16 mordred: ooh 19:55:28 yolanda: this https://review.openstack.org/#/c/254399/ causes things to blow up for TheJulia and I haven't gotten a chance to diagnose yet 19:55:30 fungi: can you tell us why it would take longer than 5 minutes in less than 5 minutes? ;) 19:55:38 can we get image builders in before we add any more featuers to nodepool? 19:55:42 ruagair: philosophical objections to letsencrypt aside, i haven't seen much discussion yet about how to handle the automated cert rotation 19:55:42 jeblair: but I cannot start thinking about it in earnest until next week 19:55:58 clarkb: yeah, i think we all agree image builders go in first 19:56:04 clarkb: I'd like to get image builders in and the shade patch landed, actually 19:56:10 it's also technically still in beta, which I'm not sure we want to rely upon for our production infrastructure 19:56:20 ok, just making sure we don't add more rebase churn to it 19:56:25 because juggling two sets of cloud interaction codebase is not awesome 19:56:34 clarkb: ++ 19:56:38 fungi: Ya, I've been trying to figure out a good approach for that too. Don't like doing crontab 19:56:48 mordred, i can take a look 19:56:49 Yes, that's an additional complication to be considered fungi. 19:57:13 ruagair: especially how to make it robust against failure to update/rotate 19:57:26 yolanda: thanks! the last error she showed was unrelated, so I'm worried something else is broken in master of occ that we're not testing right that will break bifrost if we release 19:57:35 but I just don't know 19:57:46 we don't want to rely on cert replacement automation which could cause critical services to grow new failure modes 19:58:08 Wher's your sense of adventure ;-) 19:58:46 left it in my other pants 19:58:59 That's a safe place. 19:59:49 mordred, yep, region doesn't seem to be related 19:59:56 okay, well, we're out of time now--thanks everyone! 19:59:59 #endmeeting