19:00:24 #startmeeting infra 19:00:25 Meeting started Tue Mar 24 19:00:24 2015 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:28 o/ 19:00:29 The meeting name has been set to 'infra' 19:00:31 never enough meetings 19:00:32 #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:00:32 #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-03-17-19.01.html 19:00:38 this is going to be a very fast-paced meeting today 19:00:43 #topic Schedule next project renames 19:00:52 we have some to do now, i believe 19:00:55 at least bindep 19:00:58 yes 19:01:01 o/ 19:01:05 o/ 19:01:10 is gnocchi ready to go too? 19:01:11 and gnocchi 19:01:12 O/ 19:01:15 o/ 19:01:19 Howdy. 19:01:19 i'm around friday and saturday, whenever everyone else wants it done 19:01:20 Morning 19:01:21 o/ 19:01:23 Hello 19:01:23 jeblair: I think so, they have a change up to reflect the post rename state 19:01:27 o/ 19:01:33 how about let's do the renames this friday? 19:01:37 wfm 19:01:44 o/ 19:01:45 me too 19:01:45 1900utc? later? 19:01:47 o/ 19:01:47 o/ 19:01:56 \o_ 19:02:15 party time in meeting city 19:02:24 o/ 19:02:26 would 2100 or 2200 be okay? 19:02:30 sure 19:02:39 o/ 19:02:40 yep, that's fine for me too 19:02:41 either is fine with me 19:02:42 o/ 19:03:03 * SergeyLukjanov hereo/ 19:03:04 #agreed next renames 2200 utc friday mar 27 19:03:18 someone want to send an email? 19:03:21 i can 19:03:28 #action fungi send announcement email for renames 19:03:38 i'll put together a plan etherpad for it too 19:03:38 #topic Priority Efforts 19:03:50 I'm going to try to timebox the priority efforts to just a few minutes each so that we can get to our backlog. If there are no issues blocking the effort that we need to discuss, let's just move on quickly. 19:03:52 fungi: cool, thanks 19:04:01 #topic Priority Efforts (Swift logs) 19:04:01 #link https://review.openstack.org/#/q/status:open+topic:enable_swift,n,z 19:04:10 Nothing to discuss afaik, just rolling forward 19:04:19 cool, thanks! 19:04:20 #topic Priority Efforts (Nodepool DIB) 19:04:20 #link https://review.openstack.org/#/q/status:open+topic:dib-nodepool,n,z 19:04:50 this has done well 19:04:52 mordred, fungi: anything to discuss? 19:04:52 not much here other than reviewing more tests and the bindep move and shade changes 19:04:56 gone 19:04:59 the bindep additions still need reviews (infra-core can actually +2 those as of last week), and i have a few more patches to write. no new blockers 19:05:17 ah yeah, i have forgotten to bindep. thanks for the reminder 19:05:18 fungi: does bindep have to be in a certain state before it moves? 19:05:25 anteaya: nope 19:05:27 k 19:05:44 #topic Priority Efforts (Migration to Zanata) 19:05:44 #link https://review.openstack.org/#/q/status:open+topic:zanata,n,z 19:05:59 oh hai. so we're making steady progress 19:05:59 pleia2: is away 19:06:06 I think she said just reviews 19:06:10 I just reviewed a stack of zanata related puppet 19:06:18 cinerama: ^ let me know if you have questions about m comments 19:06:18 i should do the same 19:06:19 and stevenk has started a patch to install the client via puppet 19:06:26 only thing i really want to flag at the mo is that https://review.openstack.org/#/c/165696/ needs to be deployed prior to us firing up zanata 19:06:39 thanks cinerama 19:06:45 #link https://review.openstack.org/165696 19:06:54 how do we make sure that's deployed? 19:07:02 (tldr it's a change required for openstackid.org) 19:07:10 i think mrmartin/smarcet need to tag a new release 19:07:15 i'm new to this game so i don't know how we do upgradey things 19:07:19 fungi: am I? 19:07:28 cinerama: we can fire up zanata it just won't let us login right? 19:07:31 mrmartin: smarcet i guess? 19:07:32 cinerama: we're all new to openstackid :) 19:07:44 fungi: I can, if I know what we are releasing 19:08:01 clarkb: yeah that's it. just need to let openstackid recognize a broader range of request strings so the libs zanata uses for openid will work 19:08:08 but point being 165696 needs to appear in a tag on that repo so it will get deployed into production 19:08:19 presumably it's just on master for now 19:08:24 fungi: do we deploy specific tags, or... what? 19:08:26 no rush since we need the other stuff to be ready first 19:08:37 otoh having it in place earlier won't actually hurt anything 19:09:00 * tchaypo waves 19:09:04 jeblair: yeah, production deploys updates on each new tag while teh dev server deploys tip of master if memory serves. is that correct mrmartin? 19:09:21 correct 19:09:23 that's good to know - so we can just point it at openstackid-dev.o.o for now? 19:09:31 cinerama: yes, should work 19:09:35 i wonder how we do that, but i will learn about that on my own time :) 19:09:39 fungi: super 19:09:40 hi o/ (late, sorry) 19:09:48 jeblair: puppet magics 19:10:01 cinerama: but be careful, because the profile db behind openstackid-dev.o.o can be a bit old 19:10:15 so it is not sure that your account exists there. 19:10:17 good info to capture in infra doc... 19:10:30 Rockyg: yeah, i was thinking we should update http://ci.openstack.org/openstackid.html with deployment info 19:10:58 i want to learn about it anyway so... 19:11:05 o/ 19:11:10 #action jeblair document openstack id deployment mechanism in http://ci.openstack.org/openstackid.html 19:11:17 agreed. right now it's comingled with basic documentation of the software itself 19:11:21 kewl 19:11:26 if you want to integrate openstackid I suggest to do it with a local dev version first, it is easier to track the logs 19:11:32 there is a separate change proposed to get the software documentation added and published for openstackid 19:11:37 as I know smarcet is working on an integration doc for that. 19:11:40 this was useful :) anything else on this topic? 19:11:58 #link https://review.openstack.org/165199 19:11:59 mrmartin: good thinking there. on my local setup here i have an openstackid server setup 19:12:01 for reference 19:12:31 nothing else to report on zanata - i just need to integrate and/or argue about everyone's feedback :) 19:12:41 cinerama: cool, thanks! 19:12:42 #topic Priority Efforts (Downstream Puppet) 19:12:42 #link https://review.openstack.org/#/q/status:open+topic:downstream-puppet,n,z 19:13:10 looks like just reviews required 19:13:15 yep 19:13:16 anteaya: agreed 19:13:31 so I have been reviewing them and very few actualy have the topic set 19:13:33 this one keeps needing rebases https://review.openstack.org/#/c/162830/ 19:13:39 in the same line, there are pending reviews for puppet-pupet and puppet-bandersnatch 19:13:40 clarkb: oh that'll do it 19:13:43 so would be good if you are workign on related items to use the topic 19:14:02 clarkb: + 19:14:10 clarkb, I think you can update the topic 19:14:10 puppet-puppet is our latest ouroboros i guess 19:14:14 I think asselin's make puppet-openstackci and nibalizer merge template and base are the only changes I saw there 19:14:26 nibalizer: good point, we should try to land that soon 19:14:32 fungi: openstackci-openstackci is fun one too 19:14:32 asselin: I can, but I am not going to hunt all of these changes down. I am just noticing that in my list of things to review there are very few changes 19:14:47 clarkb, ok I see 19:14:48 anteaya: it should be puppet-openstackci 19:15:02 jeblair: that's what I thought too 19:15:04 jeblair: anteaya and the puppetforge name is openstackinfra-openstackci 19:15:19 jeblair, clarkb suggested openstackinfra-openstackci 19:15:20 yolanda: set the topics to "downstream-puppet" for the puppet and bandorsnatch changes 19:15:26 #link https://review.openstack.org/#/c/165588/4/metadata.json 19:15:27 ok 19:15:41 yeah, i think our local repos have "puppet-foo" as the name, but they end up as "openstackinfra-foo" on the forge 19:15:43 asselin: anteaya the repo should be puppet-openstackci, the puppetforge name is openstackinfra-openstackci 19:15:49 jeblair, shall i do the same for changes intended to make modules more reusable? 19:15:53 clarkb: ah thank you 19:16:09 clarkb: it is correct now 19:16:29 clarkb, what about all the other modules? seems they're all wrong? 19:16:34 yolanda: i think so. 19:16:35 i can update the topics on the bandersnatch changes, yolanda 19:16:46 k 19:16:53 asselin: yes they need to be updated 19:17:00 asselin: yep 19:17:16 clarkb, asselin: where are they wrong? 19:17:27 jeblair: in the metadata.json files 19:17:29 topics changed 19:17:30 ah yeah 19:17:48 so nibalizer and i are working through publishing the httpd module 19:17:55 jeblair: http://git.openstack.org/cgit/openstack-infra/puppet-logstash/tree/metadata.json#n2 should be openstackinfra 19:18:11 let's finish working through that module before we make metadata changes to the others 19:18:16 jeblair: ok 19:18:17 in case we find we have more to do 19:18:18 fungi, still have your mass update script? 19:18:29 asselin: in my head anyway 19:18:36 safe there 19:19:09 anything else on this? 19:19:10 there's not much to iterating over a list of projects matching a regex and then running sed -i in each and commiting/pushing via git-review 19:19:21 not from me 19:19:28 no 19:19:32 jeblair: nope 19:19:33 #topic Priority Efforts (Askbot migration) 19:19:34 #link https://review.openstack.org/#/q/status:open+topic:askbot-site,n,z 19:19:42 ok, so two reviews waiting here 19:19:56 I found the Chinese solr issue, and there was a missing cron job 19:20:01 so we are on good track again 19:20:22 yeah, evgeny suggested a few additional things for us to check/test during the eval time, so that helped 19:20:27 thanks for knocking those out mrmartin 19:20:32 I guess we need to do add a backup, and as fungi suggested the local backup scripts 19:20:58 I have only one question here, do we have pgsql-backup for bup similar to mysql-backup? 19:21:00 yeah, we have an existing pattern with mysql dbs where we do a local database backup to disk, then that gets backed up by the system backup 19:21:03 aside from the chinese search problem, the rest are non-user-facing so i don't think we need to extend the eval time unless you do 19:21:28 mrmartin: if you're asking whether pgsql-backup exists yet, not to my knowledge 19:21:30 mrmartin: you would need to add postgresql specific local backups, but once you have that the bup backups will pick it up 19:21:31 mrmartin: i don't think we have backed up a pg database yet, so we will have to develop some new puppet for that 19:21:48 how's that for triangulating an answer? :) 19:21:55 nice 19:22:00 you can likely use the mysql backup manifest as a good template, just change the command and output file as appropriate 19:22:19 awesome, so we have two options here, add a simple cron script to backup, or the nice one to create the missing pgsql bup that can be reused later. 19:22:24 the basic dump command is already in the migrate spec too 19:22:50 mrmartin: i think our mysql backups actually just add a cron script 19:23:01 the original ask deployment contains a cron scheduled backup script, so we can reuse that very quickly 19:23:10 oh i see 19:23:13 and bup will backup those files too 19:23:17 jeblair: yup and logrotate to keep a few iteratons around 19:23:35 so in case of a restore situation, we have a proper sql dataset. 19:23:40 probably the one improvement i'd be interested in seeing to the original script on the current production ask.o.o is to rotate multiple db dumps like we do for mysql 19:23:59 but that's not critical to have day 1 19:24:05 ok, I check the mysql bup too, if it not seems to be too complicated, I do it quickly 19:24:13 mrmartin: sounds like a plan 19:24:32 #topic Priority Efforts (Upgrading Gerrit) 19:24:32 #link https://review.openstack.org/#/q/status:open+topic:gerrit-upgrade,n,z 19:24:49 we're on trusty now! 19:24:51 yay we're on trusty! :) 19:24:59 wooo 19:25:02 next step is "Gerrit 2.9 upgrade Saturday May 9, 2015" 19:25:04 has there been any bugs? 19:25:08 I haven' seen any 19:25:09 we still need to dig into why the utf-8 update failed 19:25:15 ah that 19:25:21 and possibly retry that on friday 19:25:21 and trusty git doesn't do shallow clones? 19:25:27 right, that 19:25:37 well, the git-http-backend cgi anyway 19:25:54 so the utf-8 thing needs a bit more planning 19:26:07 we can a script to convert tables, ready to go. 19:26:15 it seems it will involve table modifications, so we need to test with the actual production dataset 19:26:46 *we can/we have 19:26:55 zaro: can we get you a copy of that for you to work with so you can test it out? 19:27:00 can we dump/source it into a new trove instance and then try there? 19:27:10 or is going it on a local mysqld fine? 19:27:16 er, doing 19:27:30 i think a local mysql and a local gerrit should be fine 19:27:30 considering our mysql instance is mysql 5.1 we probably want something similar? 19:27:48 unsure of how much mysql version affects row data encoding 19:27:49 yes, can do 19:28:22 #action zaro test utf8 conversion with production database and local mysql/gerrit 2.8 19:28:42 zaro: i put 2.8 ^ there since we're talking about doing this before the 2.9 upgrade, so we should make sure to test it with the current version 19:28:58 sounds good 19:29:05 zaro: the review queue for the patches is just review them, yes? 19:29:07 zaro: i can get the database dump to wherever you need it 19:29:18 if that's ready to go by friday, cool; if now, we'll do it the next time -- i don't want to rush it and mess up our data 19:29:28 agreed 19:29:58 zaro: thanks! 19:30:03 #topic Election tooling (tristanC) 19:30:03 anteaya: don't understand your question 19:30:18 #link https://etherpad.openstack.org/p/scaling-election-tooling 19:30:19 tristanC is on vacation this week i believe 19:30:30 well I wrote the etherpad 19:30:36 so let me know your thoughts 19:30:55 bascially as I explain on the etherpad the current election tooling doesn't scale 19:31:12 and for the sake of both the election officials and the electorate it needs to 19:31:20 this is a proposal to address that need 19:31:44 zaro: was just saying you aren't blocked on any gerrit patches, youjust need reviews to happen 19:32:15 the foundation has a system for nominating people for board elections; i think it would be cool to use that for other elections too 19:32:21 and let's remember that when using an etherpad it is considered friendly to others to put your name in the top right hand corner 19:32:38 jeblair: great, how can we evaluate that system? 19:32:42 jeblair: I dno't know it 19:33:25 anteaya: maybe it's part of the foundation web site? 19:33:31 oh 19:33:45 I'm hesitant to use something we don't have control over in infra 19:33:57 so the specific infra items being proposed for discussion: 1. git repos for nomination and platform info, 2. some publication mechanism for the same? 19:34:08 yes 19:34:14 yeah, but i think it's worth acknowledging that there's a need that might be filled by work the foundation has already done 19:34:18 that would be a reasonable tl;dr 19:34:32 okay I'm fine with acknowledging that 19:34:38 just making sure we have actual topics to discuss rather than something nebulous 19:34:49 no idea what the next step would be to investigate it 19:34:58 fungi: yes, topics are helpful 19:35:15 fungi: i'm interested in the conversation that lead us to thinking that election nominations via git repo are a solution 19:35:21 the publication mechnisim was let a need and more an extention of using a git repo 19:35:36 well it lead me to thinking 19:35:48 jeblair: i missed the conversation myself, only commented on the etherpad after i saw it 19:35:50 it feels like git repos are the only tool we have and i'm wondering is there any other tools in that box over there 19:35:57 since this is the first opportunity it has come beofre any group that could constitute an us 19:36:17 there was no conversation 19:36:19 anteaya: are you open to conversations about other solutions, or do you want only to discuss the mechanics of the proposal in the etherpad? 19:36:33 I took a walk and realized git repos scale, and then wrote the etherpad 19:36:44 oh by all means let's discuss 19:36:49 okay, so it's more of a straw man proposal 19:36:53 but it has taken 3 weeks to get a discussion 19:37:04 definitely good as a starting point 19:37:05 and this is the only way I figured I could get the space to talk about it 19:37:06 ya, sorry, but we did better today :) 19:37:10 we did 19:37:12 thank you 19:37:18 so yes, discuss away 19:37:27 election officals need to retain their sanity 19:37:41 electorate need confidence in what is going on 19:37:51 i agree being able to reuse the nomination system would be cool, since it already has a way to link your foundation profile to your nomination 19:37:55 those are the only two stipulations in the solution 19:37:57 okay, so i've suggested the alternative of trying to adapt the foundation election system to support other elections. that is not easy because it's not (yet) an open development model and would require some code 19:38:15 we have missed this round anyway 19:38:29 as they had to make a decision to communicate to the electorate 19:38:30 probably talking with foundation folks about the progress of that and what would be involved would be useful 19:38:34 so we have until next fall 19:38:38 may not be much more to say about that here :) 19:38:39 but we do need something 19:38:53 jeblair: fair enough 19:39:00 anyone with any other thoughts? 19:39:08 you are welcome to add them to the etherpad 19:39:08 jeblair: anteaya if we send something to the infra list about that we can poke people to go read it 19:39:15 as a point of communication 19:39:20 clarkb: sounds like a plan 19:39:20 so to step back, the specific problem we're trying to address is "it's messy and time consuming to have nominations posted to the ml, confirmed, and platform statements/q&a pasted into a wiki article by a small team of election officials" 19:39:21 and/or we can point at the etherpad if we add the option there 19:39:32 * ianw will not mention storing candidates in a bitcoin side-chain :) 19:39:39 fungi: well it is overwhelming now 19:39:49 ianw: creative! :) 19:39:55 fungi: plus having stackforge use the same process confuses the electorate 19:40:08 as they keep thinking the election has started and they missed it 19:40:22 i keep thinking that too 19:40:29 so the tool needs to be clear to the electorate which are govenered elections and which aren't 19:40:29 anteaya: I don't think your proposed system fixes that though 19:40:33 wait, what, we haven't? ;) 19:40:35 clarkb: fair enough 19:40:39 let's do better 19:41:05 and yes 19:41:15 haveing more elections to administer is hard to do 19:41:25 hence the sanity for officials bit 19:41:42 thanks for listening, we can come back to this again 19:41:47 anteaya: i do think the git repo does solve a lot of problems; it may be the most expedient method of scaling this 19:41:51 and yes more ideas and communication welcome 19:41:58 jeblair: that is what I thought 19:42:03 we don't have to build a thing 19:42:07 and git scales 19:42:08 anteaya: i think it will need a lot of details worked out 19:42:15 jeblair: fair enough 19:42:22 acl permissions for one 19:42:25 maybe a list of features a solution needs/tasks it should handle is a better place to start (a requirements document of sorts) rather than jumping right into designing something 19:42:34 fungi: +1 19:42:35 sure 19:42:43 I welcome help on doing that 19:42:44 fungi: yes, i think there was some of that in the etherpad, but not organized as such 19:42:54 as I may be too close to the issue to think broadly enough 19:43:07 so let's work on a good problem statement + requirements 19:43:09 agreed. possibly with less focus on what we're doing now that doesn't work so well 19:43:14 sure 19:43:35 and then push that to the infra list and follow it up with any broad ideas we have for addressing them 19:43:36 so thanks for the time and support here 19:43:42 i mean, those are lessons learned that we need to use when considering solutions, but not necessarily requirements themselves 19:43:50 true 19:43:54 Linux Australia has a thing called memberdb that I think does most of what we need 19:44:10 but from memory it’s either a messy pile of perl, or a messy pile of python translated from perl a decade ago 19:44:10 err, I would not advocate for memberdb 19:44:11 tchaypo: cool, I'd be interested in knowing more 19:44:16 jhesketh: ah okay 19:44:20 LA themselves are trying to get rid of it :) 19:44:26 yeah, running our own election system is another interesting solution 19:44:27 ha ha ha 19:44:28 tchaypo: weren't they looking at getting rid of that? 19:44:28 well perhaps it's a contender, but it would need serious work 19:44:34 but it sounds like we have similar sets of requirements.. 19:44:38 we don't have to replace condorcet 19:44:43 it is wonderful 19:44:43 not so much get rid of as upgrade 19:44:47 just the nominations part 19:44:49 well, sounds like we might, but getting that fleshed out is really step 1 19:45:01 let’s shelve that while we investigate git though 19:45:16 to be clear just nominations is what I am looking at 19:45:27 our actual app for running the polls is wonderful 19:45:34 to be clear, memberdb is horribly broken 19:45:40 jhesketh: ack 19:45:46 nominations and position/debate details per candidate sounds like 19:45:47 I think git should work well for the nominations 19:45:49 but that's not to say it can't be fixed 19:45:55 anteaya: i know you want help on this, but i'm going to action you to at least continue driving it :) 19:46:00 we have ability for approvals and such in gerrit 19:46:01 I've taken a lot of time on this and thank you 19:46:06 jeblair: I'm fine driving 19:46:11 anteaya: feel free to interpret this as "anteaya convince someone else to..." :) 19:46:17 but we have seen what I come up with when alone 19:46:22 jeblair: will do 19:46:27 so help welcome 19:46:42 thanks all 19:46:44 #action anteaya cause a problem statement and requirements regarding election tooling to be sent to the infra list so various solutions can be discussed 19:46:55 * anteaya nods 19:47:05 anteaya: and i'll be happy to help with that :) 19:47:10 thank you :) 19:47:18 #topic Update bandersnatch (anteaya) 19:47:22 this is me again 19:47:27 mostly as driver 19:47:47 as you can see in the agenda dsufft gave us a warning to update bandersnatch 19:47:52 I don't know how to do that 19:48:00 but I do know how to copy paste to an agenda 19:48:06 so do we want to do that? 19:48:21 do we CD bandersnatch? 19:48:24 we should, its likely as simple as ensure => latest on our bandersnatch install with puppet 19:48:28 I can look into that 19:48:32 thanks 19:48:46 also, worth checking if we have "-U" on that :) 19:48:59 that's all from me 19:49:06 #action clarkb make sure bandersnatch is being updated 19:49:21 #topic Puppet Style 19:49:22 yeah, last time i manually upgraded the 4 servers 19:49:23 yea iss ensuure => present right now 19:49:37 speaking of => 19:49:39 so this https://review.openstack.org/#/c/164819/ 19:49:48 mainly so that we could downgrade again easily if it turned out to be broken 19:50:06 stylish! 19:50:06 i'm inclined to lean toward sticking with the predominant puppet style for the moment 19:50:11 i think we've got some oppinions there 19:50:17 i don't actually agree with the rules 19:50:40 but i also think we've written some terrible puppet in the past, and we're trying to write some better puppet now 19:50:40 seems like the infra cores are in agreeent, and the people more heaviliy involved with the greater puppet community are in agreement 19:51:09 and when we go out to tell people that they are doing module versioning wrong, i don't want our lack of arrow alignment to cause them to not take us seriously :) 19:51:11 I think the trouble here is the people doing the review are predominantly the cores 19:51:17 and this is a problem with reviewing code 19:51:28 I think adhereing to community style allows new people from the community to help us, rather than isolating us 19:51:37 i agree with anteaya 19:51:43 will infra-cloud puppet live in system-config or outside? 19:51:52 nibalizer: stackforge 19:51:58