19:01:26 #startmeeting infra 19:01:26 Meeting started Tue Mar 10 19:01:26 2015 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:28 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:30 The meeting name has been set to 'infra' 19:01:33 #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:33 #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-03-03-19.01.html 19:01:38 #topic Actions from last meeting 19:01:38 #action jeblair nibalizer work through openstackinfra-httpd publishing 19:01:38 #action jeblair fix openstackinfra account on puppetforge 19:01:43 Morning 19:01:46 let's just go ahead and get those out of the way ^ 19:01:53 o/ 19:01:54 o/ 19:02:05 o/ 19:02:14 o/ 19:02:17 i'm going to try to do password recovery on the puppetforge account 19:02:19 o/ 19:02:22 o/ 19:02:31 thanks. i agree the one in hiera is invalid 19:02:41 o/ 19:02:55 #topic Priority Efforts 19:03:18 last week we established gerrit topics for each effort 19:03:24 i'm going to link to them individually later 19:03:34 but also, here is a query to show you all open changes for all of our current efforts 19:03:40 #info "status:open AND (topic:enable_swift OR topic:dib-nodepool OR topic:zanata OR topic:downstream-puppet OR topic:askbot-site OR topic: gerrit-upgrade)" 19:04:03 jeblair: still in progress, ya? 19:04:04 so there's something we can all use to try to run that list of changes down to zero each week 19:04:20 oop i was scrolled up 19:04:34 nibalizer: yep, will work on that this afternoon 19:04:36 o/ 19:04:42 #topic Priority Efforts (Swift logs) 19:04:42 #link https://review.openstack.org/#/q/topic:enable_swift,n,z 19:04:46 hi 19:05:05 i should probably change these to be status:open as well 19:05:16 #link https://review.openstack.org/#/q/status:open+topic:enable_swift,n,z 19:05:20 o/ 19:05:34 for swift logs the outstanding work is double checking that we have up to date images (we should) then I can approve those two open changes 19:05:38 So nothing new here as far as I'm concerned. We just need to turn on more jobs to swift and keep rolling 19:05:41 er I guess the second needs review too 19:06:02 clarkb, jhesketh: the outstanding issues with ux are resolved? 19:06:24 jeblair: I think the only remaining one is how to display the devstack gate help footer 19:06:34 jeblair: but those changes don't touch d-g jobs yet 19:06:38 jhesketh: ^ is that correct? 19:06:59 clarkb: agreed, it seems like we should be able to proceed without that. 19:07:01 Yeah that's the only thing missing for parity as far as I know 19:07:14 for the footer, should we have the d-g jobs generate their own index pages? 19:07:38 My opinion is yes, but sdague and others disagree 19:07:59 jhesketh: where can i view this discussion? 19:08:03 I was starting to wonder if we could remove the help footer compeltely and instead try to link off to docs within the jobs themselves? but that may be too late in the deciphering of the logs 19:08:05 as i recall there were pros and cons to having the content frozen at the time when the job was run as oppsed to up to date with some reference copy 19:08:18 I think having them generated and stored will mean the documentation will stay correct as things may change 19:08:39 jhesketh: ya that is a nice benefit 19:08:40 It was irc a while ago but I can take that on notice 19:08:44 assuming it was correct to begin with 19:08:56 and harder to correct later if it was not 19:09:27 fungi: yep, that's a good summary thanks 19:09:42 * SergeyLukjanov lurking 19:09:47 OH! I know 19:09:56 i don't feel strongly either way. would just as well we flipped a coin and went with it (or chose whichever is easier to implement and moved on) 19:10:01 I was thinking about expiry, should we set an expiration date on these logs before uploading them? 19:10:09 that way we don't have to work so hard later to clean them up 19:10:17 clarkb: we can do that? 19:10:27 jeblair: yes, swift objects can have expiration dates iirc 19:10:31 Yep, it's a swift feature 19:10:42 clarkb: and tempurl passes that through? 19:10:42 and I wanted to bring this up before we have a large amount of logs in swift 19:10:44 i was wondering what the plan was for limiting retention. that's awesome! 19:10:49 jeblair: I am not sure about that 19:11:14 fungi, jhesketh: i think there are some things we can generate during the job run that should not change, and would be best saved with the run. eg "nova logs are in nova.log". 19:11:18 jeblair: I don't think it's part of the hmac, I think you just set it as an extra header when posting 19:11:37 fungi, jhesketh: other things like "here is how to use e-r" might change over time. those we should perhaps just link to. 19:11:39 the only down side is it might be harder to extend/shrink that later if we change our minds about retention of already existing data? 19:11:41 jhesketh: nice and easy then. we should probably consider getting that in early then before adding to those python jobs? 19:12:16 so i'd propose that for the d-g footer, we generate some of it in the job itself, and for things that may change over time not related to that specific run, we provide external links. 19:12:18 anyways don't need to spend a bunch of time here on that especially since we seem to agree it is a good idea 19:12:26 but yeah, i expect if we really want to update it later, we can do so by manually firing some mass update script 19:12:33 fungi: I think we can update the meta data later 19:13:09 two tasks: log expiry header and d-g footer. who want's em? :) 19:13:22 clarkb: depends if we mind cleaning up later or having a few old logs hang around 19:13:28 I can poke at the expiry header 19:13:30 jeblair: I can 19:13:36 or jhesketh :) 19:13:51 jhesketh: maybe we can look really quickly to see how much work expiry is and base our decision on that? 19:13:58 jhesketh: if we can get that in today/tomorrow probably worht waiting 19:14:04 #action jhesketh work on devstack-gate footer 19:14:10 actually expiration metadata cannot be changed later, the object must jut be deleted 19:14:17 just 19:14:21 clarkb: sounds good 19:14:27 #action clarkb jhekseth set swift expiry data 19:14:38 also, if we delete all the infra logs in swift, i'm okay with that. 19:14:44 krtaylor: oh interesting, thanks 19:14:55 it is initially set at object creation time 19:15:04 after that, it must just be deleted 19:15:11 but I wrote tools to do that :) 19:15:25 krtaylor: I guess if we were desperate we could re push them 19:15:42 can new objects be created (copied) from old ones without having to download and upload again? 19:16:24 anyway, i guess not critical to figure out such minutia in the meeting 19:16:38 yeah, good questions, but lets look into them async 19:16:39 #topic Priority Efforts (Swift logs) 19:16:39 #link https://review.openstack.org/#/q/topic:enable_swift,n,z 19:16:41 grr 19:16:43 jhesketh, it is just a matter of deleting anything that slips through the expire policy, once it is established, it takes care of itself 19:16:44 #topic Priority Efforts (Nodepool DIB) 19:16:44 #link https://review.openstack.org/#/q/topic:dib-nodepool,n,z 19:17:15 my interest here is getting better nodepool testing 19:17:20 there are several stacks of changes to nodepool up with this topic that add better testing 19:17:40 its not comprehensive but is a good start to test some behaviors we have recently noticed 19:17:45 i'd really like to get that in first because we keep breaking ourselves 19:17:50 +1 19:18:06 especailly when we start thinking about moving to shade better testing will be very valuable 19:18:07 also, ianw was going to work on yaml validation, unsure of progress there 19:18:39 i know mordred has been busy; anything else on this? 19:18:44 jeblair: out for review https://review.openstack.org/161952 <- will add job when that is approved 19:18:47 its up for review, looks large because of the yaml involved but it should be reasonable 19:18:51 ianw: neat! 19:18:55 ianw: can you update the topic too? 19:19:10 ianw: to dib-nodepool so it shows up in the query jeblair posted 19:19:18 i was working on nodepool-shade integration but no time last week 19:19:23 there is a governance patch to move bindep into infra 19:19:30 clarkb: urg, i thought i specified that with -t to git review, will look into 19:19:42 i think it will end up scheduled for next week's tc meeting 19:19:46 ianw: you can set it in gerrit ui without pushing new patchset 19:20:11 clarkb: yep, done 19:20:23 fungi: any thing in particular we should be looking at with the bindep stuff? 19:20:26 jeblair: do you want to discuss agreeing to a downtime for that move, or wait on tc? 19:20:40 jeblair: cool. i'l strive to get most of my submitted and planned features implemented before we move bindep 19:20:58 clarkb: i haven't revisited outstanding comments on my changes since i got home 19:21:05 i know i need to add some tests for some of it 19:21:05 anteaya: let's wait 19:21:10 jeblair: very good 19:21:11 My hope has been to develop a simple way to run a fake nova with a tmpfs-backed glance to point nodepool at for fast functional test runs. 19:21:40 SpamapS: aware of nova fakevirt driver? 19:21:45 But I haven't gotten beyond writing that down in a text file and breaking it up into tasks to figure out if it is even feasible or if there are other people working on the same thing. 19:21:54 jeblair: yes that is the fake part I'd use. :) 19:22:12 thought so, just checking :) 19:22:19 SpamapS: also, ++ 19:22:26 glance needs a fake-i/o 19:22:45 fungi: well you need to be able to retrieve the image. :) 19:22:56 fungi: so just throwing it in /tmp and then torching the tempdir should be fine. 19:23:15 it could accomplish that be just feeding you something, unless we need what we upload to match what we download 19:23:21 er, by 19:23:26 but anyway, sounds cool 19:23:44 it should be "web scale" 19:23:53 #topic Priority Efforts (Migration to Zanata) 19:23:53 #link https://review.openstack.org/#/q/topic:zanata,n,z 19:24:02 hi there! 19:24:04 cinerama: hi! 19:24:41 so as far as zanata goes, i have a review up https://review.openstack.org/#/c/147947/ with pleia2 which has a base functional puppet module that installs zanata 19:25:15 it needs some work. i have a further patch in the series i'm nearly done with to put an apache proxy in front 19:25:27 ++apache 19:25:37 cinerama: are we wanting to break that up into multiple changes or should we wait for a new patchset on the above change? 19:25:57 other outstanding issues - zanata can use clamav to check docs for viruses, so we could add that in the mix 19:26:02 cinerama: mostly want to avoid merging anything you think is not ready 19:26:19 and checking stuff for graceful upgrade, redundancy, etc etc 19:26:51 cinerama: i don't think clamav is necessary 19:26:55 because i am infra n00b i will need some help with best practices and methodology 19:27:27 i was thinking to break it up just to have reviewable chunks of reasonable size 19:27:29 cinerama: i hope you'll find we're very helpful :) 19:27:34 cinerama: if we have a standalone version that is running, we can focus on redundancy later 19:27:57 yeah, we're not allowing arbitrary uploads into zanata, right? this is specifically source code which has already been reviewed by humans 19:28:15 the other question is about migrating existing data 19:28:16 so virus scanning that seems entirely unnecessary 19:28:34 mrmartin, cinerama: also, in general rax has such good uptime that i think we can live with zanata being singly hosted for a while. so we can put that _waaay_ down the priority list 19:29:02 cinerama: upgrades would be cool; we should talk about how to do those, but we can also probably get the initial version up and running without it for now 19:29:12 jeblair: I agree, if we have a proper backup, we can go with a standalone install 19:29:22 cinerama: what data need to be migrated? 19:29:33 i've been super head down in "just get the thing working" mode so things like sso login etc are coming to mind 19:29:58 the strings themselves should be in the source code and probably should be "migrated" the first time we run new import jobs for them 19:30:04 cinerama: would you like to use the openstackid.org for sso login? 19:30:07 jeblair: my understanding is that we need to export stuff from transifex and import it to zanata 19:30:19 cinerama: correct 19:30:35 what is in transifex that's not in the git repos? 19:30:47 jeblair: we do not have all translations in git 19:30:51 jeblair: good question; i'm going off the spec 19:30:59 right, zanata going offline due to server trouble will be ~= to gerrit going offline 19:31:00 because we have the 75% threshold right? 19:31:06 Export all translations (not only the 75% translated ones) from Transifex and import all the files into Zanata. Also import old versions of projects with translations (for example horizon/icehouse, horizon/havana) as reference and seed for translation memory. 19:31:09 ^ from spec 19:31:11 we decided some time ago to only download files that are reasonable translated 19:31:23 jeblair: correct ;) 19:31:26 ya so there is a step but it should basically be identical to the automated tooling 19:31:34 er, the chances of them going offline will be roughly equivalent i mean, because we don't have redundant gerrit either 19:31:41 with a small tweak to get all the files 19:32:01 wrt to sso i need to take a step back and see what zanata can support 19:32:04 clarkb: just remove the limiting ;) 19:32:10 cinerama: i think clarkb and AJaeger_ should be able to help you work through what the export/import step means 19:32:32 yup happy to help 19:32:34 indeed, I can help 19:32:35 cinerama: for sso, i think openstackid supports openid or oauth2, right mrmartin? 19:32:53 jeblair: supports both 19:32:54 currently those two protocols, yes 19:33:20 we are using oauth2 because it supports of retrieving of profile pictures. 19:33:21 i think at the time we evaluated it, we were only looking at openid, so i'm fairly certain it should support that 19:33:41 yes, openid works well too 19:33:42 mrmartin: when you say "we are using" you mean for sites like group and later ask, i think. 19:33:48 groups portal 19:33:53 ya 19:34:03 for the ask I think further testing required 19:34:28 cinerama: so mrmartin can help out with sso questions. the main thing is that we want a login button, and it should take you directly to openstackid -- so no option to log in via another method 19:34:29 but the groups initially used the openid sso, so it is working well too 19:34:46 the only reason I upgraded to oauth2, that openid not provides to profile pictures of users. 19:34:53 i need to look more into what zanata/wildfly can do 19:35:16 to openstackid.org that is 19:35:22 ya 19:35:41 (openstackid is just the name of the software running on there) 19:35:43 cinerama: it feels like we're getting close to being able to stand something up, which will be exciting! :) 19:35:59 anything else on this? 19:36:09 for the openstackid testing I have a vagrant: https://github.com/fremontlabs/vagrant-openstackid 19:36:13 yes. my primary focus has been on the puppet stuff, which is in a reasonably good state. i think that's all i have for now 19:36:33 do we have a specific timeframe for this, or is it "when it's done"? 19:37:29 I like to use it for groups portal translations, I have a bunch of .po files actually 19:37:30 cinerama: i think not before the release, but if we can move over early in L that would be good 19:37:41 having it for the summit would be great if possible 19:37:55 clarkb: yeah, up by summit, in use early in L would be good 19:38:00 summit is a realistic goal 19:38:05 okay. i'm cautiously optimistic :) 19:38:11 * asselin_ has to go now. No updates from last week w.r.t. downstream-puppet 19:38:32 the translation team gets super-busy right before release, so they will have limited bandwidth to evaluate it around that time 19:38:51 starting soon, actually i think 19:39:15 cinerama: thanks! 19:39:17 #topic Priority Efforts (Downstream Puppet) 19:39:17 #link https://review.openstack.org/#/q/topic:downstream-puppet,n,z 19:39:30 i put two reviews up this week on my part of this 19:39:51 https://review.openstack.org/#/c/162830/ and https://review.openstack.org/#/c/162819/ 19:40:16 i've been sending some patches as well 19:40:35 yolanda: for which spec? 19:40:51 well, related to downstream puppet 19:40:56 * krtaylor reads 19:41:21 yolanda: ya I think I reviewed a good chunk of them today. The cleanups in the various modules to make them reconsumable right? 19:41:29 clarkb: yep 19:41:32 yes, these ones 19:41:35 er s/today/yesterday/ 19:41:43 its the great 'hey look what hp did internally, yip' 19:41:50 i had questions for https://review.openstack.org/162727 , https://review.openstack.org/161663 and https://review.openstack.org/161695 19:41:52 we're gearing up for the 'Big Sync (tm)' 19:42:12 yes, i expect to be sending more of those during the week 19:42:48 okay, so this is waiting on reviews 19:42:59 yep 19:43:14 i'm glad we're getting it moving :) 19:43:17 #topic Priority Efforts (Askbot migration) 19:43:17 #link https://review.openstack.org/#/q/topic:askbot-site,n,z 19:43:30 ok, fungi made the instance for askbot 19:43:36 fungi: hooray! 19:43:38 there's a server up and running, still need to do the data import 19:43:50 so I can access it on https, so need to move forward, and do the data import 19:43:56 do we want a separate dev server for this too? 19:44:03 that's all we have now, a good progress 19:44:16 fungi: we don't need that 19:44:35 because ask comes from pip, and only the theme lives in our repos 19:44:36 cool. then i'll try to work on the data import later today. you had those instructions written up somewhere righth? 19:44:59 fungi here: https://review.openstack.org/160693 19:45:17 aha, perfect. thanks 19:45:31 not a big deal, need to do a pgsql backup, static files backup, restore that on the other side, an rebuild the solr indexes 19:45:33 #link https://review.openstack.org/160693 19:46:06 i think that's all for this topic at the moment 19:46:15 thanks! 19:46:19 #topic Priority Efforts (Upgrading Gerrit) 19:46:26 we did not establish a gerrit topic for this... 19:46:30 "gerrit-upgrade" ? 19:46:38 ohh actually i used a different one. 19:46:45 https://review.openstack.org/#/q/status:open+topic:Gerrit-2.9-upgrade,n,z 19:46:54 i can change it if you like 19:47:00 gerrit makes that too easy 19:47:11 nothing new just need to review. 19:47:20 zaro: let's change it because it looks too hard to type :) 19:47:30 #info gerrit topic: gerrit-upgrade 19:47:50 earlier april 11 and may 9 were suggested as upgrade dates 19:47:55 should we try to settle on one of those now? 19:47:57 also we're planning to tack a switch to utf-8 onto this, sounds like 19:48:00 would like to see https://review.openstack.org/#/c/155448/ merged because it's blocking testing of gerrit/LP integration 19:48:08 april 11 is pycon, I'm for may 9 19:48:27 i don't think i have any explicit plans for either april 11 or may 9 19:48:30 fungi: you mean https://review.openstack.org/#/c/163104/ ? 19:48:34 i'm a pycon then, but i think fungi and clarkb are not... 19:48:46 correct I am not 19:48:52 nor i, no 19:49:00 however that is right during when ttx asks us to be slushy 19:49:02 i have no plans for may 9 19:49:06 which was the other reason we considered may 9 19:49:26 so let's say may 9 then? 19:49:41 wfm 19:49:54 #agreed Gerrit 2.9 upgrade Saturday May 9, 2015 19:49:58 zaro: yeah, not sure if we want to merge 163104 asap or wait until we're doing other changes. hopefully there's no negative impact from the encoding change 19:50:11 and yeah, when do we want to do the utf8 change? 19:50:12 #link https://review.openstack.org/163104 19:50:21 with the os move, or the gerrit upgrade? 19:50:32 (or immediately) 19:50:49 my only concern is effect on existing data 19:50:51 changing db url seems independent of gerrit upgrade. 19:50:56 no effect, then right away 19:51:04 or am i missing something? 19:51:12 any chance of effect, then either upgrade 19:51:27 fungi: can we do it without downtime? 19:51:39 jeblair: it needs a gerrit restart 19:51:49 i suggested one of those two because we will have downtime and database backups already in place 19:51:50 yep, just a restart 19:51:53 so in theory it could be wrapped into a the bindep move for example 19:52:06 or when we next restart gerrit to fight the memory leak :) 19:52:09 zaro: how long does it take? 19:52:33 does it involve table scans or is it just a table metadata change? 19:52:36 jeblair: since it's just a config change it appears to take just as long as a normal gerrit restart 19:52:47 just a dburi change in the config is all 19:53:08 fungi: oh wait i think i may not understand what we're talking about 19:53:15 i thought we were changing the charset of tables 19:53:35 jeblair: apparently that was being ignored by gerrit. pelix pointed zaro at changing the dburi 19:54:00 jeblair: see 163104 19:54:04 fungi: i thought zaro changed it on review-dev 19:54:05 (linked above) 19:54:16 yeah, he made that config change 19:54:26 so he did not change the tables? 19:54:36 yep, just a config change. no updates to db at all 19:54:54 zaro: the table charset modification didn't actually work, right? 19:54:58 okay, so everything about that is completely different than what we discussed before 19:55:05 i'm not at all comfortable with that 19:55:12 so we need to take this back for further discussion 19:55:13 i didn't make any changes to the db at all. 19:55:29 well, we were discussing it in #openstack-infra but i think it got lost interleaved with the zuul debugging discussion 19:55:43 yeah, something went really wrong there 19:55:47 let's work on it later 19:55:56 #topic Open discussion 19:55:56 sounds good 19:56:13 anything else in 5 mins? :) 19:56:14 oh the ansible puppet fixes 19:56:16 can we discuss the election tooling? 19:56:31 we have scalilbilty concerns and could benefit from input 19:56:41 or onto #openstack-infra ... 19:56:46 oh sorry 19:56:49 #topic Election tooling (tristanC) 19:56:50 well stuff gets lost there 19:56:56 thanks 19:57:06 First questions, what changes in ATC definition ? If ATC also means being a member of the OpenStack foundation, then we need a way to cross check gerrit list with foundation database. 19:57:24 fungi: we were told you might ahve some deveopments here 19:57:34 clarkb: link for that? 19:57:36 atc is already poorly defined compared to how we build the electorate 19:57:53 tristanC: my understanding is that being an ATC has always required being a member of the foundation 19:57:57 so i think we should get some more explicit election definitions before the tc for discussion 19:57:59 but i'm not the foundation's lawyer 19:58:02 sorry I meant the foundation db access 19:58:05 zaro: I will #link as soon as this topic is done 19:58:09 rather than discussing the charter here 19:58:22 fungi: do you have access to the foundation db 19:58:36 anteaya: right now i have some knowledge of the backend database schema which makes it _possible_ to query whether someone is a foundation member 19:58:41 jeblair: yes, but I guess the "resign button" from the foundation changed the rules 19:58:42 assuming they don't change it 19:58:46 since if you don't bringing this up again, doen't change anything 19:58:49 but a proper api is preferred 19:59:03 fungi: yes, i think if tooling is developed, it should use a proper api 19:59:16 Second questions, is the openstack-dev mailing list + wiki suited to gather nominations and present results ? 19:59:30 I feel this appraoch doesn't scale 19:59:30 but also, right now we define an atc (for election purposes) based on gerrit change ownership 19:59:36 well it didn't scale for me 19:59:43 but I'm not doing it this round 19:59:44 which is not exactly analogous to how atc is defined in the bylaws 19:59:55 but don't want to leave the election officials in the lurch 20:00:04 i think for the election definitions, someone should propose their understanding to a mailing list, and we should ask the tc and legal folks to weigh in on it and see if they agree. 20:00:17 yes, that seems like the best way forward 20:00:18 thanks 20:00:24 the infra bits are a small portion 20:00:48 as for nomination, perhaps we really do need an election platform for this 20:00:52 jeblair: sounds like a plan, thanks for the guidance :) 20:01:08 the foundation has a nomination system for the board 20:01:10 open governance enabling software 20:01:30 no tc meeting this week? 20:01:33 perhaps we could use it, or something else, to collect ptl nominations. and maybe even run condorcet elections 20:01:43 and we're out of time 20:01:54 thanks everyone! 20:01:56 #endmeeting