19:01:46 #startmeeting infra 19:01:47 Meeting started Tue Dec 16 19:01:46 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:50 The meeting name has been set to 'infra' 19:01:52 jeblair, hey, mostly ok, /me monitoring the USD-RUB :) 19:01:53 o/ 19:01:57 #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:02:00 \o/ 19:02:00 o/ 19:02:01 #link last meeting http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-12-09-19.02.html 19:02:06 o/ 19:02:11 o/ 19:02:13 ohai 19:02:17 o/ 19:02:25 o/ 19:02:28 #topic Actions from last meeting 19:02:31 o/ 19:02:36 clarkb add DENY Email Third-Party CI rule to gerrit ACLs, giev Third-Party Coordinators ownership of Third-Party CI, seed Third-Party Coordinators with third party meeting attendees 19:02:42 jeblair: done 19:02:51 yay 19:03:10 rock on. does that more or less conclude the self-service 3pci project? 19:03:23 o/ 19:03:28 jeblair: I think so. Haven't heard screaming and things seems to still be getting tested 19:03:30 just the patch to remove -requests from the servers 19:03:42 oh right the mail list 19:03:45 the patch is in, we need manual disabling 19:03:47 which is an artifact 19:03:49 cool 19:03:57 pleia2: can/will you do that? 19:04:27 jeblair: can't, I think rmlist on the server will be needed 19:04:36 oh i thought we were going to leave the archives 19:04:40 are they interesting? 19:04:46 the plan was to leave the archives 19:04:47 we should confirm though, rmlist in the man page says it won't remove archives, but I don't know where the archives end up 19:04:51 you need rmlist -a 19:04:54 not terribley but they are history 19:04:55 but we don't want that :) 19:05:00 o/ 19:05:03 and we did at one point reference them 19:05:26 okay, so rmlist without -a? 19:05:40 seems like they might be hard to find if that's the case 19:05:46 might just keep them on the filesystem, which is sub-optimal 19:05:59 if they aren't discoverable no point in keeping them 19:06:17 anyone else know anything about mailman's rmlist? :) 19:06:19 I just was hoping to avoid a blackhole if someone found an old breadcrumb trail 19:06:21 yeah. probably we should either: a) put the list defn back and just disable subscription, etc. or b) remove everything. 19:06:31 +1 to disabling subscription 19:06:36 alternatively, just switch the list to reject 19:06:43 er, that 19:06:51 I'd rather disable than reject 19:06:55 so, the list sort of stays alive 19:07:00 too much confusion in this space as it is 19:07:08 pleia2: :( 19:07:23 i don't actually think the archives are interesting, so i'm okay with either. 19:07:30 they aren't interesting 19:07:44 we can put notes up everywhere about how it's dead, but it'll still show up on lists.openstack.org and people can try to subscribe (it'll just discard them) 19:08:05 what happens if we remove everything? 19:08:06 is this something consensus from the third-party working group would be helpful to solicit? 19:08:23 i don't think so 19:08:46 in that case, i'm fine seeing it just removed, archives and all 19:08:57 ya I guess the archives aren't particularly interesting 19:09:03 just requests and calrification of requests 19:09:06 okay looks like the tendency is for it to go 19:09:07 I can see them being removed 19:09:07 i don't think we would do that with most lists, but i think this is an unusual circumstance 19:09:14 so I am fine with it going 19:09:27 #action jeblair rmlist -a third-party-requests 19:09:42 #topic Priority Specs 19:09:43 thanks 19:09:50 Migrate to Zanata: https://review.openstack.org/#/c/133222/ 19:09:59 I reviewd that one +2 19:10:02 who wants to +A? :) 19:10:13 this is ready to merge; anyone have any last minute requests to hold off on +A for approval, otherwise it's going in 19:10:34 great work pleia2 19:10:35 sorry, hold off on +A for additional review 19:10:49 pleia2: thanks a lot! 19:11:00 woo, my first spec 19:11:02 done. yaaay! 19:11:07 \o/ 19:11:09 thanks all 19:11:25 let the games begin 19:11:35 great spec, pleia2 19:12:06 this one isn't really a priority spec, but again i just want to call it out for infra review because i think we have considerable interest in the area: 19:12:07 Storyboard streaming API: https://review.openstack.org/#/c/105252/ 19:12:40 #topic Priority Efforts (Swift logs) 19:12:51 oh, so we had that rax issue last week; did that get resolved? 19:13:00 seems like it 19:13:14 i was able to browse recently uploaded logs there 19:13:37 yes I think it resolved, but I didn't track down the bug to see what they had to say 19:14:09 Thank you for the update. I was able to do a little more digging into the issue and did notice that the container in question 'infra-files' does not have the header 'X-Container-Read: infra-files-ro' associated to it. You will need to make sure that the header is added to this container. 19:14:23 also there is a proposal to add this to the dg-tempest- jobs so that we can throw a bit more load at it 19:14:29 that only prompts more questions, in my mind. 19:14:42 clarkb: I merged that patch last night I do believe 19:14:44 jeblair: hrm how did it ever work then 19:14:49 anteaya: awesome 19:14:52 if we are thinking of the same patch 19:14:53 let's debug more later 19:15:10 #topic Priority Efforts (Puppet module split) 19:15:21 asselin: ping 19:15:42 Hi, I wrote some patches to accelerate module splits 19:15:51 #link https://review.openstack.org/#/c/140523/ 19:15:54 there're two "unusual" patches: https://review.openstack.org/140523 and https://review.openstack.org/140548 19:16:02 #link https://review.openstack.org/#/c/140548/ 19:16:11 asselin: sorry, for interrupting ;) 19:16:39 would like to get ppl's opinions. It is a bit 'different' 19:16:51 im for it 19:16:54 but we can go much faster by pre-approving changes 19:17:05 than doing the same review one by one * 50 19:17:10 * anteaya notes she still has a problem with hyphens and underscores in the same repo name 19:17:13 These patches add all the infrastructure for the projects in one bulk to make the real imports easier. 19:17:27 so I disagree with the assertion that it is easier to review them all at once 19:17:31 i left some comments that i think elasticsearch is gonna split before the big patch lands so might as well rebase on it and intentionally do that 19:17:34 that change is ~800 lines and that is hard to erview 19:17:42 i actually agree with clarkb 19:17:53 nibalizer: elasticsearch is split 19:18:02 aha so i was right! 19:18:08 one of the issues i've seen is that git doesn't diff/merge the yaml files very well 19:18:10 I expressed my concerns in the patch comments 19:18:24 even if the resultant patches end up being fewer edits 19:18:30 so if you have 3 or more 'split outs' in flight and one lands the rest need rebases which slows everything down 19:18:40 I still need to check the foundation is there and spelled correctly during my review 19:18:45 mmedvede: i don't see your comments 19:18:49 diff/patch algorithms generally have problems with files containing lots of repetition 19:19:09 so while it seems like less work for the patch owner, the work for the reviewer is actually more 19:19:10 jeblair: https://review.openstack.org/#/c/140548/ 19:19:17 ah the other one :) 19:19:22 since the jobs I have to confirm aren't in the patch 19:20:03 anteaya: what jobs? 19:20:24 when I review a patch for a new repo creation, any jobs that are run on it 19:20:31 i don't think 140548 solves anything. you'll still hit merge conflicts (probably much more often) 19:20:38 I confirm the name and the job name in jenkins/jobs 19:21:03 anteaya: confirm it against what? 19:21:08 if the group decides they want this then fine, I'm just sharing my workflow when I review repo creation patches 19:21:22 the zuul/layout.yaml jobs 19:21:41 and the project.yaml files 19:21:46 I compare them 19:22:02 and make sure there is a jenkins job and taht it does what is expected 19:22:05 anteaya: there is a job that automatically ensures that every job listed in layout.yaml is defined by jjb 19:22:11 anteaya: we have a job which confirms that the jobs added in layout.yaml are actually configured for jenkins-job-builder 19:22:22 that 19:22:40 anteaya: so the only thing missing from 523 is the repo itself 19:22:46 it doesn't analyze the operation of the job itself, but it does at least make sure it exists 19:22:55 right 19:23:11 like I said if the group wants this then I will review as required 19:23:25 so when the repo is added, you can then double check that the (already defined) jobs match 19:23:27 I still am not a fan of hyphens and underscores in the repo name 19:23:35 right 19:23:44 which I do in new repo creation patches 19:23:51 is there a technical reason for hyphens/underscores? 19:24:06 uh kinda 19:24:08 it can lead to errors 19:24:10 we went throught this with apache 19:24:25 which is why we came up with puppet-httpd 19:24:27 a class in puppet, openstack_project for example, has to be letters and underscores 19:24:41 by convention we have the repos named puppet-thing 19:24:43 fyi, I kept the name the same as in module. 19:24:47 so that leads to puppe-thing_thing 19:24:56 asselin: I know 19:25:17 i am super strongly against trying to wedge a module rename into a split out 19:25:27 nibalizer: me too 19:25:32 It's 54 repositories that asselin tries to add (if my math is right). Do we want 54 single patches? Or how do we want to do it? 19:25:33 we can rename it inside of system-config before we split it out if we really need this 19:25:54 nibalizer: that would work for me 19:26:03 what would we rename them to? 19:26:21 im not sure what the offending modules are named 19:26:25 let me look 19:26:27 names without underscores in them presumably 19:26:45 yes, this is just 'base work'. We can still adjust special cases, or delete some we don't want to split out in follow up patches. 19:26:56 puppet-elastic_recheck for example 19:27:27 I think the dash might be puppet module naming thing 19:27:30 #link https://docs.puppetlabs.com/puppet/latest/reference/modules_publishing.html#a-note-on-module-names 19:28:04 this would also be easy to do in the split script that I am building, so if anyone thinks this is a good candidate for automation, I can include it. 19:28:48 for reference, the full list of affected modules is elastic_recheck, devstack_host, etherpad_lite, log_processor, mysql_backup, openstack_project, project_config, remove_nginx, ssl_cert_check, unattended_upgrades 19:29:23 i dont want to split out openstack_project or project_config 19:29:34 so I am for asselin's suggestion to remove those from this patch 19:29:41 nibalizer: which the proposed change does, fwiw 19:29:41 asselin: did you get the list from the spec? 19:29:52 fungi: yea i noticed that and -1'd 19:30:07 jeblair, no, but can update it to the spec 19:30:37 if we get agreement on the direction 19:31:22 asselin: yeah, let's work from the spec if we decide to do this 19:31:59 i'm generally not a fan of commented out functional content in revision control systems, but can get over it if the proposal has general support 19:32:02 will we decide now? or later? 19:32:28 i think it's going to take too much time to hash out the details in this meeting 19:32:40 asselin: can you work with nibalizer and anteaya to resolve remaining issues 19:32:53 * nibalizer nods 19:32:55 jeblair, yes 19:33:05 also clarkb and fungi :) 19:33:09 also be aware that new modules end up getting incubated in system-config rather than started separate (perhaps that needs more consideration for future direction too) 19:33:26 fungi: yes I think we need to start starting separate in the near future 19:33:29 fungi: oh I didn't know that 19:33:33 fungi: yeah, i think we can establish policy on that once the split is done. 19:33:43 fungi: we had been doing it that way, but now there is enough momentum behind splitting we can talk about what the future looks like 19:33:47 right now it would just be a distraction 19:33:56 a plan like this will quickly become disjoint and out of sync if we continue with that process 19:34:02 * nibalizer agrees with most of this ^ 19:34:24 fungi: well, if we merge everything quickly, it won't have much time to get out of sync. 19:34:32 this is true 19:34:56 #topic Priority Efforts (Nodepool DIB) 19:35:13 though "merge everything quickly" would mean finish the module split quickly 19:35:13 jeblair, sorry, regarding merging everything quickly 19:35:15 thanks to the debugging efforst of jeblair nodepool is running on trusty now 19:35:23 the images all work now 19:35:24 so i guess we can resume work on nodepool contos? 19:35:28 I was proposing (also in agenda) for a mini sprint 19:35:29 that reminds me, asselin - you mentioned maybe doing module-split sprint in 3rd party meeting yesterday 19:35:30 yay 19:35:32 nodepool centos works 19:35:32 jeblair: yup 19:35:39 at least dib nodepool centos works 19:35:48 and boots and everything 19:35:52 I think next steps in this space are reviewing and deploying the various bug fixes that have been pushed to nodepool 19:36:04 now I'm writing the python version of the bash scripts I've been using to do this 19:36:08 then figuring out why hpcloud didn't want to boot the images we built 19:36:09 so that I can make some patches to nodepool 19:36:48 it seems those want to go on top of the various bugfixes clarkb mentions 19:37:06 I am also currently cleaning up old nodepool so that it can be deleted 19:37:12 woot 19:37:13 and just found a new nodepool bug in the process :) 19:37:19 okay, so nothing blocking there? just resume speed and try to catch up on nodepool reviews? 19:37:23 will file that on storyboard post meeting 19:37:25 jeblair: yup 19:37:27 yup 19:37:37 who put "Explore idea of upgrading Gerrit to ver 2.9.x" on the agenda? 19:38:03 me 19:38:12 #topic Explore idea of upgrading Gerrit to ver 2.9.x (zaro) 19:38:36 so a few people suggested we upgrade. i wanted to get feedback on that. 19:38:43 ver 2.9.3 now 19:38:57 zaro: I am on board with it as long as ssh stream events works 19:39:01 I am in favour since it allows us to use a plugin that will list all users in gerrit 19:39:08 probably put it on review-dev and attach a bunch of event stream readers 19:39:23 #link https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.9.3.html 19:39:23 currently we can't list all users, so I can't search for all users with CI at the end 19:39:38 now under 2.9 I still won't be able to, but an admin can 19:39:51 zaro: is the old change screen still existing on 2.9? 19:39:53 new reports coming that it has been fixed. 19:40:01 yes it is 19:40:04 cool 19:40:11 even available in 2.10 & 2.11 19:40:13 clarkb: known bug with ssh stream events? 19:40:30 jeblair: yup it was beingworked and 2.9.3 was supposed to fix it. Just proposing we actually test that specific thing 19:40:47 i am wholly in favor of keeping in sync with latest releases of gerrit when our dev cycle activity allows (e.g. not in the middle of release crunch) and if there are no breaking bugs we know about 19:40:56 sounds like it's worthwhile to explore 19:41:34 so, yes, with old review screen still available in it and that stream bug supposedly fixed, now seems like a good time to press forward 19:41:44 the new change screen is looking _better_ than last time. i think the comments display is still too wide to read comfortably, but that's relatively minor. 19:42:06 zaro: can you get review-dev up to 2.9? 19:42:18 jeblair: yeah stream events stopped working for some people after a few days. but latest repors seems to indicate that it's actually the jenkins gerrit trigger that was the cause. 19:42:21 should we consider 2.10? 19:42:34 #link https://groups.google.com/d/msg/repo-discuss/4va1DH520to/pDGmY7pNUG4J 19:42:42 2.10 is not released yet. 19:42:43 zaro: well people do use the jenkins trigger 19:42:57 gerrit itself is running 2.10-rc1-938-gd3591cd 19:42:59 clarkb: ahh yeah, that's true. 19:43:18 and having the stream events processes pile up on on our gerrit because of their use of the plugin would be not-so-great 19:43:22 zaro: you mean jenkins gerrit-trigger alone is the cause, or jenkins-gerrit-trigger is incompatible with 2.9? 19:43:53 jeblair: jenkins gerrit-trigger alone is the cuase. 19:44:18 also, to be fair, i think we may have seen this once with our current version 19:44:38 i think clarkb and i both like the idea of using same version as gerrit-review.googlesource.com 19:44:42 this might then explain the hung stream events process pile-ups we've seen in the past (though it's been a while since i last spotted on) 19:44:42 yeah 19:45:24 that would imply much more frequent upgrades 19:45:48 the biggest downside with it is we would quickly suck down the latest weirdness that they do 19:45:57 and we seem to have a very different use case 19:46:00 yah - and we have a different set of primary use cases 19:46:02 yah 19:46:25 sorry about that last link. this one has latest reports: #link https://groups.google.com/d/msg/repo-discuss/NuFti4SVNQM/Ks8FN6xmW9AJ 19:47:23 i need more time to read about the ssh issue; let's pick this up later 19:47:42 #topic Create plugins for Gerrit <-> Storyboard integration instead of using jeepyb (zaro) 19:47:52 me again. 19:47:59 * krotscheck lurks 19:48:07 zaro: can you tell us about "its" plugins briefly 19:48:18 that's probably new to most of us 19:48:26 its is short of issue tracking system 19:48:29 also, for clarification, you're talking about the update_bug.py script specifically? or something else 19:48:38 fungi: yeah, that i believe 19:48:40 its-storyboard is gerrit storyboard integration 19:48:55 #link https://storyboard.openstack.org/#!/story/2000012 19:49:05 #link https://gerrit-review.googlesource.com/#/c/60590 19:49:20 zaro: what can an its plugin do? 19:49:26 its-storyboard will allow users to configure updates to storyboard from updates to gerrit change (via gerrit stream events_ 19:50:15 so it can add comments to storybaord and update status 19:50:28 if there's a plugin, why would gerrit stream-events come into play? 19:50:59 that seems less robust than using gerrit hooks 19:51:05 it gets the info to update from info in the stream events 19:51:34 like, from the inside-of-gerrit stream events interface, right? not by having a thing parsing stream-events over ssh 19:51:37 it's using the same info as what stream-events sends, but not directly subscribing to the event stream socket? 19:51:45 okay, that's a little more sane 19:52:00 yes, you are both correct. 19:52:40 zaro: just out of curiosity - what does its stand for? (looking at the code) 19:52:45 i'm basically done with the plugin. just need to make a change to add it to system-config 19:52:50 mordred: 'issue tracking system' 19:53:01 mordred: it seems to be a class of plugins which are beginning to appear in gerrit 19:53:05 cool 19:53:12 that makes sense 19:53:38 zaro: so maybe you could send an email to the -infra list describing the plugin and why that approach is better than jeepyb and hooks 19:53:44 i'm just finishing up with documentation and some tests. probably one final push. 19:54:01 jeblair: sure, can do. 19:54:03 one final thing - zaro, it seems that you're writing a java storyboard client library inside of that plugin - should we make a java-storyboard repo somewhere that we can release to maven and have this plugin depend on it? (in case other people want to write storyboard integration too) 19:54:32 that seems like an excellent idea. basically it could be the seed for a java storyboard sdk 19:54:34 yeah, i thought about that. would be a good idea. 19:54:35 it doesn't look like the StoryboardClient class is aware of gerrit at all 19:54:38 ++ 19:55:05 * mordred thinks this is neat 19:55:22 mordred: it's actually just used in the StoryboardItsFacade.java class 19:55:36 zaro: yep, sounds very cool. next time, be sure to let us know the cool things you're working on before they're done. :) 19:55:37 i am thrilled to see some of our hackish glue replaced by sane interfaces between consenting applications 19:56:08 aha, i don't finish every cool thing i start :) 19:56:17 * mordred is going to be amused when we have a java client lib before we have a python client lib 19:56:32 mordred: clearly we should rewrite everything in java 19:56:39 #topic Strategy for upstreaming Gerrit command to ls-members of 'Registered Users' group (also zaro?) 19:56:45 * fungi starts a version of git-review in java 19:56:53 me once again. 19:57:17 * mordred declares today as national zaro day 19:57:25 I think this one came from my request 19:57:31 so anteaya was interested in having an ls-members command that would list all users in gerrit. 19:57:34 again to be able to search all users 19:57:48 which we currently can't do 19:57:50 or I can't 19:57:59 there's a could of avenues to maybe get this just wanted to explore before i start something. 19:58:02 well, you could via the rest api... enumerate all user ids 19:58:16 but that would be slow and ugly 19:58:46 so I would like this but does anyone else care? 19:58:49 the command laready exists in a plugin. #link https://gerrit.googlesource.com/plugins/admin-console/+/master/src/main/resources/Documentation/cmd-ls-users.md 19:58:52 anteaya: what's the use case? 19:59:06 find the CI account that the owner can't remember the name 19:59:13 happened all the time 19:59:29 with bouncing ci's back to all users, I can't search that 19:59:34 since it is a virtual group 19:59:35 but clarkb and fungi suggested that it should be in Gerrit core so we should propose a change there. 19:59:49 anteaya: why do you care about the name? 19:59:58 if we disable or reenable them 19:59:59 zaro: ya, that suggestion came from the command already existing it just doesn't work right on that group 20:00:03 to find the account 20:00:12 I used to search third party group often 20:00:13 zaro: so ideally we can fix the existing command to work on the meta group thing 20:00:14 clarkb: i agree. 20:00:27 more specifically, just suggested that virtual groups should have read api behaviors similar to configured groups 20:00:29 clarkb: yep. good suggestion. 20:00:37 anteaya: but i mean you have something to go on, right? 20:00:43 sometimes 20:00:46 sometimes not 20:00:46 anteaya: either a comment in a change, or an email address? 20:00:53 I never used to 20:00:59 anteaya: how can you be interested in an account but start with nothing? 20:01:07 reenable me from trinaths 20:01:11 would be all I would get 20:01:12 anteaya: when disabling a ci account, we can use e-mail address, ssh username, account id number... 20:01:17 right 20:01:22 but I don't ahve those now 20:01:25 anteaya: trinaths does 20:01:31 this was what I used to get frm the command 20:01:38 jeblair: ha ha ha 20:01:42 his ci system uses a specific ssh account name 20:01:44 you havn't worked with him much 20:01:54 okay well if I don't need this fine with me 20:02:03 just to let you know I used to have it and don't now 20:02:08 and we are over time 20:02:27 anteaya: i would suggest insisting on requests with correct information 20:02:32 thanks all 20:02:35 #endmeeting