19:01:50 #startmeeting infra 19:01:51 Meeting started Tue Feb 5 19:01:50 2013 UTC. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:52 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:54 The meeting name has been set to 'infra' 19:02:31 old business? 19:02:41 on the agenda for today: CLA, wiki, logs, reviewday, jenkins plugins, and whatever else we want to talk about 19:02:51 sounds goof 19:02:57 good too 19:03:03 #topic CLA 19:03:18 fungi: how are things in redoing the CLA land? 19:03:33 no news is good news. also known as "i've been busy with other things" 19:03:47 we're on track 19:04:05 toddmorey and i keep missing each other to exchange the new key, but other than that good 19:04:11 the dummy endpoint for CLA checking landed on review-dev correct? 19:04:20 a few weeks back, yeah 19:04:59 other than exchanging a key is there any other work we should keep on our radar? 19:05:05 annegentle posted a follow-up to the ml, but nobody seems to have piped up past that 19:05:31 the actual cut-over will be happening on the 24th of this month, so keep that in mind i guess 19:05:50 i'll remind everybody to get some reviews done on the pending wip patches when we get closer 19:06:07 ok sounds good to me 19:06:10 they'll likely need a rebase between now and then anyway 19:06:28 I will do my best to write changes that conflict with yours :) 19:06:33 anyway, unless there are questions, we can move on to the next topic 19:06:40 none from me. 19:06:51 though i think we skipped the action items from last week 19:07:13 not that it probably matters much since most of that's on the agenda anyway 19:07:22 given that we never revised teh agenda ;) 19:07:39 #link http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-01-29-19.01.html <- last weeks notes 19:08:00 I am going to skip the wiki for now because I don't think ryan_lane is here yet 19:08:05 k 19:08:07 #topic quantal slaves 19:08:36 i've got four up and running. our automation for launching new slaves has been fixed to accommodate quantal 19:09:03 over the weekend i tested the master branch unit tests for all core and incubated projects on quantal slaves and they're fine 19:09:08 I think the tripleo folks have volunteered to guinea pig for us 19:09:30 i need to update jclouds to be capable of adding extra quantals 19:09:48 but we can press forward before that as long as we don't switch too many projects over 19:09:54 fungi: did you want to go ahead and propose changes that migrate projects one by one to quantal? (or in groups smaller than doing all at once) 19:10:24 i assume we want buy-in from the core devs on each project, right? 19:10:42 yeah I would have them +1 the change (or get the PTL to) 19:10:47 so probably separate changes, that way they don't block progress if some projects are slow reviewing 19:11:07 i can batch the server and client projects together 19:11:13 as pairs 19:11:14 ++ 19:11:39 i'll add jclouds first though, so that we can be lazy about adding persistent slaves if needed 19:12:02 sounds good to me 19:12:39 any questions about quantal slaves? 19:13:08 #topic Jenkins Build Logs 19:13:24 also known as how to lose a monday 19:13:29 heh :) 19:13:56 yesterday (Monday) static.o.o which hosts the logs.o.o vhost decided it would run out of disk space 19:14:27 #link http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=309&rra_id=all 19:14:38 this caused the gate to effectively grind to a halt as Jenkins could not copy build logs to the log server and zuul interpreted these failures as job failures 19:15:22 fungi managed to run the compression job aggressively to keep things semi sane but compressing things and cleaning up old unneeded cruft wasn't helping much 19:15:30 #link https://bugs.launchpad.net/openstack-ci/+bug/1115391 19:15:32 Launchpad bug 1115391 in openstack-ci "Root filesystem full on static.openstack.org" [High,In progress] 19:16:49 after looking at the problem more closely we realized that the tempest full runs were producing 42MB syslog which was being duplicated by screen logs (all uncompressed). That plus the other logs we were capturing put each devstack tempest full run at near 100MB of log data 19:17:28 jenkins was not compressing this data before copying it and each change runs 2 to 3 of these jobs. tl;dr it didn't take many jenkins jobs to eat up gigs of data before things got compressed 19:17:53 and we normally only compress on static.o.o every 4 hours 19:18:10 the monday rush overran that rapidly 19:18:47 the temporary solution is we are no longer copying the openstack service screen logs as syslog duplicates that data, and we are compressing the files before jenkins copies them to the log server 19:19:11 vishy pointed out that syslog truncates messages so I have written a change to increase the MaxMessageSize in rsyslog on our slaves 19:19:22 #link https://review.openstack.org/#/c/21233/ 19:20:01 this should keep things reasonable while we sort out a long term solution to this problem 19:20:50 long term options include: growing the disk space on static.o.o, storing all logs in swift, rotating logs more aggressively (there are many ways of doing this), and I am sure we can come up with many more 19:21:22 #action clarkb start discussion on long term log archival options when jeblair gets back 19:21:41 in the meantime we've got about 4gb free right now 19:22:01 we may need to come up with something we can purge between now and next week, depending 19:22:27 and if the gate queue ever quiets down we may just go ahead and grow that disk 19:22:54 my only concern there is that it comes with a bigger vm all around, and we can't as easily scale back 19:23:05 but it definitely is an easy way out 19:23:23 well, growing it a little now could give some breathing room (certainly isn't a long term solution) 19:23:28 4G doesn't seem like a lot right now 19:23:29 yeah, and we are trying to move more items to static.o.o so I don't think making it a bigger server is a bad idea in general 19:23:44 i'll buy that 19:23:53 next step up is 4g ram and 160g disk 19:24:04 right now it's at 2 and 80 19:24:59 anything else on logs? 19:25:19 #topic reviewday 19:25:40 so, last week I cleaned up the code so now it passes pep8 and pyflakes tests 19:26:02 also made it so it can take a command line argument to specify the output directory 19:26:19 puppetizing is now in progress 19:26:51 planning on putting this on static.o.o at the moment (it's small and doesn't really grow) 19:27:24 that's all I've got 19:27:35 pleia2: are there outstanding changes that need review? 19:27:49 I have been particularly bad keeping up with the review queue lately (will try to fix that today) 19:28:05 clarkb: I checked in a change, but it's more for sharing with fungi at the moment to fill in the missing pieces (not ready for proper review) 19:28:18 ok 19:28:32 though i saw pabelanger had some suggestions on it already, so you'll want to have a look at those 19:28:46 yeah, most of those are known, but it is helpful 19:28:49 and i'll start going over it after the meeting, before dinnertime 19:28:56 much appreciated 19:28:59 cool 19:29:04 #topic Jenkins Plugins 19:29:07 i need some of my java stuff reviewed if anyone wants to do it. 19:29:30 #link https://review.openstack.org/#/q/project:openstack-infra/gearman-plugin,n,z 19:29:52 zaro is a.k.a. khaido i guess. i miss nick changes sometimes 19:29:55 trying to get the number of outstanding changes down to a reasonable number to simplify git management 19:29:57 fungi: yes 19:30:30 sorry didn't inform about nick change. but it's true 19:30:37 zaro demoed gearman running jobs across multiple jenkins masters for me. it is pretty awesome 19:31:04 nice! 19:31:15 scalability, here we come 19:31:16 i've been holding off further changes until reviews come back. 19:31:35 working on documenting it and further tests. 19:31:51 zaro has also started a gearman puppet module. so there is an assortment of related items around the gearman plugin that are ready for review 19:32:14 i'll try to take a closer look in the next couple of days. i've also been very slack on reviews of late 19:32:25 me too. 19:33:09 I skimmed jeblair's LCA talk and he mentioned the ZMQ plugin there. And implied it would be available through the normal openstack-infra channels. So I will try getting that into gerrit sometime in the near future too 19:33:47 jeblair's already releasing your future work. awesome 19:33:50 have a link to his talk? 19:34:05 zaro: http://mirror.linux.org.au/linux.conf.au/2013/mp4/OpenStack_Zuul.mp4 19:34:05 the plugin is running on jenkins and jenkins-dev and is still spewing events so I think it is working now 19:34:15 zaro: watched last night, it's great :) 19:34:53 and that gets us a step closer to logstash indexing of job logs, right? 19:35:12 fungi: yes, it makes the notification of "hey logs are available" very simple 19:35:19 pleia2: got a link to talk? 19:35:25 zaro: the mp4 link above 19:35:43 pleia2: opps. thanks. 19:36:04 fungi: jeblair was suggesting zuul may use it too 19:36:18 oh, i can see where that would be helpful, yes 19:36:38 still no ryan_lane but I think we should move onto talking about the wiki move 19:36:45 * annegentle is here 19:36:51 #topic wiki 19:37:09 olaph: how are things? 19:37:15 the underlying skin on wiki-staging.o.o is different from the one I was using, but I've ported over all the style changes required for 'v1' 19:38:26 https://wiki-staging.openstack.org is not returning bytes to me. is that expected? 19:38:49 https://wiki-staging.openstack.org/wiki/Main_Page 19:38:56 i got a cert from it, but yeah, no content 19:39:12 ooh shiny 19:39:28 we should probably make apache forward you to /wiki/Main_Page when you hit / 19:39:31 I would assume that would resolve correctly when they go live... :) 19:39:35 timing out getting a response after the handshake 19:39:36 ah 19:39:38 nice 19:40:26 annegentle: olaph I know jeblair requested we move the migration up a day 19:40:27 looking good 19:40:45 I was hoping ryan_lane would be here to chime in on that 19:41:04 he works next door to where I live, shall I go knock? :) 19:41:38 pleia2: I won't stop you, but I don't think it is necessary :) 19:41:44 annegentle: how do you feel about that? 19:41:55 no problem from me, either day works 19:42:00 clarkb: o/ 19:42:00 I think both fungi and I are available on the 17th if we want to stick wit hthe original day that everyone agreed to 19:42:09 yeah, i'm cool either way 19:42:10 ttx: heelo 19:42:29 olaph: nice work on the theme 19:42:47 ok sounds like either day works for most people. if we can get a hold of ryan today and he is ok with the 16th then I think we should move it 19:42:50 ttx: ^ that work with you? 19:43:09 I'll be in jetlag mode but yes, works for me 19:43:21 if we do that, i guess we also need to follow up to the original announcement on the ml 19:43:31 fungi: yes we will need to send an update 19:43:42 yeah makes sense, I'll send an update as needed 19:44:14 is anyone aware of any additional outstanding items? I believe image upload works and the template looks good 19:45:25 I guess not 19:45:45 #action clarkb to find out if ryan_lane is ok with migrating the wiki on Feb 16 instead of Feb 17. 19:45:59 #topic open discussion 19:46:45 #action annegentle to send update to mailing list if moved to Feb 16 19:46:51 clarkb: was wondering if we could do something to get python-swiftclient 1.3.0 to Pypi 19:47:01 been chasing mordred about it to not avail 19:47:13 ttx: that was the tag that didn't push properly because the jenkins job was off? 19:47:38 that was the tag that didn't push properly because a script was missing 19:47:54 yes, we can manually push that to pypi 19:48:09 oh, sorry 19:48:16 mordred: did you want to do that? 19:48:24 * ttx blames mordred for not doing it earlier 19:48:40 clarkb: I can, unless someone beats me to it 19:48:55 clarkb: I was going to re-trigger the job in jenkins to re-test it 19:49:00 #action mordred to upload python-swiftclient 1.3.0 19:49:02 rather than pushing the tag manually 19:49:06 ++ to retriggering jenkins job 19:49:24 mordred: while you're in acceptance mode, remember to push new versioning code to projects asap 19:49:41 ttx: https://review.openstack.org/#/q/status:open+branch:master+topic:oslo-version,n,z 19:49:54 mordred: cool, thx 19:49:59 mordred: any news on the rhel slave licensing situation? 19:50:21 fungi: I learned new things at LCA, but need to follow up on them 19:50:29 okay, awesome 19:52:05 we will probably want to follow http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-0169.html 19:52:06 clarkb: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem. When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0169) 19:52:19 mordred: also I'll probably start up summit.openstack.org on the same cloudserver instance as for past summits 19:52:34 mordred: couldn't push it to infra proper and now time is running very short 19:52:40 ttx: ok 19:52:50 mordred: I kinda want to do it myself as an exercise 19:52:57 so I'll do that early in H. 19:53:07 * ttx remembers saying that 6 months ago too. 19:53:22 :) 19:53:27 we could trap ttx in a bar in portland and make him do it then 19:54:05 anything else? 19:54:30 #endmeeting