19:02:50 <fungi> #startmeeting infra
19:02:51 <openstack> Meeting started Tue Dec 29 19:02:50 2015 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:52 * nibalizer drinks 'monster zero ultra'
19:02:54 <openstack> The meeting name has been set to 'infra'
19:03:15 <fungi> nibalizer: i've not heard of that style of eggnog yet
19:03:18 <zaro> o/
19:03:20 <pleia2> hehe
19:03:21 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:28 <fungi> #topic Announcements
19:03:37 <zaro> seattle is eggnog crazy!
19:03:45 <zaro> can't seem to find any at starbucks
19:03:56 <fungi> nobody has brought anything importantly announce-worthy to my attention since last week, but i've also not been around much. anything i'm forgetting/overlooking?
19:04:14 <bkero> zaro: make some :)
19:04:33 <zaro> bkero: i'm not a fan
19:04:36 <fungi> so aside from bkero's homemade eggnog, no announcements this week
19:04:48 <fungi> #topic Actions from last meeting
19:04:54 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-12-22-19.03.html
19:05:08 <fungi> fungi remove standing Priority Effort meeting agenda item for "Gerrit 2.11 Upgrade"
19:05:17 <fungi> that's done. we won't be covering it this week
19:05:22 <kzaitsev_mb> is this thing on? =)
19:05:27 <fungi> except insofar as i just did ;)
19:05:36 <fungi> AJaeger move gerrit-2.11 spec to the implemented list
19:05:43 <AJaeger> done
19:05:44 <fungi> that's done now too. i approved it earlier today
19:05:48 <fungi> thanks AJaeger!
19:06:21 <fungi> #topic Specs approval
19:06:56 <fungi> PROPOSED: "Consolidation of docs jobs"
19:07:02 <AJaeger> #link https://review.openstack.org/246550
19:07:05 <fungi> #link https://review.openstack.org/246550
19:07:09 <fungi> #undo
19:07:10 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0xa829690>
19:07:12 <AJaeger> sorry, fungi
19:07:14 <fungi> np
19:07:23 <fungi> i have the power of #undo at my fingertips
19:07:28 <AJaeger> thanks
19:07:52 <fungi> so because people aren't probably around much this week, i'm thinking of setting voting for this to expire a week from thursday
19:08:00 <AJaeger> I'd like some feedback on the general direction of this and whether to do this for both check/gate and post.
19:08:16 <fungi> my calendar says that's january 7
19:08:42 <AJaeger> yep
19:08:49 <fungi> AJaeger: so you feel it's in sufficient condition to at least go up for a vote for approval at this stage?
19:09:00 <pabelanger> AJaeger: Nice spec!
19:09:07 <fungi> or are you trying to refine it still?
19:09:35 <pleia2> I've already voted so probably repeating myself, but I'm still liking the approach
19:09:36 <AJaeger> fungi: I hope it's good - but there's still the question whether it's worth doing it ;)
19:09:53 <fungi> that sounds like you're ready to put it up for a vote to me
19:10:14 <AJaeger> And it's easy to do for post jobs but harder to do right for check/gate, so those are two questions I'd like you to look at when reviewing
19:10:36 <fungi> #info Council voting is open on the "Consolidation of docs jobs" spec until 19:00 UTC on Thursday, January 7, 2016.
19:10:55 <fungi> i need to get myself used to typing "2016" now
19:11:20 <fungi> thanks AJaeger!
19:12:16 <fungi> i don't see any priority efforts items up for discussion, so will move right along to the one meeting topic on the agenda
19:12:58 <fungi> #topic How to setup translations? (AJaeger)
19:13:00 <pleia2> AJaeger is a popular guy this meeting (as it should be)
19:13:15 <AJaeger> ;)
19:13:29 <AJaeger> #link http://lists.openstack.org/pipermail/openstack-infra/2015-December/003610.html
19:13:36 <fungi> #link https://review.openstack.org/262073
19:13:37 <pleia2> AJaeger: so I didn't get a chance to reply to the thread, so hi! have we had recent discussion with the horizon folks about any of this?
19:13:41 <fungi> was also in the agenda
19:13:45 <AJaeger> The email ^ explains the problems I have
19:14:00 <kzaitsev_mb> this was probably brought up with some help from my side
19:14:22 <pleia2> I think part a is pretty easy and your suggestion seems fine, but b is much harder
19:14:34 <pleia2> (c is fine too)
19:14:35 <AJaeger> pleia2: no discussions with any of the projects at large - but I had discussions with each individual when the last 5 or so translation setups merged, a week ago with dougwig and dhellmann  about some networking repo
19:15:12 <AJaeger> we can do these three parts independent
19:15:15 <pleia2> I think mostly I lack history as to why dashboard things are such a mess
19:15:23 <pleia2> maybe "because django"
19:15:34 <AJaeger> but I wanted to bring up the challenges I have - when I see a new translation setup coming in...
19:15:36 <anteaya> that an dashboards don't scale
19:15:41 <anteaya> and*
19:15:46 <pleia2> anteaya: don't scale?
19:15:51 <clarkb> I think drop django.pot and use modulename.pot and modulenamejs.pot
19:15:51 <anteaya> you only have so much screen real estate
19:16:02 <fungi> django having its own translation management implementation was the only explanation i was aware of
19:16:07 <clarkb> then its not django specific and non django web thi gs can use the same setup
19:16:15 <pleia2> anteaya: ah, this is mostly about where the translations land
19:16:20 <pleia2> s/translations/translations files
19:16:20 <anteaya> pleia2: ah sorry
19:16:27 <AJaeger> clarkb: would be my preference as well, we need to figure out whether it's hardcoded by django or just configured this way by us
19:17:23 <pleia2> AJaeger: maybe that's something we can approach the horizon team about?
19:17:34 <AJaeger> pleia2: yes, we should
19:18:16 <pleia2> clarkb's suggestion would be my preference as well if it's possible
19:18:48 <AJaeger> My main question here is: Do you agree with the approach in general and how to move forward?
19:19:09 <fungi> of course, any ways to make this more consistent across projects is preferable, if some of the current inconsistency is merely something we're inflicting on ourselves unknowingly
19:19:12 <pleia2> strong yes from me, having been in the trenches with this inconsistency, it's a serious time sink
19:19:19 <kzaitsev_mb> one interesting thing is that horizon doesn't use extract messages, but rather switched to babel for that. part of the reason I'm aware of is that they needed it for angular/js things to work correctly.
19:19:28 <clarkb> yes I think better standards here will simplify the scripts and jobs
19:19:37 <AJaeger> fungi: Some of the existing inconsistencies are historic relicts.
19:19:55 <AJaeger> We used $repo/locale/$repo.pot - and this worked fine for the main projects
19:19:58 <fungi> AJaeger: great, so baggage we can stop carrying now that it's ceased to be necessary?
19:20:15 <AJaeger> But then oslo.log got renamed to oslo_log and python-novaclient wanted translations as well and now our scripts didn't work...
19:20:48 <AJaeger> fungi: It's not baggage, we've allways done it this way and never adopted
19:21:09 <AJaeger> but if we do make changes now, we might need to update a majority of the translated repos...
19:21:36 <AJaeger> Which seems easier to me than to explain twice a week why our scripts enforce some locations...
19:21:46 * pleia2 nods
19:22:00 <clarkb> well I think we still need to enforce consistency
19:22:11 <AJaeger> the move to zanata was good - now everybody wants translations ;)
19:22:19 <clarkb> but it can be more sensible consistency
19:22:38 <AJaeger> clarkb: Agreed
19:22:57 <AJaeger> it was sensible when there was only cinder, nova, swift translated
19:23:05 <AJaeger> but we're beyond that
19:23:43 <AJaeger> So, what's the next step? Refining the document to a spec, asking horizon folks especially for feedback?
19:24:05 <pleia2> +1
19:24:08 <AJaeger> Is this a cross-project spec or an infra one?
19:24:20 <pleia2> I think it's an infra one
19:24:38 <pleia2> we'll need to include i18n, but same was for zanata
19:24:49 <pleia2> and translations check site
19:25:36 <AJaeger> anybody wants to work with me on the spec - and implementation?
19:26:07 <pleia2> you're welcome to grab me for reviews and testing
19:26:30 <AJaeger> Thanks, will do, pleia2 !
19:27:34 <AJaeger> Anything else to discuss on this? Otherwise I suggest to conclude and ask fungi to give me an action to write a spec
19:27:53 <pleia2> seems like we're all on the same page, thanks AJaeger :)
19:28:04 <fungi> yeah, seems like a reasonable way forward
19:28:22 <fungi> #action AJaeger propose a spec to consolidate translation setup
19:28:36 <fungi> does that capture the next step accurately?
19:28:53 <AJaeger> it does for me
19:29:32 <fungi> great!
19:29:48 <fungi> nobody had anything else on the agenda, so...
19:29:50 <fungi> #topic Open discussion
19:30:17 <zaro> when's the next rename?
19:30:22 <zaro> project rename
19:30:29 <anteaya> do we need one?
19:30:32 <clarkb> gerrit admins can go to https://review.openstack.org/monitoring and expand the other charts link to see how much time gerrit is spending GCing
19:30:48 <AJaeger> anteaya: ceilometer-specs to openstack/telemetry-specs
19:30:48 <clarkb> if you look at it now you can see the dramatic difference between normal and slow
19:30:53 <anteaya> :(
19:31:05 <anteaya> can we call finished on renameing things for these folks?
19:31:10 <anteaya> once this is done
19:31:29 <zaro> i see 2 items in upcoming project renames.
19:31:53 <anteaya> zaro: one is the patch to governance
19:32:04 <anteaya> so two patches, one rename
19:32:37 <nibalizer> clarkb: thanks
19:33:14 <zaro> so i have a few patches up for gerrit-tuning. was wondering whether we should do that now or wait another week to see whether gerrit slows down for forced restart?
19:33:31 <clarkb> ou can also see how that maps to used memory, cpu use, active threads etc
19:33:31 <zaro> or wait until rename time.
19:33:50 <clarkb> what seems to happen is we run low on memory, this forces all threads to contend for memory which makes the garbage collector run more frequeny
19:33:53 <clarkb> *frequently
19:33:55 <anteaya> zaro: oh I'm all for discussing gerrit tuning separate from renames
19:33:58 <clarkb> nibalizer: there is a graph for perm gen too
19:34:05 <anteaya> zaro: I don't see them as related at all
19:34:08 <nibalizer> clarkb: i unsee it? where shhould I be clicking
19:34:16 <clarkb> nibalizer: what part do you unsee?
19:34:16 <zaro> clarkb: we merged a few patches increasing threads, wondering whether it's too much?
19:34:38 <nibalizer> i see 'used memory'
19:34:53 <clarkb> nibalizer: there is used memory, used non heap, and loaded classes
19:34:59 <clarkb> perm gen is laoded classes aiui
19:34:59 <nibalizer> oh other charts
19:36:54 <fungi> yeah, i've been avoiding bringing up project rename scheduling for the past few meetings because we had the gerrit upgrade maintenance to prepare for, and then went straight into a couple of weeks where people weren't going to be around
19:37:12 <fungi> so figured i'd put it on next week's agenda
19:37:21 <pleia2> anteaya: I don't think renames will ever end, there will always be one offs here and there, retirement of stackforge only gets rid of the quarterly ones
19:37:45 <crinkle> can we schedule a paste/wiki/nodepool downtime for the puppet-mysql upgrade?
19:37:46 <anteaya> pleia2: yeah, just needed to sigh a bit I guess
19:38:11 <pleia2> crinkle: I think that's a good idea
19:39:30 <fungi> now that i'm back from my travels, i want to work on switching paste.o.o back to trove too
19:39:48 <fungi> we'll see if i'm allowed time tonight/tomorrow to do it
19:40:12 <zaro> clarkb: should we not change gerrit for a little while to see if it's stable?
19:40:27 <clarkb> zaro we already bumped memory
19:40:41 <clarkb> note nodepool testing is broken by oscc
19:40:55 <clarkb> I wrote a fix which hasnt merged I dont think
19:41:28 <crinkle> bifrost is also broken by occ
19:41:30 <zaro> clarkb: yeah, so should we keep it unchanged for the next few weeks to see if thing are better?
19:41:43 <clarkb> crinkle same issue about region invalid?
19:41:45 <zaro> clarkb: or should we continue to apply other tuning?
19:42:12 <clarkb> zaro I think we should prep more tuning but apply slowly to observe results
19:42:43 <crinkle> clarkb: no it's something about unable to auth, sorry i assumed it was the same
19:42:48 <crinkle> will look into it more this afternoon
19:43:02 <AJaeger> Btw. small kind of announcement: bandit, system-config, and project-config now use the linters target that we discussed a few days ago on #openstack-infra, patch for keystone is proposed
19:43:24 <pleia2> AJaeger: nice work on those
19:44:49 <clarkb> crinkle ok, if you pass along error I cansee if it is familiar
19:45:00 <crinkle> clarkb: will do
19:45:38 <zaro> i think review-dev.o.o needs more disk space.  anybody willing to help with that?
19:46:19 <pabelanger> fungi: I think we are ready to stand up openstackid-dev.o.o under trusty.  Then we can use that to debug any failures
19:46:37 <pabelanger> only issue I found was php5 related and will upload a new patch later today
19:46:44 <fungi> pabelanger: okay, so it installs successfully now?
19:46:47 <clarkb> zaro it may just need an apt cache vleanup
19:46:54 <fungi> pabelanger: ahh, yeah, once you have that patch
19:47:03 <pleia2> zaro: only 64% full, 10/24G free
19:47:05 <clarkb> fungi we run autoremove and autoclean for that?
19:47:24 <fungi> pabelanger: my problem was that last time i tried it wouldn't puppet sufficiently to bootstrap the server
19:47:35 <pleia2> clarkb: yeah, 1.8G in /var/cache/apt
19:47:44 <pabelanger> fungi: Ya, looks to be successful, but more eyes on the prize helps.    Plus, having dev environment up and running allows for more people to debug
19:48:11 <pabelanger> fungi: only issue I had was php5 mcrypt
19:48:19 <pabelanger> but can try again
19:48:42 <fungi> clarkb: zaro: yeah, for some reason i thought someone (yolanda?) had proposed a config change to unattended-upgrades to do that automatically
19:49:00 <zaro> actually looks like there's lots of new space in /opt that can be utilized
19:49:04 <fungi> clarkb: zaro: sudo apt-get autoremove ; sudo apt-get clean
19:49:11 <clarkb> fungi yolanda did but made it co figurable for hosts that dont want autoupdates
19:49:31 <clarkb> I asked that hosts that dont want autoupdate not use the autoupdate module
19:50:04 <fungi> pabelanger: also make sure puppet isn't erroring when being applied to a new vm. if  you're testing by reapplying to an already running vm you can miss races pretty easily
19:50:31 <pabelanger> fungi: Right
19:51:01 <fungi> our launch script aborts if puppet can run through to completion in one shot
19:51:07 <fungi> er, can't
19:51:59 <zaro> clarkb: should we schedule a gerrit restart for the purpose of apply tuning now?
19:52:00 <pabelanger> only otherthing I had was the nodepool gate failures for unit test.  clarkb mentioned we need new releases I believe (os-client-config)?
19:52:27 <clarkb> pabelanger correct but fix needs merging first
19:53:08 <clarkb> zaro if gerrit gets slow again we can probably just do it as emergency restart
19:54:26 <zaro> yeah, ok.
19:56:02 <clarkb> pabelanger: need release with https://review.openstack.org/#/c/260704/ looks like it did get merged
19:59:34 <pabelanger> clarkb: thanks, was trying to find the review
20:00:50 <fungi> we're out of time
20:00:52 <fungi> thanks all!
20:00:57 <fungi> #endmeeting