19:02:24 <fungi> #startmeeting infra
19:02:25 <openstack> Meeting started Tue Mar 22 19:02:24 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:28 <openstack> The meeting name has been set to 'infra'
19:02:31 <cdent> o/
19:02:32 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:34 <dougwig> o/
19:02:41 <fungi> #topic Announcements
19:02:58 <fungi> we got our requested summit session allotment
19:03:02 <anteaya> yay
19:03:06 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:09 <pleia2> and we're sharing friday with QA
19:03:13 <jeblair> yay us!
19:03:15 <fungi> er, wrong link. fixing
19:03:17 <fungi> #undo
19:03:18 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x9fb9d10>
19:03:23 <fungi> #link https://etherpad.openstack.org/p/infra-newton-summit-planning Newton Summit Planning
19:03:27 <pleia2> (I updated the etherpad to reflect this sharing)
19:03:33 <fungi> i've tweaked that to make it clear what we have
19:03:51 <Zara> :)
19:04:08 <fungi> there we go
19:04:31 <fungi> anyway, reminder to put ideas there so we can firm them up, choose and schedule them
19:04:51 <fungi> #info Please put Infra team summit session ideas on the Etherpad
19:05:09 <fungi> #topic Actions from last meeting
19:05:17 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-03-15-19.03.html
19:05:25 <fungi> 1.
19:05:26 <fungi> (none)
19:05:33 <fungi> i'm terrible at pasting things today
19:05:43 <jeblair> we're rocking it again
19:05:45 <fungi> #topic Specs approval
19:06:13 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/publish-election-repo.html APPROVED: Add elections.openstack.org
19:06:13 <mordred> o/
19:06:47 <jeblair> fungi: what happened with the url modification of that spec?
19:06:48 <fungi> #link https://review.openstack.org/278777 PROPOSED: Add Nodepool Zookeeper spec (jeblair)
19:07:01 <anteaya> jeblair: I think it was a follow up patch
19:07:03 <fungi> jeblair: that was the final result with the url modification
19:07:17 <fungi> i technically approved teh spec and teh addendum to the spec
19:07:35 <jeblair> ah.  it's perhaps still somewhat misleading.
19:07:35 <fungi> but there was no final artifact to link to for the interim state
19:07:54 <fungi> oh, yes the title needs fixing it appears
19:08:11 <jeblair> also a bit in the 1st pgraph
19:08:18 <jeblair> but the porposed change reads as i would expect
19:08:26 <fungi> if tonyb or jhesketh or tristanC or anyone else proposes that cleanup i'll happily approve as administrivia
19:08:33 <pabelanger> So, reading the spec, zookeeper will be a hard dependency now for zuul?
19:08:45 <clarkb> pabelanger: nodepool
19:08:53 <pabelanger> oh, right
19:08:57 <jeblair> ah, yes
19:09:11 <anteaya> the name of the file changed but the title of the spec didn't: https://review.openstack.org/#/c/292666/4/specs/publish-election-repo.rst
19:09:12 <jeblair> i believe greghaynes and i have bashed this into shape
19:09:29 <jeblair> i believe greghaynes and i have bashed the nodepool zookeeper spec into shape
19:09:32 <pabelanger> I think there is zookeeper packaging issues on centos7, but think ianw was looking into that
19:09:35 <jeblair> so i think this is ready for voting
19:09:44 <pabelanger> either way, not a blocker for us
19:09:48 <jhesketh> I can update that later today
19:09:50 <jeblair> unless anyone else wants to dive in on specifics first
19:09:53 <anteaya> jhesketh: thanks
19:10:15 <ianw> yes, it is not on centos, but i'm thinking we need it for tooz/distributed locking/devstack soon
19:10:38 <jeblair> also, fyi, i had hoped to have a follow-on spec for using zookeeper as part of the nodepool-zuulv3 interface, but we were short-staffed friday and that did not go as planned.  hope to have it done soon.
19:10:59 <mordred> ianw: ++
19:11:30 <pabelanger> ianw: I'll look into that soon too. Nodepool RPM is about ready for rawhide and hope to move into epel7 shortly after
19:11:47 <ianw> (just for the minutes, i have started looking at this in -> https://etherpad.openstack.org/p/zookeeper-epel7)
19:12:56 * ianw waves hello to people googling "zookeeper" "centos" :)
19:13:10 * jeblair wishes them good luck
19:13:18 <clarkb> run a ubuntu container on centos7
19:13:21 <clarkb> problem solved _>
19:13:32 <eil397> : - )
19:13:44 <eil397> just for one java app
19:14:27 <fungi> yeah, it is worth noting this adds yet another java app into our toolchain
19:14:41 <jeblair> yep.
19:14:44 <fungi> hopefully it just works and we get to treat it as mostly a black box
19:14:45 <clarkb> but its a very easy one to deploy
19:14:53 <jeblair> i'm told zookeeper is the best at what it does
19:14:55 <pabelanger> clarkb: good to know
19:15:10 <pabelanger> hopefully we don't need 8 of them
19:15:16 <clarkb> pabelanger: the recommendation is 5
19:15:18 <jeblair> pabelanger: we may want a few of them
19:15:22 <jeblair> for quorum purposes
19:15:31 <pabelanger> indeed
19:15:33 <clarkb> 3 works, 5 is better, you don't want more than 5 apparently
19:15:47 <jeblair> but we'll be able to share it for multiple services
19:16:02 <fungi> prime numbers huh?
19:16:03 <jeblair> so if this goes well, and zuul starts using it too, we can manage one centrally
19:16:04 <mordred> yup. installing and running on ubuntu is VERY easy
19:16:11 <jeblair> https://review.openstack.org/293176
19:16:21 <jeblair> i started on adding it to nodepool's test infrastructure ^
19:16:34 <jeblair> that'll give you a minimal idea of what it takes to run
19:17:15 <fungi> that's not terrible at all
19:17:55 <fungi> nice, already has a python client lib
19:17:56 <jeblair> nah, and most of that is boilerplate that packages handle for the system installation (i had to copy some things in to run it inside the unit test runner)
19:18:07 <jeblair> yeah, it's _super_ easy to use
19:18:27 <jeblair> i wrote a quick test script to validate the ideas i wrote in the spec.  it's as easy to use as you would hope.
19:18:41 <jeblair> get() put() lock() etc
19:18:49 <mordred> and it's a library used in openstack
19:19:06 <mordred> so it's getting some hammering from that direction too
19:19:25 * jeblair wonders whether we will hammer it more :)
19:19:25 <fungi> on the servers section, the implication is that we'd run a zookeeper service on the nodepool server, or would it be on separate servers?
19:19:54 <clarkb> given nodepool's reliability requirements not sure we would need to do anything complicated to start
19:20:00 <jeblair> fungi: i was thinking we'd just start with a local zookeeper on nodepool.o.o
19:20:01 <mordred> fungi: if you want to play with one 'easily' on an ubuntu node, just apt-get install zookeeper, then run '/usr/share/zookeeper/bin/zkServer.sh start'
19:20:06 <clarkb> jeblair: +1
19:20:08 <mordred> ++
19:20:17 <jeblair> mordred: there's an init script too, in the zookeeper-init package or something
19:20:20 <mordred> fungi: and /usr/share/zookeeper/bin/zkCli.sh will then connect to that in a repl
19:20:25 <mordred> jeblair: oh - really? neat
19:20:25 <fungi> yeah, that seems like a convenient way to bootstrap our use of it
19:20:45 <cody-somerville> \o_
19:20:50 <mordred> zookeeperd
19:20:53 <mordred> for the record
19:20:57 <mordred> is where the init scripts are
19:20:58 <tristanC> fungi: about election.o.o, its likely stalled from election officials since we are quite busy with the on-going elections process
19:21:09 <jeblair> mordred: ah, thanks
19:21:11 <mordred> btw - I like having init scripts in one package and software in another
19:21:23 <fungi> tristanC: yeah, no need to worry about that for now. the spec can take as long as it takes to get implemented
19:21:49 <jeblair> mordred: it's a pattern i could get used to
19:22:01 <fungi> mordred: agreed, makes it more reusable when you don't want to run it as a service but still want the software installed
19:22:30 <clarkb> or just don't start be default
19:22:46 <fungi> clarkb: hush, you're speaking in redhatese now
19:23:15 <fungi> (granted, it's one of the only things rh does differently from debian which i actually agree with)
19:23:49 <fungi> okay, any more questions on "Add Nodepool Zookeeper" or should we put it up for final council voting this week?
19:24:27 <anteaya> I'm good for voting
19:24:40 <fungi> #info Council voting is open on the "Add Nodepool Zookeeper" spec until 19:00 UTC, Thursday, March 24
19:24:48 <jeblair> yay! thanks!
19:25:01 <fungi> #link https://review.openstack.org/287373 PROPOSED: Deploy-Stackviz spec (austin81, timothyb89)
19:25:26 <fungi> you have similarly numbered nicks
19:25:38 <timothyb89> huh, so we do :)
19:26:04 <fungi> it made me double-check to be sure i didn't havea  copy-paste error there ;)
19:26:17 <austin81> Anyone had a chance to review the spec over the last weeks?
19:26:39 <fungi> it looks like AJaeger and pabelanger were the only ones with input so far
19:27:20 <AJaeger> And I'm still not sure what's the best way to set this up is - at image build time or via puppet
19:27:39 <fungi> you've today added details on puppeting as an alternative
19:28:04 <clarkb> its puppet that would run at image build time though right?
19:28:18 <fungi> right, it happens during image build regardless
19:28:49 <AJaeger> Ah, didn't get that one before.
19:28:52 <austin81> if it happens at build time regardless, what is the benefit of doing puppet over a dib element?
19:29:18 <AJaeger> thanks for asking austin81 ! I just wondered the same...
19:29:45 <pabelanger> clarkb: yup
19:30:10 <mordred> austin81: for the log server ...
19:30:17 <mordred> is there a single stackviz on the log server?
19:30:32 <mordred> or does the static site from each run get copied to the run dir on the logs server?
19:30:39 <timothyb89> mordred: unless we change server configs, there is one per run dir
19:30:43 <mordred> cool
19:30:46 <fungi> similar things we install into our images at build time are zuul, swift-log-publisher and bindep
19:30:59 <mordred> then I do not see any need to puppet vs. dib personally
19:31:17 <fungi> bindep is done in a dib element
19:31:26 <fungi> checking to see whether the other two are
19:32:05 <fungi> yeah, nodepool-base/install.d/ has 99-install-zuul and 90-venv-swift-logs
19:32:17 <timothyb89> fungi: the subunit2sql init tasks are also similar to what we want, in a dib element
19:32:28 <fungi> so consistency would be to do it the way it's written in the specv
19:32:31 <fungi> er, spec
19:32:39 <jeblair> i think we want to minimize and eventually stop using puppet for nodepool nodes
19:32:41 <mordred> timothyb89: does npm have a thing like the virtualenv?
19:32:56 <timothyb89> mordred: it largely functions that way by default
19:33:00 <mordred> the other concern is making sure that we don't pollute the images with a ton of stuff that's globally installed
19:33:02 <mordred> awesome
19:33:11 <mordred> then that should be fine
19:33:14 <austin81> So if the consensus is dib on this... https://review.openstack.org/279317/  https://review.openstack.org/212207/ are the next things that need approval.
19:33:14 <timothyb89> mordred: 'npm install' will only touch our own directory
19:33:20 <mordred> cool
19:33:39 <fungi> yeah, i don't see any blockers for putting this up for a council vote. anyone object?
19:34:09 <anteaya> I do not object
19:34:53 <fungi> #info Council voting is open on the "Stackviz Deployment" spec until 19:00 UTC, Thursday, March 24
19:35:03 <fungi> (that appears to be its actual name in the rst)
19:35:29 <hashar> o/
19:35:40 <fungi> #topic Priority Efforts
19:35:54 <fungi> i don't see any last-minute additions for this
19:36:01 <anteaya> zaro: added something
19:36:09 <fungi> #topic Gerrit tuning - git gc (zaro)
19:36:17 <anteaya> oh sorry, yeah that
19:36:18 <fungi> that's not a priority effort afaik
19:36:27 <fungi> but adding to the agenda topics
19:36:33 <anteaya> sorry got listed there
19:36:41 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2016-January/003640.html Gerrit tuning - git gc
19:37:07 <fungi> zaro: you had updates on this?
19:37:12 <zaro> ohh yeah
19:37:32 <zaro> just added it to let you guys know that i would like ot share info i've gathered.
19:37:44 <fungi> please share ;)
19:37:47 <zaro> it can be now or outside of this meetig
19:37:55 <anteaya> I'm listening
19:38:33 <zaro> so the link outlines the concern that jeblair had with scheduleing git gc.
19:38:48 <zaro> here's somet of the things i've found: http://paste.openstack.org/show/491498/
19:39:17 <zaro> line 241 shows jgit gc command run, which give the same result as git gc.
19:40:30 <zaro> line 119 shows files before and after git gc.  which seems to reduce file sizes and puts all objects in a single file.  i believe the single file is the thing that jeblair had issue with.
19:41:12 <clarkb> the concern is that may make fetch operations slower
19:41:22 <clarkb> iirc
19:41:22 <zaro> from reading git info i think updating refs (if needed) is still possible using `git-ref-update` or `git-symbolic-ref`
19:41:53 <fungi> right, the thing i see missing from this research so far is the performance impact on common git remote operations (particularly on packed repos with many refs)
19:41:56 <zaro> clarkb: yeah, that's the other thing.  but i don't see how smaller files would make it slower?
19:42:19 <clarkb> zaro: because it is all in one file that must be operated on for every operation
19:42:32 <fungi> zaro: it's a question of fewer files making it slower because git may have to seek within the packfile rather than relying on lookups at teh filesystem level
19:42:57 <clarkb> that ^ but you are right the size diff is large so may do the opposite
19:43:20 <fungi> yep. which is why we were hoping someone would test this
19:43:46 <eil397> time git fetch ? after git gc ?
19:44:04 <fungi> so anyway, at least the indication so far is that relying on gerrit's background jgit gc task might be no worse than our cgit gc cron job
19:44:07 <clarkb> eil397: git clone, fetch a specific branch, and so on
19:44:23 <fungi> if i'm understanding your assertion, zaro
19:44:39 <zaro> fungi: yeah, that would be my conclusion
19:44:50 <jeblair> we don't run git gc
19:45:04 <zaro> let me go and test the fetch time on it and see how that works out
19:45:29 <fungi> oh, we run repack
19:45:34 <clarkb> eil397: zaro and use file:// git:// and http
19:45:47 <hashar> IIRC the git .pack files have an index and the file is mmap()  ed in memory, so it is rather faster than stat() on a myriad of files
19:45:57 <fungi> find /home/gerrit2/review_site/git/ -type d -name "*.git" -print -exec git --git-dir="{}" repack -afd \;
19:46:04 <fungi> from puppet-gerrit
19:46:05 <hashar> Wikimedia runs the git gc on all Gerrit repo once per week or so
19:46:22 <clarkb> hashar: thats another possible concern, now we have to have 500mb of memroy for every nova git op
19:46:48 <clarkb> its already not cheap so not a huge deal
19:47:59 <jeblair> clarkb: i'm not following?  what's the concern?
19:48:52 <clarkb> jeblair: if the entire file has to be mapped into memory to fetch a single ref that means the git servers all of a sudden need more memory
19:49:13 <jeblair> clarkb: ah, i don't think a memmapped file takes up that amount of memory; the opposite in fact
19:49:19 <clarkb> but nova and neutron have always been hard on memory so not sure f there will be an appreciable difference
19:49:26 <fungi> there are ways to optimize that with separate addressing/indexing, but i have no idea if cgit does that
19:49:26 <jeblair> clarkb: it usually means you only need to load the pages you access into memory
19:49:27 <hashar> clarkb: na the pack file has an index. So you map the index, then lookup the part of files that contains what you need an map that solely that part
19:49:46 <fungi> sounds like it does
19:50:03 <clarkb> ya looks like 660mb ish for a current git process
19:50:04 <hashar> all based on X years memory really. Would have to dig again in git pack source code
19:50:05 <clarkb> resident
19:51:15 <fungi> okay, anything else on this topic? we wanted to see a comparison to the actual git repack we're doing in cron now, yes?
19:51:19 <eil397> I've heard about way to accelarate clone by using git bundle. not sure. will it be good for infra. just fyi
19:51:25 <fungi> and also tests on impact to remote git operations?
19:51:29 <clarkb> hashar: ah ok, not sure why git needs so much memory today then
19:51:35 <eil397> https://www.kernel.org/cloning-linux-from-a-bundle.html
19:51:37 <clarkb> hashar: possibly this will be cheaper than current situation
19:52:24 <zaro> fungi: got it.  will will
19:52:25 <fungi> we're at 8 minutes, still 2 topics on the agenda, at least one of which may need to get pushed out to next week
19:53:00 <fungi> #topic IRC Bots (anteaya)
19:53:06 <anteaya> thanks
19:53:09 <anteaya> #link http://lists.openstack.org/pipermail/openstack-dev/2016-March/089509.html
19:53:14 <cdent> I'm here
19:53:21 <anteaya> so this email thread is indicative of a trend
19:53:27 <anteaya> cdent: thanks for being here
19:53:34 <anteaya> cdent wrote an irc bot
19:53:40 <anteaya> there are also other irc bots
19:53:51 <anteaya> as well, I think I may have narrowed the topic too much
19:54:03 <anteaya> as tehre are also gerrit bots folks are running
19:54:17 <AJaeger> notmyname has the patchbot that is used in some repos
19:54:22 <anteaya> so I thought we might be able to disuss some of the thoughts folks have
19:54:29 <anteaya> AJaeger: yes that is one of the other irc bots
19:54:51 <anteaya> as a for instance, for gerrit bots, I don't feel a bot having +2 and +A on a repo is a good idea
19:55:15 <anteaya> also for meeting channels jeblair had asked that folks add bots out our infra config files if they want bots
19:55:26 <anteaya> so sorry to already not follow my own topic
19:55:32 <anteaya> does anyone else have thoughts?
19:55:41 <fungi> we don't have a lot of control over whether people run irc bots in the channels we manage, though we do generally frown on unsanctioned bots echoing in channels shared by multiple teams
19:55:58 <mordred> I think the main concern with other bots is that when people start depending on them
19:56:05 <mordred> they then come and ask infra for help when they break
19:56:08 <fungi> or in channels used by teams who don't want those bots echoing in the channels they inhabit
19:56:09 <clarkb> I personally set up a smartfilter for anything I don't want to see and move on ...
19:56:12 <pleia2> mordred: yeah
19:56:15 <mordred> tons of people depended on a bot that soren ran for a while
19:56:20 <mordred> then soren stopped caring
19:56:22 <hashar> scale bot maintenance beyond infra team?
19:56:46 <mordred> hashar: well, we have subteams in infra for the purposes of scaling tasks and stuff
19:56:56 <mordred> so there is no reason there could not be a bot subteam
19:57:12 <fungi> it's mostly 1. noise factor, and 2. assuming maintenance for services on which a large portion of the community depends
19:57:17 <mordred> yah
19:57:32 <mordred> it's also possible that we need to do better advertisement of the features of the bots we already run
19:57:36 <angdraug> can (2) be solved by listing bot owners in its whois data?
19:58:23 <pleia2> angdraug: using the launchpad bot as an example, we all knew who ran it, but when they lost interest it was still difficult to contac them and get changes made
19:58:40 <ttx> yeah, results in a brittle env
19:58:47 <pleia2> ttx: nods
19:58:52 <cdent> In the specific case of purplerbot, I'm happy to see it run on infra
19:58:58 <cdent> I've packaged it and put it on pypi
19:58:59 <mordred> for instance: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t2016-03-22T19:44:30 is the per-line url link from the channel logs produced by the bot we run, and http://p.anticdent.org/logs/openstack-nova?dated=2016-03-22%2019:44:30#l8A is the link to the same thing from cdent's bot
19:59:08 <jeblair> dead irc server, irony of ironies :)
19:59:10 <ttx> you end up depending on something, and since you don't want overlapping functions, that featureset never ends up in the canonical bot
19:59:11 <fungi> anteaya: so we're at 1 minute remaining. did you have a specific action or decision you were looking to come out of this?
19:59:20 <cdent> mordred: the log in purplerbot is ancillary to its real purpose
19:59:35 <anteaya> fungi: I just wanted to see if folks wanted to discuss
19:59:40 <anteaya> looks like they do
19:59:41 <jeblair> also, i have looked into errbot, but the way forward is not completely clear; happy to follow up with others if they are interested
19:59:45 <anteaya> we can continue later, thank you
19:59:49 <hashar> Wikimedia is a wide range of irc bots; Most are hosted on a shared tenant for people to act on them as needed and most are documented on a wiki with basic restart/maintenance procedure. With bot code in Gerrit/Github
19:59:56 <angdraug> had a problem last week, someone launched a broken bot on #fuel-dev, had to ban it...
20:00:00 <fungi> igorbelikov: looks like we'll hit your topic at the beginning of our general topics section next week. sorry we ran out of time for the full agenda
20:00:12 <fungi> okay, we're out of time everyone
20:00:15 <fungi> thanks!
20:00:16 <hashar> so if JohnDoe bot dies, I can head to the wiki page, copy paste the command to restart it and hopefully it is back ;-}  Even if I have no root on the infra
20:00:18 <anteaya> thankyou
20:00:20 <fungi> #endmeeting