Friday, 2016-01-08

openstackgerritMerged openstack-infra/tripleo-ci: No longer split gerrit ref on " "  https://review.openstack.org/25950800:00
jeblairfungi: this change exists: https://review.openstack.org/25928100:00
jeblairi do not know whether it would have an effect on the leak00:00
jeblairfungi: we could try merging it and see00:00
*** mdenny has joined #openstack-infra00:00
fungijeblair: oh, right, i had already mostly reviewed that one00:01
jeblairother than that, i think our choices are wait for jhesketh to return and investigate it, or revert the connections branch.00:01
fungii'll run back through it real quick and approve if cool00:01
anteayais jhesketh on holidays?00:01
jeblairyes00:01
* anteaya nods00:01
fungiclarkb also seems to have looked over that some, but is probably not really around to rereview00:02
*** peristeri has quit IRC00:02
*** bpokorny has joined #openstack-infra00:02
jeblairi would offer to try to fix it myself, but even in zuulv3, i still haven't gotten to the point where i understand how connections can interact with the rest of the data model.00:02
jeblairso i'm very likely to just repeat my incomplete zuulv3 work on the v2 branch, and i don't think that's helpful.00:03
anteayawould a restart make any difference?00:03
jeblairanteaya: yes, for a while.00:03
jeblairi think the most sensible thing to do is to merge https://review.openstack.org/259281 and restart it and see if it makes a difference.00:04
*** dimtruck is now known as zz_dimtruck00:04
fungi259281 lgtm, i can restart zuul once it merges, unless someone else is around to do that00:04
* anteaya has no objections00:04
jeblairby the time we can tell if it has, we can probably hand the problem off to jhesketh00:04
anteayaheh00:04
clarkbsorry not really here right this second, but if my comments appear to be addressed I had nothing else00:04
fungiyeah, just want to keep an eye on it for a few days (assuming this patch isn't like zomg crazy revert now)00:05
*** Qiming has quit IRC00:06
jheskethHey. So yes, I'm on leave until Tuesday but am trying to take a look. However I only have my phone with me at the moment00:06
jeblairjhesketh: go back on holiday! :)00:06
jheskethSo we don't know what it is exactly, but some kind of memory leak right00:06
*** ayoung has quit IRC00:06
fungijhesketh: don't look00:06
fungijhesketh: vacation is useless if you don't commit to it ;)\00:07
jheskethHeh00:07
anteayajhesketh: see you next week00:07
fungiwe're giving your change a whirl to see if it helps, and if not it'll be here waiting for you one way or another00:07
jheskethOkay but if there's anything I can do to help or particular questions etc please ping00:07
mgagne*jhesketh has been kicked from #openstack-infra channel*00:08
fungiheh00:08
anteayaha ha ha00:08
jhesketh(I won't answer if I'm busy ;-))00:08
jheskethHeh00:08
*** sabeen3 has quit IRC00:09
jheskethYep that seems wise. When was the last time zuul was restarted? Does anybody know how long we may have been on the connection changes00:09
jheskethIe I thought we had been running it at least for a bit without any obvious regressions00:09
fungijhesketh: connection changes merged sometime between the gerrit upgrade and the last zuul restart prior to that. at the gerrit upgrade related restart we see memory usage climb to ~2x its previous steady state. zuul was restarted again on ~ december 23 for reasons only irc channel logging probably remembers00:10
fungihowever, load on our system really didn't pick back up until this past week, when memory usage climbed to >10x that00:11
krotscheckWill using puppet to include mod_rewrite do all the fancy allowoverride things as well?00:11
openstackgerritJames E. Blair proposed openstack-infra/project-config: Validate grafyaml based on master  https://review.openstack.org/26500900:11
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: Update OVH graph with axis labels  https://review.openstack.org/26501000:11
anteayafungi: the dec 23 log doesn't have the words zuul and restart in the same sentence anywhere I can find00:12
jeblairbtw, 'last' is really useful with ansible.00:13
*** SumitNaiksatam has joined #openstack-infra00:13
fungianteaya: i was approximating. looks like ~16:00 utc on december 2200:13
jheskethHmm okay.. So it could be a number of things but connections possibly most suspicious00:13
*** SumitNaiksatam has left #openstack-infra00:14
fungihttp://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=3&view_type=&graph_start=1450759057&graph_end=1450845174&graph_height=120&graph_width=500&title_font_size=1000:14
fungijhesketh: well, also lots of things about gerrit potentially changed at that point, so could also be something tickled by that00:14
jheskethI would prefer to avoid reverting if possible. I think it'd be difficult and very time consuming to find the leak and re review everything00:14
jeblairjhesketh: yeah, it's at the top of my list because it touches the Change objects and event handling.  second on my list is fungi's theory of 'unhandled gerrit events cause leaks'.00:14
anteayafungi: ah00:15
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add wheel mirror to mirrors  https://review.openstack.org/25352100:16
jeblairjhesketh: well, if we revert, it'll be a single commit, and we can un-revert with a single commit followed by a small commit that fixes the problem.  alternatively, we could pin to a prior sha.00:16
anteayafungi: yep http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-12-22.log.html#t2015-12-22T16:02:1300:16
krotscheckjeblair: Finally got around to including that rewrite rule ^^00:16
* krotscheck goes and fixes the wheel distribution job now.00:16
mgagnewas about to ask for y axis legend/unit =)00:16
jheskethSure, I just mean finding the cause before the revert might be easier, but of course that's not necessarily true00:16
jeblairjhesketh: yeah, if you want to run in a debugger, etc.00:17
openstackgerritMerged openstack-infra/project-config: Adds an API migration and docs generation tool, fairy-slipper  https://review.openstack.org/24535200:19
openstackgerritMerged openstack-infra/puppet-ansible: Stop installing the puppet inventory  https://review.openstack.org/25092600:19
openstackgerritMerged openstack-infra/zuul: Fix regression in change tracking  https://review.openstack.org/25928100:19
*** jaypipes has quit IRC00:19
krotscheckOh, drat. Made a mistake00:20
*** Keedya has joined #openstack-infra00:20
*** tzn has quit IRC00:21
*** dims has joined #openstack-infra00:22
jeblairfungi: are you doing the restart once that propagates or shall i?00:24
*** Keedya has quit IRC00:24
*** dahoo has joined #openstack-infra00:24
*** dahoo is now known as jerryz00:24
fungijeblair: it's all yours if you want it. i can stick around or fade out to evening00:24
jeblairfungi: have an evening, i'll do it00:24
fungijeblair: thanks!00:25
*** salv-orlando has quit IRC00:25
funginow i need to stop polishing this atc listing script00:25
*** Keedya has joined #openstack-infra00:25
*** salv-orlando has joined #openstack-infra00:25
anteayafungi: happy evening to you00:25
anteayathe cacti graphs have a blank at 23:4000:26
jeblair2016-01-08 00:26:11,853 DEBUG zuul.Scheduler: Project None not found00:26
jeblairthe number of those lines is also disappointing.00:26
jeblair(and may be a clue)00:27
jeblairanteaya: that means the machine was too busy to respond to cacti00:27
anteayaah makes sense00:28
jeblairanteaya: which makes sense since it was also too busy to respond to us :)00:28
anteayaright00:28
fungianteaya: cacti polls snmp over udp by default, and udp doesn't get retransmits or retries or anything, so if the machine is busy... silence00:28
anteayaah00:28
jeblairi have stopped zuul00:28
anteayaokey dokey00:28
fungias opposed to tcp which is a little more resilient (if we used snmp v3 we could opt to switch to tcp instead of udp)00:29
anteayaI got what I needed, blank space == bad00:29
anteayaothers may need something more00:29
fungiwhen you have to fill in the blanks with snmp, fill them in with all your worst fears00:29
anteayamy worst fears don't fit in that tiny space00:30
anteayamy passing fears perhaps00:30
anteayawhimsical fears00:30
*** mriedem has quit IRC00:31
*** openstackgerrit has quit IRC00:32
*** openstackgerrit has joined #openstack-infra00:32
jeblairre-enqueue complete00:34
anteayayay00:35
jeblairhi00:35
jeblaircan someone who decided to run bash8 on project-config tell me how to fix this: 2016-01-08 00:20:40.861 | [E] E011: Then keyword is not on same line as if or elif keyword: 'if [ ! -e /usr/zuul-env/bin/zuul-cloner ] &&'00:35
jerryzanteaya: is it possible to push stable/liberty code to newly created networking-*  project?00:35
fungijeblair: while not a fan of linters myself, i think it means do `if foo ; then`00:36
anteayajerryz: why would you want that?00:36
*** mriedem has joined #openstack-infra00:36
fungijeblair: rather than putting the "then" on a following line00:36
jeblairfungi: right, i'm kind of assuming, things being the way they are around here, that we have a severe allergy to lines longer than 80 chars as well.00:36
anteayajerryz: stable/liberty has passed00:36
jeblairfungi: so i'm sort of trying to reconcile the two.00:36
fungijeblair: dunno, antihistamines are working out well for me00:37
jeblairA set of rules that help keep things consistent in control blocks. These are ignored on long lines that have a continuation, because unrolling that is kind of "interesting"00:37
*** otter768 has joined #openstack-infra00:37
jeblairaccording to the README00:37
jeblairso00:37
jeblairerm, i guess i need to put a \ on the line to make bashate happy00:37
jeblairdespite the fact that the syntax doesn't require it at all00:37
fungii guess it doesn't like a natural line break in place of a ; for solving long lines00:38
jeblairand in fact, the thing it's railing against is the perfectly intentional aspect of the syntax that allows it to be on more than one line?00:38
fungiyeah00:38
jerryzanteaya: because we claim to support liberty but the project was just recently created upstream00:38
*** oomichi has joined #openstack-infra00:38
fungijeblair: perhaps the best fix is to put E011 in the ignore list00:39
jeblairhttp://docs.openstack.org/developer/bashate is a 40400:39
mriedemanyone else seeing this today? http://logs.openstack.org/80/256180/1/gate/gate-grenade-dsvm/d1909a9/logs/grenade.sh.txt.gz#_2016-01-07_19_40_08_18800:39
jeblairfungi: that seems slightly better form than what i was about to do.  any idea where we keep the ignore list?00:39
*** Keedya has quit IRC00:40
fungii had hoped it was in tox.ini... no?00:40
fungihrm, bashate is probably _not_ a flake8 plugin00:40
jeblairthis is what i mean when i say this stuff is a waste of peoples' time00:41
anteaya260966 and 260967 are the two bashate changes to project-config00:41
*** thorst has joined #openstack-infra00:41
*** otter768 has quit IRC00:42
jeblair260967 apparently explains how to ignore things00:42
anteayamriedem: I have not00:43
mriedemanteaya: bingo https://pypi.python.org/pypi/django_compressor/2.000:43
mriedem2.0 release breaks things00:43
anteayawooooo00:43
* anteaya is so surprised00:43
fungijeblair: also http://git.openstack.org/cgit/openstack-dev/bashate/tree/doc/source/man/bashate.rst#n3700:43
fungi-I apparently00:44
jeblairand, for some reason, comma separated, even though argparse allows repitition00:44
jeblair(which i mention because apparently, i guessed wrong)00:44
anteayajerryz: well it seems to me you have two options, 1) come to grips with the fact we are in mitaka and re-evaluate your project claims or 2) try to work against our release workflow to satisfy your claims00:45
anteayaI'm supportive of 1)00:45
fungiold optparse habits die hard?00:46
jeblairerm00:47
jeblairi get different output when i run bashate locally00:47
fungidifferent bashate release?00:47
jeblairfungi: using tox00:47
jeblairand i assure you, i've never run that tox env before :)00:47
fungihrm. interestinger00:47
openstackgerritJesse Keating proposed openstack/requirements: Cap django_compressor below 2.0  https://review.openstack.org/26502500:47
jeblairRunning bashate on /home/corvus/git/openstack-infra/project-config/tools/run-compare-xml.sh00:47
jeblair[W] E006: Line too long: '/usr/zuul-env/bin/zuul-cloner -m ../tools/run-compare-clonemap.yaml --cache-dir /opt/git git://git.openstack.org openstack-infra/jenkins-job-builder'00:47
jeblairoh wait00:48
jeblairmaybe that's because of the argparse thing00:48
*** piet has joined #openstack-infra00:48
fungi-I E006,E01100:48
fungi;)00:48
*** Swami_ has quit IRC00:49
timrcpabelanger: Cool.  I'll look this evening when I trick my kids to go to sleep00:49
docaedoHi, I got an lint test failure with package update failures like "W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-amd64/Packages  Hash Sum mismatch"  (http://logs.openstack.org/98/260198/7/gate/gate-app-catalog-npm-run-lint/c578247/console.html)00:49
*** zz_dimtruck is now known as dimtruck00:49
docaedoI'm not sure what to do about it?00:49
jeblairdocaedo: wait and try again00:49
*** ybathia has joined #openstack-infra00:50
openstackgerritMatt Riedemann proposed openstack/requirements: Cap django_compressor<2.0  https://review.openstack.org/26502600:50
mriedemany reqs stable peeps around? ^00:50
mriedemsdague: dhellmann: dims: mtreinish: ^00:50
docaedojeblair: thanks - I'll make some dinner and then .. for "try again" do I add a recheck comment or something? This was after giving a workflow +100:50
fungidocaedo: it's a long-winded way of saying "the us.archive.ubuntu.com mirror your job is hitting is broken"00:51
mtreinishmriedem: +200:51
dimsmriedem : +1 from me. i was just looking at pypi+django like 2 mins before i saw your message :)00:51
fungidocaedo: yeah, recheck comment in your review00:51
mriedemyeah, 2.0 drops support for django<1.800:51
*** bpokorny has quit IRC00:51
mriedemand stable/kilo g-r has django capped at 1.800:51
docaedofungi: thanks, thats what I thought but wanted to make sure I wasn't doing something wrong00:52
*** bpokorny has joined #openstack-infra00:52
docaedofungi jeblair - thank you, I'll give it a shot again in a bit :)00:52
fungiwe're all doing something wrong. just enjoy your dinner ;)00:52
docaedohahaha00:52
*** bpokorny has quit IRC00:53
openstackgerritJames E. Blair proposed openstack-infra/project-config: Validate grafyaml based on master  https://review.openstack.org/26500900:53
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: Update OVH graph with axis labels  https://review.openstack.org/26501000:53
openstackgerritJames E. Blair proposed openstack-infra/project-config: Ignore E011 in bashate  https://review.openstack.org/26502700:53
*** bpokorny has joined #openstack-infra00:53
jeblairanteaya: you're aware that the project-config repo runs a job that performs grafyaml validation, right?00:55
anteayayes00:56
anteayamy laptop is faster, plus I get to feel I am learning something00:56
*** woodster_ has quit IRC00:56
jeblairanteaya: okay; just be careful zuul doesn't complain to the union :)00:56
anteayaoh I'm a good union member00:57
* anteaya waves her I.A.T.S.E card00:57
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for django_compressor 2.0 bug 1532048  https://review.openstack.org/26502800:57
openstackbug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] https://launchpad.net/bugs/153204800:57
mriedemmtreinish: ^00:57
mtreinishmriedem: your just full of demands tonight00:57
oomichimriedem: we are facing the problem on stable/liberty also. so do you have it fo liberty?00:57
anteayamtreinish: that's different than any other night?00:57
oomichimriedem: ah, I maybe misunderstand. it is not necessary00:58
mriedemoomichi: it's failing liberty b/c of grenade00:58
mriedemgrenade in liberty fails to install on the kilo side00:58
oomichimriedem: yeah, I see the point00:58
mtreinishanteaya: well, just because I'm paying attention tonight00:58
anteayamtreinish: ah points for you00:58
*** pvaneck has quit IRC00:58
oomichimriedem: because the problem happens on grenade only00:59
*** mudassirlatif has quit IRC01:00
zarofungi: looks like there a gerrit bug. Currently the required permission to allow viewing emails is Modify Account but it should be View All Accounts.01:00
zaroand additional bug is that  modify accounts permission should automatically allow users to  view accounts.01:01
zarowill have a fix up shortly.01:01
openstackgerritMatt Riedemann proposed openstack/requirements: Cap django_compressor below 2.0  https://review.openstack.org/26502501:02
mriedemsdague: want to approve this one https://review.openstack.org/#/c/265025/ ? jlk got that up first01:02
mtreinishmriedem: done01:03
*** Keedya has joined #openstack-infra01:03
mriedemthanks01:03
fungizaro: have a link to the issue url? i'd like to track it in a code comment for future cleanup01:04
zarowriting it up now :)01:05
openstackgerritTony Breeds proposed openstack/requirements: Cap django-compressor  https://review.openstack.org/26502901:05
*** Keedya has quit IRC01:06
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add baremetal hpuswest node definition  https://review.openstack.org/20875101:07
StevenKcrinkle: My task on that story can be marked invalid, we likely are not going to deploy using packages, and we don't have any infrastructure around building them anyway.01:07
gmannclarkb: ping01:09
*** Qiming has joined #openstack-infra01:09
anteayatonyb: mriedem has 265025 up01:09
tonybanteaya: Thanks  I saw that too late.01:10
gmannclarkb: i remember we had issue with server host name in case of multinode where it gets appended with some extra number01:10
anteayaah, cool01:10
tonybanteaya: I've dropped mine01:10
gmannclarkb: we had issue in live migration tempest tests01:10
tonyb3 people all looking at grenade failures for liberty.  That's gotta be a record :)01:11
gmannclarkb: its hitting in new multinode tests too - http://logs.openstack.org/60/257660/13/experimental/experimental-tempest-dsvm-multinode-full-dibtest/153a3aa/console.html#_2016-01-07_07_55_57_81501:11
*** Swami has joined #openstack-infra01:11
anteayatonyb: well better than noone looking01:11
* anteaya rings a bell01:11
crinkleStevenK: okay thank you01:11
jlkmriedem: with this change you and I both made, does devstack complete okay? I honestly fired off the patch before my local test env could complete01:12
openstackgerritMerged openstack-infra/elastic-recheck: Add query for django_compressor 2.0 bug 1532048  https://review.openstack.org/26502801:14
openstackbug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] https://launchpad.net/bugs/153204801:14
*** piet has quit IRC01:15
anteayaseems the -meeting channel needs a topic refresh, not sure why01:15
jeblairfungi, clarkb: are you aware of tox env variable isolation?01:16
fungijeblair: with the passenv option to list what you want whitelisted?01:17
jeblairyeah01:17
fungithen... yes01:17
jeblairapparently this is a thing as of like half a year ago01:17
fungi1.7 i think01:17
jeblair2.001:17
openstackgerrithardik proposed openstack-infra/project-config: Fixed mistral experimental job  https://review.openstack.org/26503001:17
jeblairthis will prevent zuul-cloner from working in a tox env01:17
jeblairwhich happens a bit and is happening more01:17
fungiwithout whitelisting all the necessary ZUUL_.* envvars anyway01:17
*** mtanino has quit IRC01:18
jeblairright01:18
fungiworth noodling on a cleaner solution to01:18
jeblairi'm saying i guess no one realized this was happening 6 months ago01:18
jeblairthis could have a big impact on the neutron stuff01:18
Thelo2016-01-08 08:59:38,191 INFO nodepool.image.build.dpc: + sudo chown -R jenkins:jenkins /opt/git/openstack/tempest/preseed-streams01:18
Thelo2016-01-08 08:59:38,367 INFO nodepool.image.build.dpc: sudo: unable to resolve host hps1001:18
Thelo2016-01-08 08:59:38,370 INFO nodepool.image.build.dpc: chown: cannot access '/opt/git/openstack/tempest/preseed-streams': No such file or directory01:18
Thelo2016-01-08 08:59:38,373 INFO nodepool.image.build.dpc: ++ check_break after-error run_in_target bash01:18
oomichican anyone resubmit a new tempest-lib release?01:18
oomichiThe release was failed during uploading to pypi.python.org like http://logs.openstack.org/f0/f0b44f3e4292353b186ee3b32d9b89f9bfe9d894/release/tempest-lib-pypi-both-upload/92ab2a5/console.html01:19
TheloDoes anyone know what is this project for:  chown: cannot access '/opt/git/openstack/tempest/preseed-streams?01:19
fungiespecially since they tend to want to have fallbacks on missing zuul envvars assuming that they're being invoked on developer machines and not the ci01:19
fungioomichi: that's an unfortunate pypi failure01:20
fungioomichi: requeuing that tag's jobs now01:21
oomichifungi: that means it is ok that I just am waiting for the release?01:21
fungioomichi: yeah, zuul's got the jobs queued for it now01:22
openstackgerritMerged openstack/diskimage-builder: Make check for image-should-fail quiet  https://review.openstack.org/25876201:22
openstackgerritMerged openstack/diskimage-builder: Fix discoverd bug when dmidecode reports GB  https://review.openstack.org/25875701:22
mriedemjlk: i tested mine, it was fine01:23
fungioomichi: i mean, it required manual intervention. i basically told zuul to pretend it just saw the 0.13.0 tag again01:23
mriedemjlk: i'm watching yours in zuul01:23
openstackgerritJames E. Blair proposed openstack-infra/project-config: Validate grafyaml based on master  https://review.openstack.org/26500901:23
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: Update OVH graph with axis labels  https://review.openstack.org/26501001:23
openstackgerritJames E. Blair proposed openstack-infra/project-config: Pass ZUUL_* env vars through tox  https://review.openstack.org/26503101:23
fungijeblair: oh! passenv will do glob matching?01:23
openstackgerritMerged openstack/diskimage-builder: Properly account for pipefail during cleanup  https://review.openstack.org/26282701:23
oomichifungi: thanks for doing this! I see it on zuul :-)01:23
jeblairfungi: yeah... but wait till you see my next idea :)01:24
anteayafungi: zuul must like the milk and cookies you offered01:24
fungioomichi: seems it was successful on retry http://logs.openstack.org/f0/f0b44f3e4292353b186ee3b32d9b89f9bfe9d894/release/tempest-lib-pypi-both-upload/c5bdd55/console.html01:25
fungioomichi: lgtm https://pypi.python.org/pypi/tempest-lib01:25
oomichifungi: thanks so much, we can see new version on https://pypi.python.org/pypi/tempest-lib now01:25
fungiawesome01:25
oomichifungi: yeah :)01:25
*** Keedya has joined #openstack-infra01:26
openstackgerritKen'ichi Ohmichi proposed openstack/requirements: Bump minimum tempest-lib version  https://review.openstack.org/26310601:28
*** tsg_ has quit IRC01:28
anteayatopic in -meeting still shows open discussion for nova which ended some time ago <insert interpretive dance>01:30
zarofungi: https://code.google.com/p/gerrit/issues/detail?id=375401:30
openstackgerritJames E. Blair proposed openstack-infra/project-config: Have Zuul force tox to obey Zuul  https://review.openstack.org/26503201:30
jeblairfungi: ^ :)01:30
fungijeblair: 265009 lgtm but a couple ideas commented inline you could do in a new patchset or follow-on change if you're so inclined01:31
zarofungi: modify account permission will work, but you probably don't want to set that.01:31
anteayazaro: yeah, I don't want that01:31
jeblairfungi: good points, thanks01:31
fungizaro: right, that's not so great. thanks for working on improving it! i'll track that issue in a code comment for now and we can revisit if and when it gets solved01:31
*** Keedya_ has joined #openstack-infra01:33
openstackgerritMerged openstack-infra/project-config: Ignore E011 in bashate  https://review.openstack.org/26502701:33
openstackgerritPaul Belanger proposed openstack-infra/grafyaml: Add support for template query  https://review.openstack.org/25089301:33
openstackgerritJames E. Blair proposed openstack-infra/project-config: Revert "Pass ZUUL_* env vars through tox"  https://review.openstack.org/26503401:33
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Add a CONTRIBUTING details  https://review.openstack.org/26503501:34
asselinfungi, Thelo as promissed ^^01:34
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Add a CONTRIBUTING.rst document  https://review.openstack.org/26503501:35
*** apoorvad has joined #openstack-infra01:35
*** Keedya has quit IRC01:36
*** changbl has joined #openstack-infra01:36
openstackgerritMatthew Kassawara proposed openstack-infra/project-config: Add Sam-I-Am to IRC operator list  https://review.openstack.org/26503601:38
*** sputnik13_ has quit IRC01:38
*** Keedya_ has quit IRC01:38
anteayaobviously I should know what the ramifications of tox and zuul not behaving as zuul wants but I don't01:38
fungijeblair: heh, TOX_TESTENV_PASSENV almost seems like it defeats the point ;)01:38
fungianteaya: some projects are invoking zuul-cloner in scripts called under tox, and both those scripts and zuul-cloner itself will potentially have different behaviors if they don't see the ZUUL_* envvars which they expect when called by a zuul-based ci system01:40
fungianteaya: because those developers also want to be able to run tox -e whatever on their systems and get ci-free behaviors01:41
anteayaah ha01:41
anteayathank you01:41
fungianteaya: and they're looking at absence of those envvars as an indicator that they're not running in the ci when called01:41
fungiand tox's default behavior now is to filter any envvars unless you tell it to let specific ones through01:42
anteayaah01:42
anteayathank you, that helps01:42
anteayawell helps me understand that is01:42
*** Keedya has joined #openstack-infra01:43
fungiso for example, zuul might be starting a job which calls tox -e functional and then the functional job is cloning the master branch tip instead of a change zuul is suggesting via ZUUL_REF or whatever. end result is that the job isn't testing what people think it's testing and could let bugs slip through01:43
anteayaperfect01:44
anteayain the sarcastic sense of the word01:44
pabelangernibalizer: clarkb: Right, so like I said, the only reason we don't have never versions of Fedora ATM, is because each time we attempt a launch, we've exposed another issue / bug. A lot has to do with running DIBs, glean and so on, which sadly fedora's been the first to run into most of the issues.  It be cool if we could not do a DIB for fedora 22 and upload that while ian works on Fedora 23 issues.01:44
pabelangerEither way, no -1 from me, just sad we are moving our last fedora now.01:44
*** ybathia has quit IRC01:44
*** Keedya has quit IRC01:45
anteayaso how do I test 265032?01:45
fungianteaya: you could create a special "foo" tox env which echoes back the value of $ZUUL_FOO and then export ZUUL_FOO=bar in your environment before running tox -e foo01:47
*** Swami has quit IRC01:47
anteayaI'll see how far I get trying that01:47
fungiit won't directly test the parameter function script being modified in 265032 but it'll validate the tox feature that's depending on01:48
fungiwhat i suggested should fail, but then export TOX_TESTENV_PASSENV=ZUUL_FOO and try again01:49
fungiand hopefully you'll see it say "bar"01:49
* anteaya hopes it will say "bar"01:50
*** MarkAtwood1 has quit IRC01:51
pabelangertimrc: https://review.openstack.org/#/c/250893/ would be a good one to start on :)01:52
*** Sukhdev has quit IRC01:52
anteayaI'm waiting for local tox to finish with 265031 first01:52
*** grassass has joined #openstack-infra01:54
*** Sukhdev has joined #openstack-infra01:55
anteayano likey: http://paste.openstack.org/show/483290/01:56
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics  https://review.openstack.org/26397101:56
*** apoorvad has quit IRC01:57
anteayaI had just run `tox` no arguments01:57
*** rajinir has quit IRC02:05
Sam-I-Amanteaya: thx for the +202:05
anteayaSam-I-Am: welcome :) happy to see you volunteer to take on the work02:06
anteayathank you02:06
Sam-I-Amnp02:06
*** sridhar_ram has quit IRC02:06
fungianteaya: http://paste.openstack.org/show/483291/02:07
fungianteaya: note that the first tox invocation doesn't output bar, bur the second does02:07
fungier, but the second does02:08
anteayafungi: thank you I will try that02:09
fungithe bash -c is so that i could get a subprocess operating on that environment variable, and the deps = was just to make it go faster (no need to install requirements)02:09
anteayaah ye requirements take a while02:10
anteayamy system didn't fancy 26503102:10
*** bpokorny has quit IRC02:12
*** Sukhdev has quit IRC02:13
anteayaI don't seem to be able to get it to work: http://paste.openstack.org/show/483292/02:15
*** zhurong has joined #openstack-infra02:15
anteayaI also have ZUUL_FOO=bar and TOX_TESTENV_PASSENV=ZUUL_FOO in my environment variables02:18
anteayanot that I care just saying they are there02:18
*** bharathm has quit IRC02:19
fungianteaya: the \ gets interpreted out in my >> tox.ini02:20
fungiif you manually edited that into your tox.ini drop the \02:20
anteayayeah that was it02:21
anteayathough now since i have TOX_TESTENV_PASSENV as an env var it just prints bar both times02:21
anteayabut thanks okay that one works02:21
anteayawhy is my system so unhappy with 031 I wonder?02:21
fungiyeah, if you were exporting them in your env you can `unset TOX_TESTENV_PASSENV ; export TOX_TESTENV_PASSENV` to get it good and gone again02:22
anteayathanks02:22
*** jerryz has quit IRC02:27
anteayaI get this just before the xml tests finish: You have a macro ('tox') defined for 'builder' component type that is masking an inbuilt definition02:27
*** apoorvad has joined #openstack-infra02:30
anteayaokay my system is failing to build Pillow apparently02:30
anteayahttp://paste.openstack.org/show/483294/02:30
*** otter768 has joined #openstack-infra02:31
*** thorst has quit IRC02:32
clarkbanteaya: you need libjpeg-dev02:32
anteayaokay02:32
anteayashould that be downloaded for me, or this is expected I would have that?02:32
* anteaya downloads libjpeg-dev02:33
*** thorst has joined #openstack-infra02:33
*** ayoung has joined #openstack-infra02:35
*** jaosorior has quit IRC02:35
*** jaosorior has joined #openstack-infra02:35
*** dims has quit IRC02:36
*** uberjay has quit IRC02:39
*** sdague has quit IRC02:39
*** SumitNaiksatam has joined #openstack-infra02:39
*** uberjay has joined #openstack-infra02:40
*** sdague has joined #openstack-infra02:40
*** thorst has quit IRC02:41
anteayathanks clarkb that was it02:41
*** piet has joined #openstack-infra02:42
anteayajeblair: I approved 265032, I hope that was okay to merge now02:45
*** tqtran has quit IRC02:47
*** Qiming has quit IRC02:49
openstackgerritMerged openstack-infra/project-config: Have Zuul force tox to obey Zuul  https://review.openstack.org/26503202:50
*** Keedya has joined #openstack-infra02:53
openstackgerritMerged openstack-infra/puppet-ansible: Remove an ensure absent  https://review.openstack.org/25092702:56
*** piet has quit IRC02:59
anteayaI think that will get picked up by the next successful image build run, yeah?03:01
*** Keedya has quit IRC03:01
*** ybathia has joined #openstack-infra03:02
*** ybathia_ has joined #openstack-infra03:03
*** yamahata has quit IRC03:04
*** Qiming has joined #openstack-infra03:04
*** unicell has quit IRC03:04
*** gyee has quit IRC03:04
*** ybathia has quit IRC03:06
*** ybathia_ is now known as ybathia03:06
*** eil397 has joined #openstack-infra03:09
*** coolsvap|away is now known as coolsvap03:15
*** ybathia has quit IRC03:20
*** apoorvad has quit IRC03:23
*** yuanying has quit IRC03:26
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add explict start dates to meeting yaml  https://review.openstack.org/26505603:34
*** piet has joined #openstack-infra03:35
*** yuanying has joined #openstack-infra03:37
*** gongysh has quit IRC03:39
*** thorst has joined #openstack-infra03:39
*** SumitNaiksatam has quit IRC03:46
*** thorst has quit IRC03:47
*** piet has quit IRC03:49
openstackgerritMerged openstack-infra/devstack-gate: create GRE bridge interfaces only once  https://review.openstack.org/23936303:49
*** piet has joined #openstack-infra03:50
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder  https://review.openstack.org/26334903:56
bkeroHehehehe03:57
*** rguillebert has quit IRC03:57
*** Keedya has joined #openstack-infra04:02
*** edmondsw has quit IRC04:05
*** yuanying has quit IRC04:05
*** Keedya has quit IRC04:05
*** yuanying has joined #openstack-infra04:07
*** salv-orl_ has joined #openstack-infra04:09
*** salv-orlando has quit IRC04:12
*** piet has quit IRC04:12
*** piet has joined #openstack-infra04:13
*** eil397 has quit IRC04:15
jeblairanteaya: yep, and appears to be working with no ill effect: https://jenkins02.openstack.org/job/gate-keystone-pep8/1194/parameters/04:18
*** yamamoto_ has joined #openstack-infra04:19
*** mriedem has quit IRC04:20
*** gongysh has joined #openstack-infra04:21
*** Ramanjaneya has joined #openstack-infra04:23
*** links has joined #openstack-infra04:27
*** yamamot__ has joined #openstack-infra04:28
*** yamamoto_ has quit IRC04:29
*** salv-orl_ has quit IRC04:30
*** salv-orlando has joined #openstack-infra04:30
*** unicell has joined #openstack-infra04:37
*** markvoelker has quit IRC04:38
*** flwang has quit IRC04:42
*** treaki_ has joined #openstack-infra04:43
*** thorst has joined #openstack-infra04:44
*** flwang1 has joined #openstack-infra04:46
*** treaki__ has quit IRC04:47
*** thorst has quit IRC04:51
*** otter768 has quit IRC04:54
*** MarkAtwood1 has joined #openstack-infra04:57
*** MarkAtwood1 has quit IRC04:59
*** MarkAtwood has joined #openstack-infra05:00
*** MarkAtwood has quit IRC05:01
*** tlian2 has quit IRC05:04
*** Ramanjaneya has quit IRC05:05
*** yamamoto_ has joined #openstack-infra05:07
*** yamamot__ has quit IRC05:07
*** salv-orlando has quit IRC05:12
*** salv-orlando has joined #openstack-infra05:12
*** harlowja_at_home has joined #openstack-infra05:18
*** piet has quit IRC05:19
*** yamamoto_ has quit IRC05:19
*** bharathm has joined #openstack-infra05:20
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder  https://review.openstack.org/26334905:22
*** Sukhdev has joined #openstack-infra05:24
*** bharathm has quit IRC05:25
*** amotoki has joined #openstack-infra05:27
*** jamielennox is now known as jamielennox|away05:27
*** esker has joined #openstack-infra05:31
*** fawadkhaliq has joined #openstack-infra05:32
*** jaosorior has quit IRC05:47
*** thorst has joined #openstack-infra05:49
*** harlowja_at_home has quit IRC05:53
*** mrmartin has joined #openstack-infra05:55
*** thorst has quit IRC05:56
*** aeng has quit IRC05:58
*** yuanying_ has joined #openstack-infra06:00
*** yuanying has quit IRC06:01
*** yuanying_ has quit IRC06:01
*** yuanying has joined #openstack-infra06:01
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix cleanup of check-osc-plugins  https://review.openstack.org/26508506:02
openstackgerritgreghaynes proposed openstack-infra/nodepool: Add dib deletion test  https://review.openstack.org/21527506:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Trigger image delete using gearman  https://review.openstack.org/19811406:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Trigger image uploads using gearman  https://review.openstack.org/19772206:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Builders can be run on their own  https://review.openstack.org/23225106:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Decouple builders from nodepool instance  https://review.openstack.org/23213506:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Builders distinguish between failure and exception  https://review.openstack.org/24213106:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Make secure.conf optional and backwards-compat  https://review.openstack.org/23280006:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Move nodepool config loading into module  https://review.openstack.org/23159206:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman  https://review.openstack.org/19517506:03
openstackgerritgreghaynes proposed openstack-infra/nodepool: Decouple nodepool db and config from builders  https://review.openstack.org/22239806:03
greghaynesclarkb: ^ ok, hopefully that worked06:03
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix cleanup of check-osc-plugins  https://review.openstack.org/26508506:03
*** sigmavirus24_awa is now known as sigmavirus2406:03
*** sigmavirus24 is now known as sigmavirus24_awa06:05
*** AJaeger has joined #openstack-infra06:07
*** mrmartin has quit IRC06:08
*** jbryce has quit IRC06:08
*** notmyname has quit IRC06:09
*** rainya_ has quit IRC06:09
*** mgagne has quit IRC06:09
*** andreaf has quit IRC06:09
*** notmyname has joined #openstack-infra06:10
openstackgerritDimitry Ushakov proposed openstack-infra/project-config: Add solum-pythonclient to projects  https://review.openstack.org/26508806:11
*** dtroyer has quit IRC06:11
*** esker has quit IRC06:13
*** vgridnev has joined #openstack-infra06:13
*** markvoelker has joined #openstack-infra06:13
*** gongysh has quit IRC06:14
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/26509106:15
*** mgagne has joined #openstack-infra06:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Do not run openstack-api-site  https://review.openstack.org/26509306:16
*** mgagne is now known as Guest4180406:16
*** rainya has joined #openstack-infra06:16
AJaegerinfra-root, could I get a review of 265093, please? We broke the index page of developer.openstack.org and that change is part of fixing it...06:16
*** jbryce has joined #openstack-infra06:17
*** andreaf has joined #openstack-infra06:17
*** dtroyer has joined #openstack-infra06:17
openstackgerritMerged openstack/requirements: Bump upper constraint on python-novaclient to 3.2.0  https://review.openstack.org/26485506:17
greghaynesI think nodepool is totally broken in ci - getting access denied to database. Locally I don't get this...06:18
greghaynes:(06:18
greghaynesso probably not doing builder deploy tomorrow06:18
* greghaynes -> bed06:18
*** markvoelker has quit IRC06:18
*** yamamoto_ has joined #openstack-infra06:20
*** vgridnev has quit IRC06:20
*** jaosorior has joined #openstack-infra06:21
TheloHi all, does anyone know what password nodepool is talking about ? (2016-01-08 13:55:21,221 INFO nodepool.utils: Password auth exception. Try number 222...)06:22
greghaynesThelo: ssh'ing in to the node06:24
fungiAJaeger: that doesn't need infra-root. any project-config reviewers can also approve06:27
AJaegerfungi: thanks, yes - but I feared none of them was around ;/06:28
*** dimtruck is now known as zz_dimtruck06:28
AJaegerand we don't have yet project-config-cores ;)06:28
AJaegerfungi: sorry, didn't want to wake you up06:28
*** Swami has joined #openstack-infra06:29
*** Swami has quit IRC06:29
*** hdd has quit IRC06:31
*** yamamoto_ has quit IRC06:31
*** hdd has joined #openstack-infra06:33
*** jcoufal has joined #openstack-infra06:34
*** darvon has quit IRC06:38
openstackgerritMerged openstack-infra/project-config: Do not run openstack-api-site  https://review.openstack.org/26509306:38
*** darvon has joined #openstack-infra06:40
*** pahuang has quit IRC06:42
*** pahuang has joined #openstack-infra06:42
*** rraja_ has quit IRC06:48
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/26359206:48
openstackgerritSteve Martinelli proposed openstack-infra/project-config: move osc-plugin-job from experimental to check  https://review.openstack.org/26510706:50
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/26509106:50
*** coolsvap is now known as coolsvap|away06:52
Thelogreghaynes: thanks06:52
*** Sukhdev has quit IRC06:54
*** thorst has joined #openstack-infra06:54
*** otter768 has joined #openstack-infra06:55
*** ybathia has joined #openstack-infra06:59
*** pahuang has quit IRC07:00
*** otter768 has quit IRC07:00
openstackgerritTony Breeds proposed openstack-infra/project-config: Set BRANCH for periodic unittest bitrot jobs  https://review.openstack.org/24552507:00
*** pahuang has joined #openstack-infra07:01
*** thorst has quit IRC07:02
*** abregman|afk has quit IRC07:08
*** coolsvap|away is now known as coolsvap07:13
*** camunoz has quit IRC07:15
*** ozamiatin has joined #openstack-infra07:15
*** _nadya_ has joined #openstack-infra07:18
openstackgerritMerged openstack/requirements: Updated gabbi to 1.11.0  https://review.openstack.org/25559807:23
*** unicell has quit IRC07:24
*** unicell has joined #openstack-infra07:24
*** flepied1 has quit IRC07:27
*** ParsectiX has joined #openstack-infra07:30
*** ybathia has quit IRC07:39
*** mpavone has joined #openstack-infra07:40
openstackgerritMerged openstack-infra/project-config: move osc-plugin-job from experimental to check  https://review.openstack.org/26510707:40
*** mpavone has quit IRC07:40
*** _nadya_ has quit IRC07:43
*** sam_wan has joined #openstack-infra07:44
*** ihrachys has joined #openstack-infra07:45
*** yamamoto_ has joined #openstack-infra07:45
*** amotoki has quit IRC07:46
*** amotoki has joined #openstack-infra07:46
yolandamorning07:47
*** ihrachys has quit IRC07:48
*** ihrachys has joined #openstack-infra07:48
*** mrmartin has joined #openstack-infra07:52
*** hashar has joined #openstack-infra07:53
openstackgerritMerged openstack-infra/puppet-openstackci: Initial sphynx doc setup for publishing  https://review.openstack.org/25924507:53
AJaegermorning, yolanda !07:54
yolandaback to life after a long holiday :)07:54
AJaegeryolanda: could you review https://review.openstack.org/#/c/261857/ , please? I'll babysit it ...07:54
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Add command deleting *.pyc before executing ostestr  https://review.openstack.org/26512507:54
yolandasure07:54
AJaegeryolanda: great to have you back - and hope you really relaxed07:54
AJaegeryolanda: you were far too often here ;)07:55
*** ozamiatin has quit IRC07:55
yolandatoo much time without a computer near, i miss it :)07:55
openstackgerritMerged openstack-infra/puppet-storyboard: Add config template for the email plugin  https://review.openstack.org/26330407:55
AJaeger;)07:56
*** ihrachys_ has joined #openstack-infra07:56
*** oomichi has quit IRC07:56
AJaegeryolanda: could you add Anne Gentle to fairy-slipper-core, please? see https://review.openstack.org/#/c/245352/ - https://review.openstack.org/#/admin/groups/1240,members and it's annegentle@justwriteclick.com07:57
*** ihrachys has quit IRC07:57
yolandaok will do07:57
AJaegerthanks07:58
yolandaAJaeger, are you familiar with that translation scripts? i have a question07:59
AJaegeryolanda: yes, I'm familiar07:59
AJaegerbut have a call now - might answer in 30 mins...07:59
*** salv-orlando has quit IRC08:00
yolandawill add the question to the review08:00
*** thorst has joined #openstack-infra08:00
*** salv-orlando has joined #openstack-infra08:00
*** coolsvap is now known as coolsvap|away08:00
AJaegerok08:00
*** otter768 has joined #openstack-infra08:00
yolandaadded, let's talk later08:00
*** otter768 has quit IRC08:02
*** otter768_ has joined #openstack-infra08:02
*** otter768 has joined #openstack-infra08:04
*** fawadkhaliq has quit IRC08:04
*** coolsvap|away is now known as coolsvap08:05
*** otter76__ has joined #openstack-infra08:06
*** samuelBartel has joined #openstack-infra08:06
*** thorst has quit IRC08:06
*** otter768_ has quit IRC08:06
*** hdd has quit IRC08:07
*** otter768_ has joined #openstack-infra08:08
*** otter76__ has quit IRC08:08
*** ihrachys_ has quit IRC08:08
*** [patrol] has left #openstack-infra08:08
*** otter768 has quit IRC08:09
*** cschwede has joined #openstack-infra08:10
*** otter768 has joined #openstack-infra08:10
*** otter768_ has quit IRC08:10
*** otter768_ has joined #openstack-infra08:12
*** otter768 has quit IRC08:12
AJaegeryolanda: I'm back again and commented.08:13
AJaegerIf there's a better way to do this, I'm all ears...08:13
AJaegerOr should I just add a comment that this is set as side effect?08:13
*** otter768 has joined #openstack-infra08:13
*** otter768_ has quit IRC08:14
*** bexelbie has quit IRC08:14
*** bexelbie has joined #openstack-infra08:14
yolandaAJaeger, what's the effect of not having a CHANGE_ID= because in the else, we don't populate one08:14
*** oomichi has joined #openstack-infra08:14
yolandais that ok to ignore if not change id?08:14
*** markvoelker has joined #openstack-infra08:15
*** yamahata has joined #openstack-infra08:15
AJaegeryolanda: we handle today already the case where it's not set, so this is fine08:15
*** otter768 has quit IRC08:15
*** otter768_ has joined #openstack-infra08:15
*** yamamoto_ has quit IRC08:15
AJaegerlogic here is: Do something special if there exist already an old review08:16
*** jaosorior has quit IRC08:16
*** jtomasek has joined #openstack-infra08:16
*** jaosorior has joined #openstack-infra08:16
yolandaok i think it's fine. As the code is now hidden on the function, it's not obvious to see where the value comes from, but don't want to -1 for that08:17
yolandai'll approve08:17
*** otter768 has joined #openstack-infra08:17
*** otter76__ has joined #openstack-infra08:19
*** markvoelker has quit IRC08:19
*** otter768_ has quit IRC08:19
AJaegerthanks08:20
AJaegerlet me update the followup change with a comment ;)08:21
*** otter768_ has joined #openstack-infra08:21
*** otter76__ has quit IRC08:21
*** otter768 has quit IRC08:22
yolandait's worth noting that we can set CHANGE_ID and COMMIT_MSG on the nested function08:23
*** otter768 has joined #openstack-infra08:23
*** otter768_ has quit IRC08:23
*** jistr has joined #openstack-infra08:24
*** otter768 has quit IRC08:25
*** otter768_ has joined #openstack-infra08:25
*** markus_z has joined #openstack-infra08:26
*** yamamoto has joined #openstack-infra08:27
*** otter768 has joined #openstack-infra08:27
*** otter768_ has quit IRC08:27
*** arxcruz has joined #openstack-infra08:27
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Unify usages of checking for existing changes (2/2)  https://review.openstack.org/26185808:29
*** otter768_ has joined #openstack-infra08:29
AJaegeryolanda: I've added comments everywhere, please review ^08:29
*** otter768_ has quit IRC08:30
yolandadone08:30
*** otter768_ has joined #openstack-infra08:31
*** otter768 has quit IRC08:32
*** yamamoto has quit IRC08:32
*** otter768_ has quit IRC08:32
*** otter768 has joined #openstack-infra08:33
*** otter768 has quit IRC08:34
*** otter768 has joined #openstack-infra08:34
*** otter768_ has joined #openstack-infra08:36
AJaegerthanks, yolanda08:38
*** salv-orlando has quit IRC08:38
*** otter76__ has joined #openstack-infra08:38
*** otter768_ has quit IRC08:39
*** yaume has joined #openstack-infra08:39
*** otter768 has quit IRC08:39
*** salv-orlando has joined #openstack-infra08:39
*** yaume is now known as yaume[away]08:39
*** otter76__ has quit IRC08:40
*** markmcd has quit IRC08:40
*** otter768 has joined #openstack-infra08:40
*** markmcd has joined #openstack-infra08:42
*** otter768_ has joined #openstack-infra08:42
*** otter768 has quit IRC08:42
*** yaume[away] is now known as yaume08:42
*** yaume is now known as yaume[away]08:43
*** otter768 has joined #openstack-infra08:44
*** otter768_ has quit IRC08:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Support GitHub PR webhooks  https://review.openstack.org/16311708:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Make the string representation of change transparent  https://review.openstack.org/23894808:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Lower the log level in tests  https://review.openstack.org/26336408:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Merge pull requests from github reporter  https://review.openstack.org/24325008:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Encapsulate determining the event purpose  https://review.openstack.org/24748708:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Fix job hierarchy bug.  https://review.openstack.org/19245708:44
openstackgerritJan Hruban proposed openstack-infra/zuul: support github pull reqeust labels  https://review.openstack.org/24742108:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Add 'push' and 'tag' github webhook events.  https://review.openstack.org/19120708:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Add 'pr-comment' github webhook event  https://review.openstack.org/23920308:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Support for dependent pipelines with github  https://review.openstack.org/24750008:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Configurable SSH access to GitHub  https://review.openstack.org/23913808:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Allow github trigger to match on branches/refs  https://review.openstack.org/25844808:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Allow using webapp from connections  https://review.openstack.org/21564208:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Allow list values in template parameters.  https://review.openstack.org/19120808:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Add basic Github Zuul Reporter.  https://review.openstack.org/19131208:44
openstackgerritJan Hruban proposed openstack-infra/zuul: Support for github commit status  https://review.openstack.org/23930308:44
*** matrohon has joined #openstack-infra08:45
*** otter768 has quit IRC08:46
TheloHi all, I created an image with nodepool, uploaded it to a openstack cloud and now try to run jobs from jenkins however it looks like jenkins cannot ssh the jenkins slave (SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 38 ms)08:46
*** otter768 has joined #openstack-infra08:46
Thelodoes anyone know what key nodepool use by default when we follow this procedure (http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md)?08:46
*** yaume[away] is now known as yaume08:47
*** yaume is now known as yaume[away]08:48
*** otter768_ has joined #openstack-infra08:48
*** scheuran has joined #openstack-infra08:49
*** yaume[away] is now known as yaume08:49
*** yaume is now known as yaume[away]08:50
*** otter76__ has joined #openstack-infra08:50
*** otter768_ has quit IRC08:50
*** otter768 has quit IRC08:51
*** otter76__ has quit IRC08:51
*** otter768 has joined #openstack-infra08:52
*** otter768_ has joined #openstack-infra08:54
*** yaume[away] is now known as yaume08:55
*** otter76__ has joined #openstack-infra08:56
*** otter768_ has quit IRC08:56
openstackgerritMerged openstack-infra/project-config: Unify usages of checking for existing changes (1/2)  https://review.openstack.org/26185708:56
*** otter768 has quit IRC08:57
mrmartinmorning08:57
*** otter76__ has quit IRC08:57
*** otter768 has joined #openstack-infra08:57
*** shardy has joined #openstack-infra08:57
mrmartinis it ok that we have a ton of dependency warnings after deploying a clean puppet environment: http://paste.openstack.org/show/483303/08:58
openstackgerritMerged openstack-infra/project-config: Test gate-grenade-dsvm-partial-ncpu on requirements repo  https://review.openstack.org/26495708:58
*** otter768 has quit IRC08:59
*** jlanoux has joined #openstack-infra08:59
*** otter768 has joined #openstack-infra08:59
*** ifarkas has joined #openstack-infra09:00
*** otter768 has quit IRC09:01
*** otter768 has joined #openstack-infra09:02
*** savihou has joined #openstack-infra09:03
*** otter768 has quit IRC09:03
*** otter768_ has joined #openstack-infra09:03
*** thorst has joined #openstack-infra09:04
*** zeih has joined #openstack-infra09:04
*** otter768_ has quit IRC09:05
*** otter768 has joined #openstack-infra09:05
savihouHello Gents, sorry for the noob question; I'm in the process of setting a jenkinks based (with the help of zuul and gearman) CI system. I can get the gerrit upstream however, cannot post a review to ci-sandbox), any chance I'm missing something really stupid here ? I cannot see any trace of failures within zuul logs . your help will be much appreciated.09:07
*** otter768_ has joined #openstack-infra09:07
*** otter768 has quit IRC09:07
*** otter768_ has quit IRC09:08
*** otter768 has joined #openstack-infra09:09
*** sfinucan has joined #openstack-infra09:09
Thelomrmartin: I had this yesterday too, I reinstall ubuntu 14.04.2 and tried again and that time got no warning09:12
*** thorst has quit IRC09:12
*** otter768 has quit IRC09:14
mrmartinwe have those dependency warnings for example on ask-staging site, and I think on all other instances. I'm not sure that it makes any troubles, maybe just a result of not properly defined deps, or leftovers.09:14
openstackgerritMerged openstack-infra/irc-meetings: Add explict start dates to meeting yaml  https://review.openstack.org/26505609:14
*** dstanek has quit IRC09:16
*** dstanek has joined #openstack-infra09:18
*** dizquierdo has joined #openstack-infra09:18
*** rossella_s has joined #openstack-infra09:19
*** ihrachys has joined #openstack-infra09:20
*** ihrachys has quit IRC09:21
*** gchamoul is now known as gchamoul_Doc_afk09:22
*** ihrachys has joined #openstack-infra09:24
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Add min ver to extra libs for graph  https://review.openstack.org/26515309:30
*** yaume has quit IRC09:35
*** ihrachys has quit IRC09:35
*** kushal has joined #openstack-infra09:36
*** ihrachys has joined #openstack-infra09:38
*** mpaolino has joined #openstack-infra09:38
*** jordanP has joined #openstack-infra09:39
*** ihrachys_ has joined #openstack-infra09:40
*** dtantsur|afk is now known as dtantsur09:40
*** yamahata has quit IRC09:41
*** ihrachys has quit IRC09:43
*** yaume has joined #openstack-infra09:43
*** bauzas is now known as bauwser09:43
*** jtomasek has quit IRC09:45
*** ihrachys_ has quit IRC09:51
*** hashar has quit IRC09:52
*** hashar has joined #openstack-infra09:53
*** derekh has joined #openstack-infra09:53
*** rguillebert has joined #openstack-infra09:53
*** [1]Thelo has joined #openstack-infra09:54
*** ihrachys has joined #openstack-infra09:55
*** fawadkhaliq has joined #openstack-infra09:56
*** Thelo has quit IRC09:56
*** [1]Thelo is now known as Thelo09:56
openstackgerritAntoine Musso proposed openstack-infra/nodepool: tools: shell script to gracefully stop and wait  https://review.openstack.org/21944210:02
*** ociuhandu has quit IRC10:06
openstackgerritAntoine Musso proposed openstack-infra/nodepool: node deletion delay is now configurable  https://review.openstack.org/24522010:07
TheloHi all, I created an image with nodepool, uploaded it to a openstack cloud and now try to run jobs from jenkins however it looks like jenkins cannot ssh the jenkins slave (SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 38 ms)10:08
Thelo does anyone know what key nodepool use by default when we follow this procedure (http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md)?10:08
*** salv-orl_ has joined #openstack-infra10:09
*** thorst has joined #openstack-infra10:09
yolandahi Thelo, you can configure the key for the image in nodepool.yaml file10:09
yolandafor each image, you can define a private-key setting, that you can point to the ssh file you want to use10:10
yolandalook at http://docs.openstack.org/infra/nodepool/configuration.html10:10
jklareAJaeger hi and a happy new year10:11
AJaegerhappy new year, jklare !10:11
Theloyolanda: Hi, I did that, so I guess the node might not have the public key10:12
*** salv-orlando has quit IRC10:12
Theloyolanda: when is this one passed to the node10:12
jklareAJaeger i have a question regarding the process of creating eol tags before removing old branches10:12
yolandaThelo, have you also tried manually? from nodepool node, try ssh using that key as identity, with jenkins user. So you can check if the key is correct10:12
*** ihrachys has quit IRC10:13
AJaegerjklare: best talk with fungi later on.10:13
jklareAJaeger i read the wiki about the release process and how to push tags during this one, do you know if its similar for eol tags?10:13
jklareAJaeger ah ok10:13
AJaegerjklare: fungi will create a eol tag as part of removing the branch, so nothing for you to do10:13
jklareAJaeger thats great, less work for me then :)10:14
Theloyolanda: it works fine, so why jenkins cannot do it?10:14
jklareAJaeger thanks10:14
*** tzn has joined #openstack-infra10:14
*** dizquierdo has quit IRC10:14
yolandaThelo, one step is nodepool connecting by ssh to the vm10:15
*** markvoelker has joined #openstack-infra10:15
yolandathen, nodepool runs a create node script, that may add the keys for your jenkins masters and configure properly10:15
yolandaso i guess you have some problem in that creation scirpt10:15
*** thorst has quit IRC10:17
Theloyolanda: Shouldn't it be the opposite? If I can log in manually using this key, it could means that jenkins is not using the right key no?10:17
yolandalooks like that is the problem, yes10:18
yolandayou should check the scripts you use to prepare your image, and ensure that you are granting permissions to jenkins user with the right keys10:19
*** markvoelker has quit IRC10:20
Theloyolanda: I ssh-ed manually using the user jenkins and the key so my guess is that the image should be working, do you know where I can see which user/key jenkins is using to ssh the node?10:20
*** mpaolino has quit IRC10:23
yolandaThelo, if you use puppet-jenkins, it relies on jenkins::user class. This needs to be present on the slaves10:23
yolandathat key is normally managed by hiera10:24
Theloyolanda: You are very right I check the key that jenkins used and that one is not correct, my bad10:24
*** sam_wan has quit IRC10:25
yolandaglad to help10:25
Theloyolanda: You are very right I check the key that jenkins used and found that this "file" does not exits10:26
Theloyolanda: however, puppet created the user jenkins but I cannot find any /home/jenkins directory10:28
yolandaThelo, check in /var/lib/jenkins/10:30
*** zhurong has quit IRC10:30
*** zhurong has joined #openstack-infra10:31
Theloyolanda: you mean the .ssh folder?10:33
Thelothat one contains the right key apparently10:33
Theloyolanda: however through jenkins ui I can seen that for this node it is not taking this key10:34
yolandahave you checked that step? http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n34710:35
openstackgerritThierry Carrez proposed openstack/requirements: Add missing license information  https://review.openstack.org/26475410:35
*** zhurong has quit IRC10:37
openstackgerritThierry Carrez proposed openstack/requirements: Add missing license information  https://review.openstack.org/26475410:37
*** jaosorior has quit IRC10:37
*** jaosorior has joined #openstack-infra10:38
Theloyolanda: No sorry, I skipped the optional10:38
TheloI will do it10:38
yolandathat will help with your creds problem10:39
*** jlanoux has quit IRC10:40
openstackgerritChristoph Albers proposed openstack-infra/project-config: Removed testjobs and corresponding chef-jobs  https://review.openstack.org/26413010:40
*** electrofelix has joined #openstack-infra10:41
*** ihrachys has joined #openstack-infra10:42
openstackgerritMerged openstack/requirements: Cap django_compressor below 2.0  https://review.openstack.org/26502510:50
Theloyolanda: Thanks, however I still get JenkinsException: Error in request. Possibly authentication failed [401]: Invalid password/token for user: jenkins10:52
Theloyolanda: and I just retry to log in using jenkins and this key manually with success10:52
yolandaThelo, that's a different error than before right?10:53
*** ihrachys has quit IRC10:54
Theloyes10:54
yolandacheck jenkins global security and token, as listed on documentation10:55
*** mrmartin has quit IRC10:55
*** Qiming has quit IRC10:55
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements  https://review.openstack.org/24611410:56
*** gchamoul_Doc_afk is now known as gchamoul10:57
*** daemontool has joined #openstack-infra10:58
Theloyolanda : at http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n353 if I click on advanced I can see that the passphrase field is not empty, is it normal?10:59
yolandacannot tell, it may be a jenkins feature to mask password11:00
*** fawadkhaliq has quit IRC11:05
*** fawadkhaliq has joined #openstack-infra11:05
Theloyolanda: I still get this in nodepool's log (http://paste.openstack.org/show/483308/)11:05
TheloI double check the configuration and everything seems fine11:05
*** pahuang has quit IRC11:07
yolandaThelo, that looks as a problem with jenkins api key, check that the value you set for api key and credentials matches the hiera value, and is correctly set on nodepool, on /etc/nodepool/secure.conf11:07
Thelobuthe hiera value is the value define in common.yaml?11:08
*** dims has joined #openstack-infra11:09
*** otter768 has joined #openstack-infra11:10
yolandayes11:10
*** lucas-dinner is now known as lucasagomes11:12
Theloyolanda: I don't get it everything seems fine, should I restart nodepool?11:13
yolandait may pick new config automatically, but is worth trying11:13
*** otter768 has quit IRC11:14
*** thorst has joined #openstack-infra11:15
*** piet has joined #openstack-infra11:17
dimssdague : let's switch on the grenade job for o.vo please? https://review.openstack.org/#/c/264992/11:18
Theloyolanda: it stucks at restarting, does it require to connect to jenkins to start?11:19
*** jlanoux has joined #openstack-infra11:20
yolandabetter stop and start it11:20
Theloindeed11:21
odyssey4mehmm, it seems that git archive doesn't work against git.o.o - is that correct?11:22
*** thorst has quit IRC11:22
*** ociuhandu has joined #openstack-infra11:23
Theloyolanda: now the error log in nodepool didnt show up however I still get the [01/08/16 19:21:59] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 76 ms in jenkins logs11:23
*** fawadkhaliq has quit IRC11:25
*** fawadkhaliq has joined #openstack-infra11:25
*** jlanoux_ has joined #openstack-infra11:27
*** jlanoux has quit IRC11:29
yolandaif the failure is by ssh, you have a mismatch on the jenkins ssh keys. And if that on jenkins pass, it means you have the wrong jenkins pass and api key11:30
*** piet has quit IRC11:31
*** Roamer` has joined #openstack-infra11:34
Roamer`hi, are people aware of the fact that the xio-ise-iscsi-ci account has gone ballistic on https://review.openstack.org/#/c/264056?11:37
*** shardy has quit IRC11:37
*** ozamiatin has joined #openstack-infra11:38
*** dizquierdo has joined #openstack-infra11:40
*** ihrachys has joined #openstack-infra11:43
*** Qiming has joined #openstack-infra11:43
*** rfolco has joined #openstack-infra11:45
*** amotoki_ has joined #openstack-infra11:52
*** ociuhandu_ has joined #openstack-infra11:54
*** amotoki has quit IRC11:54
*** ociuhandu has quit IRC11:56
*** ociuhandu_ is now known as ociuhandu11:56
*** ihrachys_ has joined #openstack-infra11:57
*** ihrachys has quit IRC11:57
*** khappone_ has quit IRC12:00
*** markvoelker has joined #openstack-infra12:01
*** scheuran has quit IRC12:02
*** zeih has quit IRC12:05
*** zeih has joined #openstack-infra12:06
*** markvoelker has quit IRC12:06
*** matrohon has quit IRC12:10
*** ihrachys has joined #openstack-infra12:11
*** ihrachys_ has quit IRC12:12
sdaguedims: lgtm12:12
dimssdague : thanks!12:15
*** ihrachys has quit IRC12:16
*** mrmartin has joined #openstack-infra12:16
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add openstack/python-ironicclient to PROJECTS  https://review.openstack.org/26523012:17
*** sheeprine has quit IRC12:17
*** sheeprine has joined #openstack-infra12:18
openstackgerritMerged openstack-infra/project-config: Add a grenade job for testing oslo.versionedobjects  https://review.openstack.org/26499212:19
*** rhallisey has joined #openstack-infra12:19
*** thorst has joined #openstack-infra12:19
openstackgerritMarton Kiss proposed openstack-infra/puppet-askbot: Fix broken Pillow pip dependency  https://review.openstack.org/26523112:22
*** thorst has quit IRC12:27
*** ihrachys has joined #openstack-infra12:31
*** ihrachys has quit IRC12:31
*** coolsvap is now known as coolsvap|away12:34
*** khappone has joined #openstack-infra12:39
*** dims_ has joined #openstack-infra12:40
*** dims has quit IRC12:41
*** weshay_xchat has joined #openstack-infra12:41
*** weshay_xchat is now known as weshay12:42
*** markvoelker has joined #openstack-infra12:45
*** fawadkhaliq has quit IRC12:50
*** flepied has joined #openstack-infra12:53
*** fawadkhaliq has joined #openstack-infra12:55
*** alaski is now known as lascii12:59
*** thorst has joined #openstack-infra13:01
*** hashar has quit IRC13:02
*** dprince has joined #openstack-infra13:09
*** matrohon has joined #openstack-infra13:10
*** otter768 has joined #openstack-infra13:11
jordanPclarkb, fungi I am looking for another +2 here https://review.openstack.org/#/c/239363/ It's a trivial patch. Could you help me please ?13:12
*** esker has joined #openstack-infra13:13
*** tlian has joined #openstack-infra13:14
*** otter768 has quit IRC13:16
*** dims_ has quit IRC13:16
AJaegerjordanP: that change merged 9 hours ago, didn't it?13:18
jordanPAJaeger, ... wrong link, sorry ! I meant https://review.openstack.org/#/c/250879/13:18
*** esker has quit IRC13:20
anteayayolanda fungi we need X-IO OpenStack CI disabled, should be gerrit id 1496913:23
*** matrohon has quit IRC13:23
anteayayolanda fungi they are repeatedly commenting on 26405613:23
anteayaRoamer`: thank you, potentially activity like this can bring down our zuul, should you see it again in future, please make a louder fuss13:24
anteayaRoamer`: thanks for saying something13:24
anteayainfra-root: ^^13:24
*** matrohon has joined #openstack-infra13:25
*** links has quit IRC13:25
anteayayolanda fungi I believe the username on that account is xio-ise-iscsi-c13:26
*** pradk has joined #openstack-infra13:26
*** pradk has quit IRC13:26
anteayaxio-ise-iscsi-ci13:26
*** dims has joined #openstack-infra13:26
anteayasorry it is our gerrit that shows the effects, not zuul13:27
*** Guest76580 has quit IRC13:27
anteayahttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=34&rra_id=all13:27
*** dingyichen has quit IRC13:29
*** edmondsw has joined #openstack-infra13:35
anteayaI have to leave soon to run an errand but I think this is all the info any infra-root requires to disable 14969 to stop the commenting loop on 26405613:39
anteayaI'll be back in a few hours13:40
* anteaya tosses the message in a bottle into the ocean and hopes someone sees it soon13:40
*** claudiub has joined #openstack-infra13:40
Roamer`anteaya, no problem, it actually brought down my (StorPool) Zuul, that's how I found out about it :)13:43
*** sigmavirus24_awa is now known as sigmavirus2413:46
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename .log to .log.txt in run-tox  https://review.openstack.org/20309913:46
*** sileht has quit IRC13:46
AJaegerfungi, this is an old patch of you that I updated - could you review, please? ^13:47
*** rcarrillocruz has quit IRC13:47
*** sileht has joined #openstack-infra13:48
*** dkranz has joined #openstack-infra13:48
*** rcarrillocruz has joined #openstack-infra13:49
*** ujuc has joined #openstack-infra13:49
anteayaRoamer`: :(13:49
*** gordc has joined #openstack-infra13:50
anteayaAJaeger: if you are online when yolanda or fungi appears can you point them at the request to disable 14969 please as a priority?13:51
*** e0ne has joined #openstack-infra13:52
*** Hal has joined #openstack-infra13:52
*** Hal is now known as Guest2883113:53
AJaegeranteaya: yes, will do.13:53
anteayaAJaeger: thank you13:54
anteayarhedlind is the operators irc nick but he isn't online right now13:55
anteayaoperator's*13:55
AJaegeranteaya: thanks for gathering all information!13:55
anteayaAJaeger: thanks :)13:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Make gate-rally-dsvm-cinder job voting for Cinder patches  https://review.openstack.org/20368013:59
openstackgerritMerged openstack-infra/release-tools: use send-announcements-to metadata for deliverables  https://review.openstack.org/26228114:00
openstackgerritMerged openstack-infra/release-tools: use default from address for emails  https://review.openstack.org/26391214:00
openstackgerritMerged openstack-infra/release-tools: add command to actually send email announcements  https://review.openstack.org/26391314:00
*** bcourt has quit IRC14:02
*** jaosorior has quit IRC14:02
*** annegentle has quit IRC14:02
*** annegentle has joined #openstack-infra14:02
*** dtantsur is now known as dtantsur|brb14:02
anteayawill check back when I return14:03
*** baoli has joined #openstack-infra14:03
*** baoli has quit IRC14:05
*** baoli has joined #openstack-infra14:05
openstackgerritZara proposed openstack-infra/storyboard: Rudimentary Templates for Tasks  https://review.openstack.org/26425514:07
*** baoli_ has joined #openstack-infra14:07
*** salv-orl_ has quit IRC14:08
*** salv-orlando has joined #openstack-infra14:08
*** rlandy has joined #openstack-infra14:09
*** baoli has quit IRC14:10
*** jordanP is now known as jordanP|away14:14
*** lucasagomes is now known as lucas-hungry14:14
*** esker has joined #openstack-infra14:17
*** bcourt has joined #openstack-infra14:17
*** hdd has joined #openstack-infra14:21
*** amitgandhinz has joined #openstack-infra14:21
*** esker has quit IRC14:22
*** rossella_s has quit IRC14:22
*** rossella_s has joined #openstack-infra14:22
*** esker has joined #openstack-infra14:22
*** peristeri has joined #openstack-infra14:24
AJaegerdougwig, could you review https://review.openstack.org/#/c/258992/ , please?14:28
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Add python-ironicclient to PROJECTS  https://review.openstack.org/26333514:29
*** jsavak has joined #openstack-infra14:30
*** shardy has joined #openstack-infra14:31
*** ari has joined #openstack-infra14:31
openstackgerritDirk Mueller proposed openstack/requirements: Use OpenStack Foundation for setup.py copyright  https://review.openstack.org/26494914:33
*** dtantsur|brb is now known as dtantsur14:34
*** bcourt has quit IRC14:34
openstackgerritHongbin Lu proposed openstack-infra/project-config: Mark swarm test as non-voting in Magnum  https://review.openstack.org/26499814:34
openstackgerritHongbin Lu proposed openstack-infra/project-config: [WIP] Revert "Mark swarm test as non-voting in Magnum"  https://review.openstack.org/26527314:34
*** kgiusti has joined #openstack-infra14:35
openstackgerritFlavio Percoco proposed openstack-infra/project-config: Add prettytable into openstack  https://review.openstack.org/26527814:38
*** dkehn has quit IRC14:40
*** dkehn has joined #openstack-infra14:41
*** mriedem has joined #openstack-infra14:42
*** ujuc has quit IRC14:43
openstackgerritPaul Belanger proposed openstack-infra/grafyaml: Add support for template query  https://review.openstack.org/25089314:45
mriedemis the reqs proposal bot broken?14:45
pabelangermorning14:46
AJaegermriedem: https://jenkins.openstack.org/job/propose-requirements-updates/14:46
AJaegerstable/kilo failed, rest looks fine...14:46
AJaegermorning, pabelanger14:46
*** bcourt has joined #openstack-infra14:46
AJaegermriedem: why do you ask?14:47
mriedemi don't see a sync to horizon for https://review.openstack.org/#/c/265025/14:47
*** nelsnelson has quit IRC14:47
*** esker has quit IRC14:48
*** jaypipes has joined #openstack-infra14:49
mriedemand that's kilo14:49
mriedemso that's probably why14:49
AJaegeryep14:49
mriedemi'll sync horizon on kilo manually14:49
*** ParsectiX has quit IRC14:49
pabelangeryolanda: mind checking out 262017 and 262019?14:50
*** pradk has joined #openstack-infra14:50
AJaegermriedem: https://review.openstack.org/#/c/265179/14:50
*** jordanP|away is now known as jordanP14:50
*** doug-fish has joined #openstack-infra14:50
AJaegermriedem: and the kilo job has run in general - just failed on at least one repository14:51
mriedemAJaeger: ah, didn't see it on the dashboard i have since jenkins was -114:51
*** amotoki_ has quit IRC14:51
AJaegerhorizon fails with testrepository problem ;(14:51
mriedemshould be able to just backport a fix i'd think, i'm looking14:52
*** esker has joined #openstack-infra14:54
*** fawadkhaliq has quit IRC14:54
*** fawadkhaliq has joined #openstack-infra14:54
*** fawadkhaliq has quit IRC14:55
*** ari is now known as Ari14:55
*** Ari is now known as alivigni14:55
openstackgerritVincent Fran├žoise proposed openstack-infra/project-config: Enable Tempest tests for Watcher  https://review.openstack.org/26529114:57
yolandaanteaya, i was having lunch and a short break14:58
*** esker has quit IRC14:58
yolandai never disabled an account so i need to read about it14:58
Sam-I-Amanteaya: you sleep less than i do :)14:59
AJaegeryolanda: anteaya is out now and I don't know what to do either ;(14:59
*** ganso has quit IRC14:59
yolandai can check for the account in gerrit and check permissions but i'm not sure if that's the right approach15:00
AJaegerIs any other infra-root around to help here?15:00
AJaegerJust checked the situation still happens ;(15:00
openstackgerritVincent Fran├žoise proposed openstack-infra/project-config: Enable Tempest tests for Watcher  https://review.openstack.org/26529115:00
*** vgridnev has joined #openstack-infra15:01
*** esker has joined #openstack-infra15:01
yolandaAJaeger, i can mark the account as inactive in gerrit, i hope that this helps15:02
pabelangernibalizer: yolanda: crinkle: clarkb: What do you all think of 205596? Switching to puppetlabs-apache for nodepool using apache::vhost::custom15:02
asselin_Thelo, mrmartin I think those puppet warnings are ok...15:02
mrmartinasselin_ hi15:03
mrmartinok.15:03
yolandaaccount was marked as inactive now15:03
yolandapabelanger, taking a look at reviews now15:04
asselin_Thelo, yolanda setting up jenkins security is optional last time I tried. It should work without it.15:07
AJaegerthanks, yolanda15:07
*** eharney has joined #openstack-infra15:08
*** esker has quit IRC15:08
TheJuliacrinkle: when you get on, I have a question w/r/t the mysql module upgrade and db grants on bare machines15:08
yolandaasselin, seems Thelo had some mismatch on the ssh key and creds, but haven't figured the problem15:08
*** jaypipes is now known as leakypipes15:08
*** hdd has quit IRC15:09
*** zz_dimtruck is now known as dimtruck15:09
*** esker has joined #openstack-infra15:09
asselin_yolanda, Thelo I think only the jenkins ssh key needs to be correct in the file system /var/lib/jenkins/.ssh/id_rsa, which is done via puppet with value from heira15:10
*** otter768 has joined #openstack-infra15:12
yolandaanyway, infra-root, when you come online, i'd like to know if i did the right thing just setting the account to inactive to disable a third-party-ci15:12
Roamer`yolanda, thanks a lot for doing that!15:12
Roamer`yolanda, however, it doesn't seem that it's enough :(  https://review.openstack.org/#/c/264056 still gets new votes from that account :(15:13
*** zeih has quit IRC15:14
Roamer`yolanda, maybe the CI in question is still logged in via SSH or something15:14
asselin_yolanda, I remember fungi mentioning closing out some connections15:15
*** esker has quit IRC15:15
*** zeih has joined #openstack-infra15:16
*** otter768 has quit IRC15:16
yolandaok let me check15:16
pabelangeryolanda: great15:19
*** Qiming has quit IRC15:19
*** matrohon has quit IRC15:19
*** esker has joined #openstack-infra15:20
yolandaasselin, i cannot see any ssh connections to kill15:20
*** zeih has quit IRC15:20
*** lucas-hungry is now known as lucasagomes15:20
*** hashar has joined #openstack-infra15:21
openstackgerritDimitry Ushakov proposed openstack-infra/project-config: Add solum-pythonclient to projects  https://review.openstack.org/26508815:22
*** fawadkhaliq has joined #openstack-infra15:23
*** dansmith is now known as superdan15:23
openstackgerritDimitry Ushakov proposed openstack-infra/project-config: Add solum-pythonclient to projects  https://review.openstack.org/26508815:24
*** esker has quit IRC15:25
fungiyolanda: anteaya: looking at the gerrit ssh logs, xio-ise-iscsi-ci continues to leave lots of comments even now15:26
yolandafungi, what is the play for that? i just set that as inactive on gerrit15:26
yolandaand flushed caches just in case15:26
fungiossh -p 29418 review gerrit show-connections -n|grep 1496915:26
fungiossh -p 29418 review gerrit close-connection d9a23b3d15:27
fungiet cetera15:27
yolandaok, missed the -n15:27
yolandai did a gerrit show-connections and didn't find anything15:27
fungiwithout the -n you have to look for the username, and it may be truncated to fit the column15:28
fungiunfortunate that this is the same account which went ballistic and looped not so long ago15:28
*** nwkarsten has joined #openstack-infra15:29
*** esker has joined #openstack-infra15:30
yolandafungi, i closed them but new ones appear, even if the account is inactive15:30
*** esker has quit IRC15:30
*** mtanino has joined #openstack-infra15:31
*** ganso has joined #openstack-infra15:32
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin  https://review.openstack.org/26531115:34
*** savihou has quit IRC15:34
*** andrearosa has joined #openstack-infra15:35
fungiyolanda: the api claims that 14969 account to still be active15:36
fungiyolanda: how did you attempt to disable it?15:36
yolandafungi, first in the database and flushing caches, and after that i also tried a gerrit set-account --inactive15:36
fungithe db does indeed have inactive=Y for 14969 in the accounts table15:38
yolandayes, i did that update, but doesn't seem to be reflected15:39
jordanPhttp://developer.openstack.org/api-ref-compute-v2.1.html returns 404....15:39
AJaegerfungi, clarkb: could you later check https://review.openstack.org/#/c/264978/ and tell me whether moving the script to the app-catalog repo that runs from the periodic job is fine, please?15:39
AJaegerjordanP: please report on #openstack-doc15:39
*** e0ne has quit IRC15:42
fungigerrit flush-caches is taking several minutes to complete15:43
*** hashar has quit IRC15:44
fungiyolanda: i used the ssh api to toggle the account to active and back to inactive, then flushed all gerrit's caches, then looked for and killed all open connections for the account in question and the comments seem to have subsided now15:44
*** yamahata has joined #openstack-infra15:44
yolandacool, so maybe doing that with database needs a restart?15:45
fungidirectly modifying the backend database has a tendency to not get reflected in running memory, yeah15:45
fungidepends on what it is, but it seems that was probably where things went badly15:45
openstackgerritHongbin Lu proposed openstack-infra/project-config: [WIP] Revert "Mark swarm test as non-voting in Magnum"  https://review.openstack.org/26527315:45
openstackgerritHongbin Lu proposed openstack-infra/project-config: Mark swarm test as non-voting in Magnum  https://review.openstack.org/26499815:45
*** zz_zz_ja has quit IRC15:46
yolandafungi, new lesson learnt, thanks. I'll add this to the runbook for third party, so it gets reflected for another time15:46
*** zz_ja has joined #openstack-infra15:46
*** piet has joined #openstack-infra15:46
*** links has joined #openstack-infra15:47
*** sabeen has joined #openstack-infra15:47
*** abregman has joined #openstack-infra15:48
*** jlanoux_ has quit IRC15:49
jeblairpabelanger: can you approve https://review.openstack.org/264985 and https://review.openstack.org/264988 ?15:49
Roamer`fungi, yolanda, anteaya, AJaeger, thanks a lot to all of you!  Now let's see if my Zuul will be able to cope with the Gerrit events...15:49
*** jlanoux has joined #openstack-infra15:49
*** bnemec has quit IRC15:50
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Remove Python 2.6 compatibility patch.  https://review.openstack.org/26532115:51
fungiyolanda: for future reference, you're also supposed to be able to make an account inactive through the rest api by sending a delete for /a/accounts/{account-id}/active15:51
jeblairAJaeger, fungi, clarkb, docaedo, nibalizer: i tend to think that since it runs on the proposal slave which has a private key we would like to keep private, it should remain in project-config15:51
*** sabeen2 has joined #openstack-infra15:52
fungiyolanda: and a get of that returns a 200 ok if the account is active, or a 204 no content if inactive15:52
jeblairAJaeger, fungi, clarkb, docaedo, nibalizer: it is at least only a periodic job, which means that someone has to merge a malicios change to app-catalog rather than project-config (so it's harder to exploit than simply uploading a change...)15:52
*** mrmartin has quit IRC15:53
jeblairhas anyone removed the comments from https://review.openstack.org/#/c/264056 ?15:53
*** markus_z has quit IRC15:53
mordredwow. that's an exciting change15:54
fungijeblair: not yet, no15:54
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Remove Python 2.6 compatibility patch.  https://review.openstack.org/26532115:54
*** sabeen has quit IRC15:54
jeblairi will do so15:54
pabelangerjeblair: done15:55
jeblairselect count(*) from change_messages where author_id=14969;15:55
jeblair 1689415:55
jeblairoh, forgot the change id :)15:56
jeblairselect count(*) from change_messages where author_id=14969 and change_id=264056;15:56
jeblair1019815:56
*** amuller has joined #openstack-infra15:56
jeblair6k otherwise seems like a lot though, we should look into that after this.15:56
jeblairfungi, mordred: ^ that look ready to turn into a delete?15:57
electrofelixthere is a change queued up in JJB that affects the XML generated by adding a new tag to the output. it does not change the default behaviour of the jobs15:57
jeblairelectrofelix: does it change all of our jobs?15:57
electrofelixjust that it exposes the option for users to disable the default excludes when archiving15:57
electrofelixjeblair: yes15:57
electrofelixso I'm hesitant to approve until it's clear that it's ok15:58
jeblairelectrofelix: we just need to run jjb manually on all our hosts then; if you want to approve it now, we can do that today15:58
*** nwkarste_ has joined #openstack-infra15:58
electrofelixessentially it exposes the option added by https://issues.jenkins-ci.org/browse/JENKINS-2008615:58
electrofelixnew protection added?15:59
jeblairhuh, how about that15:59
*** esker has joined #openstack-infra16:00
fungijeblair: lgtm, that looks like the only change number i see it flooding earlier16:00
jeblairelectrofelix: no, that's always been the problem -- updating all our jobs takes hours, so puppet times out every time it tries and they never get updated again; we need to run jjb manually in screen to get them all to update, then puppet triggered runs will work again.16:00
*** bnemec has joined #openstack-infra16:00
jeblairfungi: complete16:00
electrofelixah16:00
*** esker has quit IRC16:00
jeblairyolanda: ^ see sql above16:00
*** piet has quit IRC16:00
mordredjeblair: yes16:00
*** fawadkhaliq has quit IRC16:01
mordredjeblair: delete from change_messages where author_id=14969 and change_id=264056;  yah16:01
yolandajeblair, looks fine to me, to purge the messages from queue16:01
rhe00I'm here now. Looking at the X-IO CI - again.16:01
*** rhe00 is now known as rhedlind16:02
*** jsavak has quit IRC16:02
yolandajeblair, deleting like that doesn't have any effect on the change? new comments can be queued again? i am interested in that for downstream as well16:02
*** nwkarsten has quit IRC16:03
fungirhedlind: did we get that zuul bug you found merged upstream? you were running with it applied on your system too, right?16:03
*** tonytan4ever has joined #openstack-infra16:03
rhedlindfungi: it was merged upstream. looking at my system now.16:04
fungiyolanda: we've done it in the past and it hasn't seemed to present an issue, though that was also before 2.11 i think so it's anybody's guess16:04
*** jsavak has joined #openstack-infra16:05
rhedlindArgh! the server in question is running zuul 2.1.1.dev109. The fix is in dev110.16:05
fungier, actually we have done it since gerrit 2.11 so should be fine16:05
rhedlindI upgraded one of the servers, but not this one.16:05
jeblairfungi: it the comments don't appear to be gone from that change, even after a flush-caches --all16:06
crinkleTheJulia: what's up?16:06
*** regXboi has joined #openstack-infra16:06
*** ujuc has joined #openstack-infra16:07
*** piet has joined #openstack-infra16:08
*** salv-orl_ has joined #openstack-infra16:08
jeblairfungi: i wonder if we need a restart, or worse, a reindex?16:08
*** dizquierdo has quit IRC16:09
*** zigo has quit IRC16:09
*** zigo has joined #openstack-infra16:10
*** nwkarste_ has quit IRC16:10
*** salv-orlando has quit IRC16:11
jeblairanteaya: can we just merge the series at https://review.openstack.org/265031 since it's already tested and +2d?16:11
*** abregman has quit IRC16:12
*** daemontool has quit IRC16:15
*** annegentle has quit IRC16:15
bknudson_gerrit seems slow today16:16
*** ujuc has quit IRC16:17
anteayajeblair: works for me16:18
jeblairbknudson_: well, there was a runaway third-party ci earlier; not sure if that would have an effect (certainly on the one change with 10k comments, unsure whether it would have follow-on effects)16:19
jeblairbknudson_: clarkb has been monitoring java gc time; we might need to check in on that and see if it's increased16:19
anteayajeblair: I had thought that the patch that merged last night made 265031 redundant16:19
anteayajeblair: I guess I mis-understood, I have changed my vote now, if you want to self-approve16:19
jeblairanteaya: it does.  i just don't want to waste any more time restacking it and waiting for reviews.16:20
anteayano problem16:20
*** _tzn has joined #openstack-infra16:20
TheJuliacrinkle: so on bare nodes, the openstack_citest user seems to be able to create users, and create databases, however grants seem to fail when they are attempted.  Does this ring any bells?16:21
fungijeblair: yeah, i can't recall whether we restarted after the last one16:21
*** mriedem is now known as mriedem_meeting16:21
openstackgerritMerged openstack-infra/grafyaml: Add support for nullPointMode  https://review.openstack.org/26498516:21
openstackgerritMerged openstack-infra/grafyaml: Add tooltip customization support  https://review.openstack.org/26498816:22
fungiTheJulia: it's possible we don't grant granting to openstack_citest. have you looked at the database setup script to see what it does?16:22
crinkleTheJulia: is this new in the last few days?16:22
pabelangerjeblair: template variables for grafyaml: https://review.openstack.org/#/c/250893/ when you have a moment16:23
*** aopincar has joined #openstack-infra16:23
crinkleTheJulia: openstack_citest should have privileges 'ALL' on 'openstack_baremetal_citest.*'16:23
docaedojeblair: regarding 264978 and the script that runs from app-catalog repo16:23
TheJuliafungi: on bare machines, its explicit in prepare scripts16:23
TheJuliacrinkle: new in the last few days16:23
electrofelixjeblair: I'll check that nothing obvious is broken with the change and jenkins 1.565, as an older version is being used by openstack, then I'll approve.16:23
*** tzn has quit IRC16:23
TheJuliapulling grants, it looks like openstack_citest has openstack_baremetal_citest.* and *.* defined16:24
TheJulias/pulling/show/16:24
docaedojeblair: I mainly took this route so I could fix/adjust/iterate quicker. But I understand the concern that we could merge something malicious or scary16:24
*** annegentle has joined #openstack-infra16:24
AJaegerjeblair: thanks for commenting!16:25
anteayarhedlind fungi jeblair yolanda: thanks for your help here, I just returned and have company so am sort of here ish, I feel an email to -announce is in order but am willing to wait until we have further details16:25
*** dims has quit IRC16:25
ShrewsTheJulia: sounds like you need "with grant option"16:26
fungiTheJulia: on a related note, in designing our new non-bare jobs i've been assuming openstack_baremetal_citest.* is redundant now16:26
*** dims has joined #openstack-infra16:26
TheJuliaShrews: this is the ansible module failing :\16:26
crinkleTheJulia: the mysql docs say ALL [PRIVILEGES]Grant all privileges at specified access level except GRANT OPTION16:26
clarkbfungi it is16:26
crinkleso maybe we need to have 'ALL', 'GRANT OPTION' instead of just 'ALL'16:26
*** tonytan4ever has quit IRC16:27
* TheJulia looks at the ansible module code16:27
fungiif we need grant option, i'd appreciate a patch to keep http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n580 in sync with what gets added16:27
*** links has quit IRC16:27
clarkbfungi it was a thing when we didnt allow db creation and nova and ironic used separate dbs16:29
*** devkulkarni has joined #openstack-infra16:29
*** salv-orl_ has quit IRC16:29
AJaegerjeblair: https://review.openstack.org/#/c/265009/6/tools/grafyaml-validate.sh has some comments by fungi - do you want to redo or should we merge as is?16:30
TheJuliafungi: It looks like the mysql puppet module is in the drivers seat w/r/t default settings for the user16:30
*** salv-orlando has joined #openstack-infra16:30
*** jsavak has quit IRC16:31
*** bpokorny has joined #openstack-infra16:31
*** jsavak has joined #openstack-infra16:32
fungiAJaeger: should be safe to approve and clean up later. it's not wrong, just could be simplified16:32
*** aopincar has quit IRC16:33
fungiTheJulia: oh, that might explain the recent change you're seeing in that casde16:33
fungicase16:33
crinkleI don't immediately see why this would be different between versions of the puppet module16:33
*** jistr has quit IRC16:33
clarkbcrinkle: maybe old added grant to all?16:34
TheJuliabasically I get errors saying openstack_citest@localhost is unable to open the database I want to grant access to.16:34
*** EricGonczer_ has joined #openstack-infra16:34
jeblairAJaeger: as is16:34
AJaegerfungi, jeblair: +2A16:35
jeblairclarkb: i think we need to try restarting gerrit to see if those extra comments go away; do you want to intervene for memory/gc issues?16:35
clarkbjeblair let me see what melody says really quick16:35
fungiyeah, now might be a good time to make sure any other tuning patches we want are in place16:36
AJaegerjeblair: I'll wait with approving https://review.openstack.org/#/c/265010 until the other changes are in...16:36
*** samuelBartel has quit IRC16:36
*** MarkAtwood has joined #openstack-infra16:36
jeblairAJaeger: i think the dependencies have merged16:36
clarkbGC is still at roughly 2% average16:36
TheJuliabingo, openstack_citest does not have grant privs16:37
AJaegerjeblair: they have - but the testing from master patch not. shouldn't we wait for that to be in?16:37
* TheJulia had to hunt through logs16:37
AJaegerWait, that was tested in this case...16:37
clarkbthe change zaro linked which schedules daily git gc of repos in gerrit is redundant with our git repack cron I think. and I do not expect that to affect java gc16:37
jeblairAJaeger: yep, depends-on :)16:37
ttxAs I suspected, people misunderstood completely the stackforge retirement as meaning that their projects were part of the big tent without requiring a TC approval16:37
clarkbunfortunately garbage collecting is highly overloaded here16:37
AJaegerbut 265010 needs another +2...16:38
ttxhttp://lists.openstack.org/pipermail/openstack-dev/2016-January/083589.html16:38
clarkbthat said we can probably merge that as redundant should be safe and if it does help then woot16:38
ttxjeblair: ^ sometimes I hate to be right16:38
*** nelsnelson has joined #openstack-infra16:38
*** EricGonc_ has joined #openstack-infra16:38
clarkbjeblair fungi https://review.openstack.org/#/c/262075/16:39
ttxAlso I'm not sure we are seeing the expected decline in things-in-gerrit-but-not-in-big-tent16:39
*** tonytan4ever has joined #openstack-infra16:40
jeblairttx: sure, that's because you don't want them in the big tent.16:40
ttxjeblair: or because they don't care enough to join16:40
*** gyee has joined #openstack-infra16:40
*** EricGonczer_ has quit IRC16:41
jeblairttx: i hope you recall that those are two orthogonal issues -- stackforge retirement alone was not expected to change anything related to the number of official openstack projects, only lowering the standards for joining openstack officially was meant to do that.  and i think we saw quite a few projects join officially since those standards were lowered.16:41
openstackgerritMerged openstack-infra/project-config: Pass ZUUL_* env vars through tox  https://review.openstack.org/26503116:42
fungittx: people with a loose grasp of english misunderstood anyway16:42
jeblairttx: anyway, people get confused.  there were lots of projects that didn't understand our governance structure before, and there are still lots of projects that don't.16:42
fungior who didn't read closely enough16:42
AJaegersmcginnis: are you around? Can we discusss on #openstack-doc?16:43
ttxjeblair: sure, I recall... it's just that the confusion I expected just appeared16:43
*** sfinucan has quit IRC16:43
jeblairttx: stick around and review the project-config repo and you'll see some more.16:43
ttxanyway, that's precisely why I derailed that thread, to have the opportunity for education16:43
ttx:)16:44
jeblairttx: you might even be able to educate people early there...16:44
zaromorning16:44
jeblairttx: maybe you could come up with a standard education message for new projects that could be posted as a comment on project-config new repo creation patches.16:44
ttxonce I'm done fixing things inside the tent (like giving all of them an accurate release model) I'll certainly move out16:44
openstackgerritMerged openstack-infra/project-config: Validate grafyaml based on master  https://review.openstack.org/26500916:45
jeblairttx: we could also more aggressively retire the _word_ 'stackforge'16:45
fungi(for example, fuel creating lots of unofficial repos for fuel plugins)16:45
jeblairttx: at the request of the tc, we still use 'stackforge' to describe unofficial projects16:45
jeblairttx: this costs us the opportunity to say "unofficial projects" in some cases16:45
ttxjeblair: to call them what... unofficial ?16:45
ttxright16:45
jeblairttx: continuing to use 'stackforge' probably just means that people think the documentation is outdated16:46
jeblairttx: see http://docs.openstack.org/infra/system-config/stackforge.html16:46
jeblairttx: i've had to -1 several patches that remove stackforge from there16:47
ttxjeblair: I think we could (more aggressively retire "stackforge" in benefit of "unofficial")16:47
fungii keep saying unofficial projects for clarity in any public communication, but our documentation still mentions stackforge (at the request of the tc)16:48
*** fawadkhaliq has joined #openstack-infra16:48
ttxnow that everyone understands that projects are created in openstack/16:48
ttxworth a try. It's all about guessing what will be the most confusing :)16:49
zaroclarkb: from reading the git documentation it doesn't seem like git gc and git repack do the same things.  therefore i don't think it's redundant.16:50
samueldmqAJaeger: ping, about https://review.openstack.org/#/c/26514816:50
AJaegersamueldmq: yes?16:50
samueldmqAJaeger: I thought only keeping 'deps = .[bandit]' would be enough16:51
AJaegersamueldmq: My understanding is that this is not enough. Did you try it?16:51
samueldmqAJaeger: what are the dependencies from the base env ('{[testenv]deps}') needed for ?16:51
clarkbzaro ah ok16:52
ttxfungi: lol at tc-approved release16:52
clarkbzaro but why would git gc affect java gc?16:52
AJaegersamueldmq: that's the test-requirements and requirements - to install hacking, flake8, bandit, etc16:52
clarkbzaro it is separate garbage collection16:52
ttxespecially given my call in 8min with the man behind the name16:52
samueldmqAJaeger: no I didn't, I am just trying to understand it, I will try now16:52
AJaegersamueldmq: I'll try right now as well ;)16:52
fungittx: well, the wording in the question was dangerously close to confusing tc-approved release with official projects16:52
samueldmqAJaeger: ah, and is that the default if you don't specify deps  ?16:53
openstackgerritColleen Murphy proposed openstack-infra/system-config: Allow openstack_citest 'grant option' privilege  https://review.openstack.org/26534816:53
samueldmqAJaeger: nice16:53
crinkleTheJulia: figured it out ^16:53
AJaegersamueldmq: exactly, the deps line overrides the default16:53
fungittx: but also, yes, amusing cosmic timing16:53
AJaegersamueldmq: so, I have to get the default added back...16:53
TheJuliacrinkle: awesome!16:53
AJaegersamueldmq: with my change: pep8 installdeps: .[bandit], -r/home/aj/Software/vcs/OpenStack/openstack/keystone/test-requirements.txt, .[ldap,memcache,mongodb]16:54
AJaegersamueldmq: your proposal: pep8 installdeps: .[bandit]16:54
AJaegerso that will be quite different environment16:54
zaroclarkb: it probably doesn't.  i don't have any other tuning changes targeted specifically for java gc atm.16:55
*** ddieterly has joined #openstack-infra16:55
samueldmqAJaeger: I agree, because it will intall bandit, but not the other things potentially needed for running pep8 itself16:55
samueldmqinstall*16:55
zaroclarkb: the only thing that i can think of that might be related is that if not running git gc gerrit may use more memory to load git changes?  just a guess.16:55
jeblairfungi, ttx: https://review.openstack.org/26535216:56
samueldmqAJaeger: you're correct, without it:16:58
samueldmqWARNING:test command found but not installed in testenv16:58
samueldmq  cmd: /usr/local/bin/flake816:58
samueldmq  env: /home/samuel/workspace/keystone/.tox/pep816:58
zaroclarkb: the perf tuning cheat sheet says that if you don't run git gc regularly then it will have an affect on "smooth source code browsing experience".16:58
samueldmqAJaeger: thanks16:58
clarkbzaro it shouldnt since gc will only kill things that arent valid refs16:59
AJaegersamueldmq: thanks for asking!16:59
jeblairttx: with that, we can update http://docs.openstack.org/infra/system-config/stackforge.html and then point all new projects requests at it to make sure they understand the implications.16:59
clarkbzaro so anything associated with a change should not get git gc'd16:59
jeblairpleia2: see fungi and my conversation with ttx above and also https://review.openstack.org/26535216:59
anteayarhedlind: I have to send something to the -announce mailing list regarding today's incident, what is your current system status?17:00
*** derekh has quit IRC17:00
openstackgerritMerged openstack-infra/system-config: Schedule Gerrit's git garbage collection  https://review.openstack.org/26207517:01
jeblairwait what? ^17:01
jeblairweren't we *just* talking about how that's not necessary?17:02
zaroclarkb: not that i can think of.17:02
*** jsavak has quit IRC17:03
clarkbjeblair zaro points out that it doesnt overlap with repack and the tuning guide says you should do it for "smooth code review" (though I didnt approve)17:03
clarkbstill trying to sort through what is what17:03
jeblairclarkb: who says it doesn't overlap with repack?17:03
*** ujuc has joined #openstack-infra17:03
clarkbjeblair zaro just a few minutes ago17:03
*** erlon has joined #openstack-infra17:03
jeblairzaro: source?17:03
rhedlindanteaya: I have properly upgraded zuul to the latest version, upgraded jenkins to the latest and working on upgrading nodepool. it should be back up in a few minutes17:03
anteayaokay thank you17:04
openstackgerritJames E. Blair proposed openstack-infra/system-config: Revert "Schedule Gerrit's git garbage collection"  https://review.openstack.org/26535617:04
*** hdd has joined #openstack-infra17:04
jeblairI'm self-approving that change now ^17:04
zarojeblair, clarkb : gc.interval http://tinyurl.com/q2aw6ba17:04
*** sputnik13 has joined #openstack-infra17:04
jeblairzaro: where in that does it say that the effect of gc is not included in repack?17:05
anteayarequest for comment, does anyone have any objection to turning rhedlind's ci system xio-ise-iscsi-ci back on once he is finished his system upgrades?17:05
anteayaif you have any hesitation, do state it please17:05
zarojeblair: ohh, i inferred it from git documentation https://git-scm.com/docs/git-repack17:05
*** jsavak has joined #openstack-infra17:06
*** sputnik13 has quit IRC17:06
*** devkulkarni has quit IRC17:06
*** mriedem_meeting is now known as mriedem17:07
*** devkulkarni has joined #openstack-infra17:07
AJaegeranteaya: let's ask rhedlind for a 24x7 emergency phone number ;)17:08
zaroclarkb: one concern i have is that we are setting ssh.threads too high relative to the cores on review.o.o17:08
anteayaAJaeger: I wouldn't call it anyway17:08
anteaya:)17:08
zaroclarkb: and memory17:08
rhedlindAJaeger: :)17:09
zaroclarkb: documentation recommends 2*<#Cores>17:09
jeblairzaro, clarkb, fungi: can we get a +1 from me on any tuning changes please?17:09
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account  https://review.openstack.org/26535917:09
clarkbzaro we have it set to 4*?17:11
openstackgerritMerged openstack-dev/pbr: Add dependencies for building wheels  https://review.openstack.org/26401017:11
zaroclarkb: i thought review.dev was only 8 core?17:11
clarkbthat sounds right17:11
clarkbbut I would have to double check17:12
zaroclarkb: and i believe sshd.threads is set to 100?17:12
openstackgerritMerged openstack-dev/pbr: Make setup.py --help-commands work without testrepository  https://review.openstack.org/26392817:12
*** otter768 has joined #openstack-infra17:13
jeblairwhere are all the comments from when i tuned gerrit before?17:13
*** ujuc has quit IRC17:13
jeblairah, found them.  system-config/modules/openstack_project/manifests/review.pp17:14
anteayayolanda: whitespace error in 26535917:15
jeblairzaro: so that value was set because we were seeing things queue up in show-queue17:15
*** AJaeger has quit IRC17:15
jeblairzaro: if we decrease it, we will be de-tuning something we have already tuned17:15
yolandaanteaya, thanks, i'll correct17:15
openstackgerritMerged openstack-infra/project-config: Grafana: Update OVH graph with axis labels  https://review.openstack.org/26501017:15
jeblairzaro: we may want to increase it if we see things queue up again17:15
jeblairzaro: but i don't think we've seen that.17:15
electrofelixjeblair: got distracted, confirmed that a jenkins instance of 1.565.2 doesn't have an issue with the extra tag added by https://review.openstack.org/259666, so should be ok to update the jobs manually after approval.17:16
*** jlanoux has quit IRC17:16
zarojeblair: i'm assuming you are referring to sshd.threads?17:16
jeblairzaro: yes17:16
jeblairzaro: so i wonder why you want to decrease it?17:16
*** esker has joined #openstack-infra17:16
zarojeblair: i'm not suggesting to decrease it.  i just have a concern that it's too high.17:17
zarofor the resources available on review.o.o.17:17
pleia2jeblair: thanks, it was indeed a confusing situation17:17
*** otter768 has quit IRC17:17
jeblairzaro: what resources are being impacted by it being too high?17:17
clarkband fwiw it is 8vcpu17:17
zarojeblair: heap issues17:18
anteayayolanda: I also added a comment about location17:18
yolandathanks17:18
anteayayolanda: I think the information is valuable and should be documented17:18
jeblairzaro: what effect does the number of sshd threads have on the heap?17:18
anteayayolanda: I would just like to see if we have a better, meant for infra root use kind of document that would work17:19
yolandaanteaya, can you review https://review.openstack.org/256389 and https://review.openstack.org/260068 as well? also documentation bits17:19
zarojeblair: ok, that's a good question that i haven't done much research into yet.  that's why i haven't proposed a change17:19
* anteaya reviews17:20
jeblairzaro: okay.  are there any proposed tuning changes i should review?17:20
zarojeblair: the collab net doc says "if you run into heap space issues, scale down this number" which is why i thought this might be an issue.17:20
yolandaanteaya, it can fit as well on gerrit page17:20
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account  https://review.openstack.org/26535917:21
zarojeblair: all gerrit tuning changes have topic 'gerrit-tuning'17:21
fungiwe can revert 262075, i was taking clarkb's comment at 16:39 to mean that we should go ahead and merge it before restarting gerrit17:21
jeblairzaro: right, but that will cause other problems like people being unable to push, fetch, query or just about anything else; i think increasing the heap size to accomodate the need of the sshd threads is better.17:22
clarkbfungi: oh sorry, I was just ointing out what had been proposed for tuning17:22
fungiahh, whoops17:22
zarojeblair: yes, i agree.  i think we need to give review.o.o more resources.17:22
fungialso rereading that now, i worry that it was conflating git gc with hava gc17:22
*** mriedem is now known as mriedem_lunch17:23
fungijava gc17:23
*** unicell has quit IRC17:23
jeblairfungi: yes, see my revert at https://review.openstack.org/26535617:23
*** annegentle has quit IRC17:24
*** nelsnels_ has joined #openstack-infra17:24
*** esker has quit IRC17:25
anteayayolanda: great, I think the gerrit page would be a better location for how to disable a third party ci account, thank you17:25
*** e0ne has joined #openstack-infra17:25
*** fawadkhaliq has quit IRC17:26
*** fawadkhaliq has joined #openstack-infra17:26
* anteaya doesn't know the difference between git gc/java gc/gerrit gc but acknowledges that they appear to be different things17:26
*** docaedo has quit IRC17:27
*** yolanda has quit IRC17:27
*** annegent_ has joined #openstack-infra17:27
*** nelsnelson has quit IRC17:27
*** pfallenop has quit IRC17:27
*** morgabra has quit IRC17:27
*** superdan has quit IRC17:27
*** unicell has joined #openstack-infra17:27
*** dansmith has joined #openstack-infra17:27
*** pfallenop has joined #openstack-infra17:27
*** docaedo25 has joined #openstack-infra17:27
fungijava garbage collection is cleaning up unreferenced memory allocations inside the jvm, git garbage collection is removing unreferenced git objects in a repository17:27
*** morgabra_ has joined #openstack-infra17:27
*** yolanda has joined #openstack-infra17:28
greghaynesclarkb: have you seen the nodepool ci fail http://logs.openstack.org/75/195175/37/check/gate-nodepool-python27/a99adbf/console.html ?17:29
*** dtantsur is now known as dtantsur|afk17:29
*** mudassirlatif has joined #openstack-infra17:29
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account  https://review.openstack.org/26535917:29
greghaynesclarkb: did the puppet mysql stuff upgrade mysql in images?17:29
clarkbgreghaynes: I haven't17:29
clarkbgreghaynes: it wasn't supposed to, but TheJulia found that grants aren't working17:29
greghaynesclarkb: that sounds like what this is17:29
clarkbgreghaynes: I think crinkle is looking at it17:30
jeblairfungi, clarkb: we've gotten severely distracted, but i believe we still need to either restart or reindex gerrit in order to fix the issue with so many comments on https://review.openstack.org/#/c/26405617:30
greghaynesclarkb: ah, ok17:30
clarkbjeblair: yes, a restart would make sense. I think the only thing worth delaying a restart for is the web session cache maxAge value if we want to update that really quickly17:30
fungijeblair: have you tried reloading the page? i don't see the comments on it17:31
jeblairfungi: yes, i've shift-reloaded.  should i quit browser reload?17:31
fungiyou may still have them cached in javascript or something17:31
*** dansmith is now known as superdan17:31
fungii had never browsed that page, so pulling it up now i only see maybe a couple dozen comments from various accounts and not the account in question17:31
*** pblaho has quit IRC17:32
fungiand that's with toggle ci expanded, for reference17:32
jeblairfungi: that has corrected the issue.  that's disturbing.17:32
jeblairso nevermind about the gerrit restart.17:32
fungiback to the sidetrack!17:32
*** dizquierdo has joined #openstack-infra17:33
anteayafungi: thank you17:33
jeblairzaro: i have looked at the other outstanding changes.  a max file size seems fine.  i don't think we should increase the diff timeout without serious considuration.17:33
anteayajeblair: comments are gone for me17:33
*** morgabra_ is now known as morgabra17:34
*** morgabra has quit IRC17:34
*** morgabra has joined #openstack-infra17:34
crinkleclarkb: greghaynes i am looking and have https://review.openstack.org/#/c/265348/1 but it is not going to work and I am getting on a plane soon17:34
*** lezbar has quit IRC17:34
anteayacrinkle: safe flight17:34
electrofelixjeblair: are you ok to update the JJB jobs manually?17:35
greghaynescrinkle: ah17:35
*** salv-orlando has quit IRC17:35
anteayacrinkle: how are your infra-cloud storyboard clean-up efforts going? https://storyboard.openstack.org/#!/story/200017517:35
clarkbthats a fun behavior, throw out the list if all is used17:35
jeblairinfra-root: ^ is someone available to run manual jjb updates after electrofelix merges an update-all-jobs jjb change?17:35
zarojeblair: sounds good.17:35
*** salv-orlando has joined #openstack-infra17:36
crinkleclarkb: yeah >.<17:36
*** ifarkas has quit IRC17:36
crinkleanteaya: i have poked some people, can't really focus on it today17:36
fungijeblair: i can take care of that, sure17:37
jeblairfungi: thanks; electrofelix ^17:37
clarkbcrinkle: will you be able to update with a list of all the options?17:37
clarkbcrinkle: or should I start working on pushing that patchset17:37
electrofelixfungi: thanks, I'll pull the trigger on it now so17:37
fungii'm getting logged in and screen sessions readied on all our masters now17:37
crinkleclarkb: i will have scattered time today so if you want to work on it now that would be awesome17:38
*** jsavak has quit IRC17:38
electrofelixjeblair: thanks17:38
clarkbcrinkle: kk17:38
*** unicell has quit IRC17:38
*** jsavak has joined #openstack-infra17:38
*** mrmartin has joined #openstack-infra17:38
greghaynesclarkb: crinkle this is going to require a new image upload so it isnt really a rush - will be a day minimum before it is fixed17:39
*** esker has joined #openstack-infra17:39
clarkbya, but better to fix now while I am thinking of it17:39
*** MarkAtwood has quit IRC17:40
*** dprince has quit IRC17:40
fungielectrofelix: for reference, all our masters currently report jenkins-job-builder 1.4.1.dev21 is installed17:40
clarkbactually I think we may be able to use two different mysql_grant calls17:40
*** dprince has joined #openstack-infra17:41
*** dprince has quit IRC17:41
clarkbone to grant all and one to grant the grant option.17:41
clarkbcrinkle: ^ does that make more sense to you or do you think one call with complete grant list is best?17:41
*** dprince has joined #openstack-infra17:41
*** jerryz has joined #openstack-infra17:41
*** docaedo25 is now known as docaedo17:41
fungielectrofelix: once i see that rise to .dev22 i'll start manual config updates on all of them17:41
docaedojeblair: did you have more thoughts/opinions about the script for the app-catalog periodic job living in app-catalog?17:42
*** odlD2 has quit IRC17:42
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for default-excludes in archives  https://review.openstack.org/25966617:43
jeblairdocaedo: i'd like other infra-root folks to weigh in on it17:43
electrofelixfungi: thanks, change has merged, so presume it shouldn't take too long17:43
docaedojeblair: roger that :) I'm around for discussion if other infra-root peeps want to discuss17:43
nibalizerI am mostly traveling today17:43
clarkbjeblair: docaedo I remember saying it should be in project-config a little while back17:43
*** esker has quit IRC17:44
*** Swami has joined #openstack-infra17:44
*** jordanP has quit IRC17:45
docaedoclarkb: I can move it back there17:46
fungijeblair: clarkb: docaedo: yes, if it's a script running on a sensitive worker with access to a gerrit push account, we need that script reviewed in project-config17:46
docaedofungi clarkb jeblair - understood, I'm on it (though this foils my botnet plans :/ )17:47
crinkleclarkb: that won't work because the grant uses the name/host/db in the title so you would have the duplicate definition problem17:47
*** sabeen2 has quit IRC17:47
clarkbugh17:47
anteayacrinkle: understood, good work there17:47
*** piet has quit IRC17:48
jeblaircrinkle: what's "name" in that triplet?17:48
fungiuser?17:48
crinklejeblair: username, excuse me17:48
jeblaircrinkle: np, thx.17:48
*** yaume has quit IRC17:49
jeblairfungi: are index.old and cache.old directories in ~gerrit2/review_site created by gerrit or by us?17:49
fungijeblair: by us i think. that is probably cruft from our 2.10 upgrade attempt in may judging from the last modified dates on them17:50
openstackgerritChristopher Aedo proposed openstack-infra/project-config: Update app-catalog link check job  https://review.openstack.org/26497817:50
jeblairthat's about 1.7G we could get back17:51
fungijeblair: yep, sudo grep 'cache.*old' ~gerrit2/.bash_history17:52
fungiconfirms it17:52
jeblairnice17:52
jeblairfungi: how about i delete those?17:52
rhedlindasselin: ping17:52
fungijeblair: please do. they're way older than we could ever hope to make use of17:52
*** sridhar_ram has joined #openstack-infra17:53
jeblairdone17:53
anteayarhedlind: I'll post to -announce about the incident today and that your system is currently disabled, acknowledging you are aware and are addressing the issue at your end17:53
asselinrhedlind, just ask17:53
fungijeblair: i'm also doing an apt-get autoremove which is freeing ~5gb17:53
anteayarhedlind: we can follow up when you are ready17:53
rhedlindanteaya: sounds good17:54
jeblairfungi: ah nice17:54
fungiand then i'll do a clean and see how much that gets us back17:54
*** odlD2 has joined #openstack-infra17:54
openstackgerritMerged openstack-infra/project-config: Revert "Pass ZUUL_* env vars through tox"  https://review.openstack.org/26503417:54
rhedlindasselin: I upgraded to the latest nodepool and now nodepool cannot authenticate with the provider. from what I understand the provider user and password still goes in the nodepool.yaml file, right?17:54
*** reed has quit IRC17:55
*** reed has joined #openstack-infra17:55
fungiunfortunately most of what autoremove is doing is cleaning out old kernel packages we're no longer using, and each of those fires a rescan of the initrds for the remaining kernels... it's slow-ish17:55
jeblairi think the bulk of / is mysql backups; we probably don't need 31 days -- iff bup is working as expected.  :)17:55
*** EricGonc_ has quit IRC17:55
fungiyeah, on another server (which one?) we backed down to 10 days retention i think17:55
fungiask.o.o probably17:55
asselinrhedlind, I haven't checked the latest nodepool....afaik that functionaliy hasn't been removed, but you'd have to look at the commit history17:56
jeblairrhedlind: use clouds.yaml17:56
asselinrhedlind, the newer way is to use a secure.conf & clouds.yaml file. the puppet scripts have been updated to do that.17:56
jeblairright, that :)17:57
fungielectrofelix: 1.4.1.dev23 (not .dev22 because of the merge commit) is starting to show up on some of our masters now17:57
fungielectrofelix: yep, propagated to all of them now, so i'll go ahead with the config reprocessing on them17:58
*** doug-fis_ has joined #openstack-infra17:58
*** doug-fis_ has quit IRC17:58
rhedlindah, so I did update to use the secure.conf. I will look into clouds.yaml17:59
*** doug-fis_ has joined #openstack-infra17:59
rhedlindasselin, jeblair: thanks!17:59
*** AJaeger has joined #openstack-infra18:00
*** piet has joined #openstack-infra18:00
*** e0ne has quit IRC18:01
*** flepied has quit IRC18:01
*** yamahata_ has joined #openstack-infra18:01
AJaegersdague: We analyzed the translation import, it's a bug in babel 2.2.0 that is fixed now (from fix to merge in 24 hours) but not yet released. Since this is part of the scripting we do after downloading, it's fine to merge translations. Please go ahead. At one time you might get the translators back ;)18:01
*** doug-fish has quit IRC18:01
fungiinfra-root: okay, manual jjb update is running on every one of our masters in screen sessions under my account (if you want to attach, `sudo su - fungi` and then `script /dev/null` to spawn a fresh tty before you `screen -x`)18:02
*** dims_ has joined #openstack-infra18:02
*** bpokorny has quit IRC18:03
fungijeblair: we're down to 24gb used on review.o.o's rootfs now18:03
AJaegersdague: https://review.openstack.org/#/c/252197/ is current import for nova if you could approve, please18:03
*** doug-fis_ has quit IRC18:03
jeblairfungi: oh good18:03
fungijeblair: 17gb of which is /var/backups/mysql_backups18:03
*** tristanC_ has joined #openstack-infra18:04
*** wfoster has quit IRC18:04
*** annegentle has joined #openstack-infra18:04
*** tristanC has quit IRC18:04
*** annegent_ has quit IRC18:04
*** yamahata has quit IRC18:04
*** dims has quit IRC18:05
*** wfoster has joined #openstack-infra18:05
*** amuller is now known as amuller_afk18:05
fungipointed out before, but jenkins, jenkins01 and jenkins02 are considerably slower than the other production masters... chances are they're on an older nova flavor and could stand a rebuild at some point in the future18:05
*** yamahata_ has quit IRC18:06
*** rhedlind has quit IRC18:06
*** rhe00 has joined #openstack-infra18:06
fungiit's less impactful to jenkins.o.o since its load is negligible but 01 and 02 crawl18:06
*** shardy has quit IRC18:07
*** unicell has joined #openstack-infra18:07
jeblairclarkb, fungi: http://grafana.openstack.org/dashboard/db/nodepool18:07
jeblairclarkb, fungi: jenkins06 is anamolous18:07
*** dprince has quit IRC18:08
fungiindeed. that usually means it's having problems18:08
*** dprince has joined #openstack-infra18:08
fungipiles of ready and deleting nodes indicate the api was unresponsive i think18:08
fungiit's probably got a runaway thread leak going again18:09
openstackgerritPaul Belanger proposed openstack-infra/grafyaml: Add support for template query  https://review.openstack.org/25089318:09
fungioh yeah, very slow to respond on the webui, possibly because i'm also hammering it with jjb right now18:09
clarkbmelody confirms >7k threads18:09
fungii can put it in prepare for shutdown for now so it doesn't get worse and then see if that gets us through the job updates18:10
jeblairfungi: ++18:10
fungidone18:11
*** doug-fish has joined #openstack-infra18:12
anteayaclarkb: in terms of things you are juggling are you full or do you have a hand free for logstash 2.0 on worker 20 for a moment?18:12
anteayaclarkb: in terms of exchanging thoughts18:13
clarkbanteaya: sure, mostly just digigng up mysql docs to generate a list of grant permissions18:13
anteayaunderstood18:13
clarkbtrying to determine if hte lists are different between versions18:13
clarkbmordred: ^ you may just know of the top of your head18:13
anteayavia cacit I see worker20's memory usage has increased in the past 3 days, it is now up to 2G18:13
mordredclarkb: aroo?18:14
greghaynesclarkb: generate a list of grant perms?18:14
clarkbgreghaynes: ya18:14
*** ujuc has joined #openstack-infra18:14
greghaynesclarkb: oh, see what the grant perms are for a given user18:14
mordredclarkb: like, somethin gmore than "show grants" ?18:14
mordreddo you want the grants a user has, or a list of possible things that can be granted?18:14
clarkbyes more than that, I basically need to see if what ALL aliases to has changed over releases18:15
clarkbmordred: the possible things that ALL is aliased to18:15
clarkbbecause  Ineed to do ALL + GRANT OPTION in puppet that doesn't break and want to double check that if we run against different versions of mysql the list isn't different18:15
*** dizquierdo has quit IRC18:15
clarkbas an alternative we could just use an exec18:16
mordredclarkb: use mysql ; show create table user \G18:16
clarkbwhich is maybe the least bad way to do this18:16
mordredclarkb: then look at columns that end in _priv18:16
*** doug-fish has quit IRC18:16
clarkbanteaya: looks like a couple worker processes have soakced up memory. python is bad at releasing memory so that doesn't surprise me18:17
mordredclarkb: grant is just sugar around entries in that table;18:17
anteayaclarkb: okay18:17
*** dimtruck is now known as zz_dimtruck18:18
*** ujuc has quit IRC18:18
clarkbmordred: greghaynes ya I think I am just going to exec mysql and do a proper grant for the case where we give all perms out18:19
* anteaya goes shopping for system-config core reviews on Logstash 2.0 compat ES output rule 247714 so that clarkb can upgrade the other 19 logstash workers to logstash 2.018:19
greghaynesclarkb: SGTM18:19
clarkbthat way I can just rely on the alias18:19
mordred++18:19
greghaynes++ for good ol sql18:19
*** thorst has quit IRC18:20
mrmartinhi18:20
AJaegerjeblair: do you want to remove your -2 from the app-catalog change https://review.openstack.org/#/c/264978/3 ?18:20
*** thorst has joined #openstack-infra18:20
*** MarkAtwood has joined #openstack-infra18:21
*** piet has quit IRC18:21
jeblairAJaeger: +518:21
anteayaoh and if someone with irc ops feels like refreshing the topic in -meeting that would be great18:22
crinkleclarkb: ++ to exec18:23
mrmartinfungi: https://ask-staging.openstack.org is a bit broken, and returns an HTTP 500. I started to check what happened.18:23
openstackgerritRamy Asselin proposed openstack-infra/project-config: Publish openstackci docs  https://review.openstack.org/25958718:23
fungianteaya: done18:24
fungimrmartin: i can take a look too18:24
*** thorst has quit IRC18:25
anteayafungi: thank you18:25
mrmartinfungi: yeap, I already found 7-8 things related to changes in askbot-devel repo.18:25
anteayafungi jeblair openstackci docs published to http://docs.openstack.org/infra/openstackci/ any objection?18:25
* anteaya prepares to +A asselin's patch18:25
asselinvery exciting!18:25
fungimrmartin: that's... exciting18:25
AJaegerjeblair: 5? You're generous today ;)18:26
fungiAJaeger: the difference between -2 and +318:26
*** sputnik13 has joined #openstack-infra18:26
anteayafungi jeblair patch I am looking at is here: https://review.openstack.org/#/c/259587/218:27
mrmartinfungi: I have a patch for most of the issues, but the setup we are using break at db sync and migrate due some change in the language handling. http://paste.openstack.org/show/483343/18:27
fungimrmartin: yeah, "IOError: sys.stdout access restricted by mod_wsgi" looks pretty bad18:27
mrmartinit is not the error, just a result of an error situation :)18:28
AJaegerfungi: Indeed...18:28
fungiand "Target WSGI script '/srv/askbot-site/config/django.wsgi' cannot be loaded as Python module."18:28
AJaegerarmax, dougwig : Could you check https://review.openstack.org/#/c/258992/ , please?18:28
fungimrmartin: did you have any ideas for how to manually work around the migration failure there?18:29
mrmartinfungi: I cannot even redeploy the production ask.o.o in a vm, because a non-explicit dependency definition broke due a missing ubuntu package. The solution lives here: https://review.openstack.org/#/c/265231/18:30
mrmartinfungi: yeap, I'll take care of it and deliver some solution18:30
*** AJaeger has quit IRC18:30
* armax looks18:30
fungimrmartin: oh, yep, we've seen pillow 3.x libjpeg integration breaking lots of other stuff too18:31
openstackgerritMerged openstack-infra/system-config: Revert "Schedule Gerrit's git garbage collection"  https://review.openstack.org/26535618:31
*** yamahata_ has joined #openstack-infra18:31
fungimrmartin: thanks for the heads up, and let me know if there's anything else you need18:31
openstackgerritClark Boylan proposed openstack-infra/system-config: Allow openstack_citest 'grant option' privilege  https://review.openstack.org/26534818:31
clarkbcrinkle: greghaynes mordred ^18:31
fungiinfra-puppet-core reviewers, 265231 solves a problem with latest pillow in puppet-askbot18:32
electrofelixfungi: just happened to merge another change as well that has no impact18:32
mordredclarkb: lgtm18:32
mrmartinnothing actually, first of all I want to bring it back alive in a vm, than deliver the patches made there.18:32
fungielectrofelix: ahh, well anyway the updates are progressing well18:32
openstackgerritMerged openstack-infra/storyboard-webclient: Improve usability of the "Edit board" form  https://review.openstack.org/26376118:33
*** kdas_ has joined #openstack-infra18:34
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix on Asymetric key thumbprint calculation:  https://review.openstack.org/26539718:34
*** Guest28831 has quit IRC18:35
*** bharathm has joined #openstack-infra18:35
*** kushal has quit IRC18:35
*** mriedem_lunch is now known as mriedem18:35
*** electrofelix has quit IRC18:38
*** reaperhulk has joined #openstack-infra18:38
*** zz_dimtruck is now known as dimtruck18:42
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add range search functionality  https://review.openstack.org/26486918:42
*** _nadya_ has joined #openstack-infra18:42
anteayaasselin: I didn't see any objections18:44
pabelangerjeblair: so, does this regex look right to you, to match for rax?  (?<=rax-).*18:44
asselinanteaya, me neither18:44
pabelangerjeblair: tested at http://pythex.org/ but grafana doesn't like it.  I wonder if it just lacks support for it18:45
jeblairpabelanger: what graphite version did groupbyregex (or whatever it's called) appear in?18:46
pabelangerjeblair: unsure18:47
anteayafungi: this is an aging patch of yours regarding constraints updates, do you feel that keeping it alive has value? https://review.openstack.org/#/c/202210/18:48
jeblairpabelanger: ah, aliasSub18:48
jeblairthat's the one i was thinking of18:48
jeblairpabelanger: and it's in 0.9.12 which is what we're running18:49
jeblair&target=aliasSub(ip.*TCP*,"^.*TCP(\d+)","\1")18:49
fungianteaya: given the raciness of it, probably needs to be brought back up from its base argument and possible solutions proposed again. i'll abandon that for now18:50
thingeefungi: there was some ping back in tuesday for something to note in this week's dev summary?18:50
pabelangerjeblair: perfect18:50
thingeefungi: I was reading the logs but couldn't make sense of what to note18:50
fungithingee: i remember the when but not the what... looking again18:50
jeblairpabelanger: that does look correct -- i expect that to match "rax-iad" and return "iad" for the match.18:51
anteayafungi: thank you18:51
fungithingee: oh, a reminder that our worker quota may be dropping sharply around the end of january with hpcloud sunsetting on the 31st18:51
openstackgerritMerged openstack-infra/project-config: Adds check-requirements job in networking-hyperv  https://review.openstack.org/25899218:52
fungithingee: but that we're in the process of bringing new providers online to absorb the impact18:52
fungithingee: so just noting that there may be slightly longer delays that usual to get job results at peak activity times18:53
fungier, than usual18:53
openstackgerritMerged openstack-infra/project-config: Publish openstackci docs  https://review.openstack.org/25958718:54
*** amuller_afk is now known as amuller18:54
*** jcoufal has quit IRC18:54
*** sabeen has joined #openstack-infra18:55
*** Sukhdev has joined #openstack-infra18:55
asselinanteaya, do you konw how to invoke the publish job?18:56
thingeefungi: got the previous mention of this on the ML to link for referene?18:57
fungithingee: yeah, let me get you a link18:57
anteayaasselin: create a change to the document, after the job has been propogated out to all the jenkins'18:57
anteayaasselin: sorry merge a change to the document18:58
clarkbanteaya: watching worker20 I don't think this is doing anything terrible, logstash is using a sane amount of memroy and the workers are each using a bit more than they were before18:58
clarkbanteaya: there is sitll quite a bit of memroy available that is being used for caches18:58
anteayaclarkb: ack, thanks for taking a look18:58
*** sabeen3 has joined #openstack-infra18:58
anteayawonderful18:58
asselinanteaya, you mean merge any change to puppet-openstackci?18:59
anteayaclarkb: I haven't been any help getting review attention on 247714, sorry18:59
asselinor it really needs to update the doc?18:59
anteayaasselin: I don't know18:59
clarkbanteaya: thats ok, though I amostly back at this point18:59
clarkbso will probably beat that drum next week18:59
anteayaasselin: I would aim for update the doc, but keep an eye on anychange18:59
anteayaclarkb: okey dokey18:59
*** sabeen has quit IRC19:00
anteayaclarkb: I need to start waving the debug the gate banner again19:00
asselinanteaya, just +A this simple change. see if it works: Phani Raju Yayavaram Narasimha19:00
asselinhttps://review.openstack.org/#/c/265035/19:00
anteayaI've been holding off until logstash is all changed, so was going to deal with it then19:00
anteayabut can try to figure out something new to generate interest19:00
*** Sukhdev has quit IRC19:01
anteayaasselin: the job might not be out to the jenkins masters yet19:01
anteayaasselin: your patch might merge before the job is available19:01
asselinoh...right19:01
anteaya-2 will prevent it from merging19:01
anteayaadding to the commit message should get it out of the gate19:01
asselini WIP -1 it19:02
anteayaasselin: that should prevent it from merging after the gate tests finish running19:02
fungithingee: http://lists.openstack.org/pipermail/openstack-infra/2015-October/003309.html is probably the closest thing we have. not so much an announcement as a discussion back at the summit about it (since that was soon after they announced their timeline)19:02
*** jsavak has quit IRC19:03
jeblairnot sure i'd link to that; it's a bit 'of the moment'.19:03
fungiright19:03
anteayamaybe link to the hp announcement: http://community.hpe.com/t5/Grounded-in-the-Cloud/A-new-model-to-deliver-public-cloud/ba-p/6804409#.VifwJJcnFz319:04
fungithingee: i have it on the agenda to discuss again on tuesday at the meeting, so maybe better for next week's round-up19:04
jeblairand the docs about how to contribute http://docs.openstack.org/infra/system-config/contribute-cloud.html19:04
anteayait says Therefore, we will sunset our HP Helion Public Cloud offering on January 31, 2016. in that post19:04
*** tonytan4ever has quit IRC19:05
clarkbpabelanger: for the fedora 21 slaves, there isn't anything critical relying on that right?19:05
clarkbpabelanger: I mean those tests basically check that we can run stuff on an EOL platform, there isn't much value there19:05
*** ybathia has joined #openstack-infra19:06
*** dprince has quit IRC19:07
thingeefungi: thanks19:07
anteayaSergeyLukjanov: how are you feeling about this patch? https://review.openstack.org/#/c/216275/ do you think we need to keep it open?19:08
*** lucasagomes is now known as lucas-dinner19:08
pabelangerclarkb: honestly, not sure. I mean, I can ask around if needed. Either way, I won't stand in the way if we remove them.  Just adding a note to the review19:08
*** thorst has joined #openstack-infra19:08
*** fawadkhaliq has quit IRC19:08
*** _nadya_ has quit IRC19:09
*** dkranz has quit IRC19:09
*** thorst has quit IRC19:10
*** thorst has joined #openstack-infra19:11
*** bpokorny has joined #openstack-infra19:11
jeblairclarkb, zaro, fungi, mordred: i did some trials, and it looks like gc does do some things repack isn't doing for us.19:12
*** otter768 has joined #openstack-infra19:13
fungii wonder whether git prune would also be useful19:14
*** reaperhulk has left #openstack-infra19:14
jeblairclarkb, zaro, fungi, mordred: nova is 7G on gerrit; git gc reduces it to 3.6G, git prune reduces it to 860M; git prune and git gc together reduce it to 850M; git prune and git gc and packing the refs reduce it to 394M.19:14
jeblairthat last essentially means all the refs are in one file and all the objects are in another file.19:14
fungijeblair: any better with pack -Ad?19:15
*** mrmartin has quit IRC19:15
zaroi believe puppet is doing -ad, correct?19:15
jeblairwhat's that?19:15
*** ddieterly has quit IRC19:15
fungijeblair: should make a following prune more efficient since it moves unreachable objects out of the resulting pack19:16
jeblairfungi: do you mean 'git repack -afd'?19:16
fungiaccording to the manpage -A is like -a unless you add -d19:16
zarohere's the repack command: https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/cron.pp#n1319:17
*** ddieterly has joined #openstack-infra19:17
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila dsvm horizon setup bug 1532300  https://review.openstack.org/26541019:17
openstackbug 1532300 in Manila "manila dsvm jobs fail to setup horizon dashboard due to Permission Denied error on lock file" [Undecided,New] https://launchpad.net/bugs/153230019:17
jeblairfungi: i'll try "repack -Ad" followed by "prune"19:17
fungiand then it goes to extra lengths to get unreachable objects out of the repacking19:17
*** aopincar has joined #openstack-infra19:17
*** otter768 has quit IRC19:18
*** thorst_ has joined #openstack-infra19:18
fungijeblair: there's also a `git prune-packed` referenced in the manpage19:18
yolandahi pleia2, can you review https://review.openstack.org/256389 and https://review.openstack.org/260068 ? you already reviewed the one about bots, these are documentation patches19:18
fungijeblair: which looks intruiging as well19:19
jeblairclarkb, zaro, fungi, mordred: at any rate -- those numbers sound great, but i'm not sure to what extent each is desirable -- it may be that having all of nova in one packfile might make things worse, especially considering all the refs that are only of historic interest.  and having all the refs in one file means we can't edit them (though, after removing the zuul refs, hopefully we won't do that again)19:19
zarofungi: more discussion, jrnieder is google guy, https://code.google.com/p/gerrit/issues/detail?id=375419:19
*** bradjones has quit IRC19:19
jeblairfungi: tried that --  had a minimal effect likely because of the "repack -afd"19:19
fungigot it19:19
zarofungi: sorry took longer than i expected, i had to fix 3 bugs related to that issue.19:19
*** bradjones has joined #openstack-infra19:19
*** bradjones has quit IRC19:19
*** bradjones has joined #openstack-infra19:19
anteayayolanda: I had a suggestion for 256389, did you see it?19:20
*** thorst has quit IRC19:20
jeblairclarkb, zaro, fungi, mordred: my feeling is that there is probably something to be gained by looking into gc and/or prune, but i think since we are in the position of running a public server based on this, we need to evaluate any changes very carefully based on that.19:21
jeblairclarkb, zaro, fungi, mordred: i think we should find a volunteer to really dig deeply into this and thoroughly evaluate the options and ramifications.19:21
yolandano sorry, i just refreshed19:21
zarojeblair: might be good to ask questions on the gerrit ML?19:22
pleia2yolanda: yep, looking now19:22
fungijeblair: agreed, most of those are focused on reducing backend storage, but very well could pose performance problems as a tradeoff19:23
anteayayolanda: okay19:23
clarkbjeblair: that seems like areaosnable appraoch.19:23
fungiand yes, definitely sounds like an interesting research project which doesn't necessarily need a root admin involved19:24
*** _diana_ has joined #openstack-infra19:24
clarkbjeblair: what about for the web sessions maxAge? should we go ahead and just bump that to 7d?19:24
*** trown|outttypeww is now known as trown19:24
fungii'm still dismayed that the session expiration is absolute from creation time and not relative to last use19:24
jeblairclarkb: that's probably the best approach19:24
jeblairfungi: yeah :(19:24
fungiso basically, your session takes a bath once a week whether it needs one or not19:25
fungiall jjb updates completed successfully except 02 (which hit a proxy error and had to be resumed) and 06 (which is still running and as noted needs a service restart soon)19:26
fungicontinuing to keep tabs on those two stragglers19:27
*** bradjones has quit IRC19:27
*** bradjones has joined #openstack-infra19:28
*** bradjones has quit IRC19:28
*** bradjones has joined #openstack-infra19:28
openstackgerritDoug Hellmann proposed openstack/requirements: Revert "Add mimic to global requirements to use in tests"  https://review.openstack.org/26541619:28
jeblairfungi: the "git repack -Ad" resulted in a 2MB smaller repo than "git repack -afd"19:28
zaroclarkb, jeblair, fungi : once i'm free i can look deeper into what affects the git-gc and git-prune will have on gerrit performance.19:29
clarkbjeblair: should we make the nova repo as on review.o.o availalbe somewhere?19:29
clarkbjeblair: so that whoever works on it can compare like for like? its hard to get that just via clone19:29
*** _diana_ has quit IRC19:29
jeblairclarkb: yeah19:29
*** ociuhandu has quit IRC19:29
jeblairi'll do that and send an email19:29
openstackgerritMerged openstack-infra/project-config: Mark swarm test as non-voting in Magnum  https://review.openstack.org/26499819:30
zarowhat is the issue with having the repo in one packfile?19:30
clarkbzaro: it means that any time you need a ref you have to unpack that packfile19:30
*** tonytan4ever has joined #openstack-infra19:30
clarkbwhich may be huge if there is just one19:30
jeblairand maybe even transmit it under some circumstances?19:30
jeblairi'm not sure19:30
clarkbI think smart http makes a new packfile for all ops19:31
clarkbdumb http will transmit the entire packfile19:31
jeblairso maybe it's only the dumb http protocol where that's a worry19:31
sdagueAJaeger done19:31
jeblairthat may still be a worry for us though :)19:31
openstackgerritMerged openstack-infra/project-config: Add solum-pythonclient to projects  https://review.openstack.org/26508819:31
openstackgerritMerged openstack-infra/elastic-recheck: Add query for manila dsvm horizon setup bug 1532300  https://review.openstack.org/26541019:32
openstackbug 1532300 in Manila "manila dsvm jobs fail to setup horizon dashboard due to Permission Denied error on lock file" [Undecided,Confirmed] https://launchpad.net/bugs/153230019:32
openstackgerritMerged openstack-infra/openstackid: Fix on Asymetric key thumbprint calculation:  https://review.openstack.org/26539719:34
*** mrmartin has joined #openstack-infra19:35
fungielectrical: oh, i just realized, i didn't use --flush-cache, should i have done for this update? it looks like in the end only reported about 15% of our job configs were updated19:35
*** Sukhdev has joined #openstack-infra19:35
electricalfungi: wrong auto complete? Lol19:36
fungielectrical: yep, sorry!19:36
electricalHaha np bud :)19:36
anteayafungi: andreas did the last update on your patch to rename .log files to .log.txt files, do you want to review? https://review.openstack.org/#/c/203099/519:36
fungilooks like electrofelix dropped out of the channel19:36
fungianyway, i suppose the concern was mainly that the change could have resulted in a very lengthy jjb update (which it did), but that the addition is a no-op so as long as the cache has updated for anything it thought was out of sync, we should be fine19:37
*** Sukhdev has quit IRC19:37
fungianteaya: he mentioned it earlier--i'll take a look in a bit19:37
*** Sukhdev has joined #openstack-infra19:38
clarkbjeblair: what we really need is a `git optimize` which reduces repo size below some threshold while still contrucitng the repo in a way friendly for network transmission19:39
jeblairfungi: that's my thinking.19:39
anteayafungi: oh sorry I didn't mean to bother you, thank you19:40
fungijenkins06 finally tripped on a 502 from the api as well, so i've resumed it now19:40
openstackgerritMerged openstack-infra/project-config: Add python-ironicclient to PROJECTS  https://review.openstack.org/26333519:41
*** alkhodos has joined #openstack-infra19:42
zaroclarkb: you think there's a way to optimize the repo without putting it all in one file?19:43
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch to templating for nodepool rackspace  https://review.openstack.org/26499419:43
*** fawadkhaliq has joined #openstack-infra19:43
pabelangerjeblair: ^ That should be what you were asking for19:44
pabelangerbut still lowercase19:44
jeblairpabelanger: oh well, thanks :)19:45
*** treaki_ has quit IRC19:46
*** yolanda has quit IRC19:46
*** eil397 has joined #openstack-infra19:46
*** bgmccollum has joined #openstack-infra19:48
EmilienMhello, can someone point me to how to have os-loganalyze in our puppet openstack jobs?19:48
*** Sukhdev has quit IRC19:48
EmilienMI haven't found much documentation about that19:48
jeblairthe tar of the nova repo is 5.1 while the directory is 7.0G.19:49
jeblairi'm guessing our 100k refs take up a couple of G of direntries.19:49
*** ujuc has joined #openstack-infra19:50
openstackgerritMerged openstack-infra/system-config: Add instructions on how to troubleshoot bots  https://review.openstack.org/26006819:52
jeblairclarkb, fungi: http://tarballs.openstack.org/ci/ has nova.git.tar.bz2 from May 1119:52
jeblairmaybe something else related to our gerrit upgrade rollback?19:53
jeblairany ojebctions to me just overwriting that?19:53
fungijeblair: yes, we put a copy there so others could help us figure out why the repo was "going bad"19:53
fungiit's fair game to clean up now19:53
fungior replace/refresh, sure19:53
*** yolanda has joined #openstack-infra19:54
jeblairat least we're consistent with hosting locations :)19:54
jeblairwhile we're waiting for bzip: https://etherpad.openstack.org/p/1GtOYrmIGN19:55
*** [1]Thelo has joined #openstack-infra19:55
*** ujuc has quit IRC19:55
pabelangerjeblair: looks like depends-on for grafyaml not working as expected: https://jenkins07.openstack.org/job/gate-project-config-grafyaml/13/consoleText19:55
jeblairpabelanger: what change is that?19:56
clarkbjeblair: etherpad message lgtm19:56
pabelangerjeblair: 26499419:56
*** jsavak has joined #openstack-infra19:57
*** Thelo has quit IRC19:57
*** [1]Thelo is now known as Thelo19:57
fungijeblair: lgtm with one tiny clarification19:58
fungijeblair: also, xz is what all the hipsters use these days. bzip2 is so last decade19:59
*** _nadya_ has joined #openstack-infra19:59
jeblairfungi: don't make me rar it.19:59
fungibwahahaha19:59
anteayaemail looks good to me20:00
clarkbugh rar20:00
jrollplease make it a split rar20:00
clarkband host it on a warez site20:00
jroll^^20:00
jeblairjroll: i'll post it to usenet for you20:00
jeblairclarkb: like ftp.gnu.org?20:00
jrollyour download will begin in 27 seconds...20:00
Shrewsnewsgroups ftw20:00
fungijroll: i'll have uuencode split it before posting to the newsgroup instead20:00
fungino need for a split archive20:01
fungiuudecode can stich just fine20:01
fungier, stitch20:01
jeblair(apparently with twitter poised to accept >140 chars, this may be our last opportunity to download split tweets)20:02
* jeblair really hopes that downloading split archives from tweets that say "[1/165478]" is a thing20:03
clarkbjeblair: you cna base64 encode the tarball and post it to twiiter 140 bytes at a time20:03
*** dkranz has joined #openstack-infra20:04
jeblairpabelanger: i agree that didn't seem to work.  the params look correct; i'm not sure why it didn't work.20:05
*** Sukhdev has joined #openstack-infra20:06
*** EricGonczer_ has joined #openstack-infra20:07
pabelangerjeblair: the sha1 zuul is using, is not correct20:07
jeblairdoh!20:07
pabelangerjeblair: that appears to be current master20:07
jeblairit's space separated, not comma separated20:07
pabelangerah20:07
jeblairfixing20:07
pleia2yolanda: I really appreciate these documentation updates, inspired me to through my own notes to figure out what I'm storing locally that should be shared20:07
anteayaEmilienM: to be honest with you, I don't know the answer, sdague wrote most of it, perhaps he can suggest a pointer to have your logs use it20:08
pleia2seeking lunch now20:08
*** amuller is now known as amuller_afk20:08
anteayapleia2: may you find20:08
clarkberror lunch not found20:08
clarkb(this is my current isutation)20:08
anteayaclarkb: may you also find20:08
openstackgerritJames E. Blair proposed openstack-infra/project-config: Correct tox env override  https://review.openstack.org/26542720:08
jeblairanteaya, pabelanger: ^20:08
EmilienManteaya: clarkb is already helping me, thanks!20:09
jeblairpabelanger: mind if we merge that and recheck to make sure it works?20:09
pabelangerjeblair: WFM20:09
anteayaEmilienM: oh great, sorry I missed that20:09
*** kzaitsev_mb has joined #openstack-infra20:09
anteayaEmilienM: do share once you figure it out so I may learn, thank you20:10
openstackgerritBrant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job  https://review.openstack.org/26499120:10
EmilienManteaya: sure20:10
anteayaEmilienM: thanks20:10
clarkbfungi: there are a bunch of ready nodes on jenkins06 according to nodepool, we should maybe delete those to shift the load to othre masters?20:10
clarkbfungi: or do you plan on restarting soon?20:10
yolandapleia2 thanks. In downstream we have that system to document every common task we have, and i missed that a bit upstream. So if you can add more of these bits, it will be really good20:10
fungiclarkb: great point, i forgot to blow away nodepool ready nodes for jenkins06. i'll do that asap20:11
fungithey're deleting now20:12
fungithanks for catching that20:12
*** EricGonc_ has joined #openstack-infra20:15
*** cbader has joined #openstack-infra20:16
*** fawadkhaliq has quit IRC20:17
*** fawadkhaliq has joined #openstack-infra20:18
clarkbfungi: no problem I had just noticed the green portion of the status graph and saw that nodepool had a bunch of ready nodes on 0620:18
*** alivigni has quit IRC20:18
*** EricGonczer_ has quit IRC20:19
fungiokay, jenkins06 updates completed successfully finally20:20
*** jsavak has quit IRC20:21
fungiit's due for a scheduled restart later today anyway, right?20:21
docaedoThis is probably obvious, but where would I look to see results of periodic jobs? I realize I only know about jobs as the relate to specific changes, so not sure where to go :)20:21
fungidocaedo: http://logs.openstack.org/periodic/20:21
*** piet has joined #openstack-infra20:21
*** jsavak has joined #openstack-infra20:21
*** yonglihe has quit IRC20:21
docaedofungi: obviously ;)  Thanks!20:21
*** fawadkhaliq has quit IRC20:22
fungidocaedo: well, it's embarrassingly non-obvious, but there are some log dashboard efforts underway to hopefully fix the discoverability issue in coming cycles20:23
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: clean up OVH graphs  https://review.openstack.org/26543120:24
*** esp has joined #openstack-infra20:24
zaroesp: check here: https://git.openstack.org/cgit/openstack/stackalytics/tree/etc/default_data.json20:24
espnice! thx zaro20:24
*** aopincar has quit IRC20:24
*** nelsnels_ has quit IRC20:25
docaedofungi: well, there's so many parts and pieces to the infra, making it all obvious is a tall order. Feel like I learn a little bit more about it every day20:25
*** piet has quit IRC20:25
fungidocaedo: me too!20:26
fungiclarkb: i think jenkins02 may also be overdue. i keep getting 502 errors trying to get the jjb update there to finish20:27
fungii'm going to put it in prepare for shutdown as well (and delete the ready nodes for it from nodepool this time)20:27
clarkbI did check and it appeared they all successfully restarted lsat friday20:27
clarkbso a week may be too long of an internval20:27
fungii'm checking the thread details on it now20:27
*** hdd has quit IRC20:28
fungiyep, lots of blocked sshlauncher.afterdisconnect threads20:29
anteayaI'm about to approve the f21 is EOL patch at 264378 if anyone else wants to review it quickly20:29
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: update bluebox dashboard like OVH  https://review.openstack.org/26543220:29
openstackgerritMerged openstack-infra/project-config: Makes non-admin job to use pre provisioned cred  https://review.openstack.org/26397620:31
fungithat looping swift job seems to be tickling an odd situation in the scp publisher plugin https://jenkins01.openstack.org/job/gate-swift-python27/4944/consoleFull20:31
*** salv-orlando has quit IRC20:32
*** salv-orlando has joined #openstack-infra20:33
openstackgerritMerged openstack-infra/project-config: Fix cleanup of check-osc-plugins  https://review.openstack.org/26508520:34
anteayacan I get one more project-config core reviewer to agree that Sam-I-Am should have irc ops by +3'ing 26503620:34
openstackgerritIsaku Yamahata proposed openstack-infra/project-config: networking-odl: enable test with ODL release of beryllium-snapshot  https://review.openstack.org/26543720:37
*** Daviey has joined #openstack-infra20:38
*** ddieterly has quit IRC20:39
openstackgerritMerged openstack-infra/project-config: Fedora 21 is EOL stop using it  https://review.openstack.org/26437820:40
*** ddieterly has joined #openstack-infra20:40
*** jcoufal has joined #openstack-infra20:41
*** nelsnelson has joined #openstack-infra20:42
*** ddieterly has quit IRC20:42
*** jcoufal has quit IRC20:43
*** jcoufal has joined #openstack-infra20:44
*** kzaitsev_mb has quit IRC20:45
*** mrmartin has quit IRC20:46
*** sridhar_ram has quit IRC20:47
*** jcoufal has quit IRC20:49
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics  https://review.openstack.org/26397120:50
*** ujuc has joined #openstack-infra20:51
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: update internap dashboard like OVH  https://review.openstack.org/26543820:51
fungiokay, jenkins02 finally finished its jjb updates, so that's the last of them20:52
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: update rackspace dashboard like OVH  https://review.openstack.org/26543920:52
fungiinfra-root: all manual jjb updating has completed, with a confirmed successful result on each production jenkins master20:52
jeblairfungi: thanks!20:52
fungino sweat20:53
openstackgerritMerged openstack-infra/project-config: Correct tox env override  https://review.openstack.org/26542720:55
jeblairpabelanger: next zuul reload we should be able to recheck ^20:55
openstackgerritMerged openstack-infra/project-config: Removed testjobs and corresponding chef-jobs  https://review.openstack.org/26413020:56
*** ujuc has quit IRC20:57
*** claudiub has quit IRC20:57
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: update bluebox dashboard like OVH  https://review.openstack.org/26543220:58
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: update internap dashboard like OVH  https://review.openstack.org/26543820:58
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: update rackspace dashboard like OVH  https://review.openstack.org/26543920:58
jeblairokay, those are ready for review if anyone wants to double check me ^20:58
jeblairpabelanger: we might need to rebase your rax template change, sorry, but i think it's worth getting them all tidied up first.20:59
*** daemontool has joined #openstack-infra20:59
openstackgerritMerged openstack-infra/puppet-askbot: Fix broken Pillow pip dependency  https://review.openstack.org/26523121:00
pc_mHi infra folks! I have a question...21:00
pc_mI added constraints based jobs for neutron-vpnaas repo as experimental and have tested them.21:01
pc_mShould I now add them to check queue as non-voting, or add to check and gate queue as voting?21:01
*** rfolco has quit IRC21:01
pleia2pc_m: that's a decision the team has to make, if the neutron-vpnaas folks think it's reliable enough to gate on, you can skip the non-voting step21:02
*** bharathm has quit IRC21:02
*** Sukhdev has quit IRC21:02
pleia2pc_m: but many teams like to see how it goes for non-voting for a bit first21:02
anteayapc_m: I would advocate for moving to check non-voting myself21:03
pc_mpleia2: anteaya: OK. Will do non-voting. One is a post (cover-constraints). Just hold on that, until voting?21:03
anteayapc_m: I don't understand what you mean21:04
anteayawhat is the name of the repo?21:04
pc_manteaya: One of the new experimental jobs, is neutron-vpnaas-cover-constraints, which would replace the neutron-vpnaas-cover in the post queue.21:04
pc_mneutron-vpnaas21:05
*** eharney has quit IRC21:06
anteayapc_m: yeah okay that post job, has that been exercised in the experimental queue at all?21:06
pc_manteaya: yes21:06
anteayait has21:06
anteayaoh right21:06
*** hdd has joined #openstack-infra21:06
anteayaokay so for everything else move to check non-voting21:06
pc_manteaya: I've verified all the jobs in experimental queue and want to move forward.21:07
anteayawe don't have a non-voting situation for post jobs21:07
anteayaright21:07
anteayathat is a coverage job, I would say move that one job to the post queue21:07
*** harlowja_at_home has joined #openstack-infra21:07
fungialso, post jobs don't break merging since they happen "post"-merge21:07
* anteaya nods21:07
pc_manteaya: fungi: Gotcha. Thanks.21:08
fungiand as docaedo discovered, blindly recommending running post jobs in the experimental pipeline doesn't always fit21:08
fungiin his case, a job which uploads the results of processing a merge as a proposed change in gerrit21:08
*** _nadya_ has quit IRC21:08
anteayafungi: sorry I missed that, which job was that?21:09
*** aopincar has joined #openstack-infra21:09
funginot really safe to try to make into an experimental job, which runs on unmerged changes (among other problems, it ends up with a divergence from the branch the job assumes it's basing new changes on)21:09
docaedoanteaya: this is the latest PR related to it, it was the dead-link checker script https://review.openstack.org/#/c/264978/21:09
*** amuller_afk is now known as amuller21:10
anteayadocaedo: thanks21:10
*** sridhar_ram has joined #openstack-infra21:10
docaedoanteaya: no problem!21:10
fungiexperimental is a change-based pipeline like check, gate and silent21:11
fungiit's often not suitable for "trying out" jobs intended to run in ref-based pipelines like post, pre-release, release or tag21:11
fungior periodic21:12
fungiwhich is neither change-based nor ref-based21:12
anteayaack, thanks21:13
fungiyou're welcome!21:13
openstackgerritEmilien Macchi proposed openstack-infra/os-loganalyze: filter: add more coverage for puppet logs  https://review.openstack.org/26544721:13
*** aopincar has quit IRC21:13
funginow to get back to making my owner-listing script increasingly inefficient21:14
EmilienMclarkb: I finally deciced to go that way ^ because log files not managed the same way in packaging than in devstack21:14
*** otter768 has joined #openstack-infra21:14
*** kzaitsev_mb has joined #openstack-infra21:17
*** edmondsw has quit IRC21:17
jeblairdoes anyone object to me self-approving changes to the grafana dashboards?21:17
clarkbEmilienM: thats probably something we should work to make consistent by default as well21:18
clarkbjeblair: I don't21:18
*** annegentle is now known as notsogentle21:18
Shrewsjeblair: if you get bored over the weekend, i'm proposing some new functionality to shade i'd love feedback on: https://review.openstack.org/264869  nothing urgent though21:18
*** otter768 has quit IRC21:19
*** ganso has quit IRC21:19
anteayajeblair: I do not object21:19
jeblairShrews: cool, we use functionality like that in both our launch script and in nodepool -- generally implemented as "sort the flavors by size and use the first that has at least X gb of ram"21:20
jeblairShrews: we probably wouldn't use a max bound, but i could see having the ability to have one being a nice safety check, especially if you actually get billed for cloud usage.  ;)21:21
Shrews++21:21
clarkbShrews: might be good to at least confirm with nova that the scale/format/types of all the serachable flavors are consistent21:21
Shrewsjeblair: yeah, that's there for balance, really21:21
clarkbI don't think we have had probablems with the RAM numbers21:21
jeblairclarkb: agreed, i don't remember any problems with ram.  somehow i feel its only a matter of time :)21:22
Shrewsclarkb: yeah, this is sort of limited to just integer fields (or convertable to integer), atm21:22
clarkbI just imagine that it could be possible for 8GB, 8192, and 8192MB to all be valid21:22
clarkbunless nova is explicit about it21:23
Shrewsbut, this is not supposed to be limited to flavors (though i can't come up with a better use case just now, either)21:23
*** jsavak has quit IRC21:24
clarkbsorting is always good fun21:26
*** daemontool has quit IRC21:27
jeblairwait, we don't have to implement our own quicksort in nodepool?21:27
jeblairpfft.  there's my csc degree wasted.21:28
openstackgerritMerged openstack-infra/project-config: Grafana: clean up OVH graphs  https://review.openstack.org/26543121:28
Shrewsjeblair: well, my patch doesn't sort, so i could steal your codez and sort the datas  :)21:28
*** tlian has quit IRC21:28
Shrewsthat may actually be useful... even as a separate review21:29
openstackgerritMerged openstack-infra/project-config: Grafana: update bluebox dashboard like OVH  https://review.openstack.org/26543221:29
openstackgerritMerged openstack-infra/project-config: Grafana: update internap dashboard like OVH  https://review.openstack.org/26543821:29
openstackgerritMerged openstack-infra/project-config: Grafana: update rackspace dashboard like OVH  https://review.openstack.org/26543921:29
*** tlian has joined #openstack-infra21:29
jeblairwell, i just called sort() so i probably failed the assignment.21:29
clarkbyou'll already need it for min/max right?21:29
fungijeblair: no objection to self-approving grafana dashboard additions/modifications as they're still sort of under construction anyway and non-impacting if they go terribly wrong21:29
clarkbI guess not21:29
Shrewsclarkb: no. straight linear search21:29
clarkbdifference between n and n log n21:30
clarkbya21:30
clarkbnext you'll be using a tree to log n21:30
jeblairyou know what would be really cool in grafana -- if you could gang graphs together and force them to have the same y scale.21:31
jeblair(but still have that scale dynamic)21:31
*** bharathm has joined #openstack-infra21:32
*** daemontool has joined #openstack-infra21:32
jeblairoh there's a light theme: http://grafana.openstack.org/dashboard/db/nodepool-rackspace?theme=light21:35
anteayajeblair: oh I like that21:36
*** MarkAtwood has quit IRC21:38
*** ociuhandu has joined #openstack-infra21:38
*** devkulkarni has left #openstack-infra21:38
*** eharney has joined #openstack-infra21:39
jeblairokay... why are the sparklines for node counts not integers?21:41
jeblair(ovh is apparently deleting 1.6 nodes right now)21:41
jeblairgraphite itself says the much more sensible 1.0 for that query21:42
jeblairso it's a grafana mutation21:42
*** rguillebert has quit IRC21:47
jeblairoh.  maxDataPoints.21:49
jeblairgrafana is apparently not the tool to use if you are interested in accurate data21:50
jeblairby default it is more concerned that data that looks good be displayed quickly.21:50
jeblairwe could increase maxdatapoints, but that may be excessive.21:59
*** esker has joined #openstack-infra21:59
*** esker has quit IRC21:59
*** jsavak has joined #openstack-infra22:02
clarkbjeblair: our data needs skinny jeans22:03
fungialso, beard22:03
*** esker has joined #openstack-infra22:03
*** rlandy has quit IRC22:06
*** jsavak has quit IRC22:06
*** mriedem has quit IRC22:07
*** jsavak has joined #openstack-infra22:08
*** pradk has quit IRC22:08
*** salv-orl_ has joined #openstack-infra22:09
*** salv-orlando has quit IRC22:12
*** salv-orl_ has quit IRC22:12
*** salv-orlando has joined #openstack-infra22:12
*** kgiusti has left #openstack-infra22:14
*** ujuc has joined #openstack-infra22:15
amullerarmax: mtreinish: Finished mapping out all modifications to Neutron API tests since the fork: https://etherpad.openstack.org/p/neutron-tempest-defork-patches-since-initial-sync22:15
amullerlinked to it from the main etherpad22:15
openstackgerritMerged openstack-infra/puppet-openstackci: Add a CONTRIBUTING.rst document  https://review.openstack.org/26503522:16
*** nelsnelson has quit IRC22:16
*** trown is now known as trown|outttypeww22:16
*** esker has quit IRC22:17
* asselin watches first puppet-openstackci-infra-docs22:17
*** leakypipes has quit IRC22:20
anteayaasselin: go go go22:21
*** rossella_s has quit IRC22:22
*** peristeri has quit IRC22:22
*** esker has joined #openstack-infra22:22
*** rossella_s has joined #openstack-infra22:22
*** ujuc has quit IRC22:25
*** esker has quit IRC22:27
clarkbgerrit GC has fallen off, not sure if related to firday or maybe the lack of thousands of comments comming in helped it22:27
*** dkranz has quit IRC22:29
openstackgerritIsaku Yamahata proposed openstack-infra/project-config: networking-odl: enable test with ODL release of beryllium-snapshot  https://review.openstack.org/26543722:31
*** FallenPegasus has joined #openstack-infra22:31
alkhodosHi guys, I have a problem with CI: it connects to gerrit stream, adds events to queue, runs everything and posts results to web server - all good. But for some reason it stopped voting on gerrit. Was fine before. I can't figure out what is causing this issue, if you have any idea where should I like - I would really appreciate it. Thanks in advance.22:32
*** amuller has quit IRC22:32
clarkbalkhodos: it is probably one of two things, either your zuul layout file doesn't instruct it to vote anymore, or the project(s) you are posting to removed your accounts voting rights22:33
openstackgerritJames E. Blair proposed openstack-infra/grafyaml: Add support for seriesOverrides  https://review.openstack.org/26545822:34
*** Sukhdev has joined #openstack-infra22:35
*** erlon has quit IRC22:36
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: Add max server line to node graph  https://review.openstack.org/26545922:36
jeblairif anyone wants to review those 2 changes, they're pretty straightforward22:38
jeblairand we'll end up with a cool line on the test nodes graph corresponding to max_servers22:39
*** yaume has joined #openstack-infra22:39
alkhodosclarkb: Thanks! will look into those things22:41
*** tlian has quit IRC22:42
anteayajeblair: alias is required in the overrides?22:43
jeblairanteaya: yeah, it's how you specify which thing you are overriding22:43
anteayaah22:43
jeblairanteaya: so if you look at my usuage in the second patch, you'll see i'm saying "don't stack 'max'"22:43
anteayaI will look22:44
anteayawas just looking at the E501 line too long error that pep8 was offering22:44
jeblair<sigh>22:44
anteayayeah22:44
*** bharathm has quit IRC22:45
*** flepied has joined #openstack-infra22:47
openstackgerritJames E. Blair proposed openstack-infra/grafyaml: Add support for seriesOverrides  https://review.openstack.org/26545822:48
jeblairupdated the commit msg too22:48
alkhodosclarkb: sorry for a stupid question, but how can I see if my voting rights were removed?22:48
clarkbalkhodos: check the $project-ci group membership in gerrit. People -> list groups -> search -> membership22:49
clarkbyou aren't there you shouldn't have voting rights22:49
*** bharathm has joined #openstack-infra22:50
*** regXboi has quit IRC22:51
asselin#success OpenStack Third Party CI documentation published http://docs.openstack.org/infra/openstackci/22:51
openstackstatusasselin: Added success to Success page22:51
*** bharathm has quit IRC22:52
pabelangerjeblair: will test your suggestions later tonight22:53
*** bharathm has joined #openstack-infra22:54
TheloHi all, I have a problem with jenkins and nodepool, to do simple jenkins fails to ssh a slave ([01/09/16 06:43:39] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 33 ms) however if I try to ssh the slave manually using the user jenkins and the key defined in the security in jekins it works fine, does anyone have  clue of what could be wrong?22:55
clarkbThelo: is that IP address reachable from nodepool?22:56
clarkbor is it jenkins failing to ssh?22:56
*** yaume has quit IRC22:56
jeblairpabelanger: thanks.. i added some more fields to graph: https://review.openstack.org/265458 and https://review.openstack.org/26545922:56
*** ddieterly has joined #openstack-infra22:56
clarkbeither way I would double check connectivity between that path if you haven't already testing ssh22:56
*** jsavak has quit IRC22:58
clarkbThelo: another thing to check if it is between jenkins and the slave is that the credentials are configured properly on the slave22:58
jeblairpabelanger: and here's something to think about:  the sparklines show the current value as a big number, but because they use "maxDataPoints=100" in the graphite query, graphite aggregates the data before returning it using the mean.22:58
Theloclarkb: yes from nodepool I can conneect manually to the slave22:58
jeblairpabelanger: that means that it ends up as a rational number rather than integer, which doesn't make sense ("Deleting 1.8 nodes" ?)22:58
Theloclarkb: only jenkins fails to ssh it22:59
asselinThelo, check the jenkins log file22:59
clarkbThelo: ok, in that case I would double check the slave credentials that are configured in Jenkins22:59
jeblairpabelanger: we could increase maxDataPoints (we can even set it to 'auto' which is just equal to the width of the graph in pixels) -- that's usually high enough that it doesn't need to consolidate22:59
jeblairpabelanger: but it also means shipping more data around and in some quick testing, does slow things down a little bit (not much)23:00
jeblairpabelanger: we could also add the 'consolidateBy' function to the query, but i'm not sure any of the other consolidation functions make much more sense -- the best would be 'max'23:00
*** thorst_ has quit IRC23:00
*** thorst has joined #openstack-infra23:01
jeblairpabelanger: but that's still inaccurate -- it means sometimes the numbers will lag a bit23:01
anteayaasselin: congratulations23:01
asselinanteaya, thanks :)23:02
Theloclarkb: the log file only shows that "([01/09/16 06:43:39] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 33 ms)" I am double checking the slave credential configuration23:02
*** ZZelle has joined #openstack-infra23:04
asselinThelo, double check the ssh private key in /var/lib/jenkins/.ssh/id_rsa23:05
asselintry to login as jenkins user using that key file23:05
*** thorst has quit IRC23:09
Theloasselin: I just did and I could ssh my node ( using jenkins user)23:12
*** dimtruck is now known as zz_dimtruck23:12
*** sigmavirus24 is now known as sigmavirus24_awa23:13
*** notsogentle has quit IRC23:13
*** notsogentle has joined #openstack-infra23:13
asselinThelo, anything in the jenkins log file?23:14
*** otter768 has joined #openstack-infra23:15
TheloINFO: Attempting to reconnect d-p-c-local_01-1123:16
Thelo[01/09/16 07:07:39] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 38 ms23:16
Theloasselin: that message23:17
*** tonytan4ever has quit IRC23:19
*** amitgandhinz has quit IRC23:19
*** otter768 has quit IRC23:20
*** _tzn has quit IRC23:21
*** vgridnev has quit IRC23:22
Theloasselin: hudson.AbortException: Cannot find SSH User credentials with id: 2f541bcd-a553-4614-9d61-ddc7d121d4aa do you know what this id refer to?23:22
*** tzn has joined #openstack-infra23:23
asselinThelo, probably a credentials id....it's part of the docs23:23
*** sridhar_ram1 has joined #openstack-infra23:26
*** notsogentle has quit IRC23:27
*** sridhar_ram has quit IRC23:28
*** sridhar_ram1 has quit IRC23:31
*** sridhar_ram has joined #openstack-infra23:32
*** tzn has quit IRC23:35
*** tzn has joined #openstack-infra23:43
Theloaselin: could it be something like the node ask the question "are you sure that you want to connect " and jenkins does not know how to anser to that?23:44
asselinThelo, when you  checked ssh, did you do as as the jenkins user? sudo su - jenkins23:46
*** gordc has quit IRC23:47
*** Sukhdev has quit IRC23:49
*** mmedvede has joined #openstack-infra23:49
*** FallenPegasus has quit IRC23:50
*** Qiming has joined #openstack-infra23:54
Theloasselin: http://paste.openstack.org/show/483352/23:55
*** tzn has quit IRC23:56
*** Sukhdev has joined #openstack-infra23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!