Tuesday, 2015-12-01

jeblairmordred, nibalizer: where are the logs that either detail the run from 2303 or indicate the failure or why it did not run?00:00
anteayapc_m: start with what you have and offer a patch00:00
fungipc_m: i would start by trying to pin asteroid (and if necessary pylint) and see what job(s) object to that, then work backwards from there (possibly pinning asteroid with a cap in global requirements, though again this is fuzzy territory if different projects are allowed to use different versions of pylint)00:00
anteayapc_m: make your intent very clear in the commit message so a reviewer can make suggestions in their review00:00
lifelessmtreinish: (e.g. I see that we test the case you say was failing)00:00
*** otter768 has joined #openstack-infra00:01
nibalizerI don't think it's run since00:01
nibalizerpossibly because ansible-puppet has been failing on rsync?00:01
fungipc_m: lifeless and Nakato and would be my go-to for questions about requirements/constraints and python packaging on an apac timeframe00:01
jeblairnibalizer: that hypothesis matches my observations00:02
pc_manteaya: I'm pretty fuzzy myself though... global-requirements has pylint=1.4.4 and no astroid. Whereas upper-constraints has astroid=1.3.8, but no pylint.00:02
*** dimtruck is now known as zz_dimtruck00:02
mordredjeblair: I apologize - "copy files to logstash.o.o" ran successful00:02
mordredjeblair: I am investigating further00:02
lifelesspc_m: upper-constraints is the transitive expansion of dependencies00:02
pc_mwe want pylint 1.4.4 and astroid 1.3.8 for adv services and neutron00:02
*** sridhar_ram has quit IRC00:02
lifelesspc_m: this is described in the README00:02
jeblairmordred: makes sense00:02
openstackgerritMerged openstack-infra/puppet-elasticsearch: Add logging configuration  https://review.openstack.org/25154000:03
jeblairmtreinish, SpamapS: ^ see nibalizer / mordred words for info on pupput runs on logstash00:03
lifelesspc_m: we blacklist lint style tools - we can't globally define them because when they release, it would cause deadlocks across the gate00:03
nibalizerinterestingly the puppetmaster is trying to run puppet on itself and failing - exit code 400:03
openstackgerritMerged openstack-infra/puppet-elasticsearch: Stop logging actions at DEBUG level  https://review.openstack.org/25154100:03
fungilifeless: however we don't enforce/sync requirements on style and static analysis checkers, right? how does that extend to different versions of those wanting different versions of transitive deps?00:03
lifelesspc_m: (or force everyone onto absolute lockstep changes, and we have no mechanism to enable a small team to do that)00:03
*** sridhar_ram has joined #openstack-infra00:03
*** sshnaidm has quit IRC00:04
mordredoh - this is the problem that we pushed  apatch up for00:04
lifelessfungi: so, we need to blacklist it from upper-constraints if its a linter style tool00:04
lifelessfungi: and then folk can do $whatevertheywant locally with it00:04
fungilifeless: not sure if you were following, but basically pylint depends on asteroid, and there's a new release of asteroid on pypi which causes some versions of pylint in use by various projects to break00:04
lifelessfungi: (e.g. use it from tox deps= lines00:04
mtreinishlifeless: hmm, ok. Well nibalizer and I both definitely observed that behavior back when we were first spinning things up00:05
mordredjeblair: https://review.openstack.org/25157700:05
mordredjeblair: which has now merged00:05
mordred(10 minutes ago)00:05
mordredjeblair: should fix the reason that puppet did not run on logstash at that time00:05
fungiso in this case asteroid is in upper-constraints.txt presumably as a dependency of something else (or we just missed filtering it like we are for pylint which is what's pulling it in)00:05
lifelessok but00:05
pc_mlifeless: So, can we, in neutron, set pylint to 1.4.4 and astroid to 1.3.8?00:05
lifelessastroid is a ast00:05
lifelessnot a linter00:05
fungiright, it's _used_ by a linter00:06
jeblairmordred, nibalizer: does nibalizer's observation about puppetmaster failing mean that is unlikely to self-correct?00:06
fungiand different versions of the linter need different versions of that it seems00:06
fungiwhich means it potentially has to vary as the linter using it varies00:06
lifelesslets see00:06
nibalizerjeblair: from the run_all log there is some conflicting information00:07
mordredjeblair, nibalizer: there is anther problem00:07
mordredone sec00:07
*** rbrndt has quit IRC00:07
nibalizerthere is the output of the puppet run on puppetmasetr.o.o where everything went fine, then there is ansible reporting that the return code was 4 and it is fatal00:07
pc_mfungi: yeah, looks like some are not playing well together, and it looks like newer version is report new errors00:07
nibalizermordred: ok00:07
fungiit's possible in this case we could blacklist the latest release of asteroid in global reqs and then get all projects using pylint to add a similar entry for asteroid from global reqs00:07
lifelessso pylint has an open dependency00:08
fungibut it may be that implies we need everyone to upgrade to a newer version of pylint or perpetually blacklist all new asteroid releases for everyone00:08
lifelessand before the update it was also open00:08
lifelesspc_m: well, your lint jobs should be getting constrained by upper-contraints no ?00:09
lifelesspc_m: whats the actual thing thats breaking for you ?00:09
lifelessfungi: it implies that our per-project linter strategy may actually be broken00:09
lifelessfungi: and we've just been ignoring it because the social issues are too hard00:09
fungilifeless: bingo00:09
lifelessfungi: and perhaps its time to revisit that00:09
fungithis was basically my point00:10
pc_mlifeless: well, we have multiple breakages in multiple repos.00:10
lifelesspc_m: I need specifics please00:10
fungilinters installing into the same environment as everything else means we can't pretend they're an isolated thing with no implications on our dependency chain00:10
pc_mlifeless: In neutron, we were getting undefined variables (thousands) for all the imports.00:10
lifelesspc_m: what gate job?00:10
lifelesspc_m: got a link to one failing?00:11
fungipc_m: but not for the constraints version of the pylint runs right?00:11
fungi(because those are still using older asteroid at the moment)00:11
openstackgerritMerged openstack-infra/project-config: Add additional iCal tests  https://review.openstack.org/25119000:11
pc_mI was looking in local environment and seeing in all repos I use. Others seeing as well. There are jobs failing. My understanding is any running pep8.00:12
pc_mLet me look...00:12
nibalizermordred: im also wondering if the dynamic inventory is working at all00:12
tsglifeless, fungi: to make a package specific to a distro rev in bindep-fallback, where do I find the OS profile names.  Also is specifying multiple OS profiles (or excluding one) supported?00:12
nibalizer2015-12-01 00:01:11,858 p=31575 u=root |  PLAY [!review.openstack.org:!git0*:!afs*:!puppetmaster*:!disabled] ************00:12
nibalizer2015-12-01 00:01:11,860 p=31575 u=root |  skipping: no hosts matched00:12
mordrednibalizer: it's not00:12
mordredit's a thing00:12
mordredI'm working on it00:12
nibalizerthat should be the world00:12
mordredone sec00:12
nibalizerand it is not00:12
nibalizeroh ok00:12
mordredwill report back with findings00:12
mordredthere is a file missing00:12
mordreddoes not exist00:12
mordredon the machine00:12
nibalizersounds important00:13
mordredI see00:13
zaromgagne: hey, finally getting around to look at the jeepyb issue with regards to pushing acl changes.  did you get it figured out?00:13
mordredthanks. that was helpful00:13
tsglifeless, fungi: wrt the question above ^^, I meant "supported OS profile names"00:13
mgagnezaro I'm using environment variables in my playbook to override git commiter00:13
lifelesstsg: on the os you want, run bindep --profiles00:13
pc_mlifeless: Just picked one... https://review.openstack.org/#/c/249864/00:13
zaronibalizer: take another look? https://review.openstack.org/#/c/241060/00:13
lifelesstsg: if you ave no file, do bindep -f /dev/null --profiles00:14
fungitsg: the os profile names are a transformation of lsb_release output00:14
mgagnezaro https://gist.github.com/mgagne/ac3f6e1bf3ebb2e587e100:14
*** ashleighfarnham has quit IRC00:14
mgagnein an Ansible handler00:14
fungitsg: so an open-ended list00:14
zaromgagne: ok.  i think jeepyb is using env vars to set committer as well.  maybe from .gitconfig?00:14
openstackgerritMonty Taylor proposed openstack-infra/system-config: Copy the file that is installed  https://review.openstack.org/25159000:14
lifelesspc_m: ok so thats kilo, which isn't constrained, and we can just lock down.00:14
mordrednibalizer, jeblair: ^^00:14
tsglifeless: got it .. is bindep a CI specific binary?00:14
zaromgagne: you can't set commit with git commit command00:14
lifelesstsg: pip install bindep00:15
mgagnezaro it's not, it's using --author00:15
tsgah ok .. thanks00:15
fungitsg: it's on pypi00:15
fungiyeah, that00:15
mordred'!disabled' shows the right hosts now00:15
mordredbtw - for folks playing along at home00:15
SpamapSmordred: I'm a little lost on this. Is that patch fixing logstash.o.o's puppet runs, or are we still not clear why it's still not running puppet?00:15
*** rebase has quit IRC00:15
mordredansible '!review.openstack.org:!git0*:!afs*:!puppetmaster*:!disabled' --list-hosts00:15
lifelesspc_m: so this is going to need separate work to fix each of juno kilo liberty and master00:16
mgagnezaro right. So someone using jeepyb should create .gitconfig or use env vars00:16
mordredis how to verify the lists of hosts that ansible will run against00:16
lifelesspc_m: and the answer is going to look a little different in juno+kilo, liberty, master00:16
zaromgagne: yeah set author seems correct to me.  can't set committer with 'git commit'.00:16
*** Kennan has joined #openstack-infra00:16
zaromgagne: i think our puppet is doing that.00:16
mordredSpamapS: that patch will fix it00:16
SpamapSmordred: kkthx00:16
tsglifeless, fungi: thanks that helps!00:16
lifelesstsg: fungi: you can also define custom profiles easily00:16
mordredSpamapS, jeblair, nibalizer: the problem was that the puppet dynamic inventory did not get installed because we had referenced the wrong filename00:16
mordredthis means that the only inventory that existed was the disabled group00:17
lifelesspc_m: in liberty or master running tox -epep8-constraints should work; I just tested master00:17
nibalizerzaro: kk00:17
pc_mlifeless: Yes, all the ones with pep8-constraints pass.00:17
lifelessthey will however break if we don't have a reverse check on them in requirements00:17
pc_mlifeless: ones with pep8 fail in neutron, neutron-lbaas, neutron-vpnaas, and maybe neutron-fwaas (have not checked).00:18
lifelesspc_m: you should add the neutron pep8-constraints job to openstack/requirements to prevent this00:18
lifeless(in liberty and master)00:18
mordredinfra-root: ansible '<ANY HOST MATCH STANZA>' --list-hosts is the best way to test those things - thus "ansible '!review.openstack.org:!git0*:!afs*:!puppetmaster*:!disabled' --list-hosts" from earlier00:19
*** tsg has quit IRC00:19
pc_mlifeless: so use the new job in those releases?00:19
lifelesspc_m: yes, on openstack/requirements00:19
lifelessfungi: I may be using the wrong words here; can you translate what I mean ?00:19
lifelessfungi: I've had the experience several times now of saying 'add a job for X to openstack/requirements' and .. nada00:20
pc_mlifeless: Why is that job passing? Is it because it is using the upper-constraints file and that has astroid=1.3.800:20
mgagnezaro maybe Gerrit is configured to allow any comitter: https://git.openstack.org/cgit/openstack/zaqar-ui/commit/?id=ba065cba4654a60a173ad756ba76b2703649728900:20
mgagnezaro committerroot <root@review.openstack.org>00:20
lifelesspc_m: yes, that is the purpose of upper-constraints.txt00:20
lifelesspc_m: has anyone filed a bug on astroid about this breakage?00:20
pc_mlifeless: I think you mean to add it to project-config repo00:21
fungilifeless: i think we gave up some time ago on being able to symmetrically gate the requirements repo (or the devstack repo, or...) on every job they might possibly break00:21
mgagnezaro This means the openstack-project-creator account should have root@review.openstack.org for its email00:21
nibalizermordred: sweet!00:21
mgagnezaro but docs say it's openstack-infra@lists.openstack.org00:21
pc_mlifeless: not AFAIK. We just discovered the problem about 4 hours ago.00:21
mgagneor user has forgeCommitter right which only comes after initial bootstrap.00:22
fungilifeless: especially if we switch to running all jobs with a constraints variant, that would imply that a change to the requirements repo would need to have thousands of jobs run against it to verify it breaks none of them. the end result would be that you could never merge any change to it because our false negative rate alone would make it impossible00:23
zaromgagne: ahh, it's in projects.ini https://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/utils.py#n1900:23
*** tsg has joined #openstack-infra00:23
*** zz_dimtruck is now known as dimtruck00:23
pc_mlifeless: So, for neutron, are all master commits using pep8-constraints job?00:23
pc_mSeems like it.00:24
*** openstackstatus has quit IRC00:24
jeblairmordred: "ansible disabled --list-hosts" is cool :)00:24
zaromgagne: looks like that's where it pulls all the configs from.  do you have one of those files in your setup?00:24
mgagnezaro I've set all the possible configs. I guess it's not possible to come up with the exact config openstack infra is using without manual work due to additional rights required by openstack-project-creator to create project with "wrong" committer.00:24
mordredjeblair: ++ I'm pleased with it00:24
jeblairmordred: (look i read your note and internalized it!)00:24
mordredjeblair: woot!00:24
mordredjeblair: you have applied the concept to a new instantiation00:24
*** otter768 has quit IRC00:24
*** openstackstatus has joined #openstack-infra00:25
*** ChanServ sets mode: +v openstackstatus00:25
mgagnezaro this only means you cannot use jeepyb if the commiter used isn't the email of the admin user or whatever user you created. Overriding the email in the jeepyb config won't help you unless you use .gitconfig or env vars. The gerrit-committer config should in fact be renamed for gerrit-author or gerrit-git-author.00:26
nibalizerit is really nice to be able to see what is disabled00:26
nibalizerone list :)00:26
mgagnezaro it does not properly reflect what it's used for by jepeyb.00:26
pc_mlifeless: Would this be an option... 1) master OK, as it is running pep8-constraints job, 2) for liberty, kilo, juno add pin in test-requirements for astroid in neutron, 3) for neutron-vpnaas pin astroid and pylint in test-requirements?00:26
pc_mRight now neutron-vpnaas doesn't pin pylint either.  I ran locally, and it worked, but it failed in the gate.00:27
lifelessfungi: well00:27
lifelessfungi: we don't need everything00:27
lifelessfungi: but not having representative jobs doesn't make sense00:27
lifelessfungi: its highly likely for instance that *everyone* will be broken by this pylint interaction00:28
fungilifeless: yep, i don't disagree if someone wants to come up with a reasonable representative set, and perhaps also limit it to run only on changes which modify upper-constraints.txt or something if the list is larger than we anticipated00:29
*** tiswanso has joined #openstack-infra00:29
fungiso that we can rely on it to block constraints updates when they bring in badly broken things which are not broeaking what's currently tested (devstack?)00:29
*** baoli has quit IRC00:29
lifelessfungi: well, I'm proposing we just add thiem one by one as we learn we're missing important coverage00:29
zaromgagne: i think the only requirement is that the jeepyb 'gerrit-user' needs to have access to refs/meta/config/* ?00:30
lifelessfungi: because we have a JFDI policy *already* on upper-constraints updates00:30
*** otter768 has joined #openstack-infra00:30
fungilifeless: sounds like the representative set is gate-neutron-pylint-constraints until further notice? ;)00:30
lifelessfungi: for this yeah00:30
lifelessI'm just paging in how to add it :000:31
*** Kennan has quit IRC00:31
*** sridhar_ram has quit IRC00:31
mgagnezaro The issue I have is: The Git commiter doesn't match any existing Gerrit accounts due to git commiter defaulting to $USER@$HOSTNAME. The gerrit-commiter doesn't override this default value as one would expect from the config name. It only updates the git author which Gerrit doesn't care about when validating the commit.00:31
fungilifeless: i'll whip up an example real quick00:31
SpamapSmtreinish: http://puppetboard.openstack.org/report/logstash.openstack.org/e097aa19fba4da229e4993b699524b10a8206def <-- 1.0.2 picked up00:31
*** Kennan has joined #openstack-infra00:32
*** sridhar_ram has joined #openstack-infra00:32
lifelessfungi: so we'll want it on for master, liberty, off for older00:33
SpamapSmtreinish: I'd have expected run_metadata to complete quickly tho00:33
zaromgagne: existing accounts?  do you mean ones that come with gerrit by default?00:33
mgagnezaro any account.00:33
fungilifeless: well, that's probably already done for us since we'd just add constraints-based jobs for this and they should already only use those branches?00:33
lifelessfungi: fingers crossed!00:34
*** tiswanso has quit IRC00:34
mgagnezaro or in fact, Gerrit refuses the commit because it doesn't match the email associated to your public SSH key.00:34
mgagnezaro "You cannot forge commiter" something like that00:34
*** Swami has quit IRC00:34
*** rebase has joined #openstack-infra00:34
mtreinishSpamapS: well we can ask someone to check if the migration is running00:34
SpamapSmtreinish: indeed00:34
pc_mfungi: So neutron master using constraint based jobs, so OK? What about liberty?00:34
lifelesspc_m: ok now for kilo. we need to add an upper cap to astroid to juno and kilo00:34
mtreinishfungi, mordred: ^^^ :)00:35
* pc_m feebly trying to follow along.00:35
zaromgagne: yeah, wouldn't you create that account on your gerrit before using it with jeepyb?  or is this a chicken and egg problem?00:36
*** mnencia has quit IRC00:36
pc_mlifeless: Let me see if I understand...00:37
lifelesspc_m: does astroid 1.4.1 also break things?00:37
pc_mlifeless: neutron master and liberty use pep8-constraints job, so all set00:37
lifelesspc_m: or was 1.4.0 the size of it ?00:37
pc_mlifeless: kilo and juno need some restriction on astroid?00:37
SpamapSfungi , mordred: specifically looking for the content of /var/log/subunit2sql_migration.log00:37
mgagnezaro let me start from the beginning: You install Gerrit on review.example.org. You create an admin account: admin <admin@example.org>. You install Jeepyb on that same machine (review.example.org). You configure your projects and gerrit-user: admin // gerrit-committer: admin@example.org. You run manage-projects and it crashes.00:38
pc_mlifeless: AFAICT, pylint 1.4.4 and astroid 1.3.8 are good.00:38
mgagnezaro Gerrit refuses your commit because git committer (root@review.example.org) doesn't match admin@example.org00:38
lifelesspc_m: what about 1.4.0 is my question00:38
*** mnencia has joined #openstack-infra00:38
pc_mlifeless: pylint 1.4.4 and astroid 1.4.1 caused tons of failures.00:38
lifelessyes I know00:38
lifelessMy question is very specific00:38
*** sfinucan has quit IRC00:38
pc_mlifeless: Have not tried any other version.00:38
mgagnezaro now you scratch your head as to why the Jeepyb config named "gerrit-committer" doesn't override the committer. And now you find out that the config doesn't update the git committer but the git author. /facepalm00:38
mtreinishSpamapS: it's definitely running the queries for openstack-health are hitting: "Waiting for table metadata lock"00:39
lifeless1.4.0 is equally broken00:39
*** hdd has quit IRC00:40
SpamapSmtreinish: oh thats actually good00:40
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Run gate-neutron-pylint-constraints on global reqs  https://review.openstack.org/25159600:40
SpamapSmtreinish: but not exactly "online" ;-)00:40
fungilifeless: maybe this ^00:40
lifelesspc_m: is there a bug open for this?00:40
lifelesspc_m: and have you informed astroid of their mistake?00:40
SpamapSmtreinish: probably means the worker is locked up too00:41
fungilifeless: limiting it to only run on changes to upper-constraints.txt will be more complicated since the files filter is per job rather than per job+project00:41
lifelessfungi: nitpicking but dvsm is such a sample too00:41
fungilifeless: fair point00:41
lifelessI'd move all three dsvm jobs under the comment00:41
fungiyep, doing now00:41
*** yonglihe has quit IRC00:41
pc_mlifeless: no, I found the issue in LBAAS, told them and we diagnosed it down to astroid, and found it in other repos, and weren't sure how to proceed, so I was asked to ping infra.00:41
lifelesspc_m: so, fungi and I are doing the infray things, please do look at the patches00:42
*** mriedem_away is now known as mriedem00:42
lifelesspc_m: can you take point on contacting the upstream astroid and telling them about this break00:42
lifelesspc_m: if its unintentional, they can fix it, if its deliberate, pylint probably needs to do a fix build of 1.4.4 with capped deps or something00:43
*** olaph has quit IRC00:43
jeblairmgagne: can you tell why 96ceaccd-69f7-424e-b7b5-bf961527b4d7 seems to be stuck in build state?00:43
*** eil397 has quit IRC00:43
*** ajmiller has quit IRC00:43
lifelesspc_m: I'd like a bug URL I can link to in my commit message to cap juno00:43
mgagnejeblair checking now00:43
pc_mlifeless: I can try... have never done that, so may need a few pointers on how to... certainly willing.00:43
lifelesspc_m: go to the astroid page on pypi - pypi.python.org/pypi/astroid00:44
lifelesspc_m: from there find the project home page00:44
lifelesspc_m: from that find the bugtracker00:44
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Run gate-neutron-pylint-constraints on global reqs  https://review.openstack.org/25159600:44
fungilifeless: ^00:44
lifelesspc_m: then file a bug saying that this release broke pylint 1.4.4 :)00:44
pc_mlifeless: OK. Will give it a shot...00:44
pc_mlifeless: Thanks for the help.00:45
lifelesspc_m: finally ping me with the url :)00:45
pc_mlifeless: We'll need to tell the neutron community (they have IRC tomorrow)00:45
pc_mlifeless: Starting on it now..00:46
*** sdake has joined #openstack-infra00:46
openstackgerritlifeless proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25159900:47
*** thorst has joined #openstack-infra00:47
*** thorst has quit IRC00:48
*** thorst has joined #openstack-infra00:48
*** spredzy has quit IRC00:48
*** spredzy has joined #openstack-infra00:49
mgagnejeblair did you delete the instance?00:50
zaromgagne: ok, i see.  so the ENV variable works as an override?00:50
mgagnezaro yes00:50
*** rebase has quit IRC00:51
zaromgagne: that's good to know actually.  I00:51
lifelessmordred: so setuptools_scm, what don't you like?00:51
mordredlifeless: having a second set of things to diagnose when they break00:51
zaromgagne: i've been doing the forge commiter by changing my .gitconfig file.00:51
mordredlifeless: the more importan tthing is my comment in the spec about the zuul regex00:52
zaromgagne: didn't know that env vars would work.00:52
mordredlifeless: specifically, I do not want to relax the constraint protections on tag triggers for non xstatic projects00:52
mgagnezaro yea, it's not what a user of jeepyb would figure out right away00:52
mordredlifeless: I think letting pbr grok 4-number versions is fine00:52
lifelessmordred: my concern is that then we'll have a raft of folk using 4 digit things locally and coming back here when it breaks00:52
*** olaph has joined #openstack-infra00:52
*** sabeen has quit IRC00:53
mordredlifeless: I'm more concerned with opening the door to being different if people feel like being different00:53
lifelessmordred: I am confident this will happen, because for instance, I know IBM have tried to use four component versions in pbr already00:53
mordredlifeless: xstatic has a specific use case00:53
mordredand it's fair and sane00:54
mordredlifeless: but it should not open the door to more people wanting to do the same thing if we can help it00:54
jeblairmgagne: no00:54
jeblairmgagne: i did delete 325c4ce9-8f98-4153-99e7-854df09da1af which is stuck in build/deleting00:55
jeblairmgagne: but 9c6e517e-eeaf-4740-804d-fd36926677c8 is in build/spawning00:55
lifelessmordred: I am agreeing00:55
openstackgerritlifeless proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25160000:55
lifelessmordred: I am saying that enabling pbr to handle it *is* opening the door00:55
mgagnejeblair 9c6e517e-eeaf-4740-804d-fd36926677c8 != 96ceaccd-69f7-424e-b7b5-bf961527b4d7.00:55
mgagnejeblair similar looking ID00:56
mordredlifeless: nod00:56
mordredlifeless: indeed.00:56
mordredoh. I said that twice :(00:56
*** salv-orl_ has joined #openstack-infra00:57
jeblairmgagne: gah, sorry, yes, that one was deleted00:57
lifelessmordred: nod00:57
mgagnejeblair so 3 instances stuck in spawning, 1 got deleted. right?00:57
jeblairmgagne: yes, 325c4ce9-8f98-4153-99e7-854df09da1af is still in building/deleting, 96ceaccd-69f7-424e-b7b5-bf961527b4d7 was deleted, and 9c6e517e-eeaf-4740-804d-fd36926677c8 is building/spawning.00:58
jeblairmgagne: none of them ever got to active00:59
jeblairmgagne: (and that's the order i launched them in)00:59
*** salv-orl_ has quit IRC00:59
mgagneok, I created one from myself (A1.8) and it went active in no time. checking yours now00:59
*** salv-orlando has quit IRC01:00
lifelesspc_m: fungi: https://review.openstack.org/251600 and https://review.openstack.org/#/c/251599/01:00
*** thorst has quit IRC01:00
*** thorst has joined #openstack-infra01:01
*** sridhar_ram has quit IRC01:03
mgagnejeblair not much in the logs. will restart a couple of services to make sure it's not related to a dropped AMQP session :-/01:05
*** thorst has quit IRC01:05
mgagnejeblair will test creating instances on my side too01:06
*** sdake has quit IRC01:06
jeblairmgagne: ok, let me know if you want me to delete / try again01:06
mgagnejeblair sure, go ahead01:06
*** dimtruck is now known as zz_dimtruck01:07
openstackgerritSean McGinnis proposed openstack-infra/reviewstats: Update Cinder project repos  https://review.openstack.org/25160301:07
*** olaph has quit IRC01:08
openstackgerritlifeless proposed openstack/requirements: Ignore more build temp dirs  https://review.openstack.org/25160401:09
jeblairmgagne: it looks like those 2 were actually deleted this time, 4c8fe3b7-c255-4c4b-a360-6b2b95b9cb80 is now in build state for 2 mins01:10
*** olaph has joined #openstack-infra01:10
*** sridhar_ram has joined #openstack-infra01:10
*** peterlisak has quit IRC01:11
mgagnemine is active now. same compute node as some of yours stuck in spawning01:11
*** peterlisak has joined #openstack-infra01:12
*** mriedem has quit IRC01:13
openstackgerritMerged openstack-infra/system-config: Copy the file that is installed  https://review.openstack.org/25159001:13
mgagnenothing stuck in amqp queues. getting interesting now01:14
mordredZOMG that took forever to merge01:14
mordrednibalizer: ^^01:14
*** pvaneck has quit IRC01:14
*** jyuso1 has joined #openstack-infra01:14
nibalizeralso i think our git farm is not getting puppeted01:15
mordrednibalizer: nothing was getting puppeted01:16
openstackgerritPedro Alvarez Piedehierro proposed openstack-infra/storyboard-webclient: RFC: Add tag/untag story to dashboard events.  https://review.openstack.org/25160501:16
pc_mlifeless: https://bitbucket.org/logilab/astroid/issues/275/140-and-141-fail-to-work-with-pylint-14401:16
nibalizerno things like status were puppettedb ~30 minutes ago01:16
*** baoli has joined #openstack-infra01:17
mordredthen we should investigate that01:17
mordrednibalizer: ansible 'git0*:!disabled' --list-hosts shows the right things01:18
anteayaarmax: nicely written01:18
*** zhenguo has joined #openstack-infra01:19
nibalizerlines like this appear in the log:01:19
nibalizer2015-12-01 00:50:50,398 p=21588 u=root |  git02.openstack.org        : ok=4    changed=2    unreachable=0    failed=001:19
*** sdake has joined #openstack-infra01:19
mordredyah. that seems like it ran01:19
mordredI'm thinking we should enable stdout/stderr output on changed successful runs01:20
pc_mlifeless: Will need to figure out how to handle neutron-lbaas, neutron-vpnaas, and probably neutron-fwaas.01:21
*** FallenPegasus has joined #openstack-infra01:21
*** openstackgerrit has quit IRC01:22
mordrednibalizer: http://puppetdb.openstack.org/report/git02.openstack.org/f6c4f6ff18328e88f9d78ca2c3c1ded76c48fcfe01:22
*** openstackgerrit has joined #openstack-infra01:22
*** dangers is now known as dangers_away01:27
*** slogan621 has joined #openstack-infra01:28
mgagnejeblair I'm enabling debugging on my side. will you be able to spawn new instances?01:28
slogan621Can someone confirm stackforge is no longer a part of OpenStack. The telemetry PTL implied this but looking for confirmation, since it is still mentioned http://docs.openstack.org/infra/manual/creators.html and there is still a wiki page01:29
mordredslogan621: the stackforge git namespace was retired01:29
slogan621so, are there "unofficial" projects still?01:30
mordredslogan621: it is still quite possible to have non-official projects and colloquially one could still refer to that as "stackforge"01:30
mordredvery many of them01:30
*** tiswanso has joined #openstack-infra01:30
mordredwe just don't segregate them by git namespace01:30
slogan621so, this implies easier for me to get something in the repo, but if I want to integrate it with something like horizon or ceilometer, would need to be official?01:31
slogan621or maybe up to the discretion of horizon PTL/reviewers?01:32
*** sdake has quit IRC01:32
mordredslogan621: depends on what you mean by "integrate"01:32
anteayaslogan621: horizon is asking folks to make their own ui repo01:33
anteayaslogan621: as they are full01:33
anteayabut best to ask the project you want to interact with how they want to structure it01:33
anteayaevery project is different01:34
anteayabut many are moving to a plug-in model01:34
slogan621well, in my case, want to contribute a project that can publish to ceilomter via OSLO. There is a yaml file I would want to modify in ceilometer to get it to notice my publications - so what I mean is maybe as an unofficial project I should have little resistance checking in my project, but I might face resistance from Ceilomter since I am changing their code - I would expect that01:34
slogan621anteaya: nod01:35
anteayasounds like you have resonable expectations01:35
anteayaI'd suggest starting by spending time with both ceilometer and oslo folks01:35
slogan621also might infer that being official might make it easier for a PTL to work woth me01:35
anteayaattend their weekly meetings01:35
anteayamight help your cause if you review some patches for them01:36
anteayathey like that01:36
slogan621so, then, what makes a project unofficial verses official, govenrance board votes you official?01:36
slogan621got it01:36
slogan621Ok, now it makes sense why stackforge is gone01:37
anteayaif you are listed here you are official: http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml01:37
slogan621it didn't really add anything01:37
anteayawell it felt like it did at the time01:37
anteayahowever with the direction of growth and tool choice it became unweildy01:37
anteayaas changing the name of a project in gerrit stackforge/ to openstack/ requires downtime01:38
anteayaand much code01:38
jeblairmgagne: 981868ed-4a37-497e-9450-68ec046c5b3901:38
anteayaso easier to just edit the file in the governance repo, no gerrit downtime01:39
mgagnechecking right now01:39
slogan621a yaml file listing the official projects seems like it might be a lot simpler to deal with01:39
anteayaturns out is is, for us anyway01:39
slogan621I assume bugs have been filed about purging references to stackforge and maybe clarifying the difference between official and unofficial01:40
anteayawell actually there is a case for keeping the reference01:40
slogan621I never would have guessed as an outsider01:40
fungislogan621: yeah, the options to become official are either to get one of the project teams listed at http://governance.openstack.org/reference/projects/index.html to accept the repo as officially part of their effort, or apply to the technical committee to form a new official project team for the repo by following http://governance.openstack.org/reference/new-projects-requirements.html01:40
*** tiswanso has quit IRC01:41
fungiand anything that doesn't do one of those is simply an unofficial repo01:41
*** pcrews has quit IRC01:41
*** thorst has joined #openstack-infra01:42
slogan621Does antyone face *any* resistance adding an unofficial repo if it is reasonable?01:42
funginot generally, no. we approve around half a dozen weekly i think01:42
fungii don't know the exact frequency at the moment but it's something like that anyway01:43
slogan621and the project owner would be the gating influence going forward I guess01:43
slogan621PTLs only apply to official projects also?01:44
fungiuntil such time as they decided to form a larger organic team or ally with some existing team anyway01:44
slogan621or would I be my own PTL?01:44
armaxthanks anteaya01:44
slogan621fungi: thanks01:45
fungifrom an openstack terminology perspective, a ptl is an elected representative leader for a particular project team. openstack doesn't officially define terms for unofficial repos, so the question is sort of moot01:45
slogan621anteaya: thanks01:45
slogan621got it01:45
fungiyou can _call_ yourself a ptl, but openstack doesn't have a say in what that means in your unofficial situation either way01:46
*** markvoelker has quit IRC01:46
anteayaslogan621: welcome, I highly encourage getting to know the folks in the projects you want to interact with01:46
jeblairmgagne: that was delelted. 981868ed-4a37-497e-9450-68ec046c5b39 is building now.  i've stopped nodepool, so should be no further actions on that one01:46
slogan621one final question: I guess other than gerrit, I wouldn't be required to set up Jenkins/CI (I might be foolish not to, but no one is going to require it)01:47
mgagneok, trying to figure out why nova-compute gets the request but does nothing01:47
anteayaslogan621: well you need tests on your repo, but you use our ci if you are an openstack repo01:47
*** pcrews has joined #openstack-infra01:47
slogan621Ok, so CI is required as are tests, got it01:48
anteayaslogan621: follow the instructions fungi linked it will tell you how to edit this file: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml01:48
anteayaslogan621: you create the tests, we run the ci01:48
anteayathat is the infra team mission01:48
fungianteaya: i didn't link the instructions you thought i did01:48
anteayafungi: oh sorry01:48
anteayaI did not01:48
fungii linked the technical committee requirements for a new project-team application01:49
slogan621http://docs.openstack.org/infra/manual/creators.html right?01:49
anteayaslogan621: well that is to create your own repo01:49
fungislogan621: you can choose to run some of our canned job templates on your changes or just run no-op jobs01:49
anteayabut I highly suggest you start with the developers guide01:49
fungior define new jobs for us to run01:49
anteayaand learn our workflow helping out ceilometer and oslo for a bit01:49
anteayaslogan621: it will help your social contacts and save you a world of pain of you try to set everything up without knowing our workflow01:50
fungiplugging in your own external ci is possible, though somewhat limited in how it can interact with our systems (it can't block your changes from merging for example), and is a lot of work so generally not recommended unless you really, really can't run all the things you need within our ci01:50
mgagnejeblair is it nodepool that is deleting the instance after X time?01:50
*** yamamoto has joined #openstack-infra01:51
anteayaspeaking of which, I need to sleep so I can wake up again and chair the next meeting01:51
slogan621I totally appreciate your CI and expect to use it.01:51
slogan621just wondering if it has to be in place for an initial contribution on day zero or can be added at a later time01:51
fungiit can always be added later01:52
slogan621Okay, that's good for us01:52
fungiwhen you get into the meat of the configuration options in the project creators guide, it'll tell you how to just use no-op jobs as placeholders for eventual jobs01:52
fungias one possible option anyway01:52
mgagnejeblair I've disabled a compute node to see if the fault is related to this one or not.01:53
*** julim has joined #openstack-infra01:53
mgagnejeblair scheduler was always sending it to the same so it's related to compute node, we might always get same result after each try01:53
slogan621so I think the no-ops get replaced once we start working with ceilometer team01:53
fungislogan621: sure, that's one possibility01:54
slogan621we can let them guide us perhaps on the set of tests that they would want us to include, and possibly, the testing happens in ceilometer01:54
fungithough you can also consider running simple ci jobs like style checks, documentation builds or unit tests. not everything we run is an integration test01:55
slogan621unit tests for sure01:55
*** rhallisey has quit IRC01:57
openstackgerritlifeless proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25159901:59
openstackgerritlifeless proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25160001:59
lifelessfungi: pc_m: ^ updated with the url01:59
*** salv-orlando has joined #openstack-infra02:00
*** shashank_hegde has quit IRC02:00
lifelessfungi: http://logs.openstack.org/96/251596/2/check/gate-project-config-layout/2c34ac5/console.html02:01
*** EricGonczer_ has joined #openstack-infra02:01
lifeless2015-12-01 00:55:06.218 | Job gate-neutron-pylint-constraints not defined02:01
lifelessfungi: isn't it pep8-constraints?02:02
fungithey run pylint in their pep8 target? fun times!02:02
lifelessfungi: oath02:02
fungiwell i'll be, they do indeed02:04
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Run gate-neutron-pep8-constraints on global reqs  https://review.openstack.org/25159602:05
lifelesspc_m: now, once the global-requirements changes are in02:06
lifelesspc_m: then you can add astroid to test-requirements.txt in neutron kilo and juno02:06
lifelesspc_m: you can add the patches for that now using Depends-On02:06
*** tiswanso has joined #openstack-infra02:07
*** salv-orlando has quit IRC02:07
lifelesspc_m: which leaves just the neturon ass jbos02:07
lifelesspc_m: for those, they need to get -constraints up02:07
*** tiswanso has quit IRC02:08
*** rlandy has quit IRC02:08
openstackgerritMerged openstack-infra/subunit2sql: Fix test attachments DB import  https://review.openstack.org/25149902:08
*** tiswanso has joined #openstack-infra02:08
*** signed8bit_ZZZzz is now known as signed8bit02:10
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Set max_microversion on Tempest for microversion testing  https://review.openstack.org/24727802:11
*** hdd has joined #openstack-infra02:13
openstackgerritTushar Gohad proposed openstack-infra/project-config: Add liberasurecode to CI slaves  https://review.openstack.org/25160702:13
mgagnejeblair leaving the office now, will continue debug tomorrow02:15
*** rfolco has joined #openstack-infra02:21
openstackgerritRamy Asselin proposed openstack-infra/puppet-elastic_recheck: Add conf options for uncategorized.py  https://review.openstack.org/25152202:22
*** tiswanso_ has joined #openstack-infra02:23
*** tiswanso has quit IRC02:23
*** tiswanso_ has quit IRC02:24
*** tiswanso has joined #openstack-infra02:25
*** EricGonczer_ has quit IRC02:27
tsglifeless: submitted https://review.openstack.org/#/c/251607/ for bindep-fallback.  let me know if this is adequate02:28
*** shashank_hegde has joined #openstack-infra02:28
*** sridhar_ram has quit IRC02:30
*** mtanino has quit IRC02:30
phschwartzAnyone familiar with dib and nodepool on? http://paste.openstack.org/show/480448/02:31
phschwartzI can't seem to figure out why it fails to build images02:31
mgagnephschwartz have more than 4GB of ram?02:34
phschwartzmgagne: the system the build is going on has 32gb02:35
phschwartzMemTotal:       32948980 kB02:35
mgagnephschwartz hehe, didn't read carefully02:36
lifelesstsg: shouldn't it be in the other patch ?02:36
*** sabeen has joined #openstack-infra02:36
tsglifeless: isn't bindep-fallback part of project-config?02:37
tsglifeless: vs system-config (the other change)?02:37
lifelessdidn'trealise yes02:37
lifelesswill look at it in a little bit02:37
tsglifeless: ok .. :) so I do need the system-config change also I assume02:38
tsglifeless: if not, I will post the project-config change w/ the original ID02:38
fungitsg: yes, both changes are needed02:38
tsgfungi: thanks.  So far I am changing bindep-fallback.txt, jenkins_params.pp, thick_slave.pp02:39
fungisystem-config is what's handling preinstallation of distro packages on the job workers, project-config will be handling job run-time distro package installation when we transition to that model02:39
fungihence the need to keep them in sync for the time being02:39
tsgfungi: got it .. that would mean the jenkins_param, thick_slave stuff would go away?02:39
tsgmakes sense02:39
*** akshai has quit IRC02:39
tsgfungi: also lifeless mentioned about "per-project" files - when is that change planned?02:40
ianwphschwartz: the problem is probably before that ... dib does a lot of cleanup after a failure so the real issue is often very far back in the logs02:40
*** thorst has quit IRC02:41
*** thorst has joined #openstack-infra02:41
*** unicell1 has quit IRC02:42
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Make Elastic Recheck Watch more reusable  https://review.openstack.org/25151502:46
*** thorst has quit IRC02:50
*** rfolco has quit IRC02:51
openstackgerritRamy Asselin proposed openstack-infra/puppet-elastic_recheck: Add e-r bot optional configurations  https://review.openstack.org/25161002:52
SpamapSphschwartz: also just FYI, diskimage-builder's official home is #tripleo02:53
phschwartzSpamapS: that I didn't know. ty for the heads up :)02:54
*** sputnik13 has quit IRC02:56
*** fawadkhaliq has joined #openstack-infra02:57
*** yamahata has quit IRC02:58
stevemarcould someone in project-config core +A this patch: https://review.openstack.org/#/c/251200/ we were just waiting for the depends-on patches to merge :)03:00
*** hdd has quit IRC03:00
openstackgerritMerged openstack-infra/reviewstats: Update Cinder project repos  https://review.openstack.org/25160303:01
*** salv-orlando has joined #openstack-infra03:01
*** tsg has quit IRC03:05
*** salv-orlando has quit IRC03:09
SpamapS| Qcache_hits                                   | 25             |03:09
SpamapS| Qcache_inserts                                | 1758354        |03:09
SpamapSyeeeahhhh... we should.. turn off that cache.03:09
*** EricGonczer_ has joined #openstack-infra03:11
mordredSpamapS: that is a very bad hit rate03:11
SpamapSmordred: you're a very bad hit rate03:11
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Allow queue regex specification  https://review.openstack.org/25162203:14
pc_mlifeless: Had more questions.03:15
pc_mlifeless: I see I can't add astroid to test-requirements.txt for VN and LB (I guess because it is not in global requirements for master)?03:16
pc_mlifeless: Also, even though pep8 runs in gate, due to constraints, local pep8 runs on neutron fail. Won't we need it in global-requirements?03:17
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Better duplicate detection for project requirements check  https://review.openstack.org/23887103:18
pc_mlifeless: lastly, I see that pyling 1.5.0 changes required (in one case for VPN) are backward incompatible with 1.4.4. If we are using constraints, won't there be breakage, until both project and constraints projects are updated?03:19
mordrednibalizer: ok. looking at the logs puppet runs seem happy now03:19
pc_mlifeless: Is there a way to have local test-requirements override the upper constraints file, as a solution for that?03:19
ruagairDoes any one recall who else in here was working on using mod_auth_openid03:20
mordrednibalizer: AND - ansible is correctly not attempting to run hosts that are disabled03:20
ruagairthe openid module for apache03:20
mordredruagair: I do not03:20
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Better duplicate detection for project requirements check  https://review.openstack.org/23887103:21
ruagairWhile it plays nicely with Phabriator and login.ubuntu, I noticed it's abandoneware.03:21
*** sabeen2 has joined #openstack-infra03:21
lifelesspc_m: run pep8-constraints locally03:21
mordredruagair: yes. if it works, we'll just adopt it03:22
*** sabeen has quit IRC03:22
ruagairCPP ?03:22
ruagairC++, I mean03:22
ruagairgluttons :-)03:22
mordredyup. we're crazy :)03:22
pc_mlifeless: Are there pointers on how to update tox to work with the new target?03:22
mordredbut if it works, then _awesome_ we don't have to write it03:22
ruagairIt *works* in all my testing so far.03:23
*** yuanying has quit IRC03:23
mordredthat's awesome03:23
mordredand one of the best things I've heard in a while03:23
ruagairAs for vuln and other issues... no idea.03:23
lifelesspc_m: I have run; Nakato can give you pointers, and in the next couple weeks we hope to have a manual around this03:23
lifelesspc_m: neutron tox is updated already03:23
ruagairThere's some issues with login.ubuntu that I'll cover off tomorrow infra meeting.03:23
pc_mlifeless: yes, looking at LB and VPN, which are broken still.03:24
*** pcrews has quit IRC03:24
pc_mlifeless: Also, have my concern on backward incompatible pylint changes.03:24
lifelesspc_m: there was a discussion here last week about that; theres some overide the aas things do to tox's install_command - basically that has to be undone03:24
lifelesspc_m: I don't have an answer to that yet, but if astroid gets fixed there is no problem. Until then, we don't migrate?03:25
*** pcrews has joined #openstack-infra03:25
pc_mlifeless: beyond the breakage. I see that fix needed for 1.5.0 pylint is not backward compatible with
pc_mFound one so far.03:26
pc_mlifeless: so I think to summarize...03:27
pc_mlifeless: Need astroid pin for kilo/juno on neutron (and likely vpnaas and lbaas, and maybe fwaas)03:27
pc_mlifeless: Need to setup pep8-constraints for LB and VPN (and FW?)03:28
pc_mNeed to confirm pep8-constraints can run locally for neutron, and add for LB and VPN.03:28
pc_mlifeless: Right?03:28
openstackgerritRamy Asselin proposed openstack-infra/puppet-elastic_recheck: Include all gate pipelines  https://review.openstack.org/25162403:30
*** thorst has joined #openstack-infra03:32
phschwartzyolanda: asselin: Have either of you noticed that the script-dir set for nodepool by default from openstackci is not the correct path for the scripts? It clones them to /etc/nodepool/scripts which has a subdir scripts containing the scripts while dib expects them in /etc/nodepool/scripts not a subdir03:32
*** thorst has quit IRC03:33
*** thorst has joined #openstack-infra03:34
*** ayoung has quit IRC03:35
asselinphschwartz, no I haven't noticed03:36
*** yamahata has joined #openstack-infra03:36
*** Kennan has quit IRC03:39
*** links has joined #openstack-infra03:39
asselinphschwartz, I see all the scripts in the right place: http://paste.openstack.org/show/480451/03:41
openstackgerritHaomeng,Wang proposed openstack-infra/project-config: Skip unnecessary tests for bifrost docs changes  https://review.openstack.org/25162503:41
SpamapSruagair: FYI I used mod_auth_openid in a POC a few years back and it went very smoothly.03:42
*** thorst has quit IRC03:42
*** Sree has joined #openstack-infra03:43
openstackgerritHaomeng,Wang proposed openstack-infra/project-config: Skip unnecessary tests for bifrost docs changes  https://review.openstack.org/25162503:43
*** aopincar has joined #openstack-infra03:44
*** EricGonczer_ has quit IRC03:44
*** Sree has quit IRC03:44
*** Sree has joined #openstack-infra03:45
mordredinfra-root: (and anyone else) https://gist.github.com/emonty/6d3c51605b5ab65adf1e is the output of the openstack inventory on puppetmaster03:45
mordred(used gist because pastebinit truncated the data)03:45
jheskethmordred: shiny, am I looking for something in particular?03:46
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use OpenStack for inventory instead of puppet  https://review.openstack.org/24993003:46
*** baoli has quit IRC03:46
mordredjhesketh: not _really_ - I've verified that it's operating like we'd want/expect03:46
mordredjhesketh: but, that's all the information that's availabe to us in playbooks, should we want to use it03:47
mordredjhesketh: there's tons of groupings of servers which are almost completely unreadable because it's all uuid based03:47
*** baoli has joined #openstack-infra03:47
nibalizermordred: success!03:48
mordred(we may want to write a wrapper utility at some point to allow us to to uuid/hostname mappings for times when we're doing host mask checking03:49
mordredfwiw, that file will be in /var/cache/ansible-inventory/ansible-inventory.cache03:49
ruagairIt's gone rather smooth;y for me to SpamapS, was just it's current status of "abandonoed" that concerned me.03:50
SpamapSruagair: yeah we have people who can maintain it03:50
ruagairbut we got over that :-)03:50
SpamapSmight even be fun :)03:50
ruagairMasochist ;-)03:50
SpamapSwe all need some C++ in our lives to remind us why we should be using Rust.03:51
phschwartzrust makes me cringe03:51
*** signed8b_ has joined #openstack-infra03:51
ruagairYou sound like Angus Lees, SpamapS03:52
ruagairAlthough others would say Haskell.03:52
SpamapSruagair: I'll take that compliment.03:52
ruagairOh, it wasn't derogatory.03:52
SpamapSNever tried Haskell.03:52
SpamapSI barely have time for Rust :-P03:52
ruagairI like it.03:52
ruagairA few friends are rather committed to it :-)03:52
ruagairAlmost religiously.03:53
phschwartzI am not a fan of functional programming after doing it for years. so I ran from haskell as fast as possible03:53
*** mahatic has joined #openstack-infra03:54
phschwartzI don't like the syntax of rust, if I was going the route of another compiled language I seem to like go much more.03:54
*** signed8bit has quit IRC03:54
SpamapSexcept go doesn't do any of the things rust does for you ;)03:54
Nakatopc_m: I'm not really across the pep8 issue you are having, but if you run $(tox -e pep8-constraints) on a developer workstation for neutron it'll run the constrained version.   Ihar has been looking at this work for neutron-*aas but there's some issues with how it installs things that needs to be worked through before that work can land.03:54
Nakatopc_m: -constraints is only stable/Liberty and Master03:54
SpamapS(it does make it easy to write compiled programs that have native concurrency built in, which is nice)03:55
phschwartzSpamapS: only thing rust really gives you is better type safety and a built in reference counted memory manager03:56
SpamapSthats a prety big only :)03:56
SpamapSI'm waiting for jo (go that compiles into the jvm) ;)03:57
SpamapSso go can have a modern GC03:57
*** rguillebert has quit IRC03:57
* SpamapS goes off to see if his family wants to talk about reference counted memory managers03:57
phschwartzlol, go forces everything not on the stack to the gc which doesn't give you the same memory control but makes your programs safer from leaks.03:57
lifelesspc_m: yes04:03
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use OpenStack for inventory instead of puppet  https://review.openstack.org/24993004:03
lifelesspc_m: pylint 5 will have to wait04:03
*** aspiers has quit IRC04:04
lifelesspc_m: 1.5 that is04:04
mordrednibalizer, jhesketh: ^^ that's updated a little bit based on empirical testing on puppetmaster04:04
*** tsg has joined #openstack-infra04:04
mordredwith a mild change to the way disabled hosts get registered, after I learned something about static/dynamic inventory mixing interactions04:05
*** aopincar has quit IRC04:05
lifelesspc_m: you can probably work around things temporarily with deps= lines in tox.ini, though thats not something we like using04:05
*** yuanying has joined #openstack-infra04:07
openstackgerritMerged openstack-infra/project-config: Update to Fedora 23  https://review.openstack.org/24790804:13
openstackgerritMerged openstack-infra/project-config: Remove F22 nodes  https://review.openstack.org/24894004:13
fungitsg: yeah, with the switch to installing packages during the job on non-thick worker types, the thick class will become unused and will eventually be cleaned up/deleted. per-project package selection will be available as soon as jobs are converted to install packages themselves (well, in reality it can already be done we're just adding a framework to simplify it and make it self-testing)04:14
mordredfungi: you might want to take a gander at https://review.openstack.org/249930 too - mild change to the earlier docs about disabling things04:14
nibalizermordred: weird04:14
mordrednibalizer: right?04:15
fungiruagair: clarkb was trying mod_auth_openid for a separate project we're working on for some foundation staff stuff04:15
tsgfungi: cool .. sounds like this would simplify things a lot04:15
mordrednibalizer: I'm also poking fokls in ansible to see if we can write a patch to move group/instance validation to _after_ inventory merge04:15
*** links has quit IRC04:17
tsgfungi: I am hoping we can get liberasurecode in all the needed places soon, so a few Swift patches can be unblocked.  It will be great if you can take a look at patch 247936 and patch 251607 to see is that's all that's needed for now.04:18
fungimordred: thanks for the heads up04:19
tsgfungi: I understand that these changes are especially hard given the Jenkins builds don't run after integrating these patches :) so let me know if there is any local testing I need to provide04:20
phschwartzWhen I do a nodepool dib-image-list it shows my 2 images in building state. Is there any way I can verify that as they have been in that state for almost 1 hour04:22
openstackgerritMonty Taylor proposed openstack-infra/system-config: Fix three nits  https://review.openstack.org/25137404:26
tsgfungi: thanks for the +2 on bindep change.  I wish there was a way to specify platform:>=fedora-22 :-)04:27
fungitsg: yeah, well we're about to not have fedora<23 or centos<7 so we can soon just switch that to platform:rpm04:28
tsgfungi: sounds good!04:29
fungisince it simply needs to define package names in terms of the platforms we actually run in our ci04:29
* tsg nods04:29
fungisimilarly, the first line could just use platform:dpkg since it's explicitly listing our only two dpkg platforms04:29
fungione of which we'll also be dropping very soon04:29
*** baoli has quit IRC04:30
tsgfungi: ah ok .. I wasn't sure about the "list of supported platforms" - I assumed all the ubuntu/debian flavors04:30
fungibut what you have there is sufficient and correct enough for now04:30
tsgfungi: that's good to hear!04:30
fungianyway, i need to get some sleep04:30
tsgfungi: do I also need to update other-requirements or another list of pkgs that pbr maintains?04:30
fungitsg: shouldn't be necessary, no04:31
tsgfungi: finally :) .. that's great to hear04:31
fungitsg: eventually swift could specify its own other-requirements.txt file so that its jobs use that instead of the fallback fil04:31
tsgfungi: that would be much nicer04:31
fungibut first we want to get things working with the fallback04:32
tsgfungi: looking forward to that - notmyname ^^^04:32
fungianyway, g'night all!04:32
tsgfungi: thanks again and have a good night!04:32
*** otter768 has quit IRC04:35
*** Kennan has joined #openstack-infra04:35
ruagairah, thanks fungi04:36
*** thorst has joined #openstack-infra04:41
phschwartzif nodepool fails on an image build does it loop and retry the build?04:44
phschwartzIf it does that accounts for the issue I am seeing04:44
mordredphschwartz: it does retry, yes04:47
mordredphschwartz: I don't remember how tight of a loop04:47
phschwartzmordred: not that tight :( It is on the 9th attempt to rebuild my 2 images04:48
phschwartzI can't seem to find out why they are failing to build04:48
*** thorst has quit IRC04:49
openstackgerritHaomeng,Wang proposed openstack-infra/project-config: Skip unnecessary tests for bifrost docs changes  https://review.openstack.org/25162504:50
phschwartzmordred: the dib run logs everything as INFO so it makes it hard to search for actual fatal errors04:50
*** achanda has joined #openstack-infra04:51
*** gyee has quit IRC04:52
*** tlian2 has quit IRC04:53
*** rkukura has joined #openstack-infra04:58
openstackgerritMerged openstack/requirements: Updates pika-pool library  https://review.openstack.org/24920604:58
*** hdd has joined #openstack-infra04:58
openstackgerritMerged openstack/requirements: Add kolla-mesos to the projects list  https://review.openstack.org/25117104:59
*** sdake has joined #openstack-infra05:00
openstackgerritMerged openstack/requirements: Add '.venv' to .gitignore  https://review.openstack.org/24968005:00
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/24759705:00
*** sridhar_ram has joined #openstack-infra05:00
*** sdake_ has joined #openstack-infra05:03
*** aeng has quit IRC05:04
*** sdake has quit IRC05:04
*** changbl has quit IRC05:07
*** sdake_ has quit IRC05:10
*** yamahata has quit IRC05:10
*** rkukura has quit IRC05:11
*** tiswanso has quit IRC05:14
phschwartzanyone want to play pin the tail on the dib error once my log is uploaded?05:24
*** aeng has joined #openstack-infra05:25
openstackgerritMerged openstack-infra/subunit2sql: Add missing relationship from orm models  https://review.openstack.org/25003405:25
*** FallenPegasus has quit IRC05:26
*** amotoki has joined #openstack-infra05:26
*** FallenPegasus has joined #openstack-infra05:26
*** Ramanjaneya has joined #openstack-infra05:27
openstackgerritMerged openstack/requirements: Bump version of requests-mock  https://review.openstack.org/24900605:34
openstackgerritMerged openstack/requirements: Add tosca-parser to the global requirments projects  https://review.openstack.org/24895005:34
*** links has joined #openstack-infra05:35
*** fawadkhaliq has quit IRC05:36
*** bexelbie_ has joined #openstack-infra05:37
*** bexelbie_ is now known as bexelbie05:37
*** gildub has joined #openstack-infra05:46
*** thorst has joined #openstack-infra05:47
*** thorst has quit IRC05:54
*** tsg has quit IRC05:55
*** kozhukalov_ has joined #openstack-infra05:57
*** harlowja_at_home has joined #openstack-infra05:57
*** abregman has joined #openstack-infra05:57
*** fawadkhaliq has joined #openstack-infra05:58
*** fawadkhaliq has quit IRC05:58
*** fawadkhaliq has joined #openstack-infra05:59
*** mrmartin has joined #openstack-infra06:04
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Congress: Add different gate job for stable branch  https://review.openstack.org/25059806:05
*** hdd has quit IRC06:08
*** salv-orlando has joined #openstack-infra06:09
*** dims has quit IRC06:09
*** sridhar_ram has quit IRC06:09
*** ildikov has quit IRC06:12
*** sdake has joined #openstack-infra06:12
*** dims has joined #openstack-infra06:14
*** salv-orlando has quit IRC06:14
openstackgerritDeepak C Shetty (deepakcs) proposed openstack-infra/project-config: Enable devstack-plugin-ceph jobs  https://review.openstack.org/25142106:15
openstackgerritMerged openstack-infra/project-config: Remove gate-neutron-python34 in favor of -constraints  https://review.openstack.org/25016906:15
openstackgerritMerged openstack-infra/project-config: Skip unnecessary tests for bifrost docs changes  https://review.openstack.org/25162506:15
*** deepakcs has joined #openstack-infra06:16
deepakcsHi, can any core give +A to https://review.openstack.org/#/c/251197/06:16
*** bexelbie has quit IRC06:16
deepakcsit already has +2 and +1, thanks in advance06:16
deepakcsSergeyLukjanov: ^^06:17
*** ihrachys has joined #openstack-infra06:18
*** ihrachys has quit IRC06:20
*** jaosorior has joined #openstack-infra06:21
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Properly date filter in get_test_run_dict_by_run_meta_key_value  https://review.openstack.org/25167306:23
*** sabeen2 has quit IRC06:28
openstackgerritFabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements  https://review.openstack.org/25167406:32
*** harlowja_at_home has quit IRC06:34
*** pcrews has quit IRC06:34
*** bexelbie has joined #openstack-infra06:34
*** otter768 has joined #openstack-infra06:36
fricklerphschwartz: are you talking about http://nodepool.openstack.org/image.log ? That seems to fail repeatedly on building pycrypto, in fact I guess some infra-root should stop it from cycling06:36
fricklergcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directo06:36
phschwartzfrickler: I was talking about on a local version of that relating to a local nodepool setup06:37
phschwartzI traced down the error and I am working to fix it now06:37
*** sdake has quit IRC06:38
deepakcsianw: ^^06:38
*** sc68cal has quit IRC06:40
*** otter768 has quit IRC06:40
*** mrmartin has quit IRC06:43
*** sc68cal has joined #openstack-infra06:44
*** aeng has quit IRC06:45
*** yamahata has joined #openstack-infra06:50
*** FallenPegasus has quit IRC06:51
*** thorst has joined #openstack-infra06:52
*** amit213 has quit IRC06:55
*** amit213 has joined #openstack-infra06:56
*** aeng has joined #openstack-infra06:57
*** thorst has quit IRC06:59
*** achanda_ has joined #openstack-infra07:00
openstackgerritGal Sagie proposed openstack-infra/irc-meetings: Add Dragonflow weekly meeting  https://review.openstack.org/25136907:00
*** gsagie has joined #openstack-infra07:01
*** gsagie has left #openstack-infra07:01
*** achanda has quit IRC07:03
*** btully has quit IRC07:04
*** aopincar has joined #openstack-infra07:05
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/25108807:05
*** nadya_ has joined #openstack-infra07:07
*** [2]Thelo has joined #openstack-infra07:12
*** ParsectiX has joined #openstack-infra07:12
*** nadya_ has quit IRC07:12
*** gildub has quit IRC07:13
*** [1]Thelo has quit IRC07:14
*** aburmashev has joined #openstack-infra07:17
aburmashevmmedvede: hi, wanted to thank you for the advice. Actually removed /opt/nodepool and rerun puppet apply -> nodepool was installed OK :)07:18
*** jaosorior has quit IRC07:25
*** jaosorior has joined #openstack-infra07:26
*** Ramanjaneya has quit IRC07:30
*** HeOS has quit IRC07:31
*** olaph has quit IRC07:31
*** kozhukalov_ has quit IRC07:35
*** achanda_ has quit IRC07:36
*** binbincong has joined #openstack-infra07:45
yolandaphschwartz, i cannot tell you, as i don't use the one node setup07:46
*** mpryc has joined #openstack-infra07:49
*** fawadkhaliq has quit IRC07:52
openstackgerritMaty Grosz proposed openstack-infra/project-config: Add Vitrage to specs  https://review.openstack.org/25169007:53
*** fawadkhaliq has joined #openstack-infra07:53
*** fawadkhaliq has quit IRC07:53
*** fawadkhaliq has joined #openstack-infra07:54
*** fawadkhaliq has quit IRC07:57
*** fawadkhaliq has joined #openstack-infra07:57
*** thorst has joined #openstack-infra07:57
*** fawadkhaliq has quit IRC07:57
*** fawadkhaliq has joined #openstack-infra07:58
*** fawadkhaliq has quit IRC08:02
*** fawadkhaliq has joined #openstack-infra08:03
*** bexelbie has quit IRC08:03
*** fawadkhaliq has quit IRC08:03
*** fawadkhaliq has joined #openstack-infra08:04
*** thorst has quit IRC08:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add liberasurecode to CI slaves  https://review.openstack.org/25160708:08
*** coolsvap has quit IRC08:09
*** FallenPegasus has joined #openstack-infra08:10
*** jlanoux has joined #openstack-infra08:10
openstackgerritMerged openstack-infra/subunit2sql: Add seperate page for v1.0.0 release notes  https://review.openstack.org/25006608:12
*** ildikov has joined #openstack-infra08:13
*** zeih has joined #openstack-infra08:13
*** aopincar has quit IRC08:13
*** aopincar has joined #openstack-infra08:15
*** arxcruz has joined #openstack-infra08:17
*** signed8b_ has quit IRC08:18
*** markus_z has joined #openstack-infra08:19
*** nadya_ has joined #openstack-infra08:20
*** placko has joined #openstack-infra08:20
*** kushal has joined #openstack-infra08:22
*** nmagnezi has joined #openstack-infra08:22
*** achanda has joined #openstack-infra08:23
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add delete *.pyc command before executing ostestr  https://review.openstack.org/25170108:26
*** HeOS has joined #openstack-infra08:27
*** hichihara has quit IRC08:27
*** nadya__ has joined #openstack-infra08:28
*** nadya_ has quit IRC08:28
*** shardy has joined #openstack-infra08:29
*** mpavone has joined #openstack-infra08:30
*** deepakcs has quit IRC08:32
*** fhubik has joined #openstack-infra08:32
*** dbite has joined #openstack-infra08:33
*** nadya_ has joined #openstack-infra08:34
*** ifarkas has joined #openstack-infra08:36
*** Hal has joined #openstack-infra08:36
*** nadya__ has quit IRC08:36
*** Hal is now known as Guest3008108:36
*** rcernin has joined #openstack-infra08:37
*** otter768 has joined #openstack-infra08:37
*** fhubik is now known as fhubik_brb08:37
*** mrmartin has joined #openstack-infra08:37
*** fhubik_brb is now known as fhubik08:38
*** sshnaidm has joined #openstack-infra08:40
*** otter768 has quit IRC08:41
*** aspiers has joined #openstack-infra08:41
*** Ramanjaneya has joined #openstack-infra08:43
*** dtantsur|afk is now known as dtantsur08:44
*** dizquierdo has joined #openstack-infra08:45
*** nadya_ has quit IRC08:45
*** jlanoux has quit IRC08:45
*** rossella_s has joined #openstack-infra08:45
*** nadya_ has joined #openstack-infra08:45
*** EricGonczer_ has joined #openstack-infra08:47
*** jlanoux has joined #openstack-infra08:48
*** dingyichen has quit IRC08:50
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Add CI coverage of deleting the deployed overcloud  https://review.openstack.org/25049808:53
openstackgerritMerged openstack-infra/subunit2sql: Add dates to x-axis for subunit2sql graphs  https://review.openstack.org/24664108:54
*** kozhukalov_ has joined #openstack-infra08:56
*** btully has joined #openstack-infra08:57
openstackgerritMerged openstack-infra/subunit2sql: Add support to subunit2sql cli to specify a run_at time  https://review.openstack.org/25145608:57
openstackgerritAndrea Frittoli proposed openstack-infra/subunit2sql: Add docstring for run_at  https://review.openstack.org/25171708:57
*** dguitarbite has quit IRC08:58
*** dbite has quit IRC08:58
*** fhubik is now known as fhubik_brb08:58
*** dguitarbite has joined #openstack-infra08:58
*** nadya__ has joined #openstack-infra08:59
*** gmann has quit IRC09:01
*** mpavone has quit IRC09:02
*** derekh has joined #openstack-infra09:02
*** fawadkhaliq has quit IRC09:03
*** gmann has joined #openstack-infra09:03
*** nadya_ has quit IRC09:03
*** fawadkhaliq has joined #openstack-infra09:03
*** ildikov has quit IRC09:04
*** thorst has joined #openstack-infra09:04
*** nadya__ has quit IRC09:04
*** coolsvap has joined #openstack-infra09:04
*** ildikov has joined #openstack-infra09:06
*** thorst has quit IRC09:09
*** jcoufal has joined #openstack-infra09:10
*** fhubik_brb is now known as fhubik09:13
*** nadya_ has joined #openstack-infra09:13
*** gmann has quit IRC09:13
*** Sree_ has joined #openstack-infra09:15
*** Sree_ is now known as Guest5589209:15
*** mpavone has joined #openstack-infra09:16
*** Sree has quit IRC09:18
*** dizquierdo has quit IRC09:18
*** btully has quit IRC09:25
*** EricGonczer_ has quit IRC09:27
*** jordanP has joined #openstack-infra09:29
*** yuanying has quit IRC09:30
*** EricGonczer_ has joined #openstack-infra09:32
*** hashar has joined #openstack-infra09:34
*** EricGonczer_ has quit IRC09:34
*** salv-orlando has joined #openstack-infra09:35
*** salv-orlando has quit IRC09:35
*** salv-orlando has joined #openstack-infra09:35
openstackgerritMerged openstack-infra/project-config: Switch Designate DSVM jobs to use a DevStack Plugin  https://review.openstack.org/25089509:36
*** achanda has quit IRC09:38
*** olaph has joined #openstack-infra09:38
*** bexelbie has joined #openstack-infra09:43
*** kozhukalov_ has quit IRC09:44
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: DNM debug stable/liberty instack-undercloud  https://review.openstack.org/25174709:52
*** jistr has joined #openstack-infra09:52
*** olaph has quit IRC09:54
*** olaph has joined #openstack-infra09:57
*** bexelbie has quit IRC09:58
*** Ramanjaneya has quit IRC09:58
*** kozhukalov_ has joined #openstack-infra09:58
*** nmagnezi has quit IRC09:59
openstackgerritMerged openstack-infra/project-config: Run gate-neutron-pep8-constraints on global reqs  https://review.openstack.org/25159610:01
*** bexelbie has joined #openstack-infra10:02
*** megm has quit IRC10:02
*** zhenguo has quit IRC10:04
*** megm has joined #openstack-infra10:04
*** rguillebert has joined #openstack-infra10:05
*** thorst has joined #openstack-infra10:07
*** nmagnezi has joined #openstack-infra10:07
*** olaph has quit IRC10:08
*** wuhg has joined #openstack-infra10:09
*** mahatic has quit IRC10:10
*** olaph has joined #openstack-infra10:11
openstackgerritMerged openstack-infra/puppet-log_processor: Switch to using the new log_processor project  https://review.openstack.org/24999310:12
*** wuhg has left #openstack-infra10:12
*** lhcheng has joined #openstack-infra10:14
*** thorst has quit IRC10:14
lhchenghi folks10:15
openstackgerritMerged openstack-infra/puppet-gerrit: Allow setting the number of threads for indexing Gerrit changes  https://review.openstack.org/24106010:15
lhchengwe added the reno job for horizon a week ago, it seems like it is still not getting kicked off10:16
lhchengany idea?10:16
openstackgerritMerged openstack-infra/irc-meetings: Add Dragonflow weekly meeting  https://review.openstack.org/25136910:16
openstackgerritMerged openstack-infra/system-config: Add metrics as a reason to contribute a cloud  https://review.openstack.org/24101110:17
*** ParsectiX has quit IRC10:18
*** ParsectiX has joined #openstack-infra10:19
*** ParsectiX has quit IRC10:19
*** rfolco has joined #openstack-infra10:19
openstackgerritMerged openstack-infra/project-config: Fix broken ENABLED_SERVICES in neutron and scalpels jobs  https://review.openstack.org/24394910:21
openstackgerritMerged openstack-infra/project-config: Use plugin for congress installation in murano-integration job  https://review.openstack.org/24787810:22
openstackgerritMerged openstack-infra/project-config: Add non-voting python34 jobs to murano-related projects  https://review.openstack.org/25145710:22
*** kiran-r has joined #openstack-infra10:22
*** abregman is now known as abregman|lunch10:23
*** Ramanjaneya has joined #openstack-infra10:23
openstackgerritMerged openstack-infra/project-config: Publish test results for murano tempest job  https://review.openstack.org/25146410:24
*** skolekonov has joined #openstack-infra10:25
openstackgerritMerged openstack-infra/project-config: Add non-voting murano-congress integration job to congress repo  https://review.openstack.org/25146910:25
openstackgerritMerged openstack-infra/project-config: [Rally] Fix devstack-rally-verify builder  https://review.openstack.org/25133410:29
*** ramineni1 has joined #openstack-infra10:29
*** jlanoux has quit IRC10:29
*** ramineni1 has left #openstack-infra10:29
*** ramineni1 has joined #openstack-infra10:29
*** skolekonov has left #openstack-infra10:30
openstackgerritMerged openstack-infra/project-config: devstack-plugin-ceph: Remove DEVSTACK_GATE_CEPH for ceph plugin job  https://review.openstack.org/25119710:31
*** jlanoux has joined #openstack-infra10:32
*** Guest55892 has quit IRC10:32
*** weshay_xchat has joined #openstack-infra10:33
*** mahatic has joined #openstack-infra10:33
openstackgerritMerged openstack-infra/project-config: Enable docs jobs for ironic-inspector  https://review.openstack.org/25129210:36
*** jlanoux_ has joined #openstack-infra10:37
*** e0ne has joined #openstack-infra10:37
*** otter768 has joined #openstack-infra10:38
*** jlanoux has quit IRC10:39
*** deepakcs has joined #openstack-infra10:40
deepakcsAJaeger: https://review.openstack.org/#/c/251421/ - sent #2 post your comments.10:40
*** scheuran has joined #openstack-infra10:41
*** otter768 has quit IRC10:42
openstackgerritMerged openstack-infra/project-config: Cleanup jobs for the project faafo  https://review.openstack.org/23801710:44
*** jcoufal has quit IRC10:45
AJaegerdeepakcs: I just had it open to comment ;)10:45
*** jcoufal has joined #openstack-infra10:46
AJaegerdeepakcs: 251421 looks fine but touches other projects check queue - even non-voting, so add flaper87, johnthetubaguy, smcginnis ...10:46
*** GheRivero has quit IRC10:47
deepakcsAJaeger: ok will do, it touches other projects check queue, but its nv, do i still need to add them ?10:47
deepakcsjust confirming, don't want to spam them un-necessarily10:47
*** GheRivero has joined #openstack-infra10:47
*** timrc has quit IRC10:47
*** hughhalf has quit IRC10:47
openstackgerritMerged openstack-infra/project-config: Add Kuryr to devstack install CI check  https://review.openstack.org/25060410:48
*** rvba has quit IRC10:48
deepakcsAJaeger: Commit msg says "Non-voting CI job is run against Cinder, Glance, Nova10:48
deepakcsservices, akin to the existing hook based Ceph CI voting job." - do i need to add more ?10:48
*** shashank_hegde has quit IRC10:48
*** hughhalf has joined #openstack-infra10:48
johnthetubaguydeepakcs: AJaeger: does it not normally stay experimental for a little bit?10:48
*** timrc has joined #openstack-infra10:49
deepakcsjohnthetubaguy: experimental could be for those who are coming in new... here we are using the same code base (lib/ceph) used in hook approach for the plugin case, so its technically not a new CI :)10:49
*** ihrachys has joined #openstack-infra10:49
deepakcsjohnthetubaguy: also its NV so it should really harm imho10:50
*** gmann has joined #openstack-infra10:50
AJaegerdeepakcs: I added them, let them speak up themselves ;)10:51
deepakcsAJaeger: sure, thanks10:51
*** e0ne has quit IRC10:51
AJaegerjohnthetubaguy: best practice is to be first experimental. But that's what you and deepakcs have to decide on. If the job is stable, you can give it more exposure with non-voting in check10:52
johnthetubaguydeepakcs: its more a call for infra than me, I am OK with a non-voting job from the Nova POV, its just going to use a lot of resources10:52
*** rvba has joined #openstack-infra10:52
*** rvba has quit IRC10:52
*** rvba has joined #openstack-infra10:52
deepakcsjohnthetubaguy: ok10:53
AJaegerjohnthetubaguy, you're right. deepakcs do we really need it on all three repos? I understand glance and cinder but why nova?10:53
deepakcsAJaeger: because this job will eventually replace the existing ceph hook based CI job, which runs against cinder, nova, glance, all 310:53
AJaegerand yes, we have already gate-tempest-dsvm-full-ceph on these that you want to remove soon...10:54
deepakcsAJaeger: exactly, thats the point, so once the gate-tempest-dsvm-full-devstack-plugin-ceph-nv is working fine, we disable full-ceph and make gate-tempest-dsvm-full-devstack-plugin-ceph-nv voting10:54
johnthetubaguydeepakcs: agreed with your end goal, but its good to do things in smaller steps sometimes, it could start experimental in all but one project to start with10:55
AJaegerdeepakcs: I'm fine from infra perspective but like to give the other projects a chance to review this - so won't +2 for a couple of days10:55
*** ParsectiX has joined #openstack-infra10:57
AJaegerdeepakcs: the job is voting in your check queue already, please add it to the gate as well10:57
deepakcsjohnthetubaguy: doing expt'l -> nv -> v for each project will take a long time, and i think we had a deadline for hook -> plugin movement as mitaka-1 :)10:57
deepakcsAJaeger: didn't get u... u mean voting in plugin projects' check queue ?10:58
AJaegerdeepakcs: add voting in the gate of plugin repo, it'S voting only in check, isn't it?10:58
deepakcsAJaeger: probably, but is there any adv of putting in gate for plugin repo ? i can do it if it adds value10:59
johnthetubaguydeepakcs: not saying all projects, just saying limiting the initial nv bit will save quite a lot of resources, at a week in the release when we will be running very short of resources10:59
*** fhubik is now known as fhubik_brb11:00
*** sputnik13 has joined #openstack-infra11:00
AJaegerdeepakcs: that'S the normal practice to ensure that we fully test the merged status11:00
*** ParsectiX has quit IRC11:00
deepakcsAJaeger: ok will do11:01
*** ParsectiX has joined #openstack-infra11:01
deepakcsjohnthetubaguy: confirming what u r saying, you r suggesting to add nv job to one of the projects check q, post mitaka-1, add to others ?11:01
johnthetubaguydeepakcs: yes11:02
johnthetubaguydeepakcs: assuming the job seems stable on the project it was added11:02
deepakcsjohnthetubaguy: yes its stable and its running the same number of tests that the existing hook based voiting CI job is running11:02
*** fhubik_brb is now known as fhubik11:03
deepakcsjohnthetubaguy: ok will send patch to enable nv on check q for cinder, to begin with11:03
*** marekd has joined #openstack-infra11:04
*** ihrachys has quit IRC11:06
*** Ramanjaneya has quit IRC11:07
*** zeih has quit IRC11:07
*** sputnik13 has quit IRC11:07
*** abregman|lunch is now known as abregman11:07
*** mpavone has quit IRC11:09
*** fhubik has quit IRC11:09
*** dtardivel has joined #openstack-infra11:10
*** nadya_ has quit IRC11:11
*** yamahata has quit IRC11:12
openstackgerritDeepak C Shetty (deepakcs) proposed openstack-infra/project-config: Enable devstack-plugin-ceph jobs  https://review.openstack.org/25142111:13
*** thorst has joined #openstack-infra11:13
deepakcsjohnthetubaguy: AJaeger ^^ posted #2 @ https://review.openstack.org/#/c/251421/11:13
*** dguitarbite has quit IRC11:14
*** nadya_ has joined #openstack-infra11:15
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/25108811:15
*** claudiub has joined #openstack-infra11:16
*** ldnunes has joined #openstack-infra11:16
*** electrofelix has joined #openstack-infra11:18
*** thorst has quit IRC11:18
*** yamamoto has quit IRC11:20
*** olaph has quit IRC11:20
*** olaph has joined #openstack-infra11:21
*** mpavone has joined #openstack-infra11:21
*** olaph has quit IRC11:25
*** Ramanjaneya has joined #openstack-infra11:27
kragnizis gerrit having issues at the moment for anyone else? requests are timing out for me11:29
openstackgerritJulien Danjou proposed openstack-infra/system-config: Rename Ceilometer to Telemetry  https://review.openstack.org/24302011:30
*** dzamboni has joined #openstack-infra11:30
*** flaper87 has quit IRC11:30
*** flaper87 has joined #openstack-infra11:30
*** ramineni1 has left #openstack-infra11:31
*** aysyd has joined #openstack-infra11:31
*** jlanoux_ has quit IRC11:31
AJaegerkragniz: looks fine here...11:31
*** jlanoux has joined #openstack-infra11:31
kragnizmight be something on my end11:32
kragnizAJaeger: thanks11:32
*** asselin has quit IRC11:34
*** chlong has quit IRC11:35
*** asselin has joined #openstack-infra11:35
*** chlong has joined #openstack-infra11:35
*** aysyd has quit IRC11:36
*** mwagner_lap has quit IRC11:39
*** flaper87 has quit IRC11:39
*** jlanoux has quit IRC11:40
*** mdbooth has joined #openstack-infra11:42
*** svasheka has quit IRC11:43
*** flaper87 has joined #openstack-infra11:47
*** flaper87 has quit IRC11:47
*** flaper87 has joined #openstack-infra11:47
*** mdbooth has quit IRC11:48
*** svasheka has joined #openstack-infra11:49
*** mdbooth has joined #openstack-infra11:49
*** aysyd has joined #openstack-infra11:49
katyaferventhi guys!11:49
*** olaph has joined #openstack-infra11:49
katyaferventMurano team again has problems with python-muranoclient bug status tracking11:50
katyaferventstatuses are not updating automatically11:50
katyaferventthis commit https://github.com/openstack-infra/project-config/commit/453feac830a42d42e382f17b4cd2ddf7ade0c68a fixed it some time ago11:50
katyaferventbut now the problem came back11:51
katyaferventdo you know if there were some changes in infrastructure?11:51
AJaegerkatyafervent: do you have an example of a change request that did not update a bug?11:51
*** ParsectiX has quit IRC11:52
*** ParsectiX1 has joined #openstack-infra11:52
*** tsg has joined #openstack-infra11:54
katyaferventAJaeger, never mind, we created them in murano project instead of python-muranocleint project :)11:54
*** erlon has joined #openstack-infra11:56
*** ParsectiX1 has quit IRC11:57
*** bexelbie has quit IRC11:57
*** ihrachys has joined #openstack-infra11:57
*** hdd has joined #openstack-infra12:03
*** yamamoto has joined #openstack-infra12:04
*** clayton has quit IRC12:06
*** achanda has joined #openstack-infra12:09
*** kaisers has quit IRC12:11
AJaegerglad to hear that it resolved itself ;)12:12
*** clayton has joined #openstack-infra12:13
*** thorst has joined #openstack-infra12:16
tsgAJaeger, just woke up and saw your comment on https://review.openstack.org/#/c/251607/ - why do you think the CI jobs are not running (for this patch)?12:16
*** jlanoux has joined #openstack-infra12:17
AJaegertsg: bug in project-config setup, see http://lists.openstack.org/pipermail/openstack-infra/2015-December/003513.html . Please wait while we figure out how to fix that one...12:17
AJaegergood morning, tsg!12:18
*** salv-orlando has quit IRC12:18
AJaegertsg, they are not running - since Jenkins did not give your change a +1.12:18
*** sshnaidm has quit IRC12:19
tsgAJaeger: good afternoon! I see - so there is no trigger associated with changes to data/* yet12:19
AJaegertsg, exactly. I'm sure we'll find a solution today and then I'll recheck and +2 your change ;)12:19
tsgAJaeger: thanks for getting the bug filed.  clarkb had mentioned to me some time ago that the Jenkins jobs are not run with the system-config/project-config changes integrated though.  Is that still true?12:20
*** Daisy has joined #openstack-infra12:20
*** kiranr has joined #openstack-infra12:20
*** kiran-r has quit IRC12:21
AJaegertsg, don't know what you mean with that comment...12:21
AJaegertsg, wait - you mean: The Jenkins job do not take into effect the change you made. This is indeed correct.12:21
*** hdd has quit IRC12:22
tsgAJaeger: yeah .. for instance, on patch 247936, I had submitted a change to thick_slave/jenkins_params where Jenkins +1'ed the change but I didn't see the change being reflected anywhere in the logs ..12:22
*** jaypipes has joined #openstack-infra12:22
AJaegertsg, indeed, that's the current status.12:23
tsgAJaeger: which is why I was asking clarkb at the time how to best test these changes and he described the process of testing them locally12:23
AJaegerand I doubt we can easily change that - and I even would argue for keeping the status quo.12:23
tsgAJaeger: thanks for confirming that's still true (I felt that might have changed given you issued a recheck :-))12:23
*** ParsectiX has joined #openstack-infra12:23
AJaegertsg: I issued the recheck to figure out why Jenkins did not +1...12:23
AJaegerand then debugged...12:24
*** ParsectiX has quit IRC12:24
tsgAJaeger: ack ..12:24
*** thorst has quit IRC12:24
*** ParsectiX has joined #openstack-infra12:24
*** Sree has joined #openstack-infra12:24
*** skraynev has quit IRC12:24
*** e0ne has joined #openstack-infra12:25
*** regXboi has joined #openstack-infra12:25
*** kaisers has joined #openstack-infra12:26
*** skraynev has joined #openstack-infra12:26
*** tdurakov has joined #openstack-infra12:29
*** EricGonczer_ has joined #openstack-infra12:29
*** claudiub has quit IRC12:29
*** edmondsw has joined #openstack-infra12:29
openstackgerritDaniel Watkins proposed openstack/python-jenkins: Fix get_info for Jenkins running on prefixed paths  https://review.openstack.org/25183912:30
*** achanda has quit IRC12:31
*** claudiub has joined #openstack-infra12:31
*** sshnaidm has joined #openstack-infra12:32
*** lhcheng has quit IRC12:33
*** rhallisey has joined #openstack-infra12:33
*** e0ne has quit IRC12:34
*** kaisers has quit IRC12:36
*** e0ne_ has joined #openstack-infra12:37
*** achanda has joined #openstack-infra12:38
*** otter768 has joined #openstack-infra12:38
*** bdemers has quit IRC12:40
*** dizquierdo has joined #openstack-infra12:41
*** ParsectiX has quit IRC12:41
*** lhcheng has joined #openstack-infra12:42
*** jaosorior has quit IRC12:42
*** jaosorior has joined #openstack-infra12:43
*** otter768 has quit IRC12:43
*** sileht has joined #openstack-infra12:47
*** jaosorior has quit IRC12:47
*** dguitarbite has joined #openstack-infra12:47
*** jaosorior has joined #openstack-infra12:47
*** jaypipes has quit IRC12:49
*** Daisy has quit IRC12:49
*** zeih has joined #openstack-infra12:50
*** kaisers has joined #openstack-infra12:50
*** deepakcs has quit IRC12:51
*** 32NAACGR7 has joined #openstack-infra12:53
*** jaypipes has joined #openstack-infra12:54
openstackgerritMerged openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/24759712:54
*** baoli has joined #openstack-infra12:55
*** dprince has joined #openstack-infra12:56
*** hdd has joined #openstack-infra12:59
*** fawadkhaliq has quit IRC13:00
*** julim has quit IRC13:01
*** achanda has quit IRC13:01
*** mwagner_lap has joined #openstack-infra13:01
*** fawadkhaliq has joined #openstack-infra13:01
*** alivigni has joined #openstack-infra13:02
*** weshay_xchat has quit IRC13:02
*** fawadkhaliq has quit IRC13:02
*** baoli has quit IRC13:02
*** weshay_xchat has joined #openstack-infra13:02
*** kgiusti has joined #openstack-infra13:03
*** fawadkhaliq has joined #openstack-infra13:03
*** fawadkhaliq has quit IRC13:03
*** sshnaidm has quit IRC13:03
*** baoli has joined #openstack-infra13:04
*** dizquierdo has quit IRC13:04
*** fawadkhaliq has joined #openstack-infra13:04
*** achanda has joined #openstack-infra13:06
openstackgerritMarcos Fermín Lobo proposed openstack-infra/project-config: Create puppet-ec2api repository  https://review.openstack.org/25185713:08
*** mase_x200 has joined #openstack-infra13:08
*** dtantsur is now known as dtantsur|brb13:09
*** EricGonczer_ has quit IRC13:09
SotKIs there any way I could possibly get ssh access to the machine running storyboard.o.o please?13:10
SotK(Preferences appear to be broken and I can't seem to reproduce the issue in my test environment, so I'd like to investigate)13:11
SotKalternatively, I guess a dump of the production db could help in my attempts to reproduce the issue locally13:12
*** salv-orlando has joined #openstack-infra13:13
*** katyafervent is now known as katyafervent_awa13:13
*** 32NAACGR7 is now known as thorst13:14
*** weshay_xchat is now known as weshay13:14
*** gordc has joined #openstack-infra13:15
*** katyafervent_awa is now known as katyafervent13:16
*** sshnaidm has joined #openstack-infra13:16
*** persia has joined #openstack-infra13:18
*** fawadkhaliq has quit IRC13:18
*** fawadkhaliq has joined #openstack-infra13:19
*** kiranr has quit IRC13:19
*** hdd has quit IRC13:20
*** ParsectiX has joined #openstack-infra13:20
*** hdd has joined #openstack-infra13:21
*** stevemar has quit IRC13:22
*** stevemar_znc has joined #openstack-infra13:23
*** dguitarbite has quit IRC13:25
*** trown|outttypeww is now known as trown13:27
*** signed8bit has joined #openstack-infra13:31
*** signed8bit is now known as signed8bit_ZZZzz13:32
*** sdake has joined #openstack-infra13:34
*** signed8bit_ZZZzz is now known as signed8bit13:34
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/elastic-recheck: add query for random network.test_neutronv2 fails  https://review.openstack.org/25186813:34
*** jswarren has joined #openstack-infra13:35
*** monester has quit IRC13:36
*** monester has joined #openstack-infra13:36
*** mrmartin has quit IRC13:39
*** gongysh has joined #openstack-infra13:39
*** yamamoto has quit IRC13:40
fungiSotK: i should be able to get you a db dump easily. just a moment13:41
*** sdake has quit IRC13:42
SotKfungi: great, thanks!13:44
*** ParsectiX has quit IRC13:45
*** links has quit IRC13:47
*** markvoelker_ has joined #openstack-infra13:49
*** tdurakov has quit IRC13:49
*** EricGonczer_ has joined #openstack-infra13:49
*** sdake has joined #openstack-infra13:50
*** tdurakov_ has joined #openstack-infra13:50
*** yamamoto has joined #openstack-infra13:50
*** tlian has joined #openstack-infra13:51
*** maishsk has joined #openstack-infra13:51
fungiSotK: i've dumped the storyboard database except for accesstokens, authorizationcodes and refreshtokens tables and uploaded the result temporarily to https://tarballs.openstack.org/storyboard/storyboard.sql.gz13:52
fungiSotK: hopefully that's sufficient to reproduce, but i'm happy to also dig into the logs on the server, check installed versions of things, et cetera13:53
*** tsg has quit IRC13:53
*** sdake_ has joined #openstack-infra13:53
SotKfungi: thank you, I'll see if that helps and get back to you :)13:54
*** sdake has quit IRC13:54
*** olaph has quit IRC13:54
*** e0ne_ has quit IRC13:55
fungiSotK: let me know once you've downloaded it successfully and i'll take it back offline just for extra safety in case i missed some sensitive data in a table somewhere. sha256sum for it is 4b51d99f888ddbfdcd9c252311917c0b8a8abedd2620d764006b88efbe83f86013:55
*** hichihara has joined #openstack-infra13:55
*** e0ne has joined #openstack-infra13:55
*** achanda_ has joined #openstack-infra13:56
SotKgot it, thanks13:56
*** olaph has joined #openstack-infra13:57
*** signed8bit is now known as signed8bit_ZZZzz13:57
*** yamamoto has quit IRC13:57
fungicool, it's offline now, though i can put it back up easily enough or generate a new one if needed13:57
*** yamamoto has joined #openstack-infra13:57
*** achanda has quit IRC13:59
*** dkranz has joined #openstack-infra14:00
*** dkranz has quit IRC14:00
*** dkranz has joined #openstack-infra14:00
*** ParsectiX has joined #openstack-infra14:02
*** ayoung has joined #openstack-infra14:05
phschwartzCan someone verify that I am seeing this correctly in my 3rd party ci that I am setting up.14:07
phschwartz2015-12-01 14:07:04,993 ERROR zuul.Gearman: Job <gear.Job 0x7fefd8482890 handle: None name: build:noop-check-communication unique: 4b293522f33d43d1964bdab88a6fc0b5> is not registered with Gearman14:07
phschwartzThat is saying the noop-check-communication job is not in the layout correct14:08
jswarrenjeblair: got a question about the manifest in puppet-graphite if you have a minute.14:08
*** bdemers has joined #openstack-infra14:08
asselinphschwartz, no, it means jenkins via gearman isn't showing that jobs exists/has any workers14:08
*** kushal has quit IRC14:09
phschwartzasselin: I have the job pointing at a label of u-t that is a label of the image ubuntu-trusty. If I check nodepool, the image is in State Ready, but how do I verify the label?14:10
asselinphschwartz, any nodes using that image?14:11
*** Sree has quit IRC14:11
jswarrenI see that http://git.openstack.org/cgit/openstack-infra/puppet-graphite/tree/templates/graphite.vhost.erb is setting the document root to /var/lib/graphite/webapp, but nothing in the manifest seems to create that directory.  Seems like a required symlink is missing from the manifest.14:12
openstackgerritLucas Alvares Gomes proposed openstack/diskimage-builder: Follow up patch for 25d3ee547176528e86d42eb026c99a134dff9452  https://review.openstack.org/25187914:12
*** gongysh has quit IRC14:12
jswarren...maybe it was created manually on the graphite server?14:12
*** zeih has quit IRC14:12
*** amuller has joined #openstack-infra14:13
*** signed8bit_ZZZzz is now known as signed8bit14:13
openstackgerritIsaac Beckman proposed openstack-infra/system-config: Allow to disable cron jobs  https://review.openstack.org/25037414:15
*** Daisy has joined #openstack-infra14:15
*** placko has quit IRC14:16
*** mahatic has quit IRC14:16
*** kushal has joined #openstack-infra14:16
*** btully has joined #openstack-infra14:17
*** baoli has quit IRC14:18
*** EricGonczer_ has quit IRC14:18
*** baoli has joined #openstack-infra14:18
*** zhenguo has joined #openstack-infra14:18
phschwartzasselin: it looks like 1 node is using the image and is in the Label u-t, but it build/deletes really fast in a loop. It has built almost 350 in the past 20 min14:19
asselinphschwartz, take a look at the nodepool log files to see what the error is14:19
*** tiswanso has joined #openstack-infra14:20
*** ajmiller has joined #openstack-infra14:20
*** peristeri has joined #openstack-infra14:21
*** akshai has joined #openstack-infra14:21
krotscheckpleia2: Howdy! I haven't been keeping up with the i18n-testing devstack instance, and Piet just pinged me asking for some help in getting the UX team a dedidcated testing instance. Can I pick your brain sometime on what that might take?14:21
*** lykinsbd has joined #openstack-infra14:22
asselinphschwartz, I should add that verification to here: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n27414:23
*** ajmiller has quit IRC14:23
*** amotoki_ has joined #openstack-infra14:24
fungiasselin: phschwartz: usually i use the gearman status dump to see what's registered and compare against the zuul debug log to see what configured job:node pairing for which it tried to find a registration14:26
fungiasselin: phschwartz: something like `echo status|nc -q 3 localhost 4730|grep noop-check-communication`14:27
openstackgerritMerged openstack/diskimage-builder: Add dynamic-login element  https://review.openstack.org/24508214:27
*** Daisy has quit IRC14:27
*** zeih has joined #openstack-infra14:27
*** yonglihe has joined #openstack-infra14:27
asselinfungi, yeah, something else I need to document.14:27
*** Daisy has joined #openstack-infra14:27
yongliheHEADS-UP, Intel PCI/SRIO CI,  from 12/1 to 12/2, the Lab is in maintenance,  these CI need to be Offline 24 Hours.14:30
*** btully has quit IRC14:32
*** btully has joined #openstack-infra14:33
*** davideagnello has quit IRC14:34
*** markvoelker has joined #openstack-infra14:34
*** eharney has joined #openstack-infra14:35
*** markvoelker_ has quit IRC14:35
mordredinfra-root: landing the move to openstack inventory14:36
*** Piet has quit IRC14:36
*** mriedem has joined #openstack-infra14:36
*** mase_x200 has quit IRC14:36
*** jyuso1 has quit IRC14:37
*** amotoki has quit IRC14:38
*** amitgandhinz has joined #openstack-infra14:39
*** otter768 has joined #openstack-infra14:39
*** Daisy has quit IRC14:41
openstackgerritDoug Hellmann proposed openstack-infra/infra-specs: add "complete the reviewable release automation"  https://review.openstack.org/24590714:41
*** achanda_ has quit IRC14:41
*** Daisy has joined #openstack-infra14:41
*** weshay has quit IRC14:42
openstackgerritMonty Taylor proposed openstack-infra/system-config: Stop running puppetmaster  https://review.openstack.org/24981514:42
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949214:42
*** dtantsur|brb is now known as dtantsur14:43
mordredinfra-root: and that ^^ is now ready for review14:43
*** otter768 has quit IRC14:44
*** abregman is now known as abregman|brb14:48
*** kozhukalov_ has quit IRC14:49
*** mtanino has joined #openstack-infra14:50
*** dizquierdo has joined #openstack-infra14:51
*** btully has quit IRC14:51
*** dshaw has joined #openstack-infra14:52
SotKfungi: I've managed to reproduce the issue with that dump, thanks :)14:53
*** btully has joined #openstack-infra14:55
dshawhey all - is this a good channel to ask stackalytics metric questions?14:55
*** Daisy has quit IRC14:59
*** Daisy has joined #openstack-infra14:59
*** baoli has quit IRC14:59
openstackgerritMerged openstack-infra/system-config: Use OpenStack for inventory instead of puppet  https://review.openstack.org/24993015:00
*** hichihara has quit IRC15:00
*** kozhukalov_ has joined #openstack-infra15:01
*** baoli has joined #openstack-infra15:01
*** amotoki_ is now known as amotoki15:01
vponomaryovHello everyone, require some additional attention from core team here - https://review.openstack.org/#/c/247551/15:01
*** igormarnat is now known as imarnat15:02
*** angdraug has joined #openstack-infra15:02
*** Daisy has quit IRC15:03
*** Daisy has joined #openstack-infra15:04
openstackgerritMerged openstack-infra/system-config: Fix three nits  https://review.openstack.org/25137415:04
*** shayneburgess has quit IRC15:04
*** achanda has joined #openstack-infra15:04
*** yamamoto_ has joined #openstack-infra15:04
*** imarnat is now known as igormarnat15:05
*** zhenguo has quit IRC15:05
*** yamamoto has quit IRC15:05
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Add parallelization options  https://review.openstack.org/18046315:05
*** shayneburgess has joined #openstack-infra15:05
*** tsg has joined #openstack-infra15:06
*** Daisy has quit IRC15:06
*** dims has quit IRC15:08
*** dguitarbite has joined #openstack-infra15:08
jswarrenphschwartz: how do you grep eavesdrop logs?15:09
phschwartzjswarren: I have a script that pulls down all the logs from the eavesdrop script to a local storage server15:10
jswarrenOh, OK.15:10
*** mrmartin has joined #openstack-infra15:11
*** ParsectiX has quit IRC15:11
phschwartzIt would be nice if they were easily searchable online but I am use to my methods. lol15:12
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Add a neutron-full variant with fwaas tests from tempest plugins  https://review.openstack.org/24292515:12
openstackgerritMerged openstack-infra/infra-specs: add storyboard link to instructions  https://review.openstack.org/24587315:13
openstackgerritBaodong (Robert) Li proposed openstack/diskimage-builder: Allow grub2 to build with opensuse  https://review.openstack.org/25191215:15
*** btully has quit IRC15:15
*** dguitarbite has quit IRC15:16
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Add a neutron-full variant with fwaas tests from tempest plugins  https://review.openstack.org/24292515:16
*** adreznec_ is now known as adreznec15:16
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds an API migration and docs generation tool, fairy-slipper  https://review.openstack.org/24535215:17
*** btully has joined #openstack-infra15:18
*** rlandy has joined #openstack-infra15:20
*** abregman|brb is now known as abregman15:21
*** sabeen has joined #openstack-infra15:21
*** weshay_xchat has joined #openstack-infra15:22
mordredphschwartz: maybe once I've finished puppet apply I can add "searchable IRC logs" to the list15:22
*** harlowja_at_home has joined #openstack-infra15:23
*** ajmiller has joined #openstack-infra15:23
*** mrmartin has quit IRC15:24
*** fawadkhaliq has quit IRC15:25
*** FallenPegasus has quit IRC15:26
*** fawadkhaliq has joined #openstack-infra15:26
*** aopincar has quit IRC15:27
*** zz_dimtruck is now known as dimtruck15:28
*** markvoelker_ has joined #openstack-infra15:28
clarkbphschwartz google "site:eavesdrop.openstack.org" wprls great for searching15:29
*** e0ne has quit IRC15:29
*** hongbin has joined #openstack-infra15:30
clarkbmordred no need ^ search engines already fo it for us :)15:30
mordredclarkb: :)15:31
hongbinHi all. I am from Magnum team. I need some help to debug the gate random failure15:31
clarkbyour favorite non google service should work too15:31
openstackgerritDenis Laxalde proposed openstack-infra/jenkins-job-builder: Add support for "tox" builder  https://review.openstack.org/25124915:31
hongbinHere is the error: http://paste.openstack.org/show/480514/15:31
*** rbrndt has joined #openstack-infra15:31
*** markvoelker has quit IRC15:32
hongbinHere is the nova compute log: http://logs.openstack.org/84/251184/2/check/gate-functional-dsvm-magnum-k8s/7210b26/logs/screen-n-cpu.txt15:32
mordredclarkb: welcome back!15:32
AJaegerpabelanger: good morning.15:32
pabelangerAJaeger: ohai15:32
clarkbhongbin it ran out of memory15:33
AJaegerclarkb, mordred, jeblair: We do not run jobs for project-config on some changes at all since we added to many files condiditions, see http://lists.openstack.org/pipermail/openstack-infra/2015-December/003513.html.15:33
*** markvoelker_ has quit IRC15:33
AJaegerDo you have a suggestion on which way to fix this?15:33
hongbinclarkb: How much memory is available in gate?15:33
clarkbhongbin you can use the dstat output to determine what was using the memory it is in the logs15:33
clarkbhongbin 8GB15:33
*** mrmartin has joined #openstack-infra15:33
clarkbAJaeger my vpte is stop selectively running jobs based on file paths15:34
pabelangerAJaeger: if we could fall back to a noop, that would be fine in my eyes15:34
pabelangerAJaeger: since there isn't actually any testing for that file15:34
hongbinclarkb: 8G should be enough. We use only 4G to create VMs15:34
AJaegerpabelanger: I doubt that noop fallback is possible...15:34
clarkbhongbin all of openstack is running too15:34
clarkbhongbin  check the dstat output15:35
hongbinclarkb: checking15:35
*** zeih has quit IRC15:35
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/elastic-recheck: add query for random network.test_neutronv2 fails  https://review.openstack.org/25186815:35
clarkbyou likely only have about 1GB of extra memory for VMs15:36
*** sdake_ has quit IRC15:36
*** sabeen has quit IRC15:36
*** btully has quit IRC15:37
pabelangerAJaeger: Ya, so then we need a gate job that always runs, regardless of the file paths.  maybe flake815:38
AJaegerpabelanger: not sure which one to choose right now, been distracted currently15:40
*** btully has joined #openstack-infra15:42
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add missing host to empty group definitions  https://review.openstack.org/25192915:43
*** tiswanso has quit IRC15:43
*** bdemers has quit IRC15:43
mmedvedeaburmashev: thanks, I am glad it was an easy fix15:43
mordredclarkb, yolanda, fungi: ^^ need to land that now - found a little thing when landing the previous patch15:44
mordredI applied that by hand and it fixes the issue15:44
*** tiswanso has joined #openstack-infra15:45
*** Ramanjaneya has quit IRC15:45
*** achanda has quit IRC15:45
mordredself-approving - I need it in by the next puppet cron15:46
*** bdemers has joined #openstack-infra15:46
*** e0ne has joined #openstack-infra15:47
clarkbmordred lgtm, is there a tldr on that inventory setup for disabled puppet? there is nothing under disable but all under disabled:children?15:47
*** tzn has joined #openstack-infra15:47
*** claudiub has quit IRC15:48
fungithe system-config docs got updated with details15:48
hongbinclarkb: Thanks for pointing the dstat output. It is helpful advice15:48
mordredclarkb: yah - docs are in system-config docs - but tl'dr is that with openstack inventory hostname entries are actually groups15:49
mordredclarkb: the host entry itself is the host uuid, because hostname is not unique - so while we're say replacing review.openstack.org, there might be two hosts - they'll both be in a group called review.openstack.org - but there will also be an ansible inventory entry for each of their uuid's, in case we need to target them more specifically than that15:50
mordredclarkb: which means that to reference a disabled host by name in the disable file, we have to add them to :children which is how you include a group in another group15:50
*** baoli has quit IRC15:51
mordredthat may not have been any more succinct than the actual docs :)15:51
*** baoli has joined #openstack-infra15:51
AJaegerpabelanger: I have to leave now, will join later again - do you have time to look further into this?15:51
*** mrmartin has quit IRC15:52
pabelangerAJaeger: sure, I can run with it15:52
*** rkukura has joined #openstack-infra15:53
openstackgerritMonty Taylor proposed openstack-infra/system-config: Stop running puppetmaster  https://review.openstack.org/24981515:54
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949215:54
mordredyolanda: ^^ fixed your -115:54
mordredclarkb: if you have review headroom, https://review.openstack.org/249492 is the next up on the list to go in15:54
*** dims has joined #openstack-infra15:54
AJaegerthanks, pabelanger15:55
yolandamordred +215:56
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Serialize combine-queued-commits for git-revision  https://review.openstack.org/12496515:56
*** btully has quit IRC15:57
* mordred has goal of being on puppet apply by infra meeting15:57
*** bpokorny has joined #openstack-infra15:57
openstackgerritJonathan Lebon proposed openstack-infra/jenkins-job-builder: Document the new {template-name} variable  https://review.openstack.org/25193615:58
*** AJaeger has quit IRC15:58
Clintgo go gadget-mordred15:58
jeblairjswarren: i don't know right what created /var/lib/graphite/webapp; is it possible the pip install did so?  it doesn't seem to be a symlink -- it's an actual dir with 'content' as a subdir.15:58
dshawI'm having issues locating the true definition of 'Resolved Bugs' as Stackalytics reports it. I'm a bit confused on if this would be commits which resolve an existing bug, filed bugs that later on are resolved, or a combination of the two?15:59
*** pcrews has joined #openstack-infra15:59
mordredClint: I'm never sure what to do with the retractable arms when I'm done with them15:59
*** harlowja_at_home has quit IRC15:59
openstackgerritMerged openstack-infra/system-config: Add missing host to empty group definitions  https://review.openstack.org/25192916:00
openstackgerritEdgar Magana proposed openstack-infra/irc-meetings: Change the date for the networking guide team meetings  https://review.openstack.org/25193816:00
*** ssancheztrujillo has joined #openstack-infra16:00
jeblairmordred: with all the arm waving you do, it's getting dangerous16:00
Clintmordred: sell them on ebay16:00
jswarrenjeblair: When I apply the manifest the directory does not get created.  Maybe it was manually copied?16:00
*** markvoelker has joined #openstack-infra16:00
*** tiswanso has quit IRC16:00
*** tiswanso has joined #openstack-infra16:01
mordredjeblair: I take out jets when I wave the gadget arms16:01
*** rkukura has quit IRC16:01
clarkbmordred: the whoami in https://review.openstack.org/#/c/249492/5/playbooks/group_vars/all.yaml isnt redundant?16:01
clarkbmordred: via some fact?16:01
jeblairjswarren: that's possible, though i can't say for sure at this point.  it's certainly not how we like to do things.  :|16:02
*** bpokorny has quit IRC16:02
*** sabeen1 has joined #openstack-infra16:02
*** nadya_ has quit IRC16:03
*** hongbin has left #openstack-infra16:03
*** achanda has joined #openstack-infra16:03
*** achanda has quit IRC16:03
mordredclarkb: there is probably a blackmagic way of getting to it - but the problem is that we run the role for each host -so when it's being run, 'ansible_fqdn' will == review.o.o16:03
*** nadya_ has joined #openstack-infra16:03
*** btully has joined #openstack-infra16:03
clarkbdshaw: I don't know that any of us will ahve the answer, quickest thing may be to look at the code really quick16:03
jswarrenjeblair: I was going to add the creation of the symlink to the manifest.16:03
mordredclarkb: we can probably refactor it once we grow better ansible zen16:04
jswarrenjeblair: I guess that won't work with the deployed instance.16:04
*** wshao has joined #openstack-infra16:04
jeblairjswarren: thanks -- we can manually fix that.16:04
clarkbmordred: I see so you wnat to know the source of the ssh connections?16:04
mordredsort of - yeah16:04
*** dangers_away is now known as dangers16:04
jeblairjswarren: we could also recursively copy directory contents16:04
*** rcernin has quit IRC16:04
clarkbfor puppet log copies?16:04
jeblairjswarren: i don't know which approach would be more appropriate16:04
mordredit's a weird task - it's running with "delegate_to: localhost"16:04
mordredbut it needs to know the fqdn of localhost because it needs to find the ssl certs to use to connect to puppetdb host16:05
mordredbut the task itself is per-remote-host16:05
mordredclarkb: I figured for now just putting that one entry in a config was clearer than whatever the magic would be of figuring out "fqdn of executing host that we delegated to"16:06
jswarrenjeblair: it seems unnecessary to copy the directory.  If a newer version of graphite-web breaks things, switching back to the old version via git should solve the problem.16:06
clarkbmordred: ya makes sense16:06
jswarrenjeblair: of course it's different if the contents are not the same...for instance something else was copied into the copy or files in the copy were altered.16:07
*** nadya_ has quit IRC16:07
*** dims_ has joined #openstack-infra16:08
jeblairjswarren: if you're considering symlinking it into the git repo, then i'd recommend the copy approach instead -- we generally try not to have our git repos be running live16:08
*** FallenPegasus has joined #openstack-infra16:08
pabelangerfungi: What do you think about some sort of bindep validation tool?16:08
jswarrenjeblair: OK16:08
mordredclarkb: I'm poking at a simple playbook with task delegation right now to see if there is a variable that can be used16:09
*** sputnik13 has joined #openstack-infra16:09
*** changbl has joined #openstack-infra16:09
fungipabelanger: whatchamean?16:09
*** asilenkov has quit IRC16:10
*** dims has quit IRC16:10
mordredclarkb: OH!16:10
mordred                "ansible_fqdn": "puppetmaster.openstack.org",16:10
fungipabelanger: bindep exercise jobs which try to install the fallback set on our various supported ci worker platforms?16:10
mordredclarkb: I'll make some followup patches to clean that up16:10
pleia2krotscheck: Daisy is working on getting us Puppet manifests to deploy it, spec is here: http://specs.openstack.org/openstack-infra/infra-specs/specs/translation_check_site.html and my last email to her about what needs to be done: http://lists.openstack.org/pipermail/openstack-infra/2015-November/003481.html16:10
pabelangerfungi: We found a hole in our project-config validation: http://lists.openstack.org/pipermail/openstack-infra/2015-December/003513.html16:10
fungipabelanger: i.e., to validate that the package names listed are relevant for the stated platforms?16:10
mordredclarkb: ansible_fqdn when a task is delegated is the fqdn of the host the task is running on16:10
pabelangerfungi: looks like we don't actually validate the binded.txt files in any way16:10
*** achanda has joined #openstack-infra16:11
pabelangerfungi: so, trying to figure out the best way to make jenkins happy again when the file changes16:11
*** kzaitsev_ws has quit IRC16:11
*** sdake has joined #openstack-infra16:11
jeblairfungi: that sounds useful to me16:11
*** dteselkin has quit IRC16:11
fungito me too16:11
pabelangerfungi: clarkb suggestion was the remove the optimizations16:11
*** tsufiev has quit IRC16:11
*** akuznetsova has quit IRC16:11
*** katyafervent has quit IRC16:11
fungii guess i'm behind on e-mail again16:11
jeblairthat's a good interim step if bindep testing will take took long16:12
jeblairtoo long16:12
jeblairooh, we magically started using bhs1 last night when ovh fixed it :)16:12
pabelangerfungi: project-config was optimized to run gate tests based on file path. in an effort to reduce the about of testing we do per commit16:12
*** kzaitsev_ws has joined #openstack-infra16:13
fungipabelanger: oh, yeah, i'm not a fan of those and try to push back whenever they're proposed. it's too easy to get it wrong16:13
*** davideagnello has joined #openstack-infra16:13
fungiand end up not running jobs16:13
dims_fungi : https://review.openstack.org/#/c/251596/ is causing failures (example https://review.openstack.org/#/c/251088/)16:13
clarkbmordred: ok will review the docs portion of the change in a few16:14
pabelangerI think in general it is working well, minus the stuff we don't actually test.  But ya, in this case, something slipped thought because there are no tests related to bindep16:14
clarkbI am always amazed a how much catching up there is to do after what should have been a holiday for many/most16:14
mordredclarkb: oh - I have found another thing that I'll do in the same patch that is more important16:14
*** arxcruz has quit IRC16:14
*** nmagnezi has quit IRC16:15
dims_fungi : what was the initial idea? i missed the discussion16:15
fungidims_: indeed, that should be reverted. i forgot that just running another project's job doesn't actually correctly check out that other project. we need to do something much more complicated there16:15
pleia2krotscheck: I'm sure she could use some help :) but she'll be the one you want to follow up with, I'm focused on other things and just pitching in when she needs infra assistance16:15
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Revert "Run gate-neutron-pep8-constraints on global reqs"  https://review.openstack.org/25194316:15
*** dizquierdo has quit IRC16:16
dims_fungi : ok here's the revert https://review.openstack.org/#/c/251943/16:16
krotscheckpleia2: Danke16:16
krotscheckpleia2: Will ping her :)16:16
*** btully has quit IRC16:17
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Clear up some issues with fqdm vs. hostname  https://review.openstack.org/25194416:18
fungidims_: thanks, and apologies for the disruption16:18
mordredclarkb: ^^ that fixes your whoami thing - as well as another problem I found that we would have hit when landing the other patch16:19
dims_fungi : no worries at all16:19
fungilifeless: Nakato: running other projects' jobs on requirements changes needs more complex solution, probably similar to what oslotest does16:19
fungisorry that didn't dawn on me last night16:19
*** akuznetsova has joined #openstack-infra16:19
clarkbmordred +216:20
mordredclarkb: thanks!16:20
mordredjeblair, fungi, pleia2, yolanda: could use a second +2 on https://review.openstack.org/25194416:21
mordredjeblair: nice re: bhs1!16:21
mordredjeblair: oh - also, while you're clouds-ing - we get a second /27 on the BB cloud so we should be able to increase quota there16:22
pc_mfungi: hi16:24
*** dprince has quit IRC16:25
*** apoorvad has joined #openstack-infra16:25
openstackgerritMerged openstack-infra/ansible-puppet: Clear up some issues with fqdm vs. hostname  https://review.openstack.org/25194416:25
openstackgerritMonty Taylor proposed openstack-infra/system-config: Stop passing whoami in group_vars  https://review.openstack.org/25194916:26
openstackgerritMehdi Abaakouk (sileht) proposed openstack/requirements: Raise Gnocchiclient version to 2.1.0  https://review.openstack.org/25195016:26
mordredsweet. that should get in by next puppet run16:26
openstackgerritMerged openstack-infra/project-config: Revert "Run gate-neutron-pep8-constraints on global reqs"  https://review.openstack.org/25194316:26
*** katyafervent has joined #openstack-infra16:26
*** btully has joined #openstack-infra16:26
*** sfinucan has joined #openstack-infra16:27
*** dteselkin has joined #openstack-infra16:27
*** changbl has quit IRC16:27
*** tsufiev has joined #openstack-infra16:27
*** r-daneel has joined #openstack-infra16:27
*** yamamoto_ has quit IRC16:28
*** asilenkov has joined #openstack-infra16:29
*** spzala has joined #openstack-infra16:30
jeblairmordred: oh nice; i'll go update bb things16:30
*** dims has joined #openstack-infra16:30
jeblairmgagne: i'm ready to continue debugging when you are16:30
*** sdake has quit IRC16:30
*** mpavone has quit IRC16:31
*** harlowja_at_home has joined #openstack-infra16:31
*** dims_ has quit IRC16:33
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Add whitespace between tasks for better readability  https://review.openstack.org/25195316:33
phschwartzso I have an instance listed as deleted when I do a nodepool list. It no longer exists in my openstack tenant. How can I clear this from the nodepool list?16:34
*** FallenPegasus is now known as MarkAtwood16:34
jeblairmordred: looks like there's nothing to do in the admin interface -- there is a second network added and we already seem to be using ips from it16:34
mordredjeblair: that's excelent16:34
jeblairphschwartz: nodepoold will remove it automatically after a couple of minutes, or you can do so from the command line with 'nodepool delete --now'16:34
phschwartzjeblair: been sitting at deleted state in nodepool list for 2 hours16:35
phschwartzjeblair: I will force it. ty :)16:35
jeblairphschwartz: if you changed the provider name or something, that would cause it to ignore it16:36
phschwartzjeblair: worked like a charm. I wish some of the extra options where documented or in the help I didn't look at the source but also would have know about --now if I did :(16:37
clarkbnodepool help delete16:37
clarkbshould be all argparse documented at least16:38
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Fix indexes in db models  https://review.openstack.org/25154716:38
fungipc_m: what's up?16:39
openstackgerritMerged openstack-infra/elastic-recheck: add query for random network.test_neutronv2 fails  https://review.openstack.org/25186816:39
pc_mfungi: wanted to talk pep816:39
*** btully has quit IRC16:39
openstackgerritJames E. Blair proposed openstack-infra/project-config: Increase bluebox to 39  https://review.openstack.org/25195616:40
fungipc_m: sure. i don't speak many languages, but am pretty fluent in pep8 ;)16:40
pc_mfungi: I pushed up neutron patches for pinning for kilo juno. The juno one failed all over the place. Looks like neutron folks don't think we should support it.16:40
*** otter768 has joined #openstack-infra16:40
pc_mfungi: Should I abandon that change?16:40
anteayayonglihe: you have changed your wikipages, thank you, there is no need to post an announcement about an offline ci system in channel, thank you16:40
clarkbthe pylint thread confused me16:41
fungipc_m: yeah, ignore juno. it's had its last point release. i'm due to abandon all open changes for stable/juno branches and then tag it juno-eol and delete16:41
clarkbwhy do wr not already pin all linters?16:41
pc_mfungi: Also, the kilo commit is not a cherry-pick (as there is no equivalent in master). Will there be an issue there?16:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a dashboard page for boards and worklists  https://review.openstack.org/23197316:41
clarkbno need for constraints to do that16:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Update $resource wrappers to use new permissions API  https://review.openstack.org/23547516:41
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add UI for complex permissions on boards and worklists  https://review.openstack.org/23547616:41
clarkbsince we have done it with pep8 for years16:41
fungipc_m: it's only an issue if the stable reviewers make it an issue16:41
pc_mfungi: gotcha.16:42
openstackgerritMerged openstack-infra/storyboard-webclient: Align story subscription stars on right  https://review.openstack.org/25041716:42
*** btully has joined #openstack-infra16:42
pc_mfungi: For VPN, I have a simple fix for master to stop breakage. I added astroid as dependency to tox.ini and pinned it and pylint to a version there.16:42
*** changbl has joined #openstack-infra16:42
pc_mfungi: Wondering if I should cherry pick that to kilo for, versus pinning in requirements, as I'm doing right now.16:43
*** dprince has joined #openstack-infra16:43
pc_mfungi: The alternative, is to remove pylint from pep8 target in Kilo (another solution).16:44
*** otter768 has quit IRC16:45
fungipc_m: adding dependencies in your tox.ini is hacky and best avoided when possible16:45
pc_mfungi: Ref: https://review.openstack.org/#/c/251556/ (master), https://review.openstack.org/#/c/251874/ (kilo)16:45
pc_mfungi: The long term goal is to get to pep8-constraints, but that'll take a while.16:46
fungithough i have no opinion on dropping pylint in your stable/kilo branch... i've seen pylint occasionally point out bugs you don't see from other analyzers, but more often i've seen it fail on perfectly correct code16:46
clarkbI still assert no -constraints required16:46
clarkblinters should be pinned for this exact reason16:46
fungiyeah, constraints isn't required to solve this problem, it just happened to conveniently not break16:47
clarkbpyflakes and hacking and pep8 all are16:47
*** tjones has joined #openstack-infra16:47
fungiclarkb: yeah, but astroid is not a linter16:47
fungiit just happens to be a dependency of a linter16:47
fungiand there we open the door to conflicts between dependencies of our normal dependencies and dependencies of our linters being incompatible16:48
clarkbthe thread said pylint is not pinned and was upgraded too16:48
clarkbmy understanding based on what I read is pinning pylint would fix too16:48
clarkbso the stuff about pylint updating is just noise16:49
fungithe problem as originally surfaced was that pylint was pinned, newer astroid release on pypi broke the pinned version of pylint16:49
clarkbits the transitove dep16:49
pc_mclarkb: In VPN, pylint is not pinned. In Neutron it is. But astroid is not pinned16:49
pc_mclarkb: constraints happens to work, because it pins astroid to 1.3.8 and global requirements pin pylint to 1.4.416:50
*** fawadkhaliq has quit IRC16:50
fungithe point about pylint versions i think, is that if we want to avoid the astroid pin either astroid has to back out their behavior change or we need to up the version of pylint we use everywhere to one which supports newer astroid16:50
*** woodster_ has joined #openstack-infra16:51
pc_mfungi: upstream astroid is talking about trying to do pylint 1.4.5 with pinned astroid version as dependency.16:51
pc_mfungi: https://bitbucket.org/logilab/astroid/issues/275/140-and-141-fail-to-work-with-pylint-14416:52
*** bdemers has quit IRC16:52
*** deepakcs has joined #openstack-infra16:52
fungigot it, and pylint 1.4.5 would basically be otherwise identical to 1.4.416:52
fungiso my comment still stands. everyone pinned to pylint 1.4.4 will need to update their dependency to 1.4.516:52
fungiif not so as to support newer astroid, then so as to avoid newer astroid16:53
deepakcsfungi: Hi, quick q, how long does it take for changes in jenkins/jobs/devstack-gate.yaml to reflect in the CI job ?16:53
*** sridhar_ram has joined #openstack-infra16:53
pc_mfungi: right.16:53
*** amuller is now known as amuller_afk16:53
deepakcsfungi: Patch @ https://review.openstack.org/#/c/251197/ was merged around 6 hours back, but I don't see the effect of it in a recent job @ https://jenkins02.openstack.org/job/gate-tempest-dsvm-full-devstack-plugin-ceph/7/consoleFull16:54
mordredoh - gross. the openstack inventory log output is less readable16:54
fungideepakcs: generally faster than that, but it's possible something broke job updates for our jenkins masters. i'll check16:54
deepakcsfungi: in short, the patch removed setting of DEVSTACK_GATE_CEPH for ceph-plugin job-template, i sent a testpatch to test it ( https://review.openstack.org/#/c/251942/ ) and i still see DEVSTACK_GATE_CEPH being set (even after 5+ hours)16:55
mordredI think we're going to want to write a log processing plugin to map hostnames16:55
*** amotoki has quit IRC16:55
deepakcsfungi: ok, wanted to confirm if i did some mistake or just need to wait longer16:56
*** skolekonov has joined #openstack-infra16:56
*** matrohon has joined #openstack-infra16:56
fungideepakcs: looks like that file last updated on jenkins02 at 10:41 utc, 10 minutes after 251197 merged16:56
*** trown is now known as trown|lunch16:57
*** sdake has joined #openstack-infra16:57
openstackgerritMerged openstack-infra/release-tools: remove release_many.sh  https://review.openstack.org/25151016:57
deepakcsfungi: which means 251197 changes should be present there, right ?16:57
openstackgerritMerged openstack-infra/release-tools: add more detail to the output  https://review.openstack.org/25156716:57
openstackgerritMerged openstack-infra/release-tools: catch more errors in external commands  https://review.openstack.org/25156816:57
openstackgerritMerged openstack-infra/release-tools: better formatting for command errors  https://review.openstack.org/25156916:57
fungideepakcs: well, it's present in the jjb config. now i'm looking at the jenkins xml for gate-tempest-dsvm-full-devstack-plugin-ceph on jenkins02 to see if it got applied16:58
*** mriedem is now known as mriedem_meeting16:58
fungideepakcs: yep, it's not applied16:58
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Add doxygen builder support  https://review.openstack.org/13133016:58
fungithat file was last modified november 26 and still has the variable set in it16:58
fungii'm going to dig in the logs now to see if jjb is failing for some reason16:59
*** hemnafk is now known as hemna16:59
*** kushal has quit IRC16:59
deepakcsfungi: great, thanks, if u find anything can u pls update https://review.openstack.org/#/c/251942/ (test patch) so that I can followup tomorrow (its 10.30 PM here, need to hit bed)17:00
deepakcsif thats ok :)17:00
*** baoli has quit IRC17:00
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Work around "duplicate preferences" issue  https://review.openstack.org/25197017:00
fungiDec  1 11:01:22 jenkins02 puppet-agent[3675]: (/Stage[main]/Jenkins::Job_builder/Exec[jenkins_jobs_update]) Failed to call refresh: Command exceeded timeout17:00
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Accept objects in name_or_id parameter  https://review.openstack.org/25104817:00
*** btully has quit IRC17:01
fungilooks like we probably have some pending job changes which take too long to complete. i'll update by hand17:01
*** matrohon has quit IRC17:01
*** wshao has quit IRC17:01
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Don't double-print exception subjects  https://review.openstack.org/25104917:01
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Enable running tests against RAX and IBM  https://review.openstack.org/25114917:01
*** bdemers has joined #openstack-infra17:01
*** baoli has joined #openstack-infra17:02
*** btully has joined #openstack-infra17:02
fungideepakcs: will do, have a good night17:02
*** f1ller is now known as filler17:03
deepakcsfungi: awesone, if u give me a go ahead via patch comment, i will fire a recheck tomoroww my time, to confirm, thanks again17:03
fungideepakcs: i'll add a recheck comment for you when this is done so you'll hopefully have real job results instead when you wake up17:04
deepakcsfungi: cool, good day ahead, bye17:04
*** kushal has joined #openstack-infra17:04
*** xyang1 has joined #openstack-infra17:05
openstackgerritJohn Warren proposed openstack-infra/puppet-graphite: Copy content to vhost document root  https://review.openstack.org/25197617:05
*** deepakcs has quit IRC17:05
jeblairfungi, clarkb, mordred: https://review.openstack.org/25195617:06
jeblairembiggening our blue box ^17:06
*** mrmartin has joined #openstack-infra17:07
*** jlanoux has quit IRC17:07
jeblairpleia2: !  (and we're still working on further embiggening)17:07
*** mrunge has quit IRC17:07
*** hdd has quit IRC17:08
*** links has joined #openstack-infra17:08
cbaderfungi, hi work on the hpe-storage ci system and replaced ramy, I am having an issue with one of my ci system and was wondering if you might have a direction I might go to resolve my issue.17:09
cbaderfungi, I am getting an mysql_config not found error17:10
fungicbader: seeing that in jobs you run?17:10
*** mrunge has joined #openstack-infra17:10
fungicbader: probably means you're not installing libmysqlclient-dev or equivalent package on your workers17:11
*** dtantsur is now known as dtantsur|afk17:11
fungiand then trying to pip install mysqldb17:11
openstackgerritMerged openstack-infra/project-config: Increase bluebox to 39  https://review.openstack.org/25195617:11
cbaderfungi, when I run the nodepool image-build dpc command.17:12
clarkbmordred: https://review.openstack.org/#/c/249492/5 needs an update now to not whoami? or are you doing that in a followup?17:12
mordredclarkb: followup17:12
*** [1]Thelo has joined #openstack-infra17:12
mordredclarkb: https://review.openstack.org/25194917:12
*** abregman is now known as abregman|brb17:12
fungicbader: oh, well regardless that error generally means you're missing a system package for libmysqlclient-dev17:13
cbaderfungi, I was told by other group members I had old ubuntu files so ran apt-get update to get current packages. it still failed17:13
cbaderfungi, I did the apt-get update and got current libmysqlclient-dev installed17:13
*** btully has quit IRC17:13
nibalizergood morning17:14
fungicbader: apt-get update doesn't install or upgrade anything. i assume you mean you also ran apt-get upgrade17:14
*** gyee has joined #openstack-infra17:14
anteayamorning nibalizer17:14
*** [2]Thelo has quit IRC17:14
anteayaso in today's meeting the question of whether we are able to upgrade to gerrit will be asked17:14
*** yamahata has joined #openstack-infra17:15
fungicbader: anyway, i'm a bit too far underwater at the moment to help troubleshooting nodepool deployment issues, but we have lots of people here who are also familiar with it (moreso than i actually)17:15
anteayaas far as I have heard from zaro the remaining issue that needs to be addressed is the double slash // on redirect17:15
*** rossella_s has quit IRC17:15
anteayathere are two patches linked at the bottom of this etherpad: https://etherpad.openstack.org/p/test-gerrit-2.1117:15
cbaderfungi, thank you for the help have a nice day17:15
nibalizermordred: how can i help with puppet apply17:15
anteayaif folks are able to review them prior to the meeting we can have an effective discussion on the topic and perhaps even get to setting a date for the upgrade17:16
*** aspiers has quit IRC17:16
*** ashleighfarnham has joined #openstack-infra17:16
*** Swami has joined #openstack-infra17:16
*** purp has quit IRC17:16
*** btully has joined #openstack-infra17:16
*** rkukura has joined #openstack-infra17:17
anteayamorning zaro17:17
anteayadid I speak correctly above?17:17
*** ZZelle has joined #openstack-infra17:17
clarkbmordred: https://review.openstack.org/#/c/249815/ is not complete17:17
clarkbmordred: commented on the change17:17
anteayazaro: the remaining issue to agree on a solution is the double slash on redirect?17:18
zaroanteaya: yes, you are correct17:18
anteayazaro: thank you17:18
anteayazaro: trying to get some reviews prior to the meeting17:18
*** purp has joined #openstack-infra17:19
mordredclarkb: thanks!17:19
mordrednibalizer: SO ....17:19
mordrednibalizer: we're doing well, but I've now got 2 issues I think we need to work through17:19
clarkbmordred: and I only +2'd (did not approve) the switch change17:20
*** nelsnelson has joined #openstack-infra17:20
clarkbsince not sure what kind of babysitting that needs17:20
mordredclarkb: thank you,yes. we want to babysit that one for sure17:20
*** ildikov has quit IRC17:20
*** Guest30081 has quit IRC17:20
mordredand I think we need to sort out at least one issue we've got currently17:20
*** lhcheng has quit IRC17:20
mordredI'm going to write it up so we can talk about it at the meeting17:20
*** Hal has joined #openstack-infra17:20
*** ashleighfarnham has quit IRC17:20
*** mrmartin has quit IRC17:21
*** Hal is now known as Guest8696317:21
*** Guest86963 has quit IRC17:21
*** mrmartin has joined #openstack-infra17:22
*** fawadkhaliq has joined #openstack-infra17:22
*** tzn has quit IRC17:22
*** doug-fish has joined #openstack-infra17:25
*** stevemar_znc is now known as stevemar17:25
*** ashleighfarnham has joined #openstack-infra17:25
*** HeOS has quit IRC17:27
*** ayoung has quit IRC17:28
*** btully has quit IRC17:28
*** dizquierdo has joined #openstack-infra17:30
*** abregman|brb is now known as abregman17:30
*** e0ne has quit IRC17:30
*** tsg has quit IRC17:31
*** btully has joined #openstack-infra17:31
*** scheuran has quit IRC17:32
*** ashleighfarnham has quit IRC17:32
openstackgerritMerged openstack/os-testr: Add colored output feature to subunit-trace  https://review.openstack.org/24900517:35
openstackgerritMerged openstack/os-testr: Add delete *.pyc command before executing ostestr  https://review.openstack.org/25170117:36
*** EricGonczer_ has joined #openstack-infra17:36
openstackgerritMerged openstack-infra/shade: Normalize volume objects  https://review.openstack.org/25104717:36
openstackgerritDoug Hellmann proposed openstack-infra/infra-specs: mark centralize-release-tagging spec as implemented  https://review.openstack.org/25199017:36
*** krtaylor has quit IRC17:37
*** hashar has quit IRC17:37
openstackgerritMerged openstack/os-testr: Change to always parsing classes from test_id  https://review.openstack.org/24663617:37
*** nadya_ has joined #openstack-infra17:37
*** nadya_ has quit IRC17:37
*** nadya_ has joined #openstack-infra17:38
*** hdd has joined #openstack-infra17:38
*** nadya_ has quit IRC17:38
*** nadya_ has joined #openstack-infra17:39
*** mpryc has left #openstack-infra17:39
phschwartzjeblair: what would cause nodepool to build a node as shown under nodepool list and then never go active but immediately delete it?17:39
jeblairphschwartz: it should log any such decision17:39
jeblairphschwartz: usually i grep for the node number in the debug log17:40
phschwartzjeblair: in the mean time how can I get it to stop the looping of the rebuild without stopping nodepool17:40
jeblairphschwartz: of the node?17:40
clarkbthe ready script may have failed, ssh may have failed, cloud may have errored17:40
jeblair(looping building a node?)17:40
phschwartzjeblair: yes17:41
jeblairphschwartz: set max-servers to 017:41
jeblairin nodepool.yaml17:41
clarkbjeblair: I have gotten through all but the last change for nodepool image builders and they lgtm. A second set of reviews would be great and maybe we can try deploying that this week17:42
phschwartzjeblair: that did it. Now to dig in the logs17:42
clarkbother cores may want to review that as well17:42
jeblairclarkb: awesome, i'm hoping i can ramp down cloud initiation and focus on that next this week.17:42
clarkbjeblair: great thanks17:42
pabelangerhaven't checked backscroll yet, but seem to be getting a tox failure on project-config master branch: http://pastebin.com/rdaR9mpt17:43
*** dzamboni has quit IRC17:43
*** mriedem_meeting is now known as mriedem17:43
*** bswartz has joined #openstack-infra17:43
phschwartzjeblair: ohh, getting closer to it actually working.  otFound: Floating ip pool not found. (HTTP 404)17:44
phschwartzNew error for me :)17:44
*** yamahata has quit IRC17:44
jeblairphschwartz: i think there's a parameter you can set to specify a fip pool name17:44
*** shashank_hegde has joined #openstack-infra17:44
jeblairpabelanger: that looks like an old jjb17:44
clarkbyes template-name is a new feature so needs new jjb iirc17:45
*** achanda has quit IRC17:45
phschwartzjeblair: does it have to use floating ip's?17:45
pabelangerjeblair: Ah, ya. that is right, I forgot to nuke my .test folder17:46
*** nadya_ has quit IRC17:46
jeblairphschwartz: no -- it only tries to do that if it does not see a public ip on an instance17:46
jeblairphschwartz: however, it "seeing" a public ip on an instance is a confusing set of ad-hoc rules based on providers we have encountered so far...17:47
clarkbjeblair: did we end up reverting the internap IP change because it hit tripleo?17:47
jeblairclarkb: yes, i have a replacement staged and will push as soon as i can actually boot an instance on internap.  :)17:47
jeblair(i can push now if folks are curious, but would probably wip-1 it until i've verified it on internap)17:48
jeblairphschwartz: 1 sec and i'll link you to the code17:48
clarkba WIP change sounds good to me17:49
pabelangerjeblair: still getting the error when I delete both tox and .test folders17:49
clarkb(but mostly just catching up from long weekend at the beach)17:49
jeblairphschwartz: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/provider_manager.py#n3917:49
jeblairphschwartz: this will get a lot better when we move nodepool to use shade17:50
phschwartzjeblair: but there is the requirement of a public ip right?17:50
phschwartzyou can't do it with a private network for the nodes17:51
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add plugin_email_enabled preference by default  https://review.openstack.org/24166517:51
jeblairphschwartz: well, 'public' in this case really means 'accessible from the nodepool instance'17:51
anteayaclarkb: let me know when you are open to discussing logstash things17:51
clarkbanteaya: now is fine17:51
openstackgerritPaul Belanger proposed openstack-infra/project-config: Do not merge!  https://review.openstack.org/25199817:51
anteayaclarkb: okay17:51
clarkbmostl just filtering through email but can pause17:51
phschwartzjeblair: then there is an issue, they are both on the same subnet but it still tried to create a floating17:51
*** harlowja_at_home has quit IRC17:51
jeblairphschwartz: the whole thing can be on a private network, as long as the nodepool server can contact the nodes it launches17:51
*** achanda has joined #openstack-infra17:52
*** sshnaidm has quit IRC17:52
pabelangerjeblair: I think we need to cut a new release of JJB, since we install from pip17:52
jeblairphschwartz: right, it's looking for any ip on a network called 'public' or 'ext-net', or one on a network called 'private' that isn't rfc1918.17:52
anteayaclarkb: line 261: http://paste.openstack.org/show/480171/17:53
phschwartzjeblair: ah, didn't notice the rfc1918 filter there. That is the issue.17:53
jeblairphschwartz: are you using neutron and setting either net-id or net-label?17:53
phschwartzTime to add a second network to the instances17:53
phschwartzjeblair: yea17:53
anteayaclarkb: however I need to make part of that regex optional as the loglevel error and deprecation statement aren't on every line17:54
jeblairphschwartz: k, then you could either change the name to 'public', or use the (not entirely tested) patch i'm about to push up17:54
*** jistr has quit IRC17:54
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add 'public' attribute to 'networks' entry  https://review.openstack.org/25200017:55
jeblairphschwartz, clarkb: ^17:55
clarkbanteaya: you should be able to change it from RemovedInDjango to a GREEDYDATA17:55
clarkbanteaya: so that it matches whatever before the loglevel17:55
phschwartzjeblair: I will take a look at the patch but don't want to try something untested before a demo. lol17:55
jeblairphschwartz: (honestly, i'd go with 'rename the network public' unless you really want to help beta-test that patch :)17:55
jeblairphschwartz: hehe, sounds like we're on the same page :)17:56
anteayayeah, stopping GREEDYDATA at loglevel was my problem but I can try again17:56
*** austin81 has joined #openstack-infra17:56
* anteaya tries again17:56
phschwartzjeblair: fingers cross "edit network -> change name to public"17:56
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ansible-role-diskimage-builder  https://review.openstack.org/25200117:57
phschwartzjeblair: if it is labeled as public for the network will it still get filtered if it is an rfc1918 ip?17:58
jeblairphschwartz: nope, that only happens if the network is called 'private'17:58
phschwartzjeblair: perfect, this build will hopefully work then17:58
clarkbI wonder if names like Ext-Net, private, public need to have stronger codification in the neutron API17:58
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix checkboxes on preferences page  https://review.openstack.org/25200217:58
clarkbsomething that is the same for every cloud that says "this is the place you ssh to from anywhere in your world"17:59
sdagueclarkb: ++ yes please.17:59
anteayaclarkb: it worked: http://paste.openstack.org/show/480546/ yay17:59
clarkband something for "this is the place you ssh to from within your tenant"17:59
jeblairclarkb: i think mordred has come to the conclusion the only global way to solve this is network annotation, and that neutron needs the ability to have the operatior indicate that a network is intended to be accessible from outside of the cloud18:00
phschwartzjeblair: yay now to troubleshoot ssh timing out. lol18:00
clarkbjeblair: and when neutron grows that feature the annotation itself shouldn't be configurable18:00
*** dmsimard is now known as dmsimard|afk18:00
clarkbit should be a magic number that is the same in every cloud new enough to have the feature18:01
jeblairclarkb: well, whoever defines the network would need to so annotate it, i think.18:01
*** ayoung has joined #openstack-infra18:01
*** sdake has quit IRC18:01
*** jaosorior has quit IRC18:02
clarkbjeblair: ya so you set the neutron network is out of cloud accessible flag and the annotation is the same everywhere insetad of each cloud updating things however they like to expose that info18:02
*** krtaylor has joined #openstack-infra18:02
clarkbwhich is no better than the current situation18:02
clarkbwhere name seems to get used a lot "public" and "Ext-Net"18:02
*** ayoung_ has joined #openstack-infra18:02
jeblairyeah, definitely need a standard flag.  we thought 'router:external' was that flag until yesterday mordred learned it means nothing of the sort18:03
clarkbjeblair: I tried telling mordred that before :)18:03
jeblair(it actually means something like 'this network does not come with a router')18:03
*** ayoung has quit IRC18:03
*** ayoung_ is now known as ayoung18:03
clarkbI went and dug around api docs and neutron code and ya that18:03
clarkbbut as used in existing clouds it was close enough so went with it18:04
jeblairit mostly worked, except that it also implies 'this network is visible to all tenants'.18:04
jeblairwhich is kind of a wtf18:04
jeblaireven with the correct definition18:05
*** amuller_afk is now known as amuller18:05
*** derekh has quit IRC18:05
anteayaclarkb: so line 77 is another line in the same file, the regex finds no matches: http://paste.openstack.org/show/480547/ question on line 8218:05
*** kushal has quit IRC18:06
*** sdake has joined #openstack-infra18:06
*** unicell has joined #openstack-infra18:07
clarkbanteaya: you can use ? to signify zero or one matches18:08
clarkbanteaya: essentially making that optional18:08
lifelessfungi: oh ?18:08
clarkbanteaya: so wrap the whole optional section in ()?18:08
*** ivar-lazzaro has joined #openstack-infra18:08
mordredclarkb, jeblair: ++18:09
anteayaclarkb: thanks I will18:09
clarkbanteaya: * is zero or more and + is one or more18:09
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a --query option to email_stats.py  https://review.openstack.org/25200418:10
anteayaclarkb: yay! it worked18:11
anteayathe ? goes _after_ the ()18:11
fungilifeless: i didn't think through the job addition. gate-neutron-pep8-constraints (as for any of our usual tox job builders) checks out whatever repo is specified in ZUUL_PROJECT, which will be the repo corresponding to the change being tested (in this case openstack/requirements, not openstack/neutron)18:11
anteayaI was trying it before the ()18:11
anteayathank you18:11
clarkbanteaya: yes () is grouping and ? says the prior group is zero or one matches18:11
lifelessfungi: can we set ZUUL_PROJECT in the point we reference it?18:11
*** yamahata has joined #openstack-infra18:11
fungilifeless: we can, but we'd need a different job with a different builder, or more parameterization for the existing one18:12
anteayaclarkb: thank you!18:12
clarkbanteaya: you don't need the ()s if you have a single item you want to zero or one match18:12
clarkbanteaya: like if it was just loglevel18:12
fungilifeless: we wouldn't actually override ZUUL_PROJECT, rather we'd use something instead of it to determine what other repos to check out18:12
lifelessfungi: ok - well step by step I guess18:12
clarkbor if it was just a single character or whatever18:12
openstackgerritMerged openstack-infra/puppet-openstackci: Ensure build-essential and python-dev are installed  https://review.openstack.org/24597318:12
lifelessfungi: so ${MANUAL_PROJECT:-ZUUL_PROJECT} ?18:12
*** ivar-lazzaro has quit IRC18:13
fungilifeless: i believe oslotest was doing something similar to run arbitrary projects' jobs on proposed oslo changes as a safeguard, so we can likely reuse some of what they've got18:13
*** ivar-lazzaro has joined #openstack-infra18:13
fungijust needs time from me or someone to look into how that works18:13
*** eil397 has joined #openstack-infra18:14
anteayaclarkb: great thanks18:14
*** sabeen1 has quit IRC18:14
anteayaclarkb: this regex appears to be working for this horizon log18:15
anteayathis horizon log: http://logs.openstack.org/79/248279/1/check/gate-horizon-dsvm-integration/6662f61/logs/apache/horizon_error.txt.gz18:15
clarkbanteaya: I think you can go ahead and push the changes to the existing config as its not logstash version dependent18:15
*** nmagnezi has joined #openstack-infra18:15
clarkband thanks18:15
*** tsg has joined #openstack-infra18:15
*** ivar-lazzaro has quit IRC18:15
anteayathis regex line 148: http://paste.openstack.org/show/480548/18:16
anteayasure I can offer a patch18:16
*** sdake has quit IRC18:16
*** ivar-lazzaro has joined #openstack-infra18:16
*** aopincar has joined #openstack-infra18:17
anteayathanks for teaching me18:17
*** ildikov has joined #openstack-infra18:19
*** jordanP has quit IRC18:20
*** shashank_hegde has quit IRC18:20
openstackgerritAnita Kuno proposed openstack-infra/system-config: Edit the logstash pattern for apacheerror  https://review.openstack.org/25200718:22
*** jed56 has quit IRC18:23
*** links has quit IRC18:24
*** markus_z has quit IRC18:24
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ansible-role-diskimage-builder  https://review.openstack.org/25200118:25
*** mancdaz has quit IRC18:26
*** sputnik13 has quit IRC18:27
*** maishsk has quit IRC18:27
*** mancdaz has joined #openstack-infra18:27
anteayaclarkb: do you have anything in mind for me to work on next or shall I just randomly try logs?18:27
openstackgerritlifeless proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25160018:29
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add functional tests for boot from volume  https://review.openstack.org/25115018:29
Shrewsmordred: that ^^^ should make the tests work18:29
openstackgerritlifeless proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25201118:30
*** unicell has quit IRC18:30
*** trown|lunch is now known as trown18:32
*** ldnunes has quit IRC18:32
*** ldnunes has joined #openstack-infra18:32
*** yamahata has quit IRC18:33
*** DavidGF has joined #openstack-infra18:33
*** austin81 has left #openstack-infra18:34
*** unicell has joined #openstack-infra18:35
*** unicell has quit IRC18:36
*** yamahata has joined #openstack-infra18:36
*** unicell1 has joined #openstack-infra18:36
*** baoli has quit IRC18:37
*** dizquierdo has quit IRC18:37
*** eharney_ has joined #openstack-infra18:37
*** rcernin has joined #openstack-infra18:38
*** eharney has quit IRC18:38
*** baoli has joined #openstack-infra18:39
*** otter768 has joined #openstack-infra18:41
*** mohankumar has joined #openstack-infra18:43
clarkbanteaya: nothing specific, I need to dive into the puppet and get the install sorted out for the new packaging18:44
clarkbI just haven't had a good point in time to do that yet, hopefully later today18:44
anteayaclarkb: okay, I'll will just look at random logs and let you know if I spot any more grokparse errors18:44
anteayas/I'll will/I will18:44
*** otter768 has quit IRC18:46
*** shashank_hegde has joined #openstack-infra18:46
*** ssancheztrujillo has quit IRC18:48
*** alex_xu has quit IRC18:48
openstackgerritIhar Hrachyshka proposed openstack/requirements: Cap astroid to unbreak pylint.  https://review.openstack.org/25201118:49
*** ifarkas has quit IRC18:50
pabelangergreghaynes: Are you planning a 1.5.0 release of diskimage-builder this week?18:51
*** sabeen1 has joined #openstack-infra18:51
*** eharney_ is now known as eharney18:52
greghaynespabelanger: Nope, is there something that needs to be released?18:52
*** alex_xu has joined #openstack-infra18:52
pabelangergreghaynes: no rush. Was looking to start using the local-config but can hold off18:53
jeblairmordred, jhesketh, ianw: on https://review.openstack.org/196252 did you remove the sync and sleep 5 from our nodepool scripts?18:53
*** signed8bit is now known as signed8bit_ZZZzz18:54
greghaynespabelanger: ah. If it can wait until Monday that would be ideal - I really like monday releasing unless there is some kind of bug18:55
jeblairmordred, jhesketh, ianw: of course, we had that problem, which is why we added them to our build scripts, so it seems strange to move that into nodepool without removing them from our scripts.18:55
pabelangergreghaynes: Yup, totally fine18:55
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Fix get_run_metadata to take UUID  https://review.openstack.org/25201918:55
*** sridhar_ram has quit IRC18:56
*** akshai has quit IRC18:56
*** ssancheztrujillo has joined #openstack-infra18:56
*** skolekonov has quit IRC18:56
*** mriedem has quit IRC18:56
*** akshai has joined #openstack-infra18:57
*** salv-orl_ has joined #openstack-infra18:58
fungiweekly infra team meeting time is now, place is #openstack-meeting18:59
*** rbrndt has quit IRC18:59
*** baoli has quit IRC19:00
*** salv-orlando has quit IRC19:00
*** signed8bit_ZZZzz is now known as signed8bit19:01
*** mriedem has joined #openstack-infra19:01
*** gyee has quit IRC19:02
*** pvaneck has joined #openstack-infra19:02
ianwjeblair: i don't remember such a corresponding change19:02
*** mohankumar has quit IRC19:02
*** baoli has joined #openstack-infra19:03
*** tiswanso has quit IRC19:03
*** amitgandhinz has quit IRC19:04
*** tiswanso has joined #openstack-infra19:05
*** binbincong has quit IRC19:05
*** binbincong has joined #openstack-infra19:06
pabelangerianw: Do fedora-23 DIBs work now? Using fedora-minimal?19:06
ianwpabelanger: maybe, it got in late my yesterday, i need to check the logs19:07
pabelangerianw: okay, I'm having some issues local with the $release variable in the repo URL. Need to dig into it more19:08
*** eliqiao has quit IRC19:08
ianwargh, looks like its hit some sort of issue with pep8?  what's gone on there...19:08
*** tsg has quit IRC19:09
ianwno, wait, wrong change19:09
*** eliqiao has joined #openstack-infra19:09
*** aopincar has quit IRC19:12
*** kozhukalov_ has quit IRC19:14
*** erikmwilson has quit IRC19:15
openstackgerritMerged openstack-infra/subunit2sql: Properly date filter in get_test_run_dict_by_run_meta_key_value  https://review.openstack.org/25167319:15
*** erikmwilson has joined #openstack-infra19:16
*** armax has quit IRC19:16
pabelangerianw: here is the error I am seeing with fedora-23: http://pastebin.com/3AUQ5JEG19:21
ianwpabelanger: hmm, strong deja-vu on that ...19:23
*** nmagnezi has quit IRC19:23
*** ihrachys has quit IRC19:23
ianwwith releasevar not being set ... digging19:24
pabelangerianw: Ya, I thought I seen that before too19:24
pabelangerianw: FWIW: I am running this on centos-719:24
mordredjeblair: sorry, no, I do not believe it was also removed from the scripts - bad oversight19:24
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add functional tests for boot from volume  https://review.openstack.org/25115019:25
mriedemlooks like the uncategorized jobs page is dead since 11/16 http://status.openstack.org/elastic-recheck/data/uncategorized.html19:25
jeblairmordred, ianw, jhesketh: can one of you take care of that please?19:26
mriedemGenerated at: 2015-11-17T05:5219:26
ianwpabelanger: it was https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=3f3850aa0f6fc0e3e10a7c1e35ec67dbeb76d53219:26
clarkbmriedem: does it work for you locally? usually those breaks are pretty reproduceable?19:26
mriedemlet me check19:26
*** HeOS has joined #openstack-infra19:26
*** kozhukalov_ has joined #openstack-infra19:26
pabelangerianw: Yup, running that patch.  I have 1.4.0 installed19:27
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove now duplicate sync;sleep 5  https://review.openstack.org/25202719:29
mordredjeblair: ^^ done19:29
*** ldnunes has quit IRC19:31
*** devananda has quit IRC19:33
*** hashar has joined #openstack-infra19:33
ianwinfra-root: when someone has time, https://review.openstack.org/#/c/247908/ added f23 nodes but i don't see any logs in http://nodepool.openstack.org/ .  presumably i did something wrong, if someone could check the main nodepool log that would be super19:35
fungiianw: maybe after we're done with our weekly team meeting19:35
*** krtaylor has quit IRC19:35
*** ssancheztrujillo has quit IRC19:36
*** sridhar_ram has joined #openstack-infra19:36
*** julim has joined #openstack-infra19:36
*** sdake has joined #openstack-infra19:36
mriedemclarkb: worked for me19:37
*** bcourt has joined #openstack-infra19:39
*** dangers is now known as dangers_away19:40
lifelessso pylint19:41
lifelessI'm not seeing a 1.4.5 yet19:41
lifelessand claudiu's message about tha was 7 hours back19:41
lifelessI think we should land the pins19:41
*** dangers_away has quit IRC19:45
*** dmsimard|afk is now known as dmsimard19:45
*** burgerk has joined #openstack-infra19:46
*** baoli has quit IRC19:46
*** baoli has joined #openstack-infra19:47
*** devananda has joined #openstack-infra19:47
*** rbrndt has joined #openstack-infra19:47
*** tjones has quit IRC19:48
*** ldnunes has joined #openstack-infra19:49
*** nmagnezi has joined #openstack-infra19:49
*** armax has joined #openstack-infra19:51
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: ensure virtualenv is created in toolsdir  https://review.openstack.org/25157019:53
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: factor out command to list deliverable changes  https://review.openstack.org/25157119:53
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rewrite release_from_yaml.py as bash script  https://review.openstack.org/25157219:53
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add the series to the release tag  https://review.openstack.org/25157319:53
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email  https://review.openstack.org/25157419:53
*** austin81 has joined #openstack-infra19:54
*** annegentle has joined #openstack-infra19:55
*** sdake_ has joined #openstack-infra19:55
*** sdake has quit IRC19:55
*** maishsk has joined #openstack-infra19:56
*** amitgandhinz has joined #openstack-infra19:57
*** electrofelix has left #openstack-infra19:58
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/glean: Run glean after networking service  https://review.openstack.org/25203719:58
*** kfarr has joined #openstack-infra19:58
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: factor out command to list deliverable changes  https://review.openstack.org/25157119:58
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rewrite release_from_yaml.py as bash script  https://review.openstack.org/25157219:58
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add the series to the release tag  https://review.openstack.org/25157319:58
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email  https://review.openstack.org/25157419:58
zaronotmorgan: some addtional info regarding apache rules and gerrit. i'm sure it makes more sense to you than me.20:00
kfarrHi all, I am working on a release for an openstack library Castellan, but after we set up the gerrit auto-publish jobs, it only seems to be available for py27, as seen on pypi:20:00
*** aysyd has quit IRC20:00
kfarractually, just py220:00
notmorgani'll poke at these things too while i'm at it, but i *think* we should be good.20:01
kfarrI noticed a few other python clients have a [wheel] universal = 1 in their setup.cfg files, can anyone confirm that this is the way to fix it?20:01
notmorganzaro: mostly we aren't worried about URL based attacks so nocanon should be fine20:01
*** dshaw has quit IRC20:02
*** weshay_xchat has quit IRC20:02
*** sdake_ is now known as sdake20:03
fungikfarr: yes, that's the way to fix it so the wheel is py2.py3 rather than just py220:04
elmikofungi: i had a question about that [wheel] entry, does pbr provide that? (i could only find [bdist_wheel] in setuptools docs, just curious to understand better)20:05
*** baoli has quit IRC20:06
kfarrOk, thanks fungi.  I was wondering the same thing as elmiko20:06
notmorganzaro: do we really need to worry about apache 2.4 vs non2.4 in the vhost? i'm fine if we keep it, just curious (cc jeblair, fungi, clarkb) [can answer post meeting]20:07
*** salv-orl_ has quit IRC20:07
fungielmiko: it's likely a pbrism, check the pbr docs if you haven't already, i don't recall whether that section is used by more than just pbr20:08
funginotmorgan: i'd be willing to drop 2.4 support, though i think there are still some downstreams using it so they might need to stop consuming our puppet module there until they find time to upgrade20:08
notmorganfungi: you mean not-2.4 support ;)20:08
mtreinishfungi: so on https://review.openstack.org/251564 is there something I should change to make it move faster?20:08
funginoryes, sorry, 2.2 support20:09
*** erlon has quit IRC20:09
funginotmorgan: ^20:09
notmorganfungi: yah.20:09
mtreinishI'll work on redoing the module to system install, but in the meantime I want to quickly fix it so we're using a compatible version20:09
fungimy typing is going badly20:09
elmikofungi: thanks, i looked around the pbr docs some but didn't see it. not a huge deal, more scratching a curiousity itch ;)20:09
notmorganfungi: i'll spin this with no-2.4 and we can add the ifmod stuff in if it's super important20:09
*** fawadkhaliq has quit IRC20:09
*** abregman has quit IRC20:10
*** baoli has joined #openstack-infra20:10
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes  https://review.openstack.org/25204120:11
*** etoews has joined #openstack-infra20:11
fungior people who still use apache 2.2 with gerrit can provide a patch20:12
*** btully has quit IRC20:12
*** btully has joined #openstack-infra20:14
*** jcoufal has quit IRC20:15
*** weshay_xchat has joined #openstack-infra20:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Add functional tests for boot from volume  https://review.openstack.org/25115020:16
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add the release metadata to the release tag  https://review.openstack.org/25157320:16
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email  https://review.openstack.org/25157420:16
*** eliqiao has quit IRC20:17
*** ldnunes has quit IRC20:18
openstackgerritMonty Taylor proposed openstack-infra/shade: Add functional tests for boot from volume  https://review.openstack.org/25115020:18
*** spzala has quit IRC20:19
notmorganzaro, fungi, have a proxypass setup that should work.20:19
notmorganspinning the review up now.20:19
*** eliqiao has joined #openstack-infra20:20
funginotmorgan: awesome!!!20:20
*** sputnik13 has joined #openstack-infra20:20
openstackgerritMerged openstack-infra/release-tools: ensure virtualenv is created in toolsdir  https://review.openstack.org/25157020:22
openstackgerritApoorva Deshpande proposed openstack-infra/ciwatch: Change ciwatch.log location from /var/data/ciwatch.log to /var/log/ciwatch/ciwatch.log  https://review.openstack.org/25205020:24
rcarrillocruzso greghaynes20:24
rcarrillocruzyeah, i can tell it's confusing20:24
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add functional tests for boot from volume  https://review.openstack.org/25115020:24
rcarrillocruzi thought initially glean init was unable of creating vlans20:24
rcarrillocruzas that was triggered on network interfaces up event20:24
rcarrillocruzand as vlans were not physical20:25
rcarrillocruzwe didn't get those20:25
rcarrillocruzbut that was a false assumption20:25
greghaynesah ok20:25
greghaynesso, what is the race now?20:25
rcarrillocruzi noticed that in a same server, multiple reboots would get different /etc/network/interfaces.d files created20:25
rcarrillocruzsometimes just eth1.cfg20:25
greghaynesglean has a global lock, so there really shouldnt be a race issue20:25
rcarrillocruzsome others eth1.cfg and eth2.cfg20:25
*** kozhukalov_ has quit IRC20:26
greghaynesrcarrillocruz: hrm20:26
rcarrillocruzso i tested if i could run glean on an event not tied up to physical interfaces20:26
rcarrillocruzas running 'glean' vs 'glean -i foointerface' consistenly create ALL interfaces out of the config drive20:26
rcarrillocruzand that's the case20:26
rcarrillocruzmaking glean run BEFORE 'networking'20:26
rcarrillocruzdid the trick20:26
rcarrillocruzwe get all of them20:26
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] fedora on centos fixes  https://review.openstack.org/25205220:26
rcarrillocruzand vlans20:26
greghaynesrcarrillocruz: ok, so there is a bug somewhere20:27
greghaynesrcarrillocruz: I would really like to figure out what exactly is going on though20:27
greghaynesrcarrillocruz: I can try logging in to this box to have a look?20:27
rcarrillocruzi suspect is on the starting network-interface20:27
rcarrillocruzthat doesn't seem to be deterministic20:28
greghaynesrcarrillocruz: or, if you can run it the way it was failing before and paste the upstart log that would be super helpful20:28
*** baoli has quit IRC20:28
rcarrillocruzalthough i still wonder why this was put in a per interface from start20:28
rcarrillocruzi mean20:28
rcarrillocruzglean is about:20:28
rcarrillocruz1. configure nics from config drive20:28
rcarrillocruz2. put ssh key20:28
greghaynesrcarrillocruz: I suspect youre on the right track and something is wonky with our upstart ordering, but I just want to be sure of what were doing20:29
rcarrillocruzif we have all interfaces on config drive (and for those that are not we fallback to dhcp), no point in going in a per-interface pattern20:29
rcarrillocruzgreghaynes: if you go to ( the jumpstart)20:30
*** baoli has joined #openstack-infra20:30
rcarrillocruzi've been testing all this thing on costkcicd-node005520:30
greghaynesrcarrillocruz: Yep. We spent a lot of time trying to get an upstart script that worked somewhat reliably though (and I think we tried a line like this before and hit some other kind of issues)20:30
greghaynesSpamapS: ^ is kind of our expert on that20:30
*** rfolco has quit IRC20:31
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes  https://review.openstack.org/25204120:33
openstackgerritMerged openstack-infra/subunit2sql: Fix get_run_metadata to take UUID  https://review.openstack.org/25201920:33
*** rockyg has joined #openstack-infra20:33
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: Update Gerrit proxy configuration.  https://review.openstack.org/25205420:35
notmorganfungi, zaro, clarkb, mordred, ^ i think that should do it20:35
notmorganlocal test confirms i get redirects, /p is not proxied through, and no double slashes20:35
notmorganzaro: made it a new change-id for "just get it done", but added you as co-author since i snagged most of the stuff from your previous patch20:36
notmorganwe just need to test this on review-dev.o.o now to be sure.20:36
notmorgan(and provided jenkins isn't cranky)20:37
greghaynesrcarrillocruz: what username on that host?20:38
*** akshai has quit IRC20:38
zaronotmorgan: awsome!  i'm getting it on review-dev.now20:38
notmorganzaro: cool20:38
notmorganzaro: not sure how much we need the <proxy> section, but i left it since it *seemed* to work/not cause headaches20:38
notmorgani tried it both with/without locally20:38
rcarrillocruzjumphost or costkcicd-node005520:38
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a preview button for story descriptions  https://review.openstack.org/25205520:39
*** sabeen1 has quit IRC20:39
*** tsg has joined #openstack-infra20:40
notmorganzaro: oh we might need to remove the <proxy> section actually.20:40
notmorganre-reading the docs20:40
*** rajinir_ has joined #openstack-infra20:40
*** nmagnezi has quit IRC20:41
notmorganzaro: so i woouldn't be surprised if this broke in a wierd way until we remove the proxy section20:41
anteayalet's see20:42
* anteaya stands by to test review-dev20:42
notmorganbut still easy.20:42
*** otter768 has joined #openstack-infra20:42
*** kozhukalov_ has joined #openstack-infra20:42
*** AJaeger has joined #openstack-infra20:43
notmorganzaro: and i have a minor update to make the exclusions better before we finalize it.20:44
*** btully has quit IRC20:47
*** otter768 has quit IRC20:47
openstackgerritMerged openstack-infra/shade: Accept objects in name_or_id parameter  https://review.openstack.org/25104820:47
*** btully has joined #openstack-infra20:48
openstackgerritMerged openstack-infra/shade: Don't double-print exception subjects  https://review.openstack.org/25104920:50
*** achanda has quit IRC20:50
openstackgerritKaitlin Farr proposed openstack/requirements: Bump Castellan version to 0.3.1  https://review.openstack.org/24891120:51
*** tiswanso has quit IRC20:52
zaronotmorgan: applied your changes to review-dev.o.o20:53
notmorgantrying it out20:53
zarolooks like i can't view without logging in.20:53
*** maishsk has quit IRC20:53
notmorgannoticing that too20:53
AJaegerpabelanger: let's revert https://review.openstack.org/#/c/250098/ unless you have a better idea...20:53
openstackgerritMerged openstack-infra/shade: Enable running tests against RAX and IBM  https://review.openstack.org/25114920:53
notmorganok let me push my second set of changes and see if that solves it.20:53
zarothat was without the proxy section20:53
notmorganit's basically stripping out the <proxy *> section20:53
notmorganlet me check something locally20:54
*** tiswanso has joined #openstack-infra20:54
anteayaI see the same20:54
pabelangerAJaeger: sure20:54
jeblairianw: did you get the f23 info you needed?20:54
AJaegerpabelanger: do you want to propose it or shall I?20:55
pabelangerAJaeger: you can, about to jump into a meeting20:55
AJaegerwill do, pabelanger !20:55
*** annegentle has quit IRC20:55
notmorganzaro: ok so something changed, because i don't have that issue locally .20:55
zaronotmorgan: i assume we should add all the ProxyPass lines?  for replicate_local, contactstore, and robots?20:55
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/glean: Run glean after networking service  https://review.openstack.org/25203720:55
notmorgannot sure what is triggering it.20:55
notmorganhere hold on20:56
mtreinishnibalizer: if you get a sec on: https://review.openstack.org/251564 do you want me to respin it to try using the pip args option on the module?20:56
mtreinishor is it good enough as is?20:56
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: Update Gerrit proxy configuration.  https://review.openstack.org/25205420:56
notmorganzaro: ^ that should be more correct anyway20:56
zaronotmorgan: oops forgot to remove rewrite rule20:56
mtreinishI just want to get the latest subunit2sql pulled in so the dashboard works correctly again20:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Optimize gate-project-config-layout"  https://review.openstack.org/25206320:57
jeblairianw: the fedora-23 build is running now; there is no log file because it's not configured in the logging config file20:57
*** baoli has quit IRC20:57
notmorganand we can remove the ifversion if that works20:58
*** mnaser has quit IRC20:58
mordredianw: I thought I remembered you adding f23 to the logging config when we landed that patch?20:59
nibalizermtreinish: im in a car... but i have no preference, we need to redo the whole yibg without virtualenvs anywyays21:01
mtreinishnibalizer: yeah, that was gonna be my next step. But, I wanted to quickly unblock things first21:02
notmorganzaro: oooh, wow, that is broken :P21:02
anteayathat is interesting21:02
mtreinishnibalizer: would you have a preference if you weren't in a car? :)21:03
anteayain a train21:03
anteayaon a plane21:03
anteayaeating green eggs and ham?21:03
zaronotmorgan: not yet21:03
notmorganzaro: ah ok21:03
*** Piet_ has joined #openstack-infra21:04
*** asilenkov has quit IRC21:04
*** SpamapS is now known as TheKettle21:04
rockyganteaya, I do not like green eggs and ham21:04
*** TheKettle is now known as SpamapS21:04
* anteaya makes a note21:05
*** sdake has quit IRC21:06
nibalizermtreinish: no, since were changing it all soon21:06
rockyganteaya, btw do we have a samIam in OpenStack?21:06
fungirockyg: that's Sam-I-Am in irc21:06
*** mnaser has joined #openstack-infra21:07
anteayahe's everywhere21:07
*** salv-orlando has joined #openstack-infra21:07
*** asilenkov has joined #openstack-infra21:08
anteayabusy answering questions as usual in #openstack21:08
*** sdake has joined #openstack-infra21:08
zaronotmorgan: https://etherpad.openstack.org/p/25205421:08
ianwjeblair: oh, doh, thanks, please write that up to it being too early in the morning for me21:09
fungimtreinish: i believe one issue you're going to find with the openstack_health puppet module is something we ran into with the askbot module... vcsrepo ensure=>latest for a branch will always notify from installed to latest even if there is no new commit, so your execs will wind up firing on every puppet run anyway21:09
notmorganzaro: looking21:09
*** AJaeger has quit IRC21:09
notmorganthat should be it21:09
zaronotmorgan: that's what i've set for review-dev21:09
mtreinishfungi: do we specify a branch? the openstack-health repo should just be master21:09
zaronot working21:09
notmorganthen let me see what is different between my setup and that one21:10
notmorganit works locally..21:10
fungimtreinish: master is a branch21:11
mtreinishthat it is21:11
fungimtreinish: i mean instead of specifying a particular commit or tag21:11
fungiknown issue with the vcsrepo module21:11
mtreinishfungi: but I don't think I've seen that behavior look at puppetboard21:11
mtreinishmaybe I'm missing something21:11
*** changbl has quit IRC21:12
*** dprince has quit IRC21:12
openstackgerritIan Wienand proposed openstack-infra/system-config: Update nodepool logging for Fedora 23  https://review.openstack.org/24790921:12
ianwmordred: yeah, had fallen into merge conflict : ^21:12
*** sdake has quit IRC21:12
notmorganzaro: maybe21:13
notmorganzaro: ProxyRequests needs to not be disableD?21:13
notmorganmy config looks almost identical (just localhost instead) and it's working21:13
fungimtreinish: well, execs don't log to puppetdb21:13
fungii think it's a security measure or something21:13
mtreinishfungi: oh, that's true, but won't it show a vcs repo event too?21:14
*** sdake has joined #openstack-infra21:14
fungiahh, yeah it always did for askbot at least21:14
mtreinishfungi: like: http://puppetdb.openstack.org/report/health.openstack.org/8ba2ea5262ca8dd4e2a728981908e29803b79a6f21:14
zaronotmorgan: turned ProxyRequests on but no difference21:14
notmorganugh. weeeirdddd21:14
notmorgansec let me try something21:15
mtreinishfungi: as opposed to: http://puppetdb.openstack.org/report/health.openstack.org/3012b53966136447b7095dc4206d3e81073a533e (which pulled in a doc update)21:15
*** abregman has joined #openstack-infra21:15
fungianyway, if it21:16
*** maishsk has joined #openstack-infra21:16
zaronotmorgan: renenabled single line 'RewriteRule ^/(.*)$ http://localhost:8081/$1 [NE,P]' and it works again21:16
notmorganthat makes sense... i am wondering why the straight proxypass is not working21:16
fungimtreinish: if it's broken then the new change at least doesn't make it any more broken21:16
notmorganthe [NE,P] should work.21:16
mtreinishfungi: heh, true21:16
mtreinishfungi: anyway I'll have to find time to sit down and rewrite the module to not use a venv21:17
*** btully has quit IRC21:17
mtreinishbut thanks for the +A, that'll at least unblock things21:17
notmorganzaro: something else funky is going on here. it's like / is not matching ProxyPass21:17
notmorganzaro: oh i see21:17
notmorganzaro:  you have   ProxyPassReverse / http://localhost:8081/ nocanon21:18
notmorganthat should be ProxyPass21:18
notmorgannot ProxyPassReverse21:18
notmorganline 73 of the etherpad21:18
notmorganand ProxyRequests off should be fine to keep21:18
*** salv-orlando has quit IRC21:19
zaronotmorgan: ok, fixed that.21:19
*** kozhukalov_ has quit IRC21:19
anteayacan click on things when not signed in21:20
*** tjones has joined #openstack-infra21:20
zaroohh yeah! works21:20
notmorganzaro: and it looks like it is working!21:20
anteayacan sign in and end up back on patch that I was on21:20
notmorgan/p/ looks like it is not passed though (is offloaded)21:20
*** btully has joined #openstack-infra21:20
openstackgerritwill soula proposed openstack-infra/jenkins-job-builder: Fix pollscm backwards compatiblity  https://review.openstack.org/25206921:21
anteayaworks on firefox 42 and 4121:22
openstackgerritMerged openstack-infra/puppet-openstack_health: Make sure we install the latest requirements  https://review.openstack.org/25156421:22
notmorganand no double slash21:22
notmorganok cool i think we may have it21:22
zaronotmorgan: awsome, not sure if that is a valid config or does gerrit still need to be fixed?21:22
notmorganthis should be 100% valid21:22
notmorganand should not require gerrit upstream fixes21:22
notmorgannow...that mod_rewrite doesn't work is still crummy21:23
notmorganbut nolonger "our" problem21:23
zaroyeah, good at least there's a workaround.21:23
anteayaand chrome21:23
*** kgiusti has quit IRC21:23
notmorgananteaya: whataboutIE (don't throw things at me :P )21:23
anteayaha ha ha21:23
notmorgananyway. so yeah21:24
anteayanotmorgan zaro nice work21:24
notmorgani'm pretty happy with this change21:24
anteayait works21:24
notmorganlet me nuke the mod_version bit21:24
anteayaas far as I can see21:24
openstackgerritEverett Toews proposed openstack-infra/project-config: Publish the Python OpenStack SDK docs  https://review.openstack.org/24659121:24
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: Update Gerrit proxy configuration.  https://review.openstack.org/25205421:24
clarkbnotmorgan: catching up but agreed on dropping not 2.4 support21:25
notmorganclarkb: turns out, wont be an issue21:25
notmorganclarkb:  https://review.openstack.org/#/c/252054/3/templates/gerrit.vhost.erb appears to work21:25
notmorganclarkb: and is up on review-dev.o.o21:26
*** dulek has quit IRC21:26
clarkbnotmorgan: neat21:26
clarkbruagair: re mod auth openid I actually did end up poking around the code base in order to debug things21:26
notmorganalso, this is a clean config afaict, so we shouldn't need to worry about carrying anything forward21:26
clarkbruagair: it uses a pattern of throw exception on sucess21:26
*** abregman is now known as abregman|ZzZzZz21:26
clarkbruagair: I wanted to tear my eyes out21:26
*** maishsk_ has joined #openstack-infra21:27
zarodimtruck: gerrit redirect issue fixed ^21:27
*** maishsk has quit IRC21:27
*** maishsk_ is now known as maishsk21:27
anteayazaro: can you update review-dev again to use notmorgan's patch so that we are sure there are no typos?21:27
ruagairWhat are your thoughts on relying on it then?21:27
*** dulek has joined #openstack-infra21:27
clarkbruagair: if it works I don't mind relying on it and from what I can tell so far it does just work21:27
clarkbruagair: its the other side of the negotiation that is being silly21:27
ruagairThat has been my experience21:27
zaroanteaya: if there are typos it would not work.21:27
*** harlowja has quit IRC21:27
notmorganand we can probably drop the 3 proxy* directives L55-57 (cc zaro ) but it doesn't hurt us to keep them21:28
anteayazaro: have you deployed notmorgan's patch then?21:28
*** apoorvad has quit IRC21:28
phschwartzjeblair: so in my cut down ubuntu-trusty image I have been trying to build the issue with nodepool now on the node creation is it can't ssh in, the jenkins users that I am telling it to use does not exists so I need to see what I left out that creates the user :)21:28
*** harlowja has joined #openstack-infra21:28
notmorgananteaya: he has21:28
anteayanotmorgan: ah okay thank you21:28
notmorganthe only major difference is he commented things out instead of removed lines21:28
notmorganand his still has the un-used mod_version21:28
notmorgananteaya: here is the vhost config: https://etherpad.openstack.org/p/25205421:29
*** amitgandhinz has quit IRC21:29
anteayayes I saw that21:29
*** amitgandhinz has joined #openstack-infra21:29
notmorganbut it is the same patch result in the vhost21:29
*** hemna is now known as hemnafk21:29
anteayajust wanted to confirm that I review-dev is using your patch21:29
anteayanot something changed by hand21:30
notmorgananteaya: ah dunno how that is confirmed/done21:30
notmorganthen will defer to zaro and the rest of -infra21:30
anteayaright, that is why I was asking21:30
notmorganbut... we have working config thing(s)21:30
zaroanteaya: yeah, that was by hand.  but essentially having a working config is the most imporant21:30
anteayanotmorgan: yes, yay21:30
*** sdake has quit IRC21:30
*** salv-orlando has joined #openstack-infra21:31
anteayazaro: okay looking forward to testing when running off the patch, just to be sure21:31
*** alivigni has quit IRC21:31
*** mancdaz has quit IRC21:31
anteayaand glad for a working config21:31
zaroanteaya: yes, forthcoming21:31
anteayazaro: thanks tell me when21:31
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rewrite release_from_yaml.py as bash script  https://review.openstack.org/25157221:31
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add the release metadata to the release tag  https://review.openstack.org/25157321:31
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email  https://review.openstack.org/25157421:31
anteayaI'll fire up 41 again21:31
*** rcernin has quit IRC21:31
*** dizquierdo has joined #openstack-infra21:32
*** mancdaz has joined #openstack-infra21:32
clarkbnotmorgan: I don't think you can change the allow encoded slashes directive21:33
clarkbnotmorgan: that makes cgit work, but if its deployed on -dev you can test easily neough21:33
notmorganclarkb: it is like that on -dev21:34
*** sdake has joined #openstack-infra21:34
clarkband the cgit url looks ok21:34
clarkbmaybe that was fixed independently21:34
clarkbzaro: ^ do you know?21:34
*** salv-orl_ has joined #openstack-infra21:35
zaroyes, i believe it was an encode issue. /me looks for the change21:35
clarkbnotmorgan: and probably worth keeping the proxypassreverse directive so that if gerrit ever breaks that apache will do the correct thing21:35
clarkb(gerrit currently uses the canonicalurl to write those headers properly itself)21:35
*** salv-orl_ has quit IRC21:35
notmorganclarkb: hm sure we can do that.21:35
clarkbnotmorgan: but otherwise lgtm and if it allows us to use new gerrit without major hacks then +121:35
notmorganyeah no major hacks21:36
clarkbya it looks straightforward I like it21:36
*** salv-orlando has quit IRC21:36
*** sabeen has joined #openstack-infra21:36
*** salv-orl_ has joined #openstack-infra21:37
zaroclarkb: this change? https://review.openstack.org/#/c/236056/21:37
notmorgani'll add the proxypass reverse back in then.21:37
clarkbzaro: I think the one it depends on21:38
zaronotmorgan: would you mind updating etherpad with final changes then i'll apply that for further testing21:38
clarkbbut ya so ++ to using this method as a solution21:38
tsgfungi: wrt https://review.openstack.org/#/c/251607/, AJaeger mentioned in the morning that jenkins triggers were not working for project-config/jenkins/data/bindep-fallback changes - any ideas on how this can be worked around?21:38
clarkb(upstream should still fix but this should get us rolling)21:38
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: Update Gerrit proxy configuration.  https://review.openstack.org/25205421:38
notmorganzaro: doing that now21:39
tsgfungi, lifeless: AJaeger filed http://lists.openstack.org/pipermail/openstack-infra/2015-December/003513.html earlier in that context21:39
clarkbtsg: the suggestion on the mailing list was to just run all the jobs againts all changes21:39
lifelesstsg: huh? I'm lost21:39
fungitsg: yeah, it was discussed some, we have a couple paths forward. that change is not blocking anything though (it's just bookkeeping for the moment) so if it sits a little it's fine21:40
*** salv-orl_ has quit IRC21:40
fungithe system-config change shouldn't be blocked on getting that merged21:40
*** rhallisey has quit IRC21:40
notmorganzaro: ok i think i have it. do we want to drop the 3 proxy* directives that look to be superfluous?21:40
clarkbnotmorgan: if they are superfluous I would remove them to reduce confusion21:40
notmorganclarkb: i *think* they are21:41
notmorganme 2x checks21:41
*** sdake has quit IRC21:41
*** salv-orlando has joined #openstack-infra21:41
*** sabeen3 has joined #openstack-infra21:41
tsgfungi, clarkb: ok :) glad to hear that patch 247936 and patch 251607 are not blocked then21:42
notmorganok you know what, i feel better keeping these21:42
notmorganclarkb: ^ the Via and ProxyPreserveHost seem sane/worth keeping21:42
notmorganthe proxyrequestsoff should have no impact but is never harmful21:42
notmorganso, lets just keep them21:42
clarkbnotmorgan: wfm21:42
*** sabeen has quit IRC21:43
tsgclarkb, lifeless, fungi: will wait for your +2 on https://review.openstack.org/#/c/247936/ and https://review.openstack.org/#/c/251607/.  thanks!21:43
*** krtaylor has joined #openstack-infra21:44
fungitsg: i already +2'd the system-config one21:45
*** salv-orlando has quit IRC21:45
*** salv-orlando has joined #openstack-infra21:46
tsgfungi: thanks! (I see that you okayed the other one as well).  Just need one more +2 now :)21:46
zaronotmorgan: ok, what's currently on the etherpad has been applied.  seems good21:46
zaroclarkb: not sure which change fixed the cgit links but they look ok to you on review-dev?21:47
*** gildub has joined #openstack-infra21:47
clarkbzaro: the change you linked's parent was the fix21:47
clarkbbefore that we needed the no encode slashes thing21:47
*** btully has quit IRC21:48
*** lykinsbd has quit IRC21:49
tsgfungi: Ah, so job filtering is still blocking from the project-config change being merged :)21:49
openstackgerritSteve McLellan proposed openstack/requirements: Cap elasticsearch client <2.0  https://review.openstack.org/25207521:50
*** mrmartin has quit IRC21:50
*** abregman|ZzZzZz is now known as abregman21:50
fungitsg: right, but the project-config change shouldn't block anything else. as i said it's purely bookkeeping for now, until we start using that file21:50
*** bswartz has quit IRC21:51
tsgfungi: ah I see what you are saying now!21:51
anteayazaro: review-dev continues to work for me21:51
zaroanteaya: review-dev has the final apache configs.  still hand applied.  we cannot merge that patch until we switch to 2.11 because that patch is for both review-dev and review.o.o21:51
tsgfungi: makes sense.  https://review.openstack.org/#/c/247936/ should be good enough for Swift.21:51
anteayazaro: okay thanks21:51
tsglifeless: can you please revert your -1 on https://review.openstack.org/#/c/247936/?21:52
*** dizquierdo has quit IRC21:52
* anteaya tests with firefox 41 now21:52
anteayafirefox 41 and chrome are both still happy with review-dev21:54
anteayalogged out and logged in21:54
*** btully has joined #openstack-infra21:54
*** mase_x200 has joined #openstack-infra21:54
*** salv-orlando has quit IRC21:54
* zaro taking a small break to celebrate this win!21:54
anteayalogged out off all three browsers and still able to navigate around21:55
anteayanice work21:55
*** salv-orlando has joined #openstack-infra21:55
zaronotmorgan: would you be able to update the associated upstream bug with any relevant info regarding your  workaround?21:57
*** thorst has quit IRC21:57
jeblairphschwartz: i think our puppet does that; clarkb did we end up with a dib element to stick in a custom ssh key?21:58
phschwartzjeblair: I will have to look, I don't know if openstackci handles doing it21:58
*** trown is now known as trown|outttypeww21:59
*** weshay_xchat has quit IRC22:00
*** apoorvad has joined #openstack-infra22:00
notmorganzaro: hm sure. i can.22:01
notmorgani mean, mostly our workaround is using their recommendation22:01
*** thorst has joined #openstack-infra22:02
*** Sukhdev has joined #openstack-infra22:02
jeblairasselin: ^ ?22:02
*** lykinsbd has joined #openstack-infra22:02
*** hashar has quit IRC22:03
clarkbjeblair: its part of the puppet run in dib builds but ya you set an env var and it gets passed through22:05
*** EricGonczer_ has quit IRC22:05
clarkbsimilar to how it works with snapshot builds22:05
*** thorst has quit IRC22:06
jeblairclarkb: so a user dropping our puppet would need to replace that, but one using our puppet can still pass in their own ssh key, yeah?  (cc phschwartz)22:07
clarkbjeblair: yup22:07
clarkbNODEPOOL_SSH_KEY is env var and its set by openstackci as nodepool_ssh_key iirc22:07
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Update Gerrit proxy configuration.  https://review.openstack.org/25205422:08
pabelangerjeblair: clarkb: I switched to devuser recently.  Trying that out for dibs22:08
pabelangerseems to work right now22:08
clarkbpabelanger: that would be another way to do it22:09
clarkbshould work fine alongside our stuff I think22:09
*** tiswanso has quit IRC22:09
*** baoli has joined #openstack-infra22:11
*** EricGonczer_ has joined #openstack-infra22:11
*** salv-orlando has quit IRC22:12
openstackgerritgordon chung proposed openstack/requirements: bump ceilometerclient to 2.1.0  https://review.openstack.org/25208222:13
*** salv-orlando has joined #openstack-infra22:13
*** weshay_xchat has joined #openstack-infra22:13
openstackgerritgordon chung proposed openstack/requirements: bump ceilometerclient to 2.1.0  https://review.openstack.org/25208222:14
*** baoli has quit IRC22:14
*** dtroyer has quit IRC22:14
*** dtroyer has joined #openstack-infra22:15
openstackgerritDoug Hellmann proposed openstack-infra/project-config: remove direct-release option from projects  https://review.openstack.org/24892322:15
*** shardy has quit IRC22:15
openstackgerritDoug Hellmann proposed openstack-infra/jeepyb: change the default from delayed-release to direct-release  https://review.openstack.org/24892222:16
*** burgerk has quit IRC22:18
*** annegentle has joined #openstack-infra22:18
*** salv-orl_ has joined #openstack-infra22:18
*** baoli has joined #openstack-infra22:18
*** thorst has joined #openstack-infra22:19
*** harlowja has quit IRC22:19
notmorganzaro: i would comment on the original bug, but it seems like the tracker is down.22:19
*** maishsk_ has joined #openstack-infra22:19
notmorganzaro: (can't connect to code.google.com atm)22:19
*** maishsk has quit IRC22:20
*** maishsk_ is now known as maishsk22:20
*** ayoung has quit IRC22:21
*** harlowja has joined #openstack-infra22:21
*** gyee has joined #openstack-infra22:22
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor: Decouple log processing from logstash  https://review.openstack.org/24785722:22
clarkbjesusaurus: ^ there the log processor users and service stuff is decoupled from logstash22:22
anteayanotmorgan: neither can I22:22
clarkbI wonder if this is the beginning of turning google code off22:23
*** salv-orlando has quit IRC22:23
anteayais that what they are doing?22:23
*** gordc has quit IRC22:24
*** thorst has quit IRC22:24
clarkbanteaya: yes google code is going away sometime soon, was announced about a year ago iirc22:24
notmorganclarkb: interesting, not sure how that works with the issue trackers and such not being elsewhere [yet]22:24
*** baoli has quit IRC22:24
*** mwagner_lap has quit IRC22:24
anteayaclarkb: oh interesting22:24
anteayait was nice of them to keep it up for our meeting22:25
*** baoli has joined #openstack-infra22:25
anteayafor some reason I see no content22:26
anteayaeven in chrome22:26
openstackgerritApoorva Deshpande proposed openstack-infra/ciwatch: Change ciwatch.log location from /var/data/ciwatch.log to /var/log/ciwatch/ciwatch.log  https://review.openstack.org/25205022:26
*** Sukhdev has quit IRC22:26
*** tsg has quit IRC22:27
anteayafound it, the link had wrapped in my weechat and took off with the file extention22:27
anteayamy fautl22:27
timrcDear lazy #openstack-infra, was Jenkins gearman-plugin ever upgraded to the latest available version?22:28
jesusaurusclarkb: awesome22:28
clarkbtimrc: we run automated rolling restarts of the masters now so if puppet was updated to install the latest version it should be running22:29
*** maishsk has quit IRC22:29
*** maishsk has joined #openstack-infra22:29
clarkbthat happens at 0000UTC saturdays ish22:30
clarkb(its rolling so doesn't happen all at once)22:30
anteayathe post says android and chrome are staying where they are22:31
anteayaI wonder where that puts gerrit development22:31
pleia2I think gerrit was in the crew of projects that were kept hosted, along with android and chrome22:34
timrcclarkb: Where are Jenkins plugins defined in puppet now?  Not seeing it in system-config (well I am, but only for jenkins_dev)22:34
*** baoli has quit IRC22:34
* timrc checks out puppet-openstackci22:35
*** EricGonc_ has joined #openstack-infra22:35
timrcSo... if I'm to believe puppet-openstackci/manifests/jenkins_master.pp we22:36
timrcwe're running a pretty old version upstream.22:36
anteayapleia2: well it said they would provide git and gerrit hosting for android and chrome22:37
anteayaI hope they mean by extention they will also continue to provide git and gerrit hosting for gerrit22:37
*** EricGonczer_ has quit IRC22:37
pleia2anteaya: yeah, what I'm saying is that I am quite sure they ultimately included gerrit as well, though it wasn't lsited on that old blog post22:38
timrcPuppet is saying 0.1.1 which came out in 2014.  Latest version is 0.1.3 which came out Oct 9, 201522:38
anteayapleia2: I hope so22:38
pleia2anteaya: well, they shut down people using google code for their personal projects some time ago, so I'd think it would have moved already if it was going to :)22:38
clarkbtimrc: we are running on jenkins0122:38
timrczaro: I vaguely remember you saying that upstream was running something > 0.1.1?22:38
pleia2ah, meeting22:38
*** peristeri has quit IRC22:38
anteayapleia2: you have a meeting now?22:39
timrcclarkb: *blink**blink*22:39
clarkbaccording to jenkins0122:39
timrcclarkb: Doesn't seem like puppet is controlling that.22:40
*** amitgandhinz has quit IRC22:40
timrcclarkb: Do you include plugins in your magical tarball of Jenkins fun?22:40
anteayapleia2: I don't see a meeting in ical, but happy meeting22:41
clarkbtimrc: they are but I thought that puppet was controlling the on disk state of the plugins just not restarting master to pull them in22:41
*** mase_x200 has quit IRC22:41
clarkband now we have ansible restarting masters safely once a week22:42
*** otter768 has joined #openstack-infra22:43
*** tsg has joined #openstack-infra22:43
*** megm has quit IRC22:44
*** edmondsw has quit IRC22:44
*** mcarden_ has joined #openstack-infra22:45
*** sflanigan has quit IRC22:45
*** pahuang has quit IRC22:45
*** davideagnello has quit IRC22:45
*** megm has joined #openstack-infra22:45
*** davideagnello has joined #openstack-infra22:45
*** sflanigan has joined #openstack-infra22:45
*** sflanigan has joined #openstack-infra22:45
*** pahuang has joined #openstack-infra22:45
*** mcarden has quit IRC22:45
timrcclarkb: Maybe the problem is 0.1.1 evaluates to < so puppet is shrugging.22:45
*** hemnafk is now known as hemna22:45
*** mcarden_ is now known as mcarden22:46
*** thorst has joined #openstack-infra22:47
*** dtardivel has quit IRC22:47
*** davideagnello has quit IRC22:48
*** otter768 has quit IRC22:48
clarkbjesusaurus: woot https://review.openstack.org/#/c/247756/ passes testing with the depends on user changes I made22:48
*** davideagnello has joined #openstack-infra22:48
clarkbjesusaurus: so ya I think thats the first step is decoupling it all like that22:48
openstackgerritTimothy R. Chavez proposed openstack-infra/puppet-openstackci: Rev gearman-plugin to latest version  https://review.openstack.org/25209422:48
*** thorst_ has joined #openstack-infra22:48
clarkbanteaya: ^ so some progress on logstash 2.0 things there22:49
* anteaya looks22:49
anteayathe deb package does config?22:50
*** davideag_ has joined #openstack-infra22:50
clarkbanteaya: ya it replaces much of our puppet stuff22:51
*** r-daneel has quit IRC22:52
*** austin81 has left #openstack-infra22:52
*** thorst has quit IRC22:52
SpamapSjeblair: If you have a moment to reset on counter-inspection.. I added some documentation to the code itself, and made sure 'tox -edocs' builds docs that reference it. https://github.com/SpamapS/openstack-qa-tools/commit/9fa4ce73a242b29922e76e42d1cb3339ab0ed27e22:53
*** davideagnello has quit IRC22:53
SpamapSjeblair: just in case you've fully purged context on that... we were talking about documenting what exactly is being collected.22:54
*** bdemers has quit IRC22:55
*** chmouel has quit IRC22:55
*** chmouel has joined #openstack-infra22:56
clarkbSpamapS: fwiw tox -e docs isn't a thing that jenkins will do, you need to make sure python setup.py build_sphinx works22:56
*** salv-orl_ has quit IRC22:56
SpamapSclarkb: do you happen to know how to make that run 'sphinx-apidoc' ?22:57
SpamapSor the equivalent?22:57
*** harlowja has quit IRC22:57
jeblairSpamapS: pbr?22:57
SpamapSI used cookiecutter22:57
*** dingyichen has joined #openstack-infra22:57
*** dchen has joined #openstack-infra22:57
*** dchen has quit IRC22:57
clarkbnot off hte top of my head but it builds using your conf.py so i think you can configure that to be part of your sphinx builds?22:58
*** mriedem has quit IRC22:58
jeblairSpamapS: i don't really understand why there's code in github?22:59
SpamapSjeblair: https://review.openstack.org/#/c/244428/ hasn't merged22:59
jeblairSpamapS: it has an unmerged depends-on22:59
SpamapSyep, the spec, because apparently, nobody reviews QA specs.23:00
SpamapSexcept the PTL :-P23:00
jeblairthey are not on my regular reading list23:00
jeblairat any rate, i feel like if you're developing code on github and asking people to review it there because a spec hasn't been approved, the train has left the rails and maybe we should put it back on :)23:01
*** lykinsbd has quit IRC23:01
*** harlowja has joined #openstack-infra23:01
*** dkranz has quit IRC23:02
*** dingyichen has joined #openstack-infra23:02
SpamapSjeblair: ok. :-P I really don't know what to do other than respond when there _are_ reviews for the spec. :-P23:02
*** regXboi has quit IRC23:02
jeblairSpamapS: maybe we can drop the depend-on and import the repo?23:03
SpamapSjeblair: if that's appropriate. I made it dependent because QA was taking on the responsibility for maintaining the repo.23:03
SpamapSthere's a fuzzy line there23:04
jeblairi see specs as a way to get agreement on work before it's done, to ensure that it doesn't need to be re-done later; if the work is already done... meh.23:04
SpamapSsince it will be primarily called by devstack-gate, and the subunit2sql worker23:04
SpamapSand potentially anything else that wants to snapshot counters whilst running OpenStack.. :-P23:05
*** annegentle has quit IRC23:05
* mordred also votes for removing the depends on23:05
mordredthe train has left the station WRT this work - it's happening and you're moving it forward - no need to block on people agreeing with all the details of the spec23:06
mordredclarkb, jeblair, SpamapS: wanna review some tests? and by that - I mean "be amazed that mordred was involved in writing tests"23:07
jeblairSpamapS: once it's imported, we can have a docs-draft job that builds docs (that will fail because of the lack of apidoc thing)23:08
jeblairuntil we fix that23:09
*** abregman has quit IRC23:09
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/project-config: Add os-performance-tools  https://review.openstack.org/24442823:10
jeblairSpamapS: just so i'm not completely being a jerk, i've taken the unusual step of attempting to read that github diff, but i'm not seeing the description of counters...23:10
*** dimtruck is now known as zz_dimtruck23:10
jeblairi see counters_ but don't understand what that would link to if i saw a real build23:10
SpamapSjeblair: hah, I'd fully understand not wanting to read the github diff. :-D23:10
*** binbincong has quit IRC23:10
jeblairSpamapS: i mean, i'm completely failing at it :)23:10
*** binbincong has joined #openstack-infra23:13
*** eharney has quit IRC23:15
*** EricGonc_ has quit IRC23:16
*** sdake has joined #openstack-infra23:16
*** rockyg has quit IRC23:16
crinklercarrillocruz: greghaynes i added my findings to https://review.openstack.org/#/c/252037/23:17
jesusaurusclarkb: was it not passing before the decoupling? or is this an advantagous time to decouple?23:18
clarkbjesusaurus: it wasnt passing because a lot of the resources it was using have been replaced by the deb package23:19
jesusaurushurray for packaging23:19
rcarrillocruzcrinkle: ++, appreciate your experiences23:20
rcarrillocruzas for the local-filesystems23:20
rcarrillocruzyou're right23:20
clarkband I don't want to install the deb package everywhere those daemons run since it will start an unnecessary logstash process on logstash.o.o23:20
rcarrillocruzi initially removed it on purpose as i thought the networking service itself depended on local-filesystem23:20
rcarrillocruzbut apparently not in all cases23:20
rcarrillocruzi see this on the /etc/init/networking.conf23:20
rcarrillocruzstart on (local-filesystems23:20
rcarrillocruz    and (stopped udevtrigger or container)) or runlevel [2345] or stopped networking RESULT=failed PROCESS=post-stop EXIT_STATUS=10023:20
rcarrillocruzso yeah, i better put local-filesystems to honour that in all events23:21
rcarrillocruzi'll push a patch tomorrow23:21
crinklercarrillocruz: cool23:22
clarkbrcarrillocruz: I would suggest changing that behavior for systemd and any other init system too if they also do per interface23:22
clarkbso that its consistent across platforms, makes debugging easier as we transition to a systemd world23:22
rcarrillocruzi'm not familiar with systemd i'm afraid, i welcome either follow-up patchset or a follow up patch for non-upstart23:23
rcarrillocruzbut let's really be sure first that's the right approach23:23
rcarrillocruzlet's put local-filesystem23:23
rcarrillocruzcrinkle:when i push the patch, i appreciate if you could try that out on your deploys23:24
*** mase_x200 has joined #openstack-infra23:24
rcarrillocruzso we can aggregate results between the two regions and determine whether it's a safe fix23:24
crinklercarrillocruz: sure, will do23:24
rcarrillocruz( i think it is, but as you said the fact that per-interface was chosen makes me wary )23:25
crinklercarrillocruz: like i mentioned, though, fixing it this way sort of breaks our dib workaround for the bridged interface23:25
crinkleso we'll have to figure something out for that23:25
*** david-lyle has quit IRC23:25
rcarrillocruzyeah, i need to read carefully what's that eth2 all about...23:25
rcarrillocruzcos i *just* was aware of it by you23:25
clarkbcrinkle: is eth2 where the weird bridge to handle neutron things is happening?23:25
crinkleclarkb: yeah, everything goes through eth223:26
clarkbI wonder if you could just have glean go first then run that after23:26
clarkbif maybe glean emits a glean specific event too23:26
clarkboh except its baked into the image23:26
crinklewe could just move the eth2.25 and br-vlan25 into its own eni file23:27
crinklei think if glean manages eth2 it'll do the right thing23:27
clarkbthat sounds reasonable, then glean can do eth2 and everything will work together23:27
*** mwagner_lap has joined #openstack-infra23:27
rcarrillocruzcrinkle: can you expand what's that element about ?23:27
rcarrillocruzcos the change is for the whole bifrost thing and i lack context i'm afraid23:28
mgagnejeblair still at the office? I'm free now for debugging23:28
jeblairmgagne: o/23:28
jeblairmgagne: do you want me to delete the current node and then launch a new one?23:29
*** xyang1 has quit IRC23:29
mgagnejeblair yea, gimme a second, enabling debug right now23:29
*** mase_x200 has quit IRC23:29
mgagnejeblair ok done23:29
crinklercarrillocruz: greghaynes or SpamapS could explain it better, but the issue is that we need this bridge created for neutron to connect to, and the only mechanism we have to do that right now is to create that static file and let the bridge dhcp23:29
jeblairmordred: root's clouds.yaml on puppetmaster has disappeared23:30
greghaynesYa, when I am at a keyboard rather than phone ;)23:30
clarkbcrinkle: rcarrillocruz aiui the reason neutron connects to it is if you have neutron create br-ex directly against eth2 it blows away all the config there23:30
*** darrenc is now known as darrenc_afk23:31
greghaynesBut yes - we need a bridge interface and a phy added to that bridge with our networking connectivity23:31
clarkbso instead you set up a virtual siwtch in front of it so that eth2 works fine when neutron attaches to the switchport23:31
greghaynesYou can't express that in glean/configdrive23:31
*** kfarr has quit IRC23:31
mordredjeblair: I moved it aside to ~/.config/openstack/clouds.yaml - but there is now an /etc/openstack/clouds.yaml that should work23:31
mgagnejeblair 975a6a12 ?23:31
mordredjeblair: which is managed by the puppet23:31
jeblairmgagne: yes23:32
mordredjeblair: I moved it aside to ~/.config/openstack/clouds.yaml.save - but there is now an /etc/openstack/clouds.yaml that should work23:32
jeblairmordred: neat; there are no openstackjenkins accounts in there23:32
mordredjeblair: yah. because we don't want ansibl einventory to look at the jenkins accounts - there are tons of nodepool instances in there23:33
* rcarrillocruz wonders why we couldn't just have untagged interface on other nic, one for prov and other for management interface to ease things23:33
mordredjeblair: there is another file in puppet23:33
rcarrillocruzanyway, too late i guess23:33
mordredjeblair: that I wasn't sure where to put23:33
mordredjeblair: that has entries for all the clouds like the one from root's homedir did23:33
mordredjeblair: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppetmaster/all-clouds.yaml.erb23:34
*** mase_x200 has joined #openstack-infra23:34
greghaynesrcarrillocruz: we would need two nics with tagged because of what clarkb described, but yes, having ops change that would fix it23:34
*** e0ne has joined #openstack-infra23:34
jeblairmordred: ah yeah; let's put it somewhere, either root's homedir or /etc are fine... just something so we can make an alias of "/path/to/venv/bin/openstack --something /path/to/all-clouds.yaml"23:35
jeblairmordred: what is --something, by the way?23:35
mordredjeblair: hrm. interesting question. there is an env var which can point to a different one ... lemme poke for a second23:36
jeblairmordred: (so that then i can do 'nifty-openstack-alias --os-cloud openstackjenkins-internap')23:36
*** e0ne has quit IRC23:36
*** sdake has quit IRC23:36
mordredjeblair: ++23:36
mordredjeblair: is an env-var good enough for short-term?23:36
*** e0ne has joined #openstack-infra23:36
jeblairmordred: yeah23:37
jeblairi don't know if we can put  "OS_SOMETHING=/path/to/all-clouds.yaml /path/to/venv/bin/openstack" in an alias, but it can be a one-liner script if needed23:37
mordredjeblair: OS_CLIENT_CONFIG_FILE23:37
*** ayoung has joined #openstack-infra23:37
mordredjeblair: OS_CLIENT_CONFIG_FILE=/path/to/all-clouds.yaml openstack23:38
mordredjeblair: also, why don't we just install python-openstackclient ?23:38
lifelessmordred: have you written your 'never remove stuff' spec yet ?23:38
mordredlifeless: no23:38
lifelessmordred: I'd like to reference it23:38
jeblairmordred: wfm23:38
mordredjeblair: I actually feel like I wrote a puppet patch to do that23:38
mordredjeblair: it's there already23:39
*** sdake has joined #openstack-infra23:39
*** e0ne_ has joined #openstack-infra23:39
mordredso just "OS_CLIENT_CONFIG_FILE=/path/to/all-clouds.yaml openstack" will work23:39
jeblairmordred: oh, whatever is installe system-wide is fubar23:39
mordredit is? AWESOME23:39
jeblairmordred: run "openstack --help"23:39
jeblairi just assumed it just accidentally got installed and made my own venv.23:40
timrcTIL glean exists.23:40
timrcSeriously behind the times here.23:40
mordredjeblair: that's not broken23:40
mordredjeblair: that's just it being stupid23:40
mordredjeblair: it's how help works if you don't have auth creds specified23:40
*** mase_x200 has quit IRC23:40
mordredjeblair: don't EVEN get me started23:40
mgagnejeblair enabling more debug on more services. The request lands on the compute node, something update network cache update and then nothing23:40
jeblairmordred: that does not happen in my venv23:40
mordredjeblair: try: openstack --os-cloud=openstackci-rax --os-region=DFW server list23:40
timrcMaybe I should have used glean when doing the node uuid stuff.23:40
greghaynestimrc: hrm?23:41
*** e0ne has quit IRC23:41
jeblairmgagne: let me know when you want me to delete/relaunch23:41
mgagnejeblair now? =)23:41
*** dstanek has quit IRC23:41
*** dstanek has joined #openstack-infra23:41
jeblairmgagne: in progress23:42
mordredjeblair: patch coming for the other tihng - how about /etc/openstack/all-clouds.yaml ?23:42
mgagnefound your instance, checking23:42
*** MarkAtwood has quit IRC23:42
jeblairmordred: as root, ". shade-venv/bin/activate; openstack --help"23:42
jeblairmordred: it does not spew that stuff23:43
mordredjeblair: awesome23:43
*** zz_ja has quit IRC23:43
mgagnejeblair this one is active23:44
jeblairmordred: /etc/openstack/all-clouds.yaml wfm23:44
mgagnejeblair if you could create more, that would be great23:44
jeblairmgagne: i agree.  apparently the problem was lack of debugging.  :)23:44
jeblairmgagne: cycling now23:44
jeblairmgagne, mordred: have we sucessfully comped this account?23:45
mgagnejeblair yea, the quantum state. not working until observed23:45
jeblairmgagne: if so, and it's not tied to mordred's credit card, i can spin up a bunch at once :)23:45
*** zz_ja has joined #openstack-infra23:45
mgagnejeblair I'll do proper follow up on that email23:45
jeblairokay, i'll restrain myself for now23:45
timrcgreghaynes: We write the instance UUID into /etc/nodepool/uuid -- I believe that info is provided by config-drive.23:46
mgagnebilling is per minute AFAIK23:46
clarkbtimrc: I would not rely on that being available on all clouds23:46
clarkbtimrc: since config drive may not be in all clouds23:46
clarkband metadata server is almost useless23:46
greghaynestimrc: ah, glean just does as little as possible to get a usable host, so probably not the right place to add it23:46
mordreddeath to metadata server23:46
timrcclarkb: Ah yeah, if we cannot assume that *our* clouds have it available, it's just as well we don't use it.23:47
*** amitgandhinz has joined #openstack-infra23:47
greghaynesThat too23:47
mordredclarkb: config-drive is available everywhere now that I know of23:47
mordredbut I agree, it's not worth it for the nodepool case23:47
timrcthe metadata server is evil... config-drive seemed decent.23:47
clarkbmordred: internap, ovh, bluebox all alow it?23:47
clarkbI just worry that tomorrow $nextcloud will show up and not support it23:47
clarkband it isn't a thing we actually need to rely on here23:47
mordredwell, I agree with that23:48
timrcAye... if it's not something that's guaranteed (or near enough) ubiquitous, relying on it is probably not a good thing to do.23:48
mordredbut I think config-drive is bog-standard these days23:48
mordredit was only not standard back in folsom or whatever hpcloud v1 was23:48
clarkbmordred: well rax only just added it recently too23:48
mgagnejeblair I don't have write access to billing, only read: Discount: 0.00%. Let me know if mordred would like more discount :D23:49
timrcI thought rax had config-drive for awhile now?23:49
*** e0ne has joined #openstack-infra23:49
clarkbtimrc: they do now but its relatively new23:49
timrcAh, okay.23:49
mordredclarkb: I'll run a test on all 18 clouds to double check23:49
pleia2anteaya: I had a call, not openstack project meeting :)23:50
*** tjones has quit IRC23:50
*** e0ne_ has quit IRC23:51
*** weshay_xchat has quit IRC23:51
openstackgerritMonty Taylor proposed openstack-infra/system-config: Install the all clouds yaml file  https://review.openstack.org/25210623:51
*** amitgandhinz has quit IRC23:51
jeblairmgagne: discount 0% sounds like 'full price'23:51
mordredjeblair: ^^ there ya go23:51
*** darrenc_afk is now known as darrenc23:52
*** e0ne_ has joined #openstack-infra23:52
mordredjeblair: it's a cinderclient version fight between shade and python-openstackclient23:53
anteayapleia2: ah okay, well I hope you had a good call23:53
jeblairmgagne: 6d615d86-e628-4728-ada6-2fb0fd18c4da23:53
mgagnejeblair stuck?23:53
jeblairmgagne: looks like it23:54
*** e0ne has quit IRC23:54
*** Kennan has quit IRC23:55
*** tjones has joined #openstack-infra23:56
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove cinderclient version pin  https://review.openstack.org/25210923:56
mordredjeblair: ^^ that fixes the issue you had with python-openstackclient on puppetmaster23:56
mordredand amusingly enough, it's time for a release anyway, so it'll be well timed to get that rolled out :)23:57
*** rbrndt has quit IRC23:57
*** Kennan has joined #openstack-infra23:57
*** jklare has quit IRC23:57
mordredclarkb: could I sucker you in to doing a review on some functional tests?23:58
mordredclarkb: 251150 ?23:58
jeblairmordred: just to be sure, cinderclient 1.2 won't work?23:58
*** signed8bit is now known as signed8bit_ZZZzz23:58
clarkbmordred: let me finish reviewing this last nodepool image builder change23:58
mordredclarkb: kk.23:58
mordredjeblair: you want me to put in a != 1.2 ?23:58
*** jklare has joined #openstack-infra23:58
jeblairmordred: no, i was asking if >=1.2 will work23:59
mordredjeblair: yes. 1.2 will not work with any public cloud23:59
*** salv-orlando has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!