Wednesday, 2016-03-16

*** mriedem has joined #openstack-infra00:01
*** sdake has quit IRC00:01
*** dchen has joined #openstack-infra00:03
*** Swami has quit IRC00:03
*** dchen has quit IRC00:03
*** pahuang has quit IRC00:04
*** camunoz has quit IRC00:04
*** aeng has quit IRC00:04
*** sflanigan has quit IRC00:05
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add zookeeper test infrastructure  https://review.openstack.org/29317600:05
openstackgerritMerged openstack/diskimage-builder: Don't cache debootstrap rootfs by default  https://review.openstack.org/28053600:06
fungiianw: clarkb: if we merge 264726 we're going to need another manual nodepool image builder restart for now, yeah?00:06
zigofungi: clarkb: Is it possible for me to actually download an image built by infra?00:07
jeblairclarkb, greghaynes: i realize i'm getting ahead of the spec, but i've been exploring zookeeper to validate my understanding before formally proposing the spec, and i wanted to make sure it was testable.  so i went ahead and wrote 293176.00:07
clarkbfungi: yes00:07
fungizigo: not without help at the moment, and also these images bake specific access keys into them so you'd need to modify it before boot to be able to log in anyway00:07
jeblairclarkb, greghaynes: i have also put the zk spec on the next infra meeting agenda00:07
greghaynesjeblair: nice!00:07
*** dims has joined #openstack-infra00:08
*** sflanigan has joined #openstack-infra00:08
clarkbfungi: zigo or use metadata service to slip your key in00:08
fungioh, true00:08
*** sdake_ has quit IRC00:08
fungii keep forgetting about the metadata service since we shun it like a plague00:08
SergeyLukjanovjeblair hey, zk == zookeeper?00:08
zigoOk.00:08
clarkbwell config drive works too but that general thing00:08
openstackgerritMerged openstack-infra/system-config: Add #osic to meetbot  https://review.openstack.org/29179800:08
fungizigo: on the other hand, once we get one built and booted i can hold it and add your key so you can log in and inspect how it comes up00:09
jeblairgreghaynes: i still want to sync up with you on the last little bit about whether the builder should have an internal scheduler; i know you're busy, so we can do that whenever.  i figure we're largely in agreement so we probably don't have to block on that.00:09
jeblairSergeyLukjanov: yes00:09
fungizigo: or rather, inspect the result once it's up00:09
prometheanfiregreghaynes: ianw: how to non-voting job?00:09
SergeyLukjanovjeblair oh, I see the change, actually if you're using kazoo there is a fake zk impl00:09
zigofungi: I was thinking of getting the image, so I could re-design the build tool I have to adapt it to the image.00:09
jeblairSergeyLukjanov: oh neat!00:09
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix issue with non-ascii characters in get_job_md5  https://review.openstack.org/28305000:09
zigoDefinitively, having a copy of the image would help me.00:09
SergeyLukjanovjeblair lemme find00:10
*** ianw has quit IRC00:10
*** MarkAtwood has joined #openstack-infra00:10
prometheanfirezigo: I'm actually going to need to do the same thing eventually (infra built images, etc)00:10
greghaynesjeblair: Ya, so what I am trying to ague against really is trying to make the scheduler be shared-nothing. It can still be internal to the builder00:10
openstackgerritMerged openstack-infra/jenkins-job-builder: Allow using lockfile per jenkins master  https://review.openstack.org/29240500:11
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix ircbot's any-failure parameter documentation  https://review.openstack.org/27170700:11
greghaynesjeblair: I worry that doing a shared-nothing scheduler is harder with the upside that itll scale a ton better, but we dont actually need that here00:11
*** sdake has joined #openstack-infra00:11
jeblairgreghaynes: by scheduler, do you mean the nodepool daemon?00:11
jeblairgreghaynes: the thing that decides how many nodes to launch?00:11
greghaynesjeblair: nope, the builder daemon00:12
SergeyLukjanovjeblair there are helpers available for zk testing - https://kazoo.readthedocs.org/en/latest/testing.html and zake is the lib providing mocks for kazoo, it actually have the same API as kazoo https://pypi.python.org/pypi/zake/ and it's supporting mostly all usually needed stuff00:12
*** Jeffrey4l has joined #openstack-infra00:12
*** yamamoto_ has quit IRC00:12
SergeyLukjanov(btw zake was written by harlowja AFAIK)00:13
jeblairSergeyLukjanov: awesome, i missed that; i should definitely look into that...00:13
*** adrian_otto has quit IRC00:13
SergeyLukjanovjeblair it's used in ceilo and tooz00:13
*** gildub has joined #openstack-infra00:13
jeblairSergeyLukjanov: the main things i want to be sure of are: 1) we get the same behavior (eg, watches on nodes, etc).  2) parallel tests can run without interfering with each other.00:13
jeblairSergeyLukjanov: i looked at tooz and saw that it set up a zookeeper before running tox.  so i based that change on that idea, except that i did it inside of the unit test framework so we get zookeeper-per-test rather than a single one00:14
*** armax has joined #openstack-infra00:14
*** armax has quit IRC00:14
jeblairgreghaynes: hrm, i'm not seeing how it's really that much harder...00:15
SergeyLukjanovjeblair I think harlowja could comment on it better, but AFAIK for unit tests it's more than enough and for the functional tests you're usually don't want to mock the db, so, it'll not replace the real zk for the functional tests00:15
*** SumitNaiksatam has quit IRC00:15
jeblairSergeyLukjanov: yeah, all of our testing here will basically be functional testing00:16
*** Qiming has quit IRC00:16
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue  https://review.openstack.org/29317800:16
SergeyLukjanovjeblair it's absolutely safe to dedicated the single prefix path for each unit test, not necessarily to run zk per unit test00:16
EmilienMpabelanger, _ody, crinkle: when you have time, feel free to review: https://review.openstack.org/29317800:16
jeblairgreghaynes: if the scheduler is external, it's basically just cron.  it doesn't seem very hard to do the same thing internally...00:17
*** pahuang has joined #openstack-infra00:17
jeblairSergeyLukjanov: i thought about that, but then we would have to have an option to nodepool to specify the zookeeper root...00:17
SergeyLukjanovjeblair when I was doing the ZK stuff, I was using some prefix like /tests/<uuid>/<unit test full name> as a root for each unit test with create/cleanup in setup/teardown00:18
jeblairSergeyLukjanov: maybe we should... is that a best practice?00:18
SergeyLukjanovjeblair you probably anyway want to have one00:18
SergeyLukjanovjeblair I would say yes00:18
fungizigo: well, if nothing else, it's probably easy enough to mount the image on a loop block device00:18
SergeyLukjanovjeblair you need at least 3 nodes Zk cluster for HA, so, you'll be end up with sharing it between different services00:18
zigofungi: Yup ! :)00:18
jeblairSergeyLukjanov: yeah, that makes sense.00:19
SergeyLukjanovjeblair is it the mentioned spec https://review.openstack.org/#/c/278777/3/specs/nodepool-zookeeper.rst ?00:19
fungizigo: i can totally manually put the image in a downloadable location temporarily for you to grab once it's building successfully on nodepool.o.o00:19
zigofungi: kpartx is my friend for such things ! :)00:19
jeblairSergeyLukjanov: that's the one.  sounds like i should add a note about the root being configurable.00:20
zigofungi: Thanks, that will be helpful indeed.00:20
greghaynesjeblair: Internal vs external isnt really the concern I have, its whether the per image scheduling algorithm has to be written to be fully distributed/shared-nothing or whether it can be written under the assumption that it is only running in one place at a given point in time. The former is always a lot harder to reason about IME00:20
zigofungi: Should it wait for another core reviewer to approve?00:20
zigo(ie: workflow...)00:20
greghaynesjeblair: And since the only benefit we want from making a distributed scheduler is fault tolerance (not crazy scsalability) we can get away with not doing a fully distributed scheduler00:21
jeblairgreghaynes: i think the algo is basically "wake up at 14:00 utc, for each image type i can build, if there is not a build in progress, try to start one"00:21
fungizigo: i'm trying to figure out whether i have time to restart the image builder process (it doesn't have on the fly reconfigure support yet), and make sure i'm not stepping on the toes of anyone else troubleshooting our existing image builds00:21
*** aeng has joined #openstack-infra00:21
*** camunoz has joined #openstack-infra00:21
zigofungi: Ok. I guess I'll go to sleep then. Thanks, much appreciated.00:22
jeblairgreghaynes: the reason i keep pushing on shared-nothing for this is that the next step (alluded to in the spec, but i'm planning on writing a spec for) is to remove the rest of the sharing in nodepool.  so then "where to run the build scheduler" actually becomes a difficult question :)00:22
*** openstack has joined #openstack-infra00:24
jeblairSergeyLukjanov: oh that's good to know00:24
*** ddieterly has joined #openstack-infra00:24
SergeyLukjanovclarkb yeah, I've been testing on many hundreds per seconds :)00:24
greghaynesjeblair: I do like the nodepoold removal btw00:25
fungiSergeyLukjanov: since you're retiring from being a ptl, will we be seeing more of you on infra work again?00:25
* fungi asks hopefully00:26
SergeyLukjanovfungi honestly I'm not sure, but I'm planning to work more on infra then on sahara on a free time00:26
fungiSergeyLukjanov: yay! ;)00:26
jeblairgreghaynes: oh, you're suggesting any builder could run the scheduling algorithm, but only one does it one at a time, by way of a "global build scheduling lock"?00:26
SergeyLukjanovI'm glad to help with some stuff, but very unreliably... I can say for sure that I can help with projects renames :)00:26
greghaynesjeblair: exactly00:26
greghaynesjeblair: but not global - per image00:26
fungiSergeyLukjanov: well, i really hope those are increasingly rare now00:27
greghaynesjeblair: since some builders might not have every image00:27
*** baoli has joined #openstack-infra00:27
*** fverbs has quit IRC00:27
jeblairgreghaynes: now i'm not seeing the difference between what you are suggesting and what i am. :)00:27
*** mtanino has quit IRC00:28
greghaynesjeblair: :)00:28
*** Hal has joined #openstack-infra00:28
*** Hal is now known as Guest2257500:29
*** ianw has joined #openstack-infra00:29
prometheanfiregreghaynes: should I add my test to https://github.com/openstack/diskimage-builder/blob/master/tests/run_functests.sh#L16 ?00:29
*** IlyaG has quit IRC00:29
*** fawadkhaliq has quit IRC00:30
greghaynesprometheanfire: Yes, and then add a nonvoting job which runs it00:30
SergeyLukjanovjeblair I have one more nit tip - if you'll be using the "lock" or "party" primitives - make sure that you're not creating any subnodes manually - it'll make acquire operations sometimes in a dead lock, I mean if you have kazoo.lock('/a/b/c'), don't create '/a/b/foobar', so, the good approach is to have lock under separated znode like '/a/b/c/.lock'00:30
*** sheeprine has quit IRC00:30
ianwfreaking linode, every time they do network maintenance my host seems to disappear (and my irc bouncers with it)00:30
greghaynesianw: I heard about this neat cloud software you should check out ;)00:30
prometheanfiregreghaynes: ok, I'll start there (ammend that review)00:31
jeblairSergeyLukjanov: did you mean "/a/b/c/foobar" in that example?00:31
fungiclarkb: looks like your ansible pr is passing travesty jobs. was the plan to wait for that to trickle down from upstream before we can puppet review.o.o/git.o.o (and safely build images again), or were you thinking there's some way to run a forked version locally while we wait?00:32
fungior option #3 (unspecified)00:32
SergeyLukjanovjeblair I mean that lock works as creating znode per each requester, so, you shouldn't mix it with other znodes00:32
clarkbfungi: well I have no idea how to test that properly so depends on how adventurous we want to be00:32
fungiheh00:32
clarkbfungi: in theory we can stop puppet on the puppetmaster, apply that delta to /etc/ansible/hosts/openstack00:32
fungiit looks like the upstream testsuite likes it, fwiw00:32
clarkband see if that fixes us00:32
*** fawadkhaliq has joined #openstack-infra00:33
clarkbfungi: I don't know that upstream testsuite is testing it at all though :) its in the contrib dir00:33
*** MarkAtwood has quit IRC00:33
jeblairgreghaynes: here's what's in my head: https://etherpad.openstack.org/p/ePcdXiqHOb00:33
fungibut i'm on the fence as to whether faster fixing is preferable to possibly breaking worse00:33
*** pvaneck has quit IRC00:33
fungiclarkb: fair point00:33
fungicoverage of that file... ?%00:33
clarkbfungi: in theory if it breaks we just remove it and rerun puppet without it00:34
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add testing for the Gentoo element  https://review.openstack.org/29089400:34
fungiif only we had a bat signal but for mordred00:34
clarkbfungi: so the rollbcak and go back to status quo is simple00:34
*** sheeprine has joined #openstack-infra00:34
prometheanfiregreghaynes: ianw: ^ I think that's what you want00:34
fungiclarkb: yeah, as long as it only breaks itself and not... you know... everything00:35
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue  https://review.openstack.org/29317800:35
prometheanfireso, in project-config, I see a check named gate-dib-dsvm-functests-devstack-centos700:35
clarkbfungi: right, I think worst case we would still run puppet on the wrong host but for those few cases it shouldn't be worse than now00:35
jeblairSergeyLukjanov: if i understand you, i think i designed it that way in the spec; if you have time to double check that, that would be great :)00:35
prometheanfirewhere is whatever that calls so I know what I need to edit for a non-voting version for gentoo?00:35
*** MarkAtwood has joined #openstack-infra00:36
clarkbfungi: but thats based on my limited understanding of this. Which is far too complicated imo00:36
ianwprometheanfire: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/diskimage-builder.yaml00:36
clarkbI almost thing we should go to a purely uuid based system and just have one list of disables00:36
ianwprometheanfire: let me start something there and you can finish it, it will be easier00:36
clarkbinput from nova less disables then run00:36
clarkbright now we have like 5 files that all interact on this its pretty crazy00:37
*** erlon has quit IRC00:37
prometheanfireianw: thanks00:37
*** eharney has quit IRC00:37
prometheanfireatm I'd just copy lines 1-27 and make it more gentoo specific00:38
*** MarkAtwood has quit IRC00:38
prometheanfireI take it the name is a match statement?00:38
clarkbbtw I couldn't find any glaring issues with the initial devstack-trusty to ubuntu-trusty switch00:40
clarkbhttps://review.openstack.org/#/c/292573/ should be a complete siwtch for the remainder00:40
openstackgerritIan Wienand proposed openstack-infra/project-config: [WIP] Add non-voting diskimage-builder jobs  https://review.openstack.org/29318300:40
clarkbI can chop tat up into many changes but it seems easier to deal with jenkins just once so I crammed it together00:40
ianwprometheanfire: we should be able to modify the template to have two jobs with suffix "-nv" (that automatically makes them non-voting)00:41
ianwand run whatever tests we specify00:41
prometheanfireright00:41
prometheanfire./diskimage-builder/tests/run_functests.sh gentoo/build-succeeds so at least that's a thing00:41
prometheanfiredidn't actually test locally with just gentoo :P00:42
ianwprometheanfire: that's exactly why i reworked run_functests.sh to do that :)00:42
prometheanfire:D00:42
*** baoli has quit IRC00:42
fungiclarkb: the only reason we went with lots of changes on the bare-trusty cutover is that it was more nuanced and different jobs had different reconfiguration needs. for devstack-trusty it's pretty much just a straight up node name replacement so one big change is probably easier. either we press forward or we revert all at once00:42
ianwwe could move logic about what to run where into dib, rather than project-config ... depends what looks neater00:43
*** angdraug has quit IRC00:43
ianwfunctionally, all the same00:43
prometheanfireI kinda like test run logic for infra existing in project-config00:43
prometheanfirerather then the endpoints (in this case dib)00:44
eliqiaohi AJaeger there?00:44
*** moravec has quit IRC00:45
clarkbeliqiao: I doubt it, he is in europe so this is very late for him00:45
*** dims has joined #openstack-infra00:45
clarkbeliqiao: if you have a question you can probably ask the channel00:45
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks  https://review.openstack.org/29139300:45
eliqiaoclarkb: oh, thanks.00:45
eliqiaoI have a question about, https://review.openstack.org/#/c/292687/ AJaeger post comments, just want to make sure I understand him correctly before I do respin again.00:46
prometheanfiremore reviews on bumping libvirt-python in liberty would be appreciated https://review.openstack.org/#/c/291859/00:47
clarkbeliqiao: basically keep the python3-jobs from https://review.openstack.org/#/c/292687/2/zuul/layout.yaml00:47
*** tphummel has quit IRC00:47
*** zz_dimtruck is now known as dimtruck00:49
greghaynesjeblair: similarly - http://paste.ubuntu.com/15398740/00:50
greghaynesjeblair: I actually like your psuedocode more00:51
greghaynesjeblair: so lets do yours00:51
jeblairgreghaynes: okay, cool.  if it starts to go south in implementation, i think it will be simple to change course.  :)00:52
greghaynesjeblair: ya00:52
*** afazekas has quit IRC00:53
*** Jeffrey4l has quit IRC00:53
*** gyee has quit IRC00:54
ianw12.158g | 526:08.80 | python -m subunit.run discover -t ./ nodepool/tests/ --load-list /tmp/tmperG9So <- so that's why i've been wondering why my vm was running a bit slow.  stray subunit.run from nodepool testing last week!00:54
*** docaedo has quit IRC00:55
prometheanfirelol00:55
jeblairisn't there a timeout?00:55
ianwjeblair: i would have ctrl-c'd it when we were looking at that mysql user stuff last week.  clearly it doesn't get everything, and gets in some sort of terrible loop00:56
greghaynesianw: I wonder if its a dead thread issue00:57
greghayneswhere we have a non-daemon thread hanging around even though the interpreter was killed by timeout00:58
*** Kennan has quit IRC00:58
*** fawadkhaliq has quit IRC00:58
greghaynesor ctrlc00:58
*** sputnik13 has quit IRC00:58
*** afazekas has joined #openstack-infra00:59
*** docaedo has joined #openstack-infra01:01
*** fawadkhaliq has joined #openstack-infra01:02
*** fawadkhaliq has quit IRC01:02
*** dchen has joined #openstack-infra01:03
prometheanfireianw: let me know when you submit that project-config review01:03
*** zhurong has joined #openstack-infra01:03
*** ZZelle has quit IRC01:03
ianwprometheanfire: oh, i thought you were going to finish it?  that's just the gist01:03
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Revert "Tolerate both new and old setuptools"  https://review.openstack.org/29318701:04
prometheanfireoh, I thought you were going to do a review and I would make a gentoo version or something01:04
SergeyLukjanovjeblair yeah, it seems correct in a spec, if you want - ping me when you'll have code and I'll double check the zk usage01:04
prometheanfirek, I can do it01:04
*** david-lyle has joined #openstack-infra01:04
SergeyLukjanovis there somewhere the description of the specs concept?01:06
*** fawadkhaliq has joined #openstack-infra01:06
*** fawadkhaliq has quit IRC01:06
*** dims has quit IRC01:07
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Update to oslo.messaging 4.5.1  https://review.openstack.org/29256701:08
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Revert "Tolerate both new and old setuptools"  https://review.openstack.org/29318701:08
EmilienMfungi: did we figure why centos7 images failed to upload on internap cloud?01:08
*** SumitNaiksatam has joined #openstack-infra01:09
*** Kennan has joined #openstack-infra01:10
prometheanfireianw: this is what I have https://gist.github.com/8b5b3dc5ec42fce7a31e01:11
prometheanfireI prepended devstack because {node} included it (based on the match at least)01:11
fungiEmilienM: i didn't hear back from mgagne but his suspicion was that the token was expiring before the upload completed01:11
prometheanfirethis allows {node} to be used in the script01:12
EmilienMfungi: ok no prob01:12
fungiEmilienM: was that yesterday01:12
EmilienMfungi: I guess image upload still fails01:12
fungi?01:12
*** dims has joined #openstack-infra01:12
EmilienMyeah, I believe01:12
fungitoday has been so crazy it seems like it was last week01:12
EmilienMyeah :) Tuesday is busy01:12
fungiwell, today's images didn't upload anywhere due to other issues we're tracking down01:12
EmilienMall the meetings, etc01:12
prometheanfireatm that's all I think is needed01:13
ianwprometheanfire: you should be able to just add two new jobs to https://review.openstack.org/#/c/293183/1/jenkins/jobs/projects.yaml using the existing template01:13
fungididn't even build actually01:13
*** yamamoto has joined #openstack-infra01:13
ianwprometheanfire: except the new jobs specify -nv as the job-suffix, and pass in an arg for "tests"01:13
*** tphummel has joined #openstack-infra01:14
prometheanfireok01:15
prometheanfirewhy two jobs?01:16
*** Qiming has joined #openstack-infra01:16
prometheanfireone for cent one for trusty?01:16
*** Guest22575 has quit IRC01:17
prometheanfirewill I need to depend on your review or should I just replace it?01:18
ianwprometheanfire: yep, although honestly i'm not sure that the centos job is really testing that much, realistically whatever nodepool is doing is what's important01:18
*** fawadkhaliq has joined #openstack-infra01:18
ianwprometheanfire: just replace it01:18
prometheanfirek01:18
*** fawadkhaliq has quit IRC01:19
*** yamamoto has quit IRC01:20
clarkbEmilienM: I did some basic triage on https://bugs.launchpad.net/glance/+bug/1557814 it is a tempest bug not a glance bug01:22
openstackLaunchpad bug 1557814 in tempest "Tempest tests fail on Glance: "Endpoint not found" when IPv6" [Undecided,New]01:22
EmilienMright01:22
EmilienMclarkb: mtreinish just told me01:22
mtreinishclarkb: yeah it's also likely a bug in glanceclient (the code in tempest that's failing there was originally based on glanceclient)01:23
*** boris-42 has quit IRC01:24
prometheanfireianw: separate reviews for adding nv and adding gentoo jobs based on it I assume?01:24
clarkbtl;dr is socket.socket(AF_INET,...) means ipv4 only01:24
*** asselin_ has joined #openstack-infra01:24
clarkbbut I updated the bug and all that01:24
*** docaedo has quit IRC01:24
fungiclarkb: one other workaround option for the ansible openstack inventory bug... we could nova rename the replacement review.o.o temporarily?01:25
EmilienMclarkb: thanks for looking!01:25
clarkbI guess getaddrinfo should also give you the args01:25
clarkbfungi: is that a thign?01:25
* clarkb checks nova client help01:25
*** afazekas has quit IRC01:25
fungier, it's called nova update now01:25
ianwprometheanfire: no we can just do it together; the point of the nv job is to add gentoo01:25
prometheanfireok01:26
clarkbfungi: ya I never knew that was a thing, seems like something we could try01:26
clarkbfungi: I worry it won't let us rename it back due to the name collision :)01:26
mtreinishclarkb: oh, I didn't realize it was opening a raw socket with AF.INET01:26
mtreinishI thought that code path was all just httplib01:26
clarkbmtreinish: yup01:26
mtreinishoh, it's in the https code01:27
mtreinishhmm01:27
clarkbmtreinish: EmilienM so I think the fix is to socket.socket(socket.getaddrinfo(addr))01:27
*** kingia_ has joined #openstack-infra01:27
clarkbmtreinish: EmilienM that should select the correct family based on the name/addr01:27
clarkbmaybe01:28
clarkboh that always does a name lookup hrm01:28
*** ianw_ has joined #openstack-infra01:28
*** asselin_ has quit IRC01:28
*** afazekas has joined #openstack-infra01:28
*** dims has quit IRC01:29
clarkbI was wrong the []s confused it01:29
clarkbmtreinish: EmilienM socket.getaddrinfo('::1', 443)01:29
clarkbthat works so thats the real fix01:29
*** dims has joined #openstack-infra01:29
*** fawadkhaliq has joined #openstack-infra01:30
*** kingia has quit IRC01:30
clarkbcommented01:30
*** docaedo has joined #openstack-infra01:30
*** Daisy has joined #openstack-infra01:31
*** fawadkhaliq has quit IRC01:31
*** akshai has quit IRC01:33
prometheanfireianw: ok, https://gist.github.com/9a8e4caa58febdbad9d101:33
*** jiangliucheng has joined #openstack-infra01:33
prometheanfireI think that's it01:34
clarkbyou know we have a service to review these changes ;)01:35
*** esikachev has joined #openstack-infra01:35
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319301:35
prometheanfireclarkb: you don't say :P01:35
prometheanfireI just didn't want to waste cycles :P01:35
jiangliuchengDoes anyone know what's going on with zuul? My patch in workflow for one day, but no info be found in status.openstack.org/zuul01:35
prometheanfirebut look at that, I'm wasting cycles :P01:35
clarkbjiangliucheng: what change01:35
clarkbjiangliucheng: always provide the change :)01:35
jiangliuchengreview : 29203701:36
jiangliuchenga bug01:36
jiangliuchengaboute replication v2.101:36
Clinthmm01:36
*** tsg has quit IRC01:36
EmilienMmtreinish, clarkb: thx a lot, I'm working on a patch01:36
jiangliuchenghttps://review.openstack.org/#/c/292037/01:37
*** esker has joined #openstack-infra01:37
*** Daisy has quit IRC01:37
*** kingia_ has quit IRC01:37
*** kingia has joined #openstack-infra01:37
*** Daisy has joined #openstack-infra01:37
clarkbjiangliucheng: it might have gotten lost when gerrit was restarted earlier today01:37
clarkber zuul01:37
clarkbits actually being tested now, looks like it has failed.01:38
clarkbyou should be able to recheck it again once that reports back01:38
prometheanfireianw: just realized, I still depend on https://review.openstack.org/#/c/290894/01:38
prometheanfirenot sure how to do cross project deps01:38
*** esikachev has quit IRC01:39
openstackgerritSachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout  https://review.openstack.org/29319401:39
SamYapleprometheanfire: depends-on: <changeid>01:39
prometheanfirek01:39
jiangliuchengNow i find it in check list, will it be move to gate list?01:39
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319301:40
EmilienMclarkb, mtreinish: https://review.openstack.org/29319501:41
*** IlyaG has joined #openstack-infra01:41
*** fawadkhaliq has joined #openstack-infra01:42
*** Daisy has quit IRC01:42
clarkbEmilienM: I think you don't need an if else there01:42
*** mriedem has quit IRC01:42
EmilienMclarkb: I re-used glanceclient code01:42
clarkbEmilienM: just pass getadderinfo()[0] details striaght through01:42
clarkbheh glanceclient code is still broken then :)01:42
clarkbbut ok01:42
*** kaisers1 has joined #openstack-infra01:44
*** kaisers has quit IRC01:45
*** asselin_ has joined #openstack-infra01:50
*** fawadkhaliq has quit IRC01:51
*** fawadkhaliq has joined #openstack-infra01:51
*** dims has quit IRC01:52
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319301:52
*** yamahata has joined #openstack-infra01:53
*** baoli has joined #openstack-infra01:55
*** dims has joined #openstack-infra01:56
*** IlyaG has quit IRC01:57
Nakatolifeless: AJaeger: This is my initial take on what needs to be supported for post jobs.  https://review.openstack.org/#/c/293194/01:57
NakatoIt's quite a different take than the one listed in the meeting logs, which I don't think is going to make post jobs works.01:57
*** ddieterly is now known as ddieterly[away]01:59
*** aeng has quit IRC01:59
*** adrian_otto has joined #openstack-infra02:01
*** dims has quit IRC02:03
*** apoorvad has quit IRC02:03
*** dims has joined #openstack-infra02:05
*** akshai has joined #openstack-infra02:05
*** fawadkhaliq has quit IRC02:06
*** woodster_ has quit IRC02:07
*** akshai_ has joined #openstack-infra02:07
*** akshai has quit IRC02:10
fungijesusaur: ^02:10
*** ddieterly[away] is now known as ddieterly02:11
prometheanfireianw: jjb?02:11
*** baoli_ has joined #openstack-infra02:11
prometheanfireianw: gentoo/build-succeeds is the name that run_functests.sh -l returns02:11
*** fitoduarte has quit IRC02:12
*** aeng has joined #openstack-infra02:12
*** baoli has quit IRC02:14
*** ddieterly is now known as ddieterly[away]02:16
ianwprometheanfire: i mean jenkins job builder, the thing that, well, builds the jobs02:17
ianwthe template is in diskimage-builder.yaml, and then that is instantiated into jobs in projects.yaml02:17
prometheanfireok, so I need to add another entry to that02:18
*** thorst_ has quit IRC02:18
*** yamamoto_ has joined #openstack-infra02:19
*** nwkarsten has joined #openstack-infra02:20
*** yamahata has quit IRC02:21
prometheanfireianw: isn't it already instantiated by the review I'm doing?02:21
prometheanfirethe job template was already edited02:22
EmilienMclarkb, mtreinish: puppet CI now works with ipv6 \o/ https://review.openstack.org/#/c/287521/02:22
EmilienMclarkb: ty again for your help on tempest02:23
*** baoli_ has quit IRC02:23
*** baoli has joined #openstack-infra02:23
*** pahuang has quit IRC02:24
*** amrith is now known as _amrith_02:24
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services  https://review.openstack.org/27112802:24
prometheanfireianw: I know I'm missing something, but I don't see it02:25
jamielennoxAJaeger: around? what's my chance of getting ^ merged without a +1 from every project?02:25
*** fawadkhaliq has joined #openstack-infra02:25
prometheanfireah, see your comment02:26
jamielennoxif i need it then i'm going to split it up into multiple patches, the base then one per project02:26
jamielennoxi can't get that many approvals between a merge conflict :p02:26
*** ddieterly[away] has quit IRC02:26
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319302:28
*** Daisy has joined #openstack-infra02:28
jamielennoxbah, going to do it preemptively02:29
*** Daisy_ has joined #openstack-infra02:29
*** baoli has quit IRC02:29
*** yamahata has joined #openstack-infra02:30
*** Daisy__ has joined #openstack-infra02:31
*** sridhar_ram has joined #openstack-infra02:31
*** fawadkhaliq has quit IRC02:32
*** katyafervent_awa has quit IRC02:32
*** tqtran-afk has quit IRC02:32
*** Daisy has quit IRC02:33
*** Daisy has joined #openstack-infra02:33
*** Daisy_ has quit IRC02:34
*** baoli has joined #openstack-infra02:35
*** Daisy__ has quit IRC02:37
*** pahuang has joined #openstack-infra02:37
*** katyafervent_awa has joined #openstack-infra02:37
*** Daisy has quit IRC02:37
*** Daisy has joined #openstack-infra02:39
*** yamahata has quit IRC02:39
*** sridhar_ram has quit IRC02:41
openstackgerritJamie Lennox proposed openstack-infra/project-config: Refactor the v3 only job  https://review.openstack.org/27112802:43
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the v3 only job in glance  https://review.openstack.org/29319902:43
*** rhallisey has quit IRC02:45
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319302:45
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the v3 only job in neutron  https://review.openstack.org/29320002:46
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319302:46
*** dchen has quit IRC02:52
*** fitoduarte has joined #openstack-infra02:55
*** Jeffrey4l has joined #openstack-infra02:57
*** krtaylor has quit IRC02:58
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319302:58
*** aeng has quit IRC03:01
*** pahuang has quit IRC03:06
*** hichihara has joined #openstack-infra03:06
*** krtaylor has joined #openstack-infra03:10
*** esker has quit IRC03:11
*** aeng has joined #openstack-infra03:12
prometheanfireianw: wooo it passed03:15
*** binbincong has quit IRC03:15
*** fitoduarte has quit IRC03:15
prometheanfireianw: thanks for all the hand holding03:16
*** thorst has joined #openstack-infra03:19
*** Daisy has quit IRC03:19
*** yuanying has quit IRC03:20
*** pahuang has joined #openstack-infra03:23
*** yamamoto_ has quit IRC03:25
*** dimtruck is now known as zz_dimtruck03:25
*** thorst has quit IRC03:27
*** binbincong has joined #openstack-infra03:27
ianwcool03:27
prometheanfireguess I still need +2s to be readded to https://review.openstack.org/#/c/290894/03:30
prometheanfiregreghaynes: ^03:30
prometheanfiregreghaynes: mind re-reviewing that?03:30
*** gongysh has joined #openstack-infra03:34
*** esikachev has joined #openstack-infra03:35
*** dims has quit IRC03:37
*** adrian_otto has quit IRC03:37
*** baoli has quit IRC03:39
*** esikachev has quit IRC03:40
*** cody-somerville has quit IRC03:46
*** rguillebert has quit IRC03:47
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fixed doble form submision issue  https://review.openstack.org/29324103:51
*** akshai_ has quit IRC03:52
*** adrian_otto has joined #openstack-infra03:54
*** Daisy has joined #openstack-infra03:54
*** kingia_ has joined #openstack-infra03:55
*** watanabe_isao has joined #openstack-infra03:57
*** kingia has quit IRC03:58
*** cody-somerville has joined #openstack-infra04:01
*** esker has joined #openstack-infra04:01
*** ayoung is now known as ayoung_Zzz__04:01
*** gongysh has quit IRC04:04
*** kingia has joined #openstack-infra04:05
*** kingia_ has quit IRC04:05
*** yamamoto_ has joined #openstack-infra04:06
*** esker has quit IRC04:06
*** fedexo has joined #openstack-infra04:07
*** yuanying has joined #openstack-infra04:07
*** links has joined #openstack-infra04:10
*** ayoung_Zzz__ has quit IRC04:12
prometheanfireianw: thanks and nn04:13
*** [HeOS] has joined #openstack-infra04:13
*** fawadkhaliq has joined #openstack-infra04:14
*** HeOS has quit IRC04:15
*** flwang1 has quit IRC04:16
*** nwkarsten has quit IRC04:18
*** Daisy has quit IRC04:20
*** kingia_ has joined #openstack-infra04:21
*** kingia has quit IRC04:22
*** Daisy has joined #openstack-infra04:24
*** thorst has joined #openstack-infra04:25
*** kingia has joined #openstack-infra04:26
*** kingia_ has quit IRC04:26
*** rkukura has quit IRC04:28
*** tqtran-afk has joined #openstack-infra04:29
*** rkukura has joined #openstack-infra04:29
*** Daisy has quit IRC04:29
*** treaki__ has joined #openstack-infra04:30
*** thorst has quit IRC04:32
*** tqtran-afk has quit IRC04:33
*** treaki_ has quit IRC04:34
*** [1]Thelo has joined #openstack-infra04:34
*** pahuang has quit IRC04:35
*** akshai has joined #openstack-infra04:36
*** Thelo has quit IRC04:37
*** [1]Thelo is now known as Thelo04:37
*** roxanagh_ has joined #openstack-infra04:38
*** asettle has quit IRC04:50
*** sdake has quit IRC04:50
*** roxanagh_ has quit IRC04:58
*** pahuang has joined #openstack-infra05:03
openstackgerritTony Breeds proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes  https://review.openstack.org/27027305:03
*** akshai_ has joined #openstack-infra05:04
*** akshai has quit IRC05:07
*** yamahata has joined #openstack-infra05:08
tonybjhesketh: Can you help me understand why https://review.openstack.org/292657 isn't progressing to the gate pipeline?  COUld it have been missed with the zuul/gerrit restart last night?05:08
*** adrian_otto1 has joined #openstack-infra05:08
*** maishsk has joined #openstack-infra05:09
*** adrian_otto has quit IRC05:10
jheskethtonyb: yeah, looking at the time it's possibly one of the one results were lost from05:11
jheskethtonyb: shouldn't be any harm in rechecking05:11
*** asettle has joined #openstack-infra05:11
tonybjhesketh: Thanks. done.05:12
*** boris-42 has joined #openstack-infra05:14
*** Daisy has joined #openstack-infra05:15
openstackgerritMerged openstack-infra/irc-meetings: Update Nova API sub-team meeting time  https://review.openstack.org/29222205:16
*** FallenPegasus has joined #openstack-infra05:18
openstackgerritMerged openstack-infra/irc-meetings: Reverting the openstack-salt chair back to Jay Clark.  https://review.openstack.org/29212905:19
openstackgerritMerged openstack-infra/irc-meetings: As per discussion, extend Quota WG time to 60 mins  https://review.openstack.org/29262505:20
tonybjhesketh: recheck came back with a +1 from jenkins but that review still isn't prgressing to the gate pipeline05:23
jheskethtonyb: hmm right you, give me a sec and I'll take a look05:23
tonybjhesketh: Thanks.05:24
*** alex_xu has quit IRC05:25
*** sc68cal has joined #openstack-infra05:26
*** hichihara has quit IRC05:27
*** tqtran-afk has joined #openstack-infra05:30
*** thorst has joined #openstack-infra05:30
*** aeng has quit IRC05:31
*** alex_xu has joined #openstack-infra05:31
*** sridhar_ram has joined #openstack-infra05:31
*** Jeffrey4l has quit IRC05:32
*** esker has joined #openstack-infra05:33
*** tqtran-afk has quit IRC05:35
jheskethtonyb: so in the zuul logs I can see where jenkins reports back the +1 from the check, but it doesn't appear to trigger it into the gate pipeline05:36
jheskethand it's not clear why05:36
*** esikachev has joined #openstack-infra05:36
jheskethI reviewed it myself and now it's in the gate pipeline05:36
*** adrian_otto1 has quit IRC05:36
jheskethlets see if that posts a result back05:36
*** thorst has quit IRC05:37
*** infra-red has joined #openstack-infra05:38
*** esker has quit IRC05:38
openstackgerritMerged openstack-infra/release-tools: Actually use the URL specific in --project-list  https://review.openstack.org/29265705:39
jheskethtonyb: ^ seemed to work.. not sure why recheck didn't put it back in the gate though :-(05:40
*** esikachev has quit IRC05:41
*** infra-red has quit IRC05:41
*** HeOS_ has joined #openstack-infra05:41
*** fawadkhaliq has quit IRC05:42
*** infra-red has joined #openstack-infra05:42
*** sridhar_ram has quit IRC05:42
*** fawadkhaliq has joined #openstack-infra05:43
*** aeng has joined #openstack-infra05:43
*** [HeOS] has quit IRC05:43
*** akshai_ has quit IRC05:44
*** david-lyle has quit IRC05:45
*** vgridnev has joined #openstack-infra05:45
*** david-lyle has joined #openstack-infra05:45
clarkbif the vote doesnt change gerrit doesnt emit an event for it05:46
clarkbbecause an event isnt really an event its a delta05:46
*** maishsk has quit IRC05:48
*** Daisy has quit IRC05:48
*** Daisy has joined #openstack-infra05:48
jheskethclarkb: oh that's right, thanks!05:51
*** mrmartin has joined #openstack-infra05:52
jheskethclarkb: maybe we need to add the reverify comment to trigger the gate queue specifically.. it should be protected by the require approvals to not queue without an existing jenkins +1 and a +w05:52
*** fawadkhaliq has quit IRC05:52
*** fawadkhaliq has joined #openstack-infra05:53
clarkbjhesketh: I think zaro has a fix to make events events we might be able to upgrade to as well05:53
jheskethagainst gerrit or zuul?05:53
clarkbgerrit05:54
*** camunoz has quit IRC05:56
*** camunoz has joined #openstack-infra05:56
*** _nadya_ has joined #openstack-infra06:00
*** flwang1 has joined #openstack-infra06:01
*** _nadya_ has quit IRC06:01
*** mrmartin has quit IRC06:03
*** FallenPe_ has joined #openstack-infra06:09
*** flwang1 has quit IRC06:09
AJaegerNakato: Thanks for working on this! Let's discuss with jesusaur and jeblair about these06:09
*** FallenPegasus has quit IRC06:10
*** infra-re_ has joined #openstack-infra06:10
*** Daisy_ has joined #openstack-infra06:10
AJaegereliqiao: I'm here now - are you still around?06:12
*** infra-red has quit IRC06:12
eliqiaohi AJaeger ,ya06:12
*** Daisy has quit IRC06:13
*** infra-red has joined #openstack-infra06:14
NakatoAJaeger: Sounds good.  Though I don't plan to be around much longer tonight.06:14
*** infra-re_ has quit IRC06:17
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/29326806:17
*** gongysh has joined #openstack-infra06:19
*** HeOS_ is now known as HeOS06:21
AJaegerNakato: Try to catch them tomorrow morning your time, they are gone already I guess. Thanks06:24
AJaegereliqiao: Your previous revision used the python3-jobs template, use that one again and your change is fine.06:25
*** camunoz has quit IRC06:25
*** Daisy_ has quit IRC06:26
*** Daisy has joined #openstack-infra06:27
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/28884906:31
*** Daisy has quit IRC06:31
*** infra-red has quit IRC06:32
tonybjhesketh: Thanks.06:32
*** thorst has joined #openstack-infra06:35
*** sc68cal has quit IRC06:36
*** jtomasek has joined #openstack-infra06:36
*** vgridnev has quit IRC06:39
*** camunoz has joined #openstack-infra06:39
*** harlowja_at_home has quit IRC06:41
*** pahuang has quit IRC06:41
*** thorst has quit IRC06:42
*** FallenPe_ has quit IRC06:42
*** vgridnev has joined #openstack-infra06:45
*** kushal has joined #openstack-infra06:45
*** abregman has joined #openstack-infra06:46
AJaegerianw: please review 293193 again and reconsider your +2.06:49
*** asettle has quit IRC06:49
*** esker has joined #openstack-infra06:49
ianwAJaeger: added some comments06:54
*** esker has quit IRC06:54
*** gildub has quit IRC06:54
*** Daisy has joined #openstack-infra06:55
prometheanfireAJaeger: I'm in irc as well so we can all talk about https://review.openstack.org/#/c/293193/06:55
tobiashjhesketh: This patch got lost in zuul during connection refactoring which breaks our gate with new zuul versions: https://review.openstack.org/#/c/138132/106:56
AJaegerprometheanfire: Ah, that's you - didn't know ;)06:56
prometheanfire:P06:56
AJaegerianw: I'm flagging the ubuntu-trusty already since we will do the switch today/tomorrow and not have several iterations on them...06:56
prometheanfireI really need to add my nick to my account06:56
AJaegerUsing -extra or -gentoo works for me.06:57
*** asselin_ has quit IRC06:57
ianwAJaeger; yeah that's cool.  i think we can switch the name if you feel strongly.  i'd prefer "-extra", unless we can think of something else even more descriptive but not too verbose06:57
prometheanfirethere a way to edit your 'full name'?06:57
AJaegerI know the job is not gentoo specific, but you parametize it in two different ways with the same name.06:58
*** vgridnev has quit IRC06:58
AJaegerOnes as X and then as X-nv and change paremeters to use gentoo. And that's my major complaint06:58
prometheanfireI can probably add a comment to that section06:59
prometheanfireI know thierry wanted to do debian stuff that could fit there as well (at first)07:00
prometheanfireas far as adding the -extras stuff, does that designation already exist as a non-voting job?07:00
*** tqtran-afk has joined #openstack-infra07:01
prometheanfiredoesn't look like it07:01
ianwprometheanfire: no, if we switch it to "-extras" then we'll have to add a regex to make it non-voting.  lots of examples at the top of the zuul file07:01
AJaegerprometheanfire: if debian is another job, then let's use -debian, -gentoo...07:02
AJaegerunless we can differentiate that in the node.07:02
ianwAJaeger: they will all be under the same job.  it's testing we can *build* gentoo, debian images, etc, on a trusty/centos node07:02
prometheanfiredebian would go in the same -extras job07:02
prometheanfireya, the regex for -nv is simple, I'll copy that07:03
*** fawadkhaliq has quit IRC07:03
AJaegerianw: Ah... right now it has only "gentoo/build-succeeds" and that's why I thought it would be separate ones.07:03
*** ihrachys has joined #openstack-infra07:03
AJaegerThen you might need to change the parameter and make it more general.07:04
*** fawadkhaliq has joined #openstack-infra07:04
prometheanfireya, it's simply the first07:04
prometheanfirethe parameter is a list of tests, gentoo being the first added07:04
ianwAJaeger: yeah, we might move the logic into dib if it becomes impractical07:04
AJaegerprometheanfire: better give one parameter "extra" and pass that in - then there's no need to touch the job everytime07:04
AJaegerianw: agreed07:04
*** tqtran-afk has quit IRC07:05
ianwok, i gotta disappear, will check later07:05
prometheanfireI'll do the work to make it just renamed to -extras for now07:05
prometheanfirewe'll see about extra dib work in the morning07:06
AJaegerianw: but I'm fine with starting the parameter in for now and refine later - but let's keep that in mind07:06
*** ihrachys has quit IRC07:06
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting  https://review.openstack.org/29268707:07
prometheanfireI don't think it'd be a hard thing07:07
prometheanfirethere's already a list of ignored tests, making another list and specifying a parameter to trigger that list shouldn't be too bad07:07
*** exploreshaifali has joined #openstack-infra07:12
*** vgridnev has joined #openstack-infra07:12
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319307:13
*** flwang1 has joined #openstack-infra07:13
prometheanfireianw: AJaeger ^07:13
*** HeOS_ has joined #openstack-infra07:15
*** infra-red has joined #openstack-infra07:15
*** infra-red has quit IRC07:16
*** tesseract has joined #openstack-infra07:16
prometheanfireneeds to pass of course07:17
*** tesseract is now known as Guest6496707:17
AJaegerprometheanfire: commented - better ;)07:17
AJaegerNote, you can also name it: 'gate-dib-dsvm-functests-{node}{sth}{job-suffix}' and pass an empty something in07:18
*** jaosorior has joined #openstack-infra07:18
*** _nadya_ has joined #openstack-infra07:19
prometheanfiresth?07:20
*** flepied1 has joined #openstack-infra07:20
prometheanfireextras-nv does make sense though07:20
AJaegersth=something. I didn't come up with a good name07:21
*** _nadya_ has quit IRC07:21
AJaegerI'm fine with trying extras-nv ;)07:21
*** Guest64967 has quit IRC07:21
prometheanfireya, one change at a time :P07:22
*** flepied has quit IRC07:22
*** esker has joined #openstack-infra07:23
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add a core group for zaqar-ui  https://review.openstack.org/29316607:25
prometheanfireso slow07:25
*** fawadkhaliq has quit IRC07:25
*** fawadkhaliq has joined #openstack-infra07:26
*** fawadkhaliq has quit IRC07:26
openstackgerritkevin proposed openstack-infra/project-config: New Project - jacket  https://review.openstack.org/26350007:26
*** fawadkhaliq has joined #openstack-infra07:27
*** esker has quit IRC07:28
*** eric_lopez has quit IRC07:28
*** flwang1 has quit IRC07:30
*** elo has joined #openstack-infra07:30
prometheanfiregerrit is timing out for git reviews07:30
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319307:30
prometheanfireand there it goes07:30
prometheanfirejust had to complain about it07:30
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add a core group for zaqar-ui  https://review.openstack.org/29316607:30
*** akaszuba has joined #openstack-infra07:32
AJaegerianw, could you review https://review.openstack.org/292135, please?07:32
*** fedexo has quit IRC07:32
*** FallenPegasus has joined #openstack-infra07:33
*** tesseract- has joined #openstack-infra07:36
openstackgerritMerged openstack-infra/project-config: Magnum: enable py34 voting  https://review.openstack.org/29268707:37
*** FallenPegasus has quit IRC07:38
*** thorst has joined #openstack-infra07:40
openstackgerritAlexey Ovchinnikov proposed openstack-infra/project-config: Add publishing capability to buildimage-lxd job  https://review.openstack.org/29041807:41
*** chlong has quit IRC07:42
*** maishsk has joined #openstack-infra07:45
*** amotoki has joined #openstack-infra07:46
yolandagood morning07:47
*** llu has joined #openstack-infra07:47
*** thorst has quit IRC07:48
prometheanfirenn :P07:51
*** nmagnezi has joined #openstack-infra07:52
*** dizquierdo has joined #openstack-infra07:53
*** llu has left #openstack-infra07:53
*** vgridnev has quit IRC07:54
penguinologGood morning!07:55
penguinologReview, please: https://review.openstack.org/#/c/292746/07:55
*** scheuran has joined #openstack-infra07:56
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319307:58
*** xiangxinyong has joined #openstack-infra07:58
*** maishsk_ has joined #openstack-infra07:59
jheskethtonyb: oh sorry about that.. there was so much rebasing and refactoring going on it must have gotten lost. Do you want to re-propose it?07:59
*** maishsk has quit IRC08:00
*** maishsk_ is now known as maishsk08:00
yolandaAJaeger, so no resolution yet on projects not being created on gerrit?08:00
*** kingia_ has joined #openstack-infra08:01
yolandaseems that they are not even present on review.openstack.org list of projects :(08:01
jheskethtonyb: woops, wrong tab completion08:03
jheskethtobiash: oh sorry about that.. there was so much rebasing and refactoring going on it must have gotten lost. Do you want to re-propose it?08:03
*** kingia has quit IRC08:04
*** ihrachys has joined #openstack-infra08:04
tobiashjhesketh: no problem, a collegue of mine will do this today or tomorrow08:10
*** matrohon has joined #openstack-infra08:12
*** dizquierdo has quit IRC08:12
*** rossella_s has joined #openstack-infra08:13
*** flepied1 has quit IRC08:18
*** exploreshaifali has quit IRC08:18
*** MCoLo has joined #openstack-infra08:19
*** mikelk has joined #openstack-infra08:21
*** FallenPegasus has joined #openstack-infra08:28
jklaremorning08:28
*** fhubik has joined #openstack-infra08:29
jklareit might be too early for me, but i can not figure out why gerrit keeps mapping my changes to the wrong branch08:30
jklaresomebody got a minute to help me here?08:30
jklarehttps://review.openstack.org/#/c/293308/08:31
*** mrmartin has joined #openstack-infra08:32
openstackgerrityolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce  https://review.openstack.org/29279208:32
*** FallenPegasus has quit IRC08:32
*** yamahata has quit IRC08:33
*** vincentll has joined #openstack-infra08:33
*** dingyichen has quit IRC08:36
*** oanson has joined #openstack-infra08:37
*** abregman has quit IRC08:37
*** vincentll has quit IRC08:39
*** ig0r_ has quit IRC08:39
*** acruz has joined #openstack-infra08:43
*** vincentll has joined #openstack-infra08:43
*** esikachev has joined #openstack-infra08:45
*** andymaier has joined #openstack-infra08:45
*** thorst has joined #openstack-infra08:46
AJaegeryolanda: please read backscroll, I didn't follow that part ;(08:47
AJaegeryolanda, jhesketh, ianw : A review of https://review.openstack.org/#/c/292135/ would be appreciated to move us forward wiht api jobs, please08:47
* AJaeger sends coffee to jklare08:47
AJaegerjklare: not sure what you do there - but we really suggest to use git.openstack.org since that'S our master server, github is just a mirror...08:48
*** flepied has joined #openstack-infra08:49
yolandaAJaeger i don't see that repo being created. Going to check if that's present on manage_projects log08:50
AJaegeryolanda: problem seems to be the non-running puppet on review.o.o08:50
yolandai ran puppet manually there and worked08:50
AJaegeraccording to backlog - but I have no clue how that all works ;(08:50
yolandabut i still don't see that created, even not on review.openstack.org08:51
AJaeger:(08:51
*** abregman has joined #openstack-infra08:51
*** watanabe_isao has quit IRC08:52
*** vgridnev has joined #openstack-infra08:52
jklareAJaeger thanks for the coffee :D i solved i by specifiying the branch in the git review command, not sure why this is needed while it works for most other patches without08:52
yolandaAJaeger, do you have the link for that change handy? i don't even see that project on projects.yaml08:52
*** thorst has quit IRC08:52
*** schipiga has joined #openstack-infra08:53
*** zeih has joined #openstack-infra08:54
yolandaAJaeger, i'd say that even if that ansible-puppet reports as run, something is wrong. I don't see content in review.openstack.org updated since 10 Mar :(08:55
AJaegernew projects are: Ie0c7440e88c918045f1c5ca8288d826397599099 (tripleo-ui) and Icd3810574b72f98da8f51c26e46c527bdd89680b (openstack-ansible...designate)08:55
AJaegerthat's from git log, so no direct review links08:55
*** dalgaaf has joined #openstack-infra08:55
yolandaso they were copied to git, but not to gerrit08:55
*** ianychoi has quit IRC08:55
yolandaso... more troubleshooting fun today08:56
yolandaso... looks to me that rsync is trying to copy content to the wrong review.openstack.org :(08:59
AJaegerArgh ;(09:00
yolandathe other parts of the ansible roles seem to be applying correctly but for rsync i see address 2001:4800:7819:103:be76:4eff:fe05:852509:00
yolandathat's new gerrit09:00
yolanda:(09:01
yolandai also don't know so much about how this works, but going to try to understand09:01
*** HeOS has quit IRC09:01
*** HeOS has joined #openstack-infra09:02
*** HeOS has quit IRC09:03
*** HeOS has joined #openstack-infra09:03
*** kingia has joined #openstack-infra09:03
*** nijaba has quit IRC09:05
*** kingia_ has quit IRC09:06
*** hashar has joined #openstack-infra09:06
*** jlanoux has joined #openstack-infra09:07
*** takashi has quit IRC09:07
*** ifarkas has joined #openstack-infra09:07
yolandasomething is totally broken in ansible inventory cache. It has the id of the current review.openstack.org, but with ip addresses of the new one09:08
yolandamordred `09:08
*** _david_ has joined #openstack-infra09:09
*** nijaba has joined #openstack-infra09:11
rcarrillocruzdo you use a dynamic inventory?09:11
rcarrillocruzwhat's the ansible command you run09:11
rcarrillocruz?09:11
*** exploreshaifali has joined #openstack-infra09:11
*** ianychoi has joined #openstack-infra09:11
rcarrillocruzif it's dynamic, the dynamic inventory scripts typically have a flag for caches09:11
yolanda ansible-playbook รง-vvv -f 10 /opt/system-config/production/playbooks/remote_puppet_git.yaml --limit localhost,review.openstack.org09:11
*** salv-orlando has joined #openstack-infra09:11
*** esker has joined #openstack-infra09:12
yolandai'm seeing the entries totally wrong on /var/cache/ansible-inventory/ansible-inventory.cache09:12
yolandaso they have the right id of the vm in rax, but the ips are pointing to the new vm09:12
*** kushal has quit IRC09:12
*** ianychoi has quit IRC09:12
*** ianychoi has joined #openstack-infra09:12
rcarrillocruzif you are not passing explicitly the inventory from command line, that must be put on the ansible configuration...09:13
rcarrillocruzdo you have an /etc/ansible/ansible.cfg09:13
rcarrillocruzor your own ansible.cfg at ~09:13
rcarrillocruz?09:13
*** ianychoi2 has joined #openstack-infra09:14
yolandathere is an ansible.cfg, with hostsfile just pointing at /etc/ansible/hosts09:14
rcarrillocruzk09:14
rcarrillocruzwhat you have on /etc/ansible/hosts09:14
rcarrillocruzis that a folder09:14
rcarrillocruzor a plain file09:14
*** sbelous_ has joined #openstack-infra09:14
rcarrillocruz?09:14
yolandaok inside /etc/ansible/hosts there is an openstack python file09:15
yolandai'd say that this is buggy09:15
rcarrillocruzvi the openstack python file09:15
rcarrillocruzthere must be a flag for cache09:15
yolandai can see the entries for both review.openstack.org, different ids, and both point to the new ips09:15
rcarrillocruzat least the upstream openstack.py has it09:15
yolandait's shade openstack inventory09:16
*** esker has quit IRC09:16
rcarrillocruzthat one has a --refresh09:17
rcarrillocruzbut not sure if that will fix it09:17
rcarrillocruzissue is you have two servers with same name09:17
yolandayes but that was supposed to work, according to documentation and instructions on how to setup the replacement server...09:17
rcarrillocruzyou can do this:09:17
*** ZZelle has joined #openstack-infra09:17
yolandain theory the ansible inventory shall manage it properly09:17
rcarrillocruzpython /etc/ansible/hosts/<whatever shade name>.py --list09:18
*** ianychoi has quit IRC09:18
rcarrillocruzthat will show the hostvars09:18
rcarrillocruzwhich can help you troubleshoot the problem09:18
rcarrillocruzsorry i can't look further, without getting my hands on it i'm a bit limited09:18
yolandayep, that's buggy09:18
yolandait reports the id of the current review.openstack.org vm, but with the ips of the new one09:19
rcarrillocruzthe upstream openstack.py inventory switched to uuids to circumvent the issue of servers with duplicated names09:19
rcarrillocruzmordred did09:19
rcarrillocruznot sure if that's on shade inventory09:19
yolandathere is something wrong there, with that uuids09:19
yolandai'm double checking ids and ips, with the entries on rax, and is totally mismatching09:19
yolandagoing to try the refresh09:20
rcarrillocruzlook09:20
rcarrillocruzhttps://github.com/ansible/ansible/commit/c737bd48bc4ee246c378898abb80bacdd80c0e2f09:20
rcarrillocruzhttps://github.com/ansible/ansible/commit/9f61144401a16c9d610193522c71e8852addf63e09:20
rcarrillocruzi'm not sure if there's that in shade inventory09:21
rcarrillocruzbut you get the idea, there's logic needed to handle duplicate hostnames09:21
*** kingia_ has joined #openstack-infra09:21
yolandagoing to check in shade, it should be, but i'm worried about the logic not being the right one09:21
yolandait seems to always pick the first server for the two ids09:22
yolandaboth vms are configured in the inventory with same ips09:22
yolandawe are on shade 1.5.1 on puppetmaster09:23
*** kingia has quit IRC09:23
yolandathat looks as the latest anway09:23
*** kzaitsev_mb has joined #openstack-infra09:24
*** jistr has joined #openstack-infra09:24
*** Daisy has quit IRC09:25
*** jordanP has joined #openstack-infra09:25
*** Daisy has joined #openstack-infra09:26
yolandathe code in puppetmaster looks similar but it may be doing something nasty09:27
yolandagoing to debug on my environment..09:28
odyssey4memorning everyone - do we have any progress on resolving the empty repo http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate , the fact that it hasn't mirrored to github and the fact that it's empty even though https://review.openstack.org/292326 specified that it should be seeded?09:29
rcarrillocruzyolanda: if you want to quickly fix it09:29
rcarrillocruzwhat you can do09:29
rcarrillocruzis create an inventory file09:29
rcarrillocruzthat points review.openstack.org09:29
rcarrillocruzto the IP you need09:29
rcarrillocruzand pass that to ansible-playbook -i <my new inventory>09:29
yolandaodyssey4me, we are having problems with puppet in review.openstack.org09:29
odyssey4meoh dear :/ I'll leave you to it then yolanda - hope it's resolved ASAP09:30
rcarrillocruzyou'd need to put something like09:30
yolandarcarrillocruz, i thought about that yes...09:30
rcarrillocruzreview.openstack.org ansible_host=<the ip i need>09:30
*** Daisy has quit IRC09:30
yolandahowever the inventory i see in ansible cache has some other facts, i was doubting because of it09:31
*** BobBall has joined #openstack-infra09:31
yolandai'll check if there is an obvious bug and if that's more complicated, we can try that way09:31
*** tzn has joined #openstack-infra09:31
yolandamm09:32
yolanda                for server in servers:09:32
yolanda                    append_hostvars(09:32
yolanda                        hostvars, groups, server['id'], servers[0],09:32
yolanda                        namegroup=True)09:32
yolandathat always picks the first entry09:32
yolandaand in our case, the first entry is the new vm :(09:32
*** esker has joined #openstack-infra09:32
yolandagoing to debug to be sure09:32
rcarrillocruzyep09:32
yolanda!!!09:33
openstackyolanda: Error: "!!" is not a valid command.09:33
yolandait is :)09:33
*** kushal has joined #openstack-infra09:33
*** derekh has joined #openstack-infra09:34
*** zhurong has quit IRC09:35
*** _nadya_ has joined #openstack-infra09:37
*** lucas-dinner is now known as lucasagomes09:38
*** esker has quit IRC09:39
yolandarcarrillocruz yes, the issue is confirmed. But i'm looking at how can we enforce that match properly, not relying on picking the first entry09:40
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu  https://review.openstack.org/29301209:42
*** vincentll has quit IRC09:42
*** dtantsur|afk is now known as dtantsur09:42
*** amotoki has quit IRC09:43
*** rguillebert has joined #openstack-infra09:44
*** abregman has quit IRC09:45
yolandaok i have something..09:45
*** vincentll has joined #openstack-infra09:46
yolandaso... this needs to be fixed on ansible repo09:49
*** esikachev has quit IRC09:50
*** thorst has joined #openstack-infra09:50
*** dims has joined #openstack-infra09:51
*** abregman has joined #openstack-infra09:53
yolandai fixed on puppetmaster by moving the fixed file to the ansible cache, but i'll send a PR to ansible09:53
*** openstackgerrit has quit IRC09:53
*** openstackgerrit_ is now known as openstackgerrit09:53
*** openstackgerrit has quit IRC09:53
*** openstackgerrit_ has joined #openstack-infra09:53
openstackgerrit_Sebastian Schuberth proposed openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory  https://review.openstack.org/23769409:54
*** openstackgerrit_ is now known as openstackgerrit09:54
*** openstackgerrit_ has joined #openstack-infra09:55
*** openstackgerrit has quit IRC09:55
*** openstackgerrit has joined #openstack-infra09:56
*** thorst has quit IRC09:57
yolandaAJaeger, puppet is running on the right review.openstack.org now... i hope that this fixes the problems09:59
*** esikachev has joined #openstack-infra09:59
*** placko_ has joined #openstack-infra10:01
*** placko_ has quit IRC10:01
*** tqtran-afk has joined #openstack-infra10:01
*** Qiming has quit IRC10:02
*** placko_ has joined #openstack-infra10:02
*** _david_ has quit IRC10:03
AJaegeryolanda: woot!10:03
*** sdague has joined #openstack-infra10:03
openstackgerritTimur Sufiev proposed openstack-infra/project-config: Fix the location of devstack_gate_rc script at horizon job  https://review.openstack.org/29307710:04
*** placko_ has quit IRC10:04
yolandathx rcarrillocruz for the pointers10:05
*** tqtran-afk has quit IRC10:06
*** tpsilva has joined #openstack-infra10:07
*** sdague has quit IRC10:11
*** sdague has joined #openstack-infra10:12
sbelous_AJaeger: ping10:14
Zarahi, re: storyboard email notifications, in light of the comments on https://review.openstack.org/#/c/293080/ , where else would we be able to set the sender address? I thought this manifest was consumed directly?10:15
*** electrofelix has joined #openstack-infra10:15
AJaegeryolanda: http://git.openstack.org/cgit/openstack/tripleo-ui/ has content!10:15
yolandawoot!10:15
yolandaand i see projeccts created on gerrit now10:16
AJaegerand also http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/ is imported!10:16
AJaegerwoot10:16
yolandathat bug has been challenging :)10:16
AJaegersbelous_: please speak up, I'll read backscroll if necessary and others can chime in as well...10:16
AJaegeryolanda: thanks for fixing it!10:16
rcarrillocruzyolanda: np10:17
sbelous_AJaeger: ok :)10:17
* AJaeger sends a virtual ansible-mastering-pledge to rcarrillocruz and yolanda ;)10:17
sbelous_AJaeger: can you please take a look at my patch to infra when you have a time? https://review.openstack.org/#/c/286085/10:17
yolandai'll check with mordred before asking a pull request for that case, it may have some reason for that logic but i don't understand10:17
AJaegersbelous_: that's a nova change, best work with sdague further on it...10:18
rcarrillocruzheh10:18
sbelous_AJaeger: okay, will ask10:18
sdagueAJaeger / sbelous_ +210:18
sbelous_sdague: thanks!10:19
AJaegersbelous_: just commented on a few issues ;)10:20
AJaegerbut not so critical to revoke sdague 's +A ;)10:20
sdagueyeh, we can do as a follow up10:20
AJaegeryolanda: github syncing worked as well ;)10:21
AJaegerhttps://github.com/openstack/openstack-ansible-os_designate/10:21
yolandanice.. i feel happy now10:21
sbelous_AJaeger: thanks for comments, yeah, I'll consider it in further reworks10:22
*** dizquierdo has joined #openstack-infra10:22
zaroclarkb, jhesketh :at gerrit hackathon this week. i'm hopig to get the changes to stream events merged this week.10:22
*** sambetts|afk is now known as sambetts10:22
AJaegeryolanda, sdague : A review of https://review.openstack.org/#/c/292135/ would be appreciated to move us forward wiht api jobs, please10:23
* yolanda is going with the review queue after the daily troubleshooting session :)10:23
AJaegersdague: could I trouble you with a nova review as well - https://review.openstack.org/292137 , please?10:23
AJaegersdague: and then there's another compute api review: https://review.openstack.org/29288810:24
openstackgerritChristoph Gysin proposed openstack/gertty: Add support for auth-type=form  https://review.openstack.org/16946710:24
yolandaah sdague, care reviewing https://review.openstack.org/292792 again? the devstack-gate fix10:25
openstackgerritMerged openstack-infra/project-config: Initial job for Nova with os-vif  https://review.openstack.org/28608510:25
*** claudiub has joined #openstack-infra10:27
*** daemontool has joined #openstack-infra10:28
yolandaAJaeger, another thing that merged today is the changes in jjb to use lockfiles. So if you see any problems with jenkins jobs in masters let me know...10:28
sdagueyolanda: that's complicated enough, could you put a test together for it?10:29
AJaegeryolanda: will do - thanks for the heads up10:29
yolandaoh tests :)10:29
yolandaok10:29
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/29326810:31
*** rhallisey has joined #openstack-infra10:31
openstackgerritMerged openstack-infra/project-config: Fix the location of devstack_gate_rc script at horizon job  https://review.openstack.org/29307710:40
odyssey4meyolanda AJaeger aweomse, great to see the openstack-ansible-os_designate repo is properly imported and mirrored to github :)10:49
odyssey4methank you for persisting!10:49
yolandaglad to help10:50
AJaegerodyssey4me: the cookies and coffee you send to Spain have helped ;)10:52
openstackgerritZara proposed openstack-infra/system-config: Add sender email for storyboard.openstack.org  https://review.openstack.org/29336710:52
yolanda:)10:53
openstackgerritMerged openstack-infra/project-config: Add a core group for zaqar-ui  https://review.openstack.org/29316610:53
*** dims has quit IRC10:53
kozhukalovguys, could you please review https://review.openstack.org/#/c/292760/ ?10:55
openstackgerritMerged openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible  https://review.openstack.org/29277110:55
*** thorst has joined #openstack-infra10:55
*** ociuhandu has quit IRC10:55
openstackgerritMerged openstack-infra/project-config: Update Zuul Layout for Puppet OpenStack  https://review.openstack.org/29255510:56
*** salv-orlando has quit IRC10:56
*** exploreshaifali has quit IRC10:57
odyssey4meAJaeger yolanda I'd like to make arrangements for a repository rename - how do I go about doing that?10:57
rcarrillocruzAJaeger, odyssey4me: i love cookies :-)10:57
yolandaodyssey4me, check in http://docs.openstack.org/infra/manual/creators.html, Project Renames section10:58
openstackgerritMerged openstack-infra/project-config: Pre-release stable/mitaka ACLs  https://review.openstack.org/29278110:58
*** flwang1 has joined #openstack-infra10:59
AJaegerodyssey4me: renames are really expensive, it needs complete gerrit downtime and manual steps. Please reconsider10:59
*** dims has joined #openstack-infra10:59
*** kzaitsev_mb has quit IRC10:59
openstackgerritMerged openstack-infra/puppet-openstackid: Update Email Configuration file  https://review.openstack.org/29260510:59
AJaegerodyssey4me: but if it's needed, follow the steps and expect several weeks until this gets done.10:59
*** gongysh has quit IRC11:01
odyssey4meAJaeger will see what we can do11:02
*** FallenPegasus has joined #openstack-infra11:02
*** thorst has quit IRC11:02
openstackgerritMerged openstack-infra/project-config: fuel-qa: move python 2.7 checks to community  https://review.openstack.org/29274611:03
*** esker has joined #openstack-infra11:04
*** yamamoto_ has quit IRC11:05
*** Jeffrey4l has joined #openstack-infra11:06
*** FallenPegasus has quit IRC11:07
rcarrillocruzZara: sorry, i just realized https://review.openstack.org/#/c/293367/1 will fail11:07
rcarrillocruzthe sender_email_address param is not exposed in openstack_project::storyboard :/11:07
*** esker has quit IRC11:08
rcarrillocruzpedroalvarez: ^11:08
pedroalvarezouch11:09
rcarrillocruzhttp://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/storyboard.pp11:09
rcarrillocruzput sender_email_address as undef to ^ class signature11:10
rcarrillocruzthen pass in to ::storyboard::app11:10
pedroalvarezright, sounds easy. Thanks for spotting that!11:11
rcarrillocruznp11:13
*** _amrith_ is now known as amrith11:16
*** openstackgerrit has quit IRC11:18
*** openstackgerrit has joined #openstack-infra11:18
jheskethjeblair: more feedback/discuss on the logs in afs (https://review.openstack.org/#/c/269928/1).. I wonder if we should move it to the mailing list. Anyway, talk to you about it later.11:19
* penguinolog happy, now waiting for the first start of newly enabled jobs11:20
penguinologThanks to igorbelikov , yolanda and AJaeger11:20
openstackgerritMerged openstack-infra/openstackid: Fixed doble form submision issue  https://review.openstack.org/29324111:21
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891511:22
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenStackId: Update Email configuration  https://review.openstack.org/29337911:23
*** kushal has quit IRC11:25
*** Daisy has joined #openstack-infra11:26
*** smarcet has joined #openstack-infra11:27
openstackgerritZara proposed openstack-infra/system-config: Add sender email for storyboard.openstack.org  https://review.openstack.org/29336711:27
smarcetfungi: morning ! :) its seems that puppet deployment for openstackid-dev is failing, take a look at http://puppetboard.openstack.org/report/openstackid-dev.openstack.org/773192c582fcceaec2482ccd2359f1b41df0b4dd11:27
Zararcarrillocruz: thanks for your help! hopefully I did it right this time...11:28
*** hieulq has joined #openstack-infra11:29
*** hieulq has quit IRC11:31
openstackgerritSebastian Schuberth proposed openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory  https://review.openstack.org/23769411:31
*** Daisy has quit IRC11:31
openstackgerritZara proposed openstack-infra/system-config: Add sender email for storyboard.openstack.org  https://review.openstack.org/29336711:32
*** weshay has joined #openstack-infra11:34
*** vgridnev has quit IRC11:34
*** dtantsur is now known as dtantsur|brb11:34
*** thorst has joined #openstack-infra11:35
*** ldnunes has joined #openstack-infra11:38
*** rfolco has joined #openstack-infra11:39
*** exploreshaifali has joined #openstack-infra11:41
*** ihrachys has quit IRC11:42
*** placko_ has joined #openstack-infra11:42
*** ihrachys has joined #openstack-infra11:42
*** ihrachys has quit IRC11:42
*** gordc has joined #openstack-infra11:44
*** erlon has joined #openstack-infra11:44
*** kzaitsev_mb has joined #openstack-infra11:44
*** fverbs has joined #openstack-infra11:45
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281411:48
*** zeih has quit IRC11:49
*** zeih has joined #openstack-infra11:49
fungismarcet: looking in the logs, i have Mar 16 09:54:55 openstackid-dev puppet-user[25843]: (/Stage[main]/Openstackid/Exec[update-site]/returns) executed successfully11:50
openstackgerritSrushti Gadadare proposed openstack-dev/hacking: Add check for mutable objects as method arguments  https://review.openstack.org/29338611:50
smarcetfungi: tnx11:51
fungismarcet: if you suspect the update isn't actually working but is still returning a success exit code, i can try running it by hand to get more detailed output11:51
fungii should look to see if we're logging the output for that somewhere else too11:52
smarcetfungi: well not sure11:53
smarcetyesterday i was waiting for an update that never ran11:53
smarcetwell nows works11:53
smarcetalso11:53
fungiroot@openstackid-dev:~# /opt/deploy/deploy.sh status openstackid11:54
fungiINSTALLED [16-Mar-2016 11:31]11:54
smarcetfungi: i noticed that openstackid.org dns server does not has a AAAA record11:54
*** zeih has quit IRC11:54
smarcetfungi: ok, i will take note of that11:54
*** zeih has joined #openstack-infra11:54
fungifungi@dhole:~$ host -t aaaa openstackid.org11:54
fungiopenstackid.org has IPv6 address 2001:4800:7818:103:53f7:aea1:b964:d61c11:54
smarcetfungi: we have some dns issues yesterday when one tester wanted to access from a cell phone11:55
smarcetweird11:55
smarcetok will dig on that11:55
smarcettnx for all :)11:55
smarcethave a good day :)11:55
fungiyou too!11:55
fungii'm happy to dig deeper on ipv6 related stuff too if you need help11:56
smarcetok cool thx fungi for your offer :)11:57
*** salv-orlando has joined #openstack-infra11:57
smarcetwill let u know11:57
fungismarcet: i know rackspace said they were having some issues with their dns servers yesterday that broke reverse dns, so maybe something related to that11:57
smarcetoh rite, we have some issues with db host yesterday too11:59
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: builders: update cmake to match CMake plugin versions 2.x  https://review.openstack.org/27975411:59
*** dtantsur|brb is now known as dtantsur12:01
*** _degorenko|afk is now known as degorenko12:05
openstackgerrityolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce  https://review.openstack.org/29279212:06
*** baoli has joined #openstack-infra12:07
*** jaosorior has quit IRC12:08
*** yamamoto has joined #openstack-infra12:08
*** jaosorior has joined #openstack-infra12:09
*** jaosorior has quit IRC12:09
*** zeih has quit IRC12:09
*** aysyd has joined #openstack-infra12:09
*** kgiusti has joined #openstack-infra12:10
*** vgridnev has joined #openstack-infra12:10
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Set default locale to image in ubuntu-minimal  https://review.openstack.org/29078912:10
*** jaosorior has joined #openstack-infra12:11
*** zeih has joined #openstack-infra12:11
smarcetfungi: from JP : "It is not openstackid that doesn't have,12:13
smarcetIt is the DNS servers I think, Try host -t aaaa dns1.stabletransit.com"12:13
fungismarcet: right, those are rackspace's dns servers12:13
fungismarcet: jpmaxman: those are the same nameservers as for openstack.org12:14
fungikinda sad state of affairs, honestly. i get around it for my personal domains by running my own bind 10 on instances in different rackspace regions (iad and syd)12:15
smarcetfungi: ok, on another subject, there is any change to get ssh access to openstackid.org, we really wanna get access to production log file live due summit app upcomming12:15
*** ihrachys has joined #openstack-infra12:16
openstackgerritPaul Belanger proposed openstack-infra/system-config: Regenerate nodepool.logging config  https://review.openstack.org/29339612:16
jpmaxmanIt was a weird situation.12:18
jpmaxmanOpenstackid won't work on the T-Mobille LTE network in Austin12:18
fungismarcet: that runs counter to the separation of concerns we'd established for servers where we're providing system support. we can hand control over them back to tipit, but we give up hope of sharing an authentication system in the future with our infrastructure12:18
jpmaxmanFungi we were trying to follow the procedure described for gaining root access to an infra server12:19
fungithough that's worth discussing. i think now that we have gotten agreement from the board that we can ignore cla enforcement, we no longer need to plan for a strong tie between the community infrastructure resources and whatever the foundation and its contractors are developing12:19
jpmaxmanWe were going to have smarcet submit a patch similar to12:20
jpmaxmanhttps://review.openstack.org/#/c/285641/112:20
jpmaxmanOn the ipv6 thing I am not sure why T-Mobile is running in pure ipv6 mode and why even its own nameservers would query for host names in ipv412:21
jpmaxmanhttps://usercontent.irccloud-cdn.com/file/2uxySLPr/Screenshot_20160315-222520.png12:21
fungiyep, we've done that in the past for some outlying systems (wiki, pbx, lists), and given that we don't currently rely on openstackid.org for authenticating anything besides community.o.o and the www site right now it's worth discussing with the team12:22
fungiwe mostly want to be sure, in cases like this, that anyone with direct access to the servers will still work through configuration management to implement fixes for problems rather than patching around them locally12:23
jpmaxmanAnyway not sure what we can do about it but definitely would not be good at the summit in Austin if people roaming in T-Mobile could access the app12:23
fungijpmaxman: i agree, though as i pointed out we'd also need to fix it for www.openstack.org since its nameservers are also not ipv6-connected12:23
vkramskikhhey folks, could you please review https://review.openstack.org/#/c/292760/ ? this is javascript jobs enablement for fuel-ui project12:24
*** kzaitsev_mb has quit IRC12:24
pabelangerneed a better way to keep nodepool-builder.logging.conf in sync12:25
pabelangermaybe we should move that into project-config12:25
smarcetfungi: so i am good to go , and submit a patch like this one  https://review.openstack.org/#/c/285641/1 to get ssh to production server ?12:25
*** ociuhandu has joined #openstack-infra12:25
*** kzaitsev_mb has joined #openstack-infra12:25
jpmaxmanRight. Any rackspace site. Though I would equally blame T-Mobile. Anyway like I said may not be anything we can do...12:25
fungijpmaxman: moving all of our dns would be a significant undertaking on relatively short notice, though perhaps a support request to rackspace letting them know their nameservers are no longer serving anything t-mobile can resolve would be helpful12:26
fungismarcet: that's how access would be implemented, but would still be pending some majority of the existing root sysadmins agreeing to approve that access12:27
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add docs publishing jobs for fuel-docs  https://review.openstack.org/29339912:27
*** salv-orlando has quit IRC12:28
fungianyway, i need to disappear for a bit. need to take my car to the mechanic this morning12:28
*** pcaruana has quit IRC12:28
jpmaxmanIf is working on T-Mobile now12:29
*** akshai has joined #openstack-infra12:29
jpmaxmanI am chalking it up to a tmobile glitch of not having their nameservers support ipv4 queries12:30
jpmaxmanBut will keep an eye on it12:30
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add docs publishing jobs for fuel-docs  https://review.openstack.org/29339912:30
jpmaxmanRackspace is aware12:31
jpmaxmanhttps://feedback.rackspace.com/forums/298158-network/suggestions/5977739-ipv6-addresses-for-the-nameservers-cloud-dns12:31
*** fawadkhaliq has quit IRC12:31
fungii hope there's a similar one for their lack of dnssec support12:32
fungiwhich is the other main reason i run my own nameservers instead of using theirs12:33
*** boris-42 has quit IRC12:34
*** thiagop has joined #openstack-infra12:34
openstackgerrityolanda.robla proposed openstack-infra/jeepyb: Allow create/update projects with description into Gerrit  https://review.openstack.org/11000912:34
*** esikachev has quit IRC12:35
*** lucasagomes is now known as lucas-hungry12:35
pabelangerWhere did the fix to ansible inventory go that clarkb did lastnight?12:38
pabelangerAJaeger: yolanda: Ah, looks like github sync is working again12:39
jheskethspeaking of ansible inventory fixes, if anybody knows how/who to get eyes on https://github.com/ansible/ansible/pull/14699 that'd be handy12:40
openstackgerritMerged openstack-infra/project-config: Templatize api guide jobs  https://review.openstack.org/29213512:40
yolandapabelanger there is a problem on ansible inventory, when there are duplicated hosts with same name12:40
yolandait always assumes that the first server is the one on the list12:40
yolandaand for review.openstack.org that was the new vm created, so puppet was applied there12:40
yolandai did a fix on puppetmaster, but i want to check with mordred, before sending a pull request12:41
openstackgerritMerged openstack-infra/project-config: Only publish API guides from master  https://review.openstack.org/29288812:42
*** ramishra has quit IRC12:43
*** rabi has joined #openstack-infra12:44
*** esikachev has joined #openstack-infra12:44
*** yamamoto has quit IRC12:45
pabelangeryolanda: actually, clarkb did a fix last night12:45
pabelangeryolanda: and pull request12:45
pabelangerhttps://github.com/ansible/ansible/pull/1498412:46
yolandapabelanger i saw the inventory in a bad state this morning on review.openstack.org, and puppet had not run since 10 mar12:46
pabelangeryolanda: right, see above12:46
*** rabi is now known as ramishra12:46
yolandai did the same fix :)12:46
pabelangerand check eavesdrop.o.o from yesterday, we had a discussion abou tit12:46
yolandaso maybe he ran it, but not flushed cache?12:46
pabelangerabout it*12:46
pabelangerI don't actually think he ran it12:47
pabelangerbut will have to wait for clarkb to confirm12:47
yolandaah ok, so i didn't see that conversation, but i debugged and arrived the same conclusion12:47
yolandawhat i did is to test locally and once i had the inventory working properly, applied it with refreshing cache on puppetmaster, to fix review.openstack.org12:47
pabelangerokay, not sure the path moving forward. I suspect mordred can confirm the fix in PR and merge12:48
pabelangerI think we run devel branch too12:48
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental centos-7 puppet-apply  https://review.openstack.org/29340912:49
*** pradk has joined #openstack-infra12:51
*** FallenPegasus has joined #openstack-infra12:51
fungiyolanda: right, we decided not to fix it in production until the ansible reviewers bless the solution12:52
fungiin case our understanding was flawed12:52
*** csaba has quit IRC12:52
*** yamamoto has joined #openstack-infra12:52
*** jlvillal has quit IRC12:52
*** jlvillal has joined #openstack-infra12:52
*** smarcet has quit IRC12:53
pabelangeryolanda: AJaeger: when you have a moment, ^ brings online a centos7-minimal dib for experimental testing12:53
fungididn't want to risk breaking everything everywhere, when the current impact is really just new project creation remains in limbo and we can't update nodepool images at the moment12:53
pabelangerfungi: Do you want to hold off on debian-jessie DIB until we fix inventory 100%?12:54
fungifixing minor inconveniences like that at the risk of much broader breakage is more reckless than we like to be in our systems administration practices12:55
*** ddieterly has joined #openstack-infra12:56
*** FallenPegasus has quit IRC12:56
fungipabelanger: well, it sounds like maybe yolanda went ahead and applied an unsanctioned patch to ansible on the puppetmaster to work around the situation, so there's presumably nothing stopping us from approving the debian nodepool images now12:56
*** csaba has joined #openstack-infra12:56
anteayaI'm away this morning running errands12:57
openstackgerritThiago Paiva Brito proposed openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst  https://review.openstack.org/29308712:57
pabelangerfungi: Okay, I'll bring debian online and monitor12:57
*** eharney has joined #openstack-infra12:57
pabelangerwill wait for nodepool to autobuild at 14hrs12:58
fungianyway, dropping back offline again to deal with car stuff12:59
anteayahappy car stuff13:00
openstackgerritFrank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite  https://review.openstack.org/27470113:00
*** Jeffrey4l has quit IRC13:02
*** dtardivel has joined #openstack-infra13:02
*** dprince has joined #openstack-infra13:02
*** rcernin has joined #openstack-infra13:03
*** Hal has joined #openstack-infra13:04
*** Hal is now known as Guest5118413:05
yolandaah fungi i lost that conversation... i checked that the results were the expected before doing that, and i also did not update the code in puppetmaster, i was running a local copy on my directory13:06
*** kaisers1 has quit IRC13:09
pabelangerokay, relocating to the coffee shop in town.  should be back online in a hour13:09
pabelangerwill continue with debian jessie then13:09
*** e0ne has joined #openstack-infra13:10
*** ddieterly has quit IRC13:10
*** bdemers has joined #openstack-infra13:11
yolandafungi, i'm thinking on a better way to pass information between infra-roots on different shifts. In Gozer, for some time, we were putting notes on  a wiki page, about the different pending tasks, important information... that needed to be passed across shifts13:11
yolandadoing that, it was easier that the important information was well transmitted, rather than having to read the whole backlog13:11
yolandawhat do you think?13:12
*** zeih has quit IRC13:12
*** placko_ has quit IRC13:13
AJaegeryolanda: yes, having something like that would be great. Whether it's wiki page, etherpad,...13:13
*** Jeffrey4l has joined #openstack-infra13:14
*** kro_sinus_hell is now known as krot_sinus_purga13:15
*** krot_sinus_purga is now known as kro_sniffles13:15
*** nmagnezi has quit IRC13:15
*** smcginnis_away is now known as smcginnis13:15
*** moravec has joined #openstack-infra13:16
openstackgerritMerged openstack-infra/release-tools: Add update_reviews tool  https://review.openstack.org/29085313:16
*** kaisers has joined #openstack-infra13:19
*** smarcet has joined #openstack-infra13:19
*** xyang1 has joined #openstack-infra13:19
openstackgerritMerged openstack-infra/project-config: Add debian-jessie image to nodepool  https://review.openstack.org/26472613:20
*** zeih has joined #openstack-infra13:20
stevemarAJaeger: ughhhh why wasn't UC updated! https://review.openstack.org/#/c/293042/13:21
stevemarAJaeger: thanks for the detective work13:21
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add curl package to puppet nodepool element  https://review.openstack.org/29341713:22
stevemarAJaeger: it was updated at one point: https://review.openstack.org/#/c/288124/1/upper-constraints.txt13:23
stevemarlooks like https://github.com/openstack/requirements/commit/7953162c5b297f326a975a2520afb05f605ba67b reverted it13:24
stevemarthats unsettling13:24
*** ig0r_ has joined #openstack-infra13:24
*** _ari_ has joined #openstack-infra13:24
AJaegerstevemar: ask dims, maybe he made an error...13:25
openstackgerritMerged openstack/requirements: Revert "Tolerate both new and old setuptools"  https://review.openstack.org/29318713:25
stevemarAJaeger: yeah, i will post a change updating upper-constraints and explain it all, seems like it was a mistake13:25
openstackgerritMerged openstack/requirements: Update to oslo.messaging 4.5.1  https://review.openstack.org/29256713:25
*** _nadya_ has quit IRC13:25
*** lucas-hungry is now known as lucasagomes13:26
smarcetfungi:yolanda: please when u get a chance review https://review.openstack.org/#/c/293379/ thx13:26
dimsstevemar : what did i do? :)13:26
*** kaisers has quit IRC13:26
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/29342313:26
*** Jeffrey4l has quit IRC13:27
yolandafungi, AJaeger, and easy thing for that... why not send status log messages on the things we need to transmit on shifts? by that way , it will be logged on the wiki page, and when we come at EMEA any problems, status updates, etc... will be easily reachable13:27
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test  https://review.openstack.org/28944513:27
*** e0ne has quit IRC13:27
*** placko_ has joined #openstack-infra13:28
yolandasmarcet, what about hiera values? are they set on puppetmaster?13:28
openstackgerritJulien Danjou proposed openstack-infra/project-config: Do not Gnocchi coordinator backend  https://review.openstack.org/29342413:28
*** kaisers has joined #openstack-infra13:28
*** Guest51184 has quit IRC13:28
*** mriedem has joined #openstack-infra13:28
openstackgerritSteve Martinelli proposed openstack/requirements: bump uc for congressclient to 1.2.3  https://review.openstack.org/29342513:29
*** fawadkhaliq has joined #openstack-infra13:29
stevemardims: ^13:29
*** automagically_ has joined #openstack-infra13:29
*** salv-orlando has joined #openstack-infra13:29
*** _nadya_ has joined #openstack-infra13:29
*** nmagnezi has joined #openstack-infra13:30
stevemardhellmann: ^13:31
*** automagically_ has left #openstack-infra13:31
igorbelikovpabelanger: just caught an issue with trying to generate debian image with all the infra elements - missing curl/wget:( hotfix: https://review.openstack.org/#/c/293417/ could you take a look?13:32
AJaegerdims, you downgraded congressclient by accident (I guess)13:32
dimsstevemar : hmm, i watch the proposal bot daily as well and it did not catch this either :( https://review.openstack.org/#/c/288849/13:32
dimsstevemar : apologies, bad merge on my part13:32
dhellmannstevemar : +2a13:33
dimsAJaeger : ^^13:33
*** sdake has joined #openstack-infra13:33
*** jswarren has quit IRC13:34
AJaegerdims, dhellmann: patch 2 comes from the proposal bot - is there a bug in our scripts? see https://review.openstack.org/#/c/287613/2/upper-constraints.txt13:34
dimsdhellmann : looks like proposal bot thinks 2015 is newer than 1.x so it may re-propose13:34
clarkbstevemar: it is going to keep doing that because 2015.1.0 is "newer"13:34
AJaegerdims: I agree13:35
dhellmannugh13:35
dimswhew!!!13:35
dimsnot entirely my fault :)13:35
*** d0ugal has quit IRC13:35
AJaegerdims: ;)13:35
dhellmannwell, we won't be approving any proposal bot changes until the freeze lifts so we have some time to fix it13:35
*** d0ugal has joined #openstack-infra13:35
dimsright dhellmann13:35
* dims adds a post it on my wall13:36
dhellmannI wonder if hiding the 2015 release would help? I'm not sure how the bot finds new releases13:36
*** vgridnev has quit IRC13:36
ttxor special case it in the bot13:36
*** jpr has joined #openstack-infra13:36
dhellmannyeah, that would be fairly simple, too13:37
ttxso taht it knows 2015 is actually < 113:37
*** vgridnev has joined #openstack-infra13:37
ttxat least in python-congressclient13:37
clarkbpreferably we dont need to special case just do <2000 in the reqs13:37
stevemarthat should work for most cases13:37
clarkbor something like that and let existing dep resolution filter it out13:37
*** ddieterly has joined #openstack-infra13:38
dhellmannnoted in https://etherpad.openstack.org/p/newton-relmgt-plan13:38
*** vgridnev has quit IRC13:38
ttxright is there is a 20xx but no 19xx version13:38
dhellmannclarkb : oh, that's a good idea, too13:38
clarkbI want to say hiding a release doesnt affect pip only web ui browsing13:38
ttxhumans are good at 2015 < 1. Computers aren't13:39
dhellmannI think you're right, but capping the requirement spec would fix it13:39
clarkbnot sure what we did for  the other clients13:39
*** doug-fish has quit IRC13:39
dhellmannstevemar : do you want to add a <2000 to global requirements in that patch?13:39
clarkbthere were long email threads about it to go back and remember13:39
dimsso python-congressclient<=2015 ?13:39
*** salv-orlando has quit IRC13:40
dimsah <200013:40
dims+113:40
*** edmondsw has joined #openstack-infra13:41
stevemardhellmann: it's +a'ed still want to change g-r?13:42
*** doug-fish has joined #openstack-infra13:42
fungiyolanda: AJaeger: we do have "#status log" which will update https://wiki.openstack.org/wiki/Infrastructure_Status without doing noisier things in irc channels. we could start relying on that more for cross-timezone coordination (that's what it was mostly intended for, if memory serves)13:42
dhellmannstevemar : a follow up?13:42
stevemardhellmann: rgr13:42
dhellmannI thought if it hadn't even passed check it wouldn't matter, but whatever you like13:42
*** shardy has joined #openstack-infra13:42
yolandafungi, that's what i said as well, i think it's a good idea13:42
*** tonytan4ever has joined #openstack-infra13:42
*** yamamoto has quit IRC13:43
yolandafungi it will be great for us in EMEA if you start applying on the US timezone, because it's hard to know about the status sometimes13:43
openstackgerritSteve Martinelli proposed openstack/requirements: prevent installing old releases of congressclient  https://review.openstack.org/29344113:44
stevemardhellmann: ^ just in case that one is more contentious :)13:45
dhellmannstevemar : +2, maybe ttx will +2a13:45
*** tzn has quit IRC13:45
*** ddieterly is now known as ddieterly[away]13:45
*** doug-fish has quit IRC13:46
*** zeih has quit IRC13:47
fungiyolanda: i already added the hiera keys for 293379 yesterday in the corresponding fqdn files, yes13:47
*** nwkarsten has joined #openstack-infra13:48
ttxdims did +2a13:48
dimsttx : oops, did not see dhellmann 's response to wait for yours13:48
yolandacool13:48
dhellmanndims, ttx: either way, a second person to share the blame ;-)13:49
*** yuval has joined #openstack-infra13:49
dimshaha13:49
ttxdims: yep, I prefer people to blame you so awesome13:49
dimslol13:49
stevemari blame dims for everything :]13:50
yuvalHey, who should I contact to add my details to Stackalytics default_data.json?13:50
dimssingle point of failure :)13:50
dimsyuval : just log a review just like any other project13:50
stevemaryuval: you can post a change13:50
yuvaldims: thanks13:50
stevemarAJaeger: got time for another question? :)13:50
AJaegerstevemar: a quick one - more details in a few hours...13:51
stevemarAJaeger: a few days ago, this change merged: https://review.openstack.org/#/c/290351/13:52
fungittx: dims: dhellmann: clarkb: if we start adding <2000 in our requirements lists, the people who insisted on us adding python version epochs will be coming back out of the woodwork in very short order13:52
clarkbfungi: do you remember how we worked around that?13:52
stevemarbut i'm not seeing the change take effect for all jobs, like this one: https://review.openstack.org/#/c/293042/ (the osc-check-plugins) output didn't change13:53
clarkbas far as I know this isnt a problem for the other clients so we did something13:53
stevemarAJaeger: do the servers need to refresh their script or something like that?13:53
*** amitgandhinz has joined #openstack-infra13:53
clarkband I am sure it is in one of those long epoch threads13:53
fungiclarkb: novaclient was the only other library to suffer this fate, and it was pre-pypi13:53
clarkbah13:53
*** zeih has joined #openstack-infra13:54
openstackgerritMatt Riedemann proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes  https://review.openstack.org/27027313:54
dhellmanndo setuptools, pbr, and pip all handle epochs?13:54
fungiour argument for _not_ epoching things is that it was only affecting servers which are not published on pypi13:54
*** mtanino has joined #openstack-infra13:54
*** mriedem is now known as mriedem_meeting13:54
*** ddieterly[away] is now known as ddieterly13:54
openstackgerritMerged openstack-infra/system-config: OpenStackId: Update Email configuration  https://review.openstack.org/29337913:54
*** armax has joined #openstack-infra13:54
AJaegerstevemar: let's ask fungi and clarkb - this is ubuntu-trusty-ovh-bhs1 image. I think it needs to be build with the content of jenkins/scripts. And if that image wasn't updated since then ;(13:55
fungiso maybe an epock for congressclient is actually in order13:55
ttxdhellmann: humans don't handle epochs13:55
*** vgridnev has joined #openstack-infra13:55
fungier, epoch13:55
ttxspok13:55
ttxspock13:55
* AJaeger will be back later13:55
stevemarAJaeger: take care sir13:55
*** jswarren has joined #openstack-infra13:55
fungilive long and version13:55
mordredclarkb: I merged your inventory patch in ansible - thanks!13:56
mordredclarkb: also, I have core on ansible/ansible now13:56
ttxthose Ansible people clearly don't know what they opening themselves to13:57
*** exploreshaifali has quit IRC13:57
ttxthey *are* opening13:57
fungistevemar: AJaeger: yeah, it looks like the last time we got a successful ubuntu-trusty upload to ovh-bhs1 was nearly 9 days ago13:57
*** rossella_s has quit IRC13:58
mordredfungi: that's no good13:58
stevemarfungi: what does one need to do to trigger a newer version?13:58
*** zeih has quit IRC13:58
fungistevemar: figure out why glance in ovh hates us? or determine whether we've got a bug with our uploading13:58
mordredfungi: I wonder if maybe we should add a thing to nodepool to not boot nodes on providers whose image is older than X days13:58
mordredfungi: there is an age point where continuing to use the image is likely worse overall for the gate13:59
stevemarfungi: should it happen daily?13:59
fungimordred: it's worth considering. maybe set it to 4 days or a week or something13:59
fungistevemar: yep, daily like clockwork when things are working correctly13:59
*** tonytan4ever has quit IRC14:00
stevemarfungi: i see. sounds like this one is beyond capabilities of a mere mortal like myself, and needs an infra demi-god to fix >.<14:00
*** doug-fish has joined #openstack-infra14:00
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Enable ceph and netiso for the nonha job  https://review.openstack.org/29344614:00
*** tonytan4ever has joined #openstack-infra14:00
*** doug-fish has quit IRC14:01
fungistevemar: what's odd is that we have no trouble in ovh's gra1 region, it's just uploads to the bhs1 region that seem to be impacted by this14:01
openstackgerritJulien Danjou proposed openstack-infra/project-config: Do not set Gnocchi coordinator backend  https://review.openstack.org/29342414:01
*** HeOS has quit IRC14:01
*** doug-fish has joined #openstack-infra14:01
stevemarfungi: that is odd, i wonder whats different with that region14:01
fungistevemar: sad considering the bits don't have nearly as far to go to reach bhs114:01
fungiwhile they cross the atlantic to reach gra114:02
*** dmellado is now known as dmellado|lunch14:02
clarkbI think ovh has trouble with those 17GB uploads...14:03
clarkbnot a bug on our side as far as I know14:03
*** tqtran-afk has joined #openstack-infra14:03
*** jpr1 has joined #openstack-infra14:03
fungiclarkb: sounded like we're very close on having a dib with SamYaple's fstrim patch?14:03
SamYapleim here14:03
fungimaybe that will improve dramatically once we're using it14:04
SamYapleit was merged into dib, i dont think dibs been tagged14:04
clarkbthese are raw images so wont be affected that much14:04
fungioh14:04
clarkbalso this is why using raw/vhd as transport format is silly14:04
fungiovh, y u no qcow2?14:04
SamYapleclarkb: do you have access to glancev2 stuff with ovh? you can upload qcow2 and convert on thier backend if you do14:05
*** jpr has quit IRC14:06
clarkbuh I am not sure that properly exists yet14:06
*** sigmavirus24_awa is now known as sigmavirus2414:06
*** nwkarste_ has joined #openstack-infra14:06
clarkbaiui the only way to support that is via the broken tasks interface whoch is deploymwnt specific14:06
*** roaet- has quit IRC14:06
*** HeOS has joined #openstack-infra14:06
*** nwkarsten has quit IRC14:06
*** mgagne has quit IRC14:07
*** hughsaunders has quit IRC14:07
*** mgagne has joined #openstack-infra14:07
*** jbryce has quit IRC14:07
SamYapleclarkb: in ovh or in general? because thats waht i do for my cluster and ceph14:07
*** tqtran-afk has quit IRC14:08
*** hughsaunders has joined #openstack-infra14:08
*** jbryce has joined #openstack-infra14:08
*** roaet_ has joined #openstack-infra14:09
SamYapleit requires glance tasks, but maybe we are crossing wires?14:09
clarkbin general14:09
clarkbits not a glance feature14:09
clarkbit is a thing you can hide behind glance tasks ina non portable or discoverable manner that may or may not exost depending on deployment14:09
SamYaplebut we are talking about ovh specifically requiring raw, so if that can be abstracted....14:10
clarkbbecause tasks as an api dont do these things14:10
clarkbSamYaple: I am not going to use tasks for more stuff14:10
SamYaplefair enough, but this _is_ a feature of glance14:11
*** nwkarste_ has quit IRC14:11
clarkbit is probably the worst part of using opensta k for the infra team right now14:11
SamYaplethat you can use in vanilla glance14:11
SamYaplehey you dont have to tell me. im just offering suggestions14:11
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Deploy ceph in the nonha job  https://review.openstack.org/29305114:11
clarkbSamYaple: sort of, tasks are part of glance but there is no standard set of tasks aiui14:11
clarkbSamYaple: so image convert isnt really part of glance14:12
*** yamamoto has joined #openstack-infra14:12
mordredwe've talked about adding an image convert option to glance server14:12
clarkband you cant really find out if it is available as a user, or know what data to pass a task or what the return data means or any of it14:12
mordredbut that won't happen soon14:12
mordredalso - the spec for fixing image upload is pretty much done now, so I expect that to get good implementation in newton14:13
SamYapleclarkb: glance task to import from a url. the url is hosted here with infra and its a qcow2 file, it spits out a raw image14:13
SamYaplethat should be completely possible as a user14:14
clarkbSamYaple: where is it documented? that isnt in the api docs14:14
clarkbbecause tasks are not standard only the minimal api to start tadks is14:14
mordredalmost no clouds have tasks14:15
pabelangerigorbelikov: do you have a log file of the problem?  Curious when you get the error. We recently added curl into puppet manifests, but that could be too late in the build14:15
mordredand no clouds have tasks that do image type conversions14:15
*** salv-orlando has joined #openstack-infra14:15
mordredand to my knowledge there are no clouds that have a task for import-from-url14:15
*** Hal has joined #openstack-infra14:17
igorbelikovpabelanger: the interesting part is here http://paste.openstack.org/show/490714/, though I can provide a full log if you need it14:17
SamYapleclarkb: where is it documented, i dont know. but this is what i use glance --os-image-api-version 2 task-create --type import --input '{"import_from_format": "qcow2", "import_from": "http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img", "image_properties": {"name": "cirros-0.3.4-raw", "disk_format": "raw", "container_format": "bare"}}'14:17
*** Hal is now known as Guest5005114:17
*** rbrndt has joined #openstack-infra14:17
SamYaplemordred: if its not available its not availble, this was just a suggestion14:17
clarkbSamYaple: right thats not part of glance14:17
pabelangerigorbelikov: Odd, did you have an issue before with url / wget on debian?14:18
clarkbthats part of a particular glance deployment that has chosen to support a task that takes this input14:18
clarkbthis is the problem.with tasks14:18
mordredSamYaple: what cloud do you use that with?14:18
SamYaplemordred: vanilia deployed services in a cloud i manage (minimal configs, using defaults)14:19
clarkbbut with ceph14:19
SamYapleit requires v2 glance obviously14:19
igorbelikovpabelanger: think I may have tested this with a set of patches which included curl at some point14:19
SamYapleclarkb: that doesnt matter here14:19
clarkbI am guessing the ceph glance bacjend currenetly supports that task14:19
SamYapleno i can do it without ceph too14:19
SamYaplewith file backend14:19
mordredSamYaple: weird. is this mitaka?14:20
mordredmaybe this got fixed/added recently14:20
SamYaplethis would be mitaka yes, but its in liberty14:20
mordredk14:20
SamYapleim no glance expert here. more testing is needed14:20
mordredI will go investigate the clouds we have access to14:20
SamYaplei thought this was a known thing, i certainly use it14:20
mordredand see if that thing works14:20
mordredon them14:20
*** jpr1 has quit IRC14:20
clarkbSamYaple: it isnt documented in the api14:20
sdaguesome version of this has been around for a while14:20
SamYapleclarkb: well i found it by digging through code so that makes sense14:20
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Bug Fixing  https://review.openstack.org/29345914:21
*** zeih has joined #openstack-infra14:21
pabelangerigorbelikov: ok14:21
clarkbthe only api you get is create list and show tasks14:21
clarkbbut no info on what available tasks are14:21
clarkband what inputs they accept and outputs they write14:21
SamYapleclarkb: technically this is just a create task14:22
clarkbyes but the import from task is part of the api14:22
*** tonytan4ever has quit IRC14:22
clarkbbasically its how not to api14:22
*** tonytan_brb has joined #openstack-infra14:22
SamYapleis it? it looks like normal glance stuff to me (ignoring tasks all together import_from is a glance thing)14:22
clarkbwe could rewrite all our apis to be "do random undocumented thing"14:23
openstackgerritMerged openstack-infra/openstackid-resources: Bug Fixing  https://review.openstack.org/29345914:23
SamYapleclarkb: im not saying youre wrong, but the import_from_format and import_from are all valid glance things, without tasks14:24
sdagueah, we only test the v1 version of this upstream - http://logs.openstack.org/67/271467/2/check/gate-tempest-dsvm-full/a17f519/console.html.gz#_2016-02-25_02_17_31_79414:24
sdagueyes, glance v2 is definitely super undocumented14:24
clarkbSamYaple: but they arent formally part of the api14:24
clarkbSamYaple: instead its hidden behind tasks which completely breaks the whole.system for users14:24
SamYapleclarkb: im not arguing there :)14:24
sdagueclarkb: payload is part of the API14:24
sdagueallowed tasks should be documented14:25
clarkbsdague: yes thats the problem14:25
clarkbsdague: it is arguably part of the api but it isnt treated that eay14:25
sdagueso, I agree, the API is not documented here14:25
sdagueright, well that's garbage :)14:25
clarkbinstead we have documentation for run a function14:25
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Enable voting for gate jobs in sahara-tests  https://review.openstack.org/29346414:25
clarkbthe names and args to the function are neglected14:25
fungituring-complete api14:25
clarkbPOST dosoemthing14:25
sdaguethat being said, updating API docs is hard today14:25
clarkbthats ehat we get14:25
SamYapleare you guys telling me you dont dig through the projects looking for cool undocumented thigns?14:26
SamYaplecause thats like 40% of what i do14:26
sdagueSamYaple: :)14:26
yolandasdague, i fixed https://review.openstack.org/292792, added some test as well14:26
*** Daisy_ has joined #openstack-infra14:26
sdagueyolanda: thanks14:27
sdague+214:27
yolandagreat14:27
*** sdake has quit IRC14:27
clarkbthen on top of post dosomething other deployments can add extra random undocumented functions14:27
*** zeih has quit IRC14:27
clarkbwhy bother having a rest api just give me a generic computing tpc interface14:27
*** nwkarsten has joined #openstack-infra14:27
SamYaplei think clarkb might have a heart attack over this14:27
clarkbSamYaple: its something we have struggled witha lot in infra14:28
clarkbSamYaple: rax has/had their own tasks that we cant easily test against because the code isnt upstream14:28
clarkbbut these tasks are the only eay to upload to rax14:28
*** Swami has joined #openstack-infra14:29
SamYaplei worked at the RAX, i know about the special snowflake stuff14:29
clarkband you get errors back in json like "error 396"14:29
clarkbin an http 20014:29
fungiwe get the very distinct impression that the number of openstack users who use more than 1 or 2 different openstack public clouds are rare enough not to provide feedback on the terribly inconsistencies we have in our ecosystem14:29
SamYapleits a bit annoying, i give you that14:29
clarkbso the api says "success! error"14:29
*** zz_dimtruck is now known as dimtruck14:29
fungier, terrible14:29
SamYaplethats kinda like getting a pizza with a supoena inside14:30
*** fawadkhaliq has quit IRC14:30
clarkbas a resukt my opinion of the tasks api is very bad14:30
fungiSamYaple: subpoena pizza with anchovy is the best14:30
*** fawadkhaliq has joined #openstack-infra14:30
*** Daisy_ has quit IRC14:31
SamYaplei like bonchovy better14:31
*** nwkarsten has quit IRC14:31
vkramskikhhi, AJaeger, anteaya, could you please help with review of https://review.openstack.org/#/c/292760/ ? this patch will add linting jobs for fuel-ui project14:32
clarkbthe best part is glance has an upload api too14:32
fungifor the record, the majority of images in ovh bhs1 are over a week old. i'm tempted to delete all images over a week old and get nodepool to aggressively reupload until new ones stick. any reasons i shouldn't?14:32
clarkbso making users suffer through an undocumented task to upload is pretty terrible14:32
clarkbfungi: if we need the capacity that region is one of our biggest14:33
*** sc68cal has joined #openstack-infra14:33
*** ayoung has joined #openstack-infra14:33
fungiyeah, we're maxxed out on our quotas for the past hour or two, so maybe now is not the best time14:34
pabelangerbut if uploads haven't happened for a week, do we want jobs to run in that zone on stale images?14:34
*** [1]Thelo has joined #openstack-infra14:35
fungi"stale" is not necessarily bad. they will be lacking slave script updates or anything that we adjusted in puppet/elements but they still update their packages and git repos when they run14:35
clarkbright caches get updated and are generally fine14:36
pabelanger /me nods14:36
fungialso we're running a pretty large band of ready and building nodes so i'm going to make sure something's not hobbling us somwhere14:37
pabelangerI want to get a grafana dashboard to better track dib builds and image uploads14:37
pabelangerso we can easy see what each zone is running14:37
*** Thelo has quit IRC14:37
*** [1]Thelo is now known as Thelo14:37
clarkbfungi: jenkins07 needs to be restarted, the playbook died on it yssterday14:37
fungii'll take care of that unless you're already on it14:37
fungilooks like it's to blame for a lot of these ready nodes actually14:38
fungii'm clearing those away now14:38
pabelangerfungi: clarkb: when you have a second, could use some feedback on 293417 regarding curl on dibs14:39
pabelangerspecific minimal images14:39
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-changes option  https://review.openstack.org/29347214:39
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test  https://review.openstack.org/27921814:40
*** _nadya_ has quit IRC14:41
clarkbpabelanger: I almost wonder if we should have a make-minimal-useable element that installs curl and sshd and so on14:41
clarkbbut since the puppet element needs curl thats a decent locatio  for now14:42
jeblairpabelanger: i think just a simple nodepool status interface would be more useful for image status than a dashboard.  i think those data are best in tabular form.14:42
*** roxanagh_ has joined #openstack-infra14:42
*** yuval has left #openstack-infra14:43
clarkb"make minimal images great again" element14:43
fungiwould be nice to slowly move content from the puppet element into a shim element14:43
openstackgerritAlexey Ovchinnikov proposed openstack-infra/project-config: Reverting Manila LXD driver changes  https://review.openstack.org/29347414:43
fungias we un-puppet bits our job workers need14:43
fungiokay, jenkins07 is cleaned up and on its way back into service now14:44
*** asselin_ has joined #openstack-infra14:44
*** _nadya_ has joined #openstack-infra14:45
*** zeih has joined #openstack-infra14:45
pabelangerclarkb: Right, I would agree with that14:45
pabelangerjeblair: Ya, only issue with that, my html foo is weak.  Grafana makes it easy for me to render stuff :)14:46
jeblairpabelanger: i don't think it needs to be html :)14:46
pabelangerAhh14:46
fungibrowsers are great at rendering preformatted text/plain14:46
fungior it could be json and let someone else who cares write the renderer14:47
*** asselin__ has joined #openstack-infra14:47
pabelangerfungi: I was thinking using minimal2 as the element name :D14:48
fungi`wget | flamel` works for me as a renderer ;)14:48
*** nwkarsten has joined #openstack-infra14:48
*** asselin_ has quit IRC14:48
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job  https://review.openstack.org/26046614:49
*** roxanagh_ has quit IRC14:50
jeblairfungi: i'm still a little bleary eyed from waking up, and was wondering why you're so positive about "wet flannel"14:50
fungiit's a seattle thing14:51
*** akaszuba has quit IRC14:52
*** nwkarsten has quit IRC14:53
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue  https://review.openstack.org/29317814:54
*** cloudtrainme has joined #openstack-infra14:55
*** tosky has joined #openstack-infra14:55
*** ogelbukh has quit IRC14:55
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the container CI job  https://review.openstack.org/29348114:56
openstackgerritMerged openstack-infra/project-config: Add curl package to puppet nodepool element  https://review.openstack.org/29341714:56
*** mriedem_meeting is now known as mriedem14:56
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fixed DH error  https://review.openstack.org/29348214:57
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891514:58
EmilienMjeblair: where can I report a but to Gertty?14:58
*** zeih has quit IRC14:58
jeblairEmilienM: here or storyboard14:58
*** nwkarsten has joined #openstack-infra14:58
EmilienMjeblair: since recently, when I open latest Gertty and go on a review and press on "Diff" to read code, Gertty crash with http://paste.openstack.org/show/nvFDq0eCgkeIYeiYDK2U/14:59
jeblairEmilienM: if you update to latest master, it should be fixed15:00
fungido we still need the LaunchpadSync account to be a member of the Administrators group in gerrit? i'm thinking the only thing it's doing now is bug/bp updates, and the reason we had it in administrators previously was for auto-abandon (and further back, also group creation/membership updates from lp)15:01
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-projects option  https://review.openstack.org/29347215:01
EmilienMjeblair: so 1.3.1 is "broken" ?15:01
fungiall the cool kids continuously deploy gertty from master ;)15:01
EmilienMlol15:02
EmilienMok ok get it!15:02
jeblairEmilienM: gitpython shipped a breaking change :/15:02
fungiwhich they also seem unwilling to acknowledge as a problem, much less fix15:02
jeblairwell, i think they acknowledged the problem15:02
fungiEmilienM: you _can_ also downgrade gitpython to similar effect15:02
jeblairbut are completely baffeled by the idea of fixing it15:02
EmilienMfungi: I want to be cool kid.15:03
jeblairi guess we could make a brnach and tag a 1.3.2, or i could see if master is ready to be 1.4.015:03
fungijeblair: i may have not read closely enough, but it seemed like they acknowledged it was a behavior change, they just didn't acknowledge that behavior changes like that in a library are a problem from their perspective15:03
EmilienMfungi, jeblair: works fine on master, thanks guys15:04
jeblairfungi: he said something like "something is broken either way"15:04
fungiand apparently new broken is superior to old broken15:04
jeblairin a tone that made it seem like there was some vague sense of regret15:04
jeblairand that's about as far as it goes15:04
*** dizquierdo has quit IRC15:05
*** vgridnev has quit IRC15:05
*** Guest50051 has quit IRC15:05
*** yamamoto has quit IRC15:06
jeblairi'll work on 13215:06
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fixed DH error  https://review.openstack.org/29348215:07
*** sfinucan has joined #openstack-infra15:07
*** asselin__ has quit IRC15:08
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319315:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: WIP: Add python-db jobs - for nova initially  https://review.openstack.org/29247215:11
jeblairEmilienM: I tagged gertty 1.3.2 with the fix15:12
EmilienMjeblair++15:12
openstackgerritsebastian marcet proposed openstack-infra/system-config: Add user smarcet to openstackid production server  https://review.openstack.org/29349115:12
EmilienMjeblair: thx, I love this tool really15:12
jeblairEmilienM: thanks! :)15:12
*** dizquierdo has joined #openstack-infra15:13
*** mrmartin has quit IRC15:13
pabelangerokay, relocating again. Should be back in a hour15:13
*** david-lyle has quit IRC15:15
*** david-lyle has joined #openstack-infra15:15
*** yamahata has joined #openstack-infra15:15
AJaegerclarkb: how were today's ubuntu-trusty dsvm runs? Are we ready to merge https://review.openstack.org/#/c/292573/ ?15:15
*** dmellado|lunch is now known as dmellado15:15
*** roxanagh_ has joined #openstack-infra15:16
*** ociuhandu has quit IRC15:16
openstackgerritMerged openstack-infra/openstackid: Fixed DH error  https://review.openstack.org/29348215:16
*** jpr has joined #openstack-infra15:16
clarkbAJaeger: when I looked yesterday they seemed fine. I think we can merge that change15:17
zarofungi: i think you were interested in something like this? https://gerrit.googlesource.com/plugins/review-strategy/+/master/src/main/resources/Documentation/about.md15:19
AJaegerclarkb: I suggest a followup to change nodepool min-ready for ubuntu-trusty and devstack-trusty15:19
AJaegerclarkb: ONe new devstack-trusty sneaked in - do you want to rebase - or fix afterwards?15:20
AJaegerclarkb: +2 - let's see whether somebody else wants to +2 and then we need a volunteer to mass update jenkins...15:21
clarkbAJaeger: we can fix after15:21
clarkbits also only 400 something job updates so may not need manual jjb15:21
AJaegerclarkb: I fear we need it on jenkins02 at least - but let's see15:22
*** placko_ has quit IRC15:22
*** ociuhandu has joined #openstack-infra15:23
openstackgerritMerged openstack-infra/project-config: Reverting Manila LXD driver changes  https://review.openstack.org/29347415:23
fungiclarkb: AJaeger: i'll babysit some manual jjb updates once it merges15:24
fungijust to be sure15:24
*** vgridnev has joined #openstack-infra15:25
fungionce we're down to 0 references to devstack-trusty in jjb we'll also want another zuul restart so that we wipe out the old registrations15:25
*** sdake has joined #openstack-infra15:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Change min-ready for ubuntu-trusty/devstack-trusty  https://review.openstack.org/29350615:26
AJaegerclarkb, fungi, here's the nodepool update ^15:26
AJaegerfungi, we still need somebody to +2A https://review.openstack.org/#/c/29257315:27
jeblairzaro: i can tell you are the author of the change in that screenshot, since you are the first person listed in the history.15:27
AJaegerthanks fungi for babysitting this!15:29
SamYaplemordred: any luck with the clouds?15:30
SamYapleand glance specifically15:30
AJaegerfungi, is https://review.openstack.org/292472 (the python-db change) still on your list of changes to review in a quiet minute?15:30
*** pvaneck has joined #openstack-infra15:31
fungiAJaeger: yep15:31
fungiadding that now15:31
*** nmagnezi has quit IRC15:31
AJaegerthanks15:31
*** zeih has joined #openstack-infra15:32
toskyquestion about code moving between repositories (for example, tests from or to tempest): usually people simply drop the last version of the code in a review15:32
SamYapleso dare i ask for reviews on this patch https://review.openstack.org/#/c/293021/15:32
toskyand then remove the old code from the source repository15:32
*** fawadkhaliq has quit IRC15:33
*** ianychoi3 has joined #openstack-infra15:33
toskywould/did you allow or executed in the past code moving which kept the history? Like pushing a filtered subset of commits which affected the moved code15:33
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Use m1.small for ping test  https://review.openstack.org/29351115:34
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Boot from volume in ping test  https://review.openstack.org/29351215:34
*** fawadkhaliq has joined #openstack-infra15:34
*** david-lyle_ has joined #openstack-infra15:34
fungiAJaeger: what do you think about splitting the jjb and zuul parts of 292472 into separate changes? i'm worried that we'll have a bunch of nova devs coming in here asking why all their jobs are coming back NOT_REGISTERED15:34
AJaegerSamYaple: ?>! - really strange...15:34
AJaegerfungi: sure, let me do that...15:34
*** david-lyle has quit IRC15:34
fungiAJaeger: though i guess that means double-instantiating some jobs if you keep the job groups so that may not be possible15:35
SamYapleAJaeger: whats strange?15:35
SamYapleyou mean ?<!15:35
AJaegerSamYaple: I learned something new ;)15:35
fungiAJaeger: we could instead shoot for applying it at a low-impact time15:35
SamYaplethats teh look behind operator15:35
SamYaplewell thats the dont match if exists behind operator (thanks to !)15:36
fungiAJaeger: also, should probably get a +1 from a nova liaison or ptl since their job names are changing (they might have tools tracking/trending some results by job name or something)15:36
clarkbtosky you can always push and approve a stack of changes. If your goal is to push a git filtered tree into the repo bypassing review we tend to avoid that15:36
*** ianychoi2 has quit IRC15:36
*** jaosorior has quit IRC15:36
prometheanfireAJaeger: I think ubuntu-trusty isn't so happy here https://review.openstack.org/#/c/293193/15:37
prometheanfireAJaeger: it was passing before that patchest15:37
toskyclarkb: I understand the reason for that (not bypassing review); the idea would be to have the review containing the last version of the filtered git tree, test it, but then push the set15:37
toskyclarkb: but I understand that it would be non-standard15:37
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Add javascript jobs to fuel-ui project  https://review.openstack.org/29276015:38
*** fawadkhaliq has quit IRC15:38
prometheanfireAJaeger: it was patchset 9->10 that started failing, the only change in that interdiff was ubuntu-trusty15:39
fungitosky: also, splitting repo histories is fairly trivial, but grafting different repos together when they lack a common history is not15:39
prometheanfirepatchest 11 was a commit-message change15:39
openstackgerritMerged openstack-infra/project-config: Switch rest of devstack-trusty to ubuntu-trusty  https://review.openstack.org/29257315:39
toskyfungi: I know, we did it sometime in another project (kde.org)15:39
toskythere are (very) few repos with two "heads" joining at some point15:40
*** jroll has quit IRC15:41
*** jroll has joined #openstack-infra15:41
*** jroll has quit IRC15:41
*** jroll has joined #openstack-infra15:41
AJaegerprometheanfire: red the log file, it says " gate-dib-dsvm-functests-devstack-trusty not defined"15:42
AJaegerYou renamed the job to  gate-dib-dsvm-functests-ubuntu-trusty, so fix it in layout.yaml15:42
*** bpokorny has joined #openstack-infra15:42
prometheanfireah15:42
prometheanfirethanks15:43
*** adrian_otto has joined #openstack-infra15:45
clarkbin theory you can push all the changes to a brach with a parent of 0 then merge them into master15:45
mordredSamYaple: I have not tried them yet15:45
clarkband thats doable without infra doing anything I think15:45
clarkbwell maybe an acle update for the merge commit push15:46
toskyclarkb: I was going to ask about that point, yes :)15:48
toskyclarkb: so, right now, no one can do merge commits?15:48
clarkbtosky: no, many people can just not by default15:49
clarkbit requires a special permission be added to the project acl15:49
fungimainly because the number of times people accidentally create a merge commit and then try to push that in gerrit far outnumbers the number of times people legitimately need to push a merge commit into gerrit15:50
toskythinking about it, putting the changes in a separate branch means no use of gates, as the code will be different before the merging operation15:50
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281415:50
fungimuch like how we restrict the ability of most users to push tags into gerrit15:51
fungitosky: well, jobs still run on the merge commit and it gets reviewed like any other commit, though gerrit's display of merge commits is a bit... strange15:51
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially  https://review.openstack.org/29247215:51
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs  https://review.openstack.org/29352615:51
AJaegerfungi, updated ^15:51
AJaegerjohnthetubaguy, sdague, could you review the two changes above, please? ^15:52
fungiAJaeger: awesome. and we should get nova feedback on the second one15:52
fungiAJaeger: hah, you just asked for it--thanks!15:52
toskyfungi: oh, I see, so the ACL for merge should set up before pushing the review for the merge commit - got it15:52
AJaegerjohnthetubaguy, sdague: This renames gate-nova-python27 to gate-nova-python27-db etc. So, that in a followup we can remove database setup from the normal python27 jobs (similar for python34 etc)15:52
*** apoorvad has joined #openstack-infra15:52
toskythanks for the hints - I'm going to see if I can make it working, and in case I would kindly ping you for the ACL15:53
fungitosky: yep, it's generally restricted to just the people who will regularly need to push merge commits for a particular project, or even just on a temporary basis15:53
AJaegerfungi, clarkb, could you review the nodepool update, please?  https://review.openstack.org/29350615:53
fungiAJaeger: i will once i get these jjb updates running15:54
AJaegerthanks, fungi15:54
*** gyee has joined #openstack-infra15:55
AJaegersdague, johnthetubaguy I'll be offline for the next hour or two. If you have any questions about 293526 and 292472, please ask and I'll answer later - or discuss with fungi.15:55
*** tosky has quit IRC15:57
AJaegerfungi, pinged them on #openstack-nova as well...15:58
fungiyep, totally happy to discuss that change with them as needed15:58
*** maishsk has quit IRC15:59
*** cdent has joined #openstack-infra15:59
*** tosky has joined #openstack-infra16:00
cdentper request, I'm showing up to say "if there's desire to get purplerbot on infrastructure, I'm happy to push" but also to say "I'm fine leaving it run on my own stuff"16:00
cdentIt will need some work to be correctly deployable (some static paths at the moment) but that's easy enough16:01
*** e0ne has joined #openstack-infra16:01
*** jpr has quit IRC16:01
mriedemAJaeger: why do we need a database setup for pep8/docs jobs?16:01
AJaegermriedem: only for python27 jobs, not for pep8/docs16:02
mriedemAJaeger: well, on https://review.openstack.org/#/c/292472/6/zuul/layout.yaml16:02
mriedempep8 and docs are under python-db-jobs16:02
AJaegermriedem: we change the template from python-jobs which includes docs and pep to python-db-jobs16:02
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Create a test flavor for the pingtest VM  https://review.openstack.org/29353316:02
AJaegermriedem: scroll up, the few liens above16:03
mriedemAJaeger: but they don't use a db?16:03
*** sridhar_ram has joined #openstack-infra16:03
ddieterlyneed some help with https://github.com/openstack-infra/devstack-gate16:03
AJaegermriedem: The job names are not changed, so those will not use a database16:03
AJaegerbut I don't see a better way of doing it.16:03
mriedemoh i see16:03
ddieterlywhen i checkout my changes to workspace-cache, the changes don't ever get copied into /opt/stack/new16:03
ddieterlythe changes just get deleted16:03
AJaegermriedem: but if you have ideas for better naming to not confuse people - and you have been confused - please bring them up.16:04
*** fawadkhaliq has joined #openstack-infra16:04
*** matrohon has quit IRC16:05
cdentbiab16:05
*** cdent has quit IRC16:05
AJaegermriedem: let me update the comments in zuul/layout16:05
Zaracorrected storyboard sender email patch in review: https://review.openstack.org/#/c/293367/ ; *this* one should be the last one, argh.16:06
*** vgridnev has quit IRC16:06
*** yamamoto has joined #openstack-infra16:07
mriedemAJaeger: clarification in the comments would be helpful16:07
mriedemi added some more16:07
jeblairmordred: in the _create method in https://review.openstack.org/#/c/226751/60/nodepool/fakeprovider.py why are you setting public_v6 but not public_v4?16:07
*** harlowja_at_home has joined #openstack-infra16:07
mordredjeblair: looking16:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially  https://review.openstack.org/29247216:09
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs  https://review.openstack.org/29352616:09
fungizaro: does that plugin also mask comment authors on inline comments? it's an interesting feature i might use if we had it (though i'd be more likely to benefit from an implementation in gertty than the webui)16:10
*** IlyaG has joined #openstack-infra16:10
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs  https://review.openstack.org/28944516:10
AJaegermriedem: I added comments and TODOs, better now?16:10
openstackgerritIgor Belikov proposed openstack-infra/system-config: Regenerate nodepool.logging config  https://review.openstack.org/29339616:10
mordredjeblair: I cannot come up with a good reason - it seems like a mistake to do so16:10
jeblairmordred: those used to be something set later by get_openstack_vars(), but that seems to have been dropped...16:10
mordredyah. because you get them filled out by default now16:11
jeblairmordred: does shade ...ok yeah, so shade does that for all created servers?16:11
jeblairand all servers you "get'?16:11
AJaegerwill be back online later...16:11
mordredyah. it does all the things it can do without additional API calls all the time16:11
jeblaircool16:11
*** tzn has joined #openstack-infra16:11
mugsiehey - is there a way for me to take ownership of a group in gerrit? I cannot add people to designate-stable-maint16:12
*** cloudtrainme has quit IRC16:12
fungimugsie: i believe the *-stable-maint groups in projects under official release management are owned by the stable-maint-core team16:12
mugsiethe stable-maint team don't look after designate16:13
fungimugsie: the idea was that changes to those stable subteams would be vetted by the main stable team for openstack16:13
*** nwkarsten has quit IRC16:13
fungimugsie: so that they could confirm the reviewers being added were familiar with stable maintenance guidelines16:13
mriedemAJaeger: yeah16:13
*** nwkarsten has joined #openstack-infra16:14
mugsieok. so #openstack-stable is my best bet then?16:14
fungimugsie: it's worth discussing with the current and/or future stable maint ptl on whether they want to change teh process too, yes16:14
*** harlowja_at_home has quit IRC16:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Barbican & Magnum roles to OpenStack-Ansible  https://review.openstack.org/29354016:15
*** yamamoto has quit IRC16:16
*** nwkarsten has quit IRC16:18
*** Swami has quit IRC16:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Barbican & Magnum roles to OpenStack-Ansible  https://review.openstack.org/29354016:19
pabelangerand back online16:19
*** tphummel has quit IRC16:20
*** david-lyle_ is now known as david-lyle16:20
*** esikachev has quit IRC16:21
pabelangerAJaeger: are we ready for 293506 to land? Didn't know we did the devstack-trusty migration already16:21
*** cloudtrainme has joined #openstack-infra16:21
*** jpr has joined #openstack-infra16:22
pabelangerjust found the commit :)16:22
*** IlyaG has quit IRC16:22
*** harlowja_at_home has joined #openstack-infra16:22
*** IlyaG has joined #openstack-infra16:23
pabelangerfungi: actually if you are good with 293506 I can help monitor it landing16:24
pabelangerthen see what is needed to clean up devstack-trusty images16:24
fungipabelanger: well, i wanted the job updates to finish first. i'm running/monitoring those still16:25
pabelangerfungi: Right, that's what I am looking at too16:26
fungithe telltale sign will be once we see the devstack-trusty used count fall to 016:26
pabelangerand where do you monitor that?16:26
fungiat that point, switching the min-ready to 0 seems fine to me16:26
pabelangergraphite.o.o?16:26
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Fix two grafana entries to point at the right zuul pipeline  https://review.openstack.org/29354416:26
fungipabelanger: that's one place to do it. or spot checks of nodepool list16:27
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the all CI jobs  https://review.openstack.org/29348116:27
pabelangerfungi: great16:27
*** sbelous_ has quit IRC16:27
pabelangerwe should setup grafana too :)16:27
fungiyeah, node states by label would be an interesting set of graphs16:28
*** nwkarsten has joined #openstack-infra16:28
*** yuval has joined #openstack-infra16:28
fungii wouldn't put all the node labels on one graph because ubuntu-trusty is going to dominate by orders of magnitude16:29
fungiso it would be pretty hard to read16:29
pabelangerya, we could add a template var to the nodepool dashboard, allowing you to limit the singleton panel by node label16:30
pabelangercould be the easiest16:30
fungibut a graph per node label, trending each of the states for that label, would be useful16:30
clarkboh good east is in houston now, I suppose its just a matter of time for them to give it power and networking16:30
*** ramishra has quit IRC16:30
pabelangernice16:30
fungiclarkb: except we wanted west because the hardware was better16:30
*** cdent has joined #openstack-infra16:30
clarkbfungi: ya....16:30
fungiand that still hasn't left the hangar16:30
clarkbbut at least we have some progress16:30
clarkbI really want to be able to track down the io performance issues16:31
clarkband that requires having clouds of some sort16:31
*** ramishra has joined #openstack-infra16:32
fungii thought the i/o performance issues boiled down to the lack of bbwc cards for the raid controllers16:32
*** fawadkhaliq has quit IRC16:32
*** fawadkhaliq has joined #openstack-infra16:33
*** fawadkhaliq has quit IRC16:33
openstackgerritClark Boylan proposed openstack-infra/system-config: Apt mirror for ceph hammer release  https://review.openstack.org/29315516:34
openstackgerritClark Boylan proposed openstack-infra/system-config: Make reprepro useable on multiple mirrors  https://review.openstack.org/29315416:34
clarkbfungi: aiui not really. bbwc cards allow you to more safely reduce the caching in kvm which is conservative by default16:34
*** yuval has quit IRC16:35
clarkbfungi: the theory was making kvm pretend writes were complete before they were would give us the erformance we wanted but we never got to test that16:35
clarkband that bbwc cards would protect our VMs with that setting in place16:35
clarkbmordred: ^ the above reprepro changes are a thing you may be interested in16:35
mordredlooking16:36
smarcetfungi: i am waiting for a new deployment on openstackid-dev, but its been a while that i am looking PENDING STATUS16:36
smarceti am not seeing any error on puppet board16:36
*** Swami has joined #openstack-infra16:37
fungismarcet: deploy status says "INSTALLED [16-Mar-2016 15:24]"16:38
*** links has quit IRC16:38
fungiin the log i find: Mar 16 15:54:55 openstackid-dev puppet-user[22721]: (/Stage[main]/Openstackid/Exec[update-site]/returns) executed successfully16:39
*** tphummel has joined #openstack-infra16:39
fungismarcet: you were waiting for https://review.openstack.org/293482 to deploy presumably?16:40
clarkbthat was long enough ago that puppetboard should know about it16:40
fungimerged at 15:16 utc16:41
clarkbhttp://puppetboard.openstack.org/report/openstackid-dev.openstack.org/c66b8649704781c0466dc390adfb11e7fe302cd3 appers to be the run that ran that exec16:41
clarkbjust doesn't tell you it did so which may be a bug in the ansible-puppet puppetdb support16:41
smarcetfungi: yeah i am waiting for it16:41
*** matrohon has joined #openstack-infra16:42
*** kzaitsev_mb has quit IRC16:42
fungismarcet: is there an easy way for me to confirm which version of it is deployed?16:42
pabelangerclarkb: so now that we have nodepool-builders in place, I actually thing nodepool dib-image-list should have a queued state that is displayed.  Since building isn't technically correct with 1 builder16:42
*** Daisy has joined #openstack-infra16:42
clarkbpabelanger: its the same with image-list too16:43
*** eil397 has joined #openstack-infra16:43
*** hashar has quit IRC16:43
clarkbpabelanger: nodepools state machine considers them all the same once submitted to gearman though16:43
clarkbanything sent to gearman is "building"16:43
mgagnefungi: so latest centos7 uploaded fine to internap :-/16:44
*** jlanoux has quit IRC16:44
smarcetfungi: not really, i am just checking with the status script16:44
openstackgerritMatthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs  https://review.openstack.org/29319316:44
smarcetversion label is only available for production16:44
fungimgagne: looks like it. most recent two uploads were ~12 days ago and ~16 hours ago16:45
mgagnefungi: looks like trusty failed16:45
*** cloudtrainme has quit IRC16:46
*** tphummel has quit IRC16:46
mgagnefungi: I have much more logs now. checking16:46
fungismarcet: for me the status script is saying it installed something from 15:24 utc, which is about 8 minutes after your change merged16:46
*** e0ne has quit IRC16:47
*** e0ne_ has joined #openstack-infra16:47
smarcetfungi : ok16:47
*** Daisy has quit IRC16:47
fungismarcet: INSTALLED [16-Mar-2016 15:24]16:48
pabelangerzigo: I started https://review.openstack.org/#/c/185680/ a while back.  He was my first attempt at creating some chroots.16:48
pabelangerzigo: it was using puppet to do it, which happened during dib build16:48
*** scheuran has quit IRC16:49
zigopabelanger: Thanks, I'm bookmarking it for later tonight ! :)16:49
*** vgridnev has joined #openstack-infra16:49
zigopabelanger: Is using puppet mandatory? I'm dumb with puppet ...16:49
pabelangerzigo: for you, I would update your JJB template to call your tooling on job launch, get everything working as expected, then deal with optimizing it maybe as a DIB element16:49
*** flwang1 has quit IRC16:49
pabelangerzigo: As long as it is something infra can read, you'll be fine.16:49
zigopabelanger: Then only thing I can write is a shell script to setup sbuild properly, Monty said it was ok to do that.16:50
*** e0ne has joined #openstack-infra16:50
zigopabelanger: What I don't know is how the script will get access to the git repo + the reviewed patch.16:50
fungismarcet: though i agree, for example /opt/openstackid/app/start/global.php does not match the new version in change 29348216:50
zigopabelanger: How does that work?16:50
pabelangerzigo: Yup, should be fine.  Add it as a JJB template to start, then move it to the optimized place16:50
*** fawadkhaliq has joined #openstack-infra16:50
*** e0ne_ has quit IRC16:50
fungismarcet: the last time that file was updated seems to be 5 weeks ago16:51
*** adrian_otto has quit IRC16:51
*** fawadkhaliq has quit IRC16:51
zigopabelanger: What do you mean a JJB template?16:51
zigo(note: I know JJB, I use it and even maintain the package in Debian)16:51
smarcetfungi: yeah looking at server code, disregard sorry its already deployed :)16:51
mgagnefungi: is a new token used for each upload?16:51
*** fawadkhaliq has joined #openstack-infra16:51
fungismarcet: oh, right i keep forgetting /opt/openstackid isn't actually used for deploying16:51
*** vgridnev has quit IRC16:52
mgagnefungi: or is a single token fetched and used to upload all images?16:52
fungismarcet: i was looking for the version in /srv/openstackid/slot016:52
zigopabelanger: What I wont know is where to hook in project-config to get my script started...16:52
*** tonytan_brb has quit IRC16:52
clarkbfungi: those ubuntu-trusty updates are happening?16:52
*** vincentll has quit IRC16:52
zigopabelanger: The script, I know how to write it (I already have it... just need to adapt it a bit)16:52
clarkbstarting to see some of those jobs run on ubuntu-trusty now fwiw16:52
fungimgagne: i don't recall whether we refresh the token between uploads but clarkb or greghaynes may16:52
fungiclarkb: still underway16:52
*** tonytan4ever has joined #openstack-infra16:53
pabelangerzigo: you'll need to create a new jenkins job, so jenkins/jobs/debian-packaging.yaml (for example)16:53
clarkbfungi: mgagne: I believe that shade + glance should use keystoneauth sessions and refresh the token if necessary16:53
pabelangerzigo: you likely have a noop job for your project ATM16:53
clarkbfungi: mgagne but that is something that should be double checked16:53
mgagneclarkb: when?16:53
*** e0ne has quit IRC16:53
mgagneclarkb: before I suspect the token expires before the upload completes16:53
mgagnebecause*16:53
clarkbmgagne: prior to ever request keystoneauth should check the token expiry and if it is soon it is supposed to get a new one first16:53
clarkbI do not know what happens if the token expires during a very long request16:54
mgagneclarkb: if an image upload operation is done, I suggest to always renew it16:54
mgagneclarkb: glance crashes because it tries to contact registry and token is expired16:54
*** tphummel has joined #openstack-infra16:54
clarkbmgagne: is it erroring on the next request?16:54
clarkbbecause that should get a new token with keystoneauth if that is all working16:55
mgagneclarkb: yes16:55
*** nwkarsten has quit IRC16:55
mgagneclarkb: it's not glean that crashes16:55
mgagneclarkb: it's glance-api itself16:55
*** FallenPegasus has joined #openstack-infra16:55
*** kzaitsev_mb has joined #openstack-infra16:55
*** nwkarsten has joined #openstack-infra16:55
clarkbmgagne: so it is happening server side?16:56
mgagneyes16:56
clarkbhrm ok16:56
fungiugh. jjb on our jenkins masters is no longer able to update its cache16:56
fungino, wait, that's a lockfile16:56
fungidid jjb recently grow a (broken) lockfile feature?16:57
clarkbI think yolanda proposed a thing not sure if it merged16:57
fungii need to go digging in the source, but getting "ValueError: I/O operation on closed file"16:57
*** vgridnev has joined #openstack-infra16:58
*** tonytan_brb has joined #openstack-infra16:58
fungiyep, "Allow using lockfile per jenkins master" merged today (utc)16:58
*** tonytan4ever has quit IRC16:58
fungihttps://review.openstack.org/29240516:59
fungiyolanda: zxiiro: electrofelix: ^16:59
fungii'll get a full traceback up momentarily16:59
mgagnehttps://bugs.launchpad.net/glance/+bug/1182536 exists but there is an OSSN associated to it https://wiki.openstack.org/wiki/OSSN/OSSN-006017:00
*** maishsk has joined #openstack-infra17:00
*** maishsk has quit IRC17:00
openstackLaunchpad bug 1182536 in Glance "Glance api fails to set image to active after upload when user token expires or is revoked." [Medium,Fix released] - Assigned to Alex Meade (alex-meade)17:00
openstackgerritsebastian marcet proposed openstack-infra/system-config: Openstackid Production Release 1.0.14  https://review.openstack.org/29357317:00
*** IlyaG has quit IRC17:00
*** e0ne has joined #openstack-infra17:00
*** FallenPegasus is now known as MarkAtwood17:00
*** nwkarsten has quit IRC17:00
*** piet has joined #openstack-infra17:01
*** fhubik has quit IRC17:01
*** IlyaG has joined #openstack-infra17:01
*** HeOS has quit IRC17:01
*** HeOS has joined #openstack-infra17:02
*** sigmavirus24 is now known as sigmavirus24_awa17:02
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NodeJS and NPM to single-use-slave  https://review.openstack.org/24216117:02
smarcetfungi: yolanda: when u get a chance please review https://review.openstack.org/#/c/293573 :)17:03
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NodeJS and NPM to single-use-slave  https://review.openstack.org/24216117:03
*** ifarkas has quit IRC17:03
openstackgerritMerged openstack-infra/shade: Add wait_for_server API call.  https://review.openstack.org/29263117:03
openstackgerritZara proposed openstack-infra/storyboard-webclient: Fix setting new task status and priority  https://review.openstack.org/29357717:05
clarkbfungi: that change should probably use pylockfile17:05
fungiyolanda: zxiiro: electrofelix: does http://paste.openstack.org/show/490758/ look like a reentrancy issue? are there multiple threads that might be calling the _unlock() method?17:06
*** ddieterly is now known as ddieterly[away]17:06
fungiclarkb: agreed, it does some wheel reinventing17:07
*** _nadya_ has quit IRC17:07
clarkbit also reopens the file over and over every second without properly closing it17:07
*** Nepoc has joined #openstack-infra17:07
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Removed npm-install macro  https://review.openstack.org/24219317:08
mgagnefungi: looks like multiple images are also uploaded at the same time. will check on that side17:08
clarkbhoneslty I think a better way of handling that entire thing is to write the cache to a tmp file then mv it in place when done17:08
fungiimmediately before the traceback, there is another "INFO:jenkins_jobs.builder:Cache saved" message so the jjb cache is being written more than once i think17:08
clarkbthe races between processes will still exist but be tiny17:08
kro_snifflesWell, _that_ was just plain wrong17:09
pabelangerI thought ansible was just doing to use flock for JJB17:09
*** infra-red has joined #openstack-infra17:09
NepocHello, I'm setting up a mirror of git.openstack.org for openstack-ansible and I was hoping someone could help me replicate the server settings you have so git clone/checkout work as expected with the version/commit hash in the url17:09
* kro_sniffles is talking to himself.17:09
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu  https://review.openstack.org/29301217:09
pabelangerNepoc: we use puppet-cgit have you looked at it?17:09
Nepocpabelanger: I have not17:10
fungiNepoc: our git.openstack.org mirror servers are passive mirrors, not active mirrors. our gerrit server pushes into them continually17:10
pabelangerNepoc: so, you'll be able to look at system-config, which will then call puppet-cgit17:11
pabelangerNepoc: all under the openstack-infra namespace17:11
fungikro_sniffles: i hope you get over your sniffles (and now whatever psychopathy has you muttering to yourself) soon!17:11
Nepocpabelanger: I don't need the web front end that cgit provides. I only need the git interactions to work.17:12
*** aspiers has quit IRC17:12
kro_snifflesfungi: THanks :)17:12
kro_snifflesfungi: The latter ls less likely17:12
*** mikelk has quit IRC17:12
*** jistr has quit IRC17:12
pabelangerNepoc: like fungi says, we use gerrit as our upstream git repos, which are replicated to git.o.o and github.com17:13
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Removed npm-install macro  https://review.openstack.org/24219317:13
pabelangerNepoc: but all the configuration will be in system-config, if you are up for digging into it17:13
fungiNepoc: pabelanger: so using our puppet-cgit module alone won't actually get you complete mirrors of our repositories. you'll still need to design something to do the active mirror updating piece17:13
pabelangerYa, that is true17:13
kro_snifflesThere, that seems a little more sane.17:13
fungiand puppet-cgit is mostly just going to get you a web frontend to browsing git repositories17:14
*** kzaitsev_mb has quit IRC17:14
openstackgerritThomas Maddox proposed openstack-infra/project-config: testing with fedora-23  https://review.openstack.org/29358517:14
Nepocpabelanger: fungi: I have the mirroring "done" just need to make sure I'm serving it the same way.17:14
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Added mising link to config file  https://review.openstack.org/29358617:14
Nepocwithout the web frontend17:14
fungithough i think we do have the vhost stuff on there set up for doing smart git http backend cgi as well17:14
fungiNepoc: so, yeah, maybe you just want to crib our apache configuration for the serving part (and ignore the parts of that vhost config for cgit's webui)17:15
Nepocsounds good, that's under system-config?17:16
fungiNepoc: https://git.openstack.org/cgit/openstack-infra/puppet-cgit/tree/17:16
Nepocfungi: thank you17:16
fungiNepoc: specifically https://git.openstack.org/cgit/openstack-infra/puppet-cgit/tree/templates/git.vhost.erb17:16
Nepocfungi: excellent already looking at it17:17
*** aspiers has joined #openstack-infra17:17
*** ddieterly[away] is now known as ddieterly17:18
*** flepied has quit IRC17:18
fungiwe've also had some requests for setting up grokmirror support on our mirrors, to make it easier for others to mirror from them17:18
fungithough i don't recall if anyone has started working on that17:19
*** dtantsur is now known as dtantsur|afk17:20
*** sridhar_ram1 has joined #openstack-infra17:21
*** sridhar_ram has quit IRC17:22
*** piet has quit IRC17:22
*** nwkarsten has joined #openstack-infra17:23
clarkbas an alternative you could watch the event stream for replication events and do pulls that way17:23
clarkbprobably more work than just adding grokmirror to puppet17:23
fungifor now i'm just going to roll back jjb to pre-292405 and rerun to make sure everything's succeeding17:23
*** smarcet has quit IRC17:24
clarkbfungi: sounds good, is there a revert change yet?17:24
clarkbI can push taht through but didn't catch it going by in here if so17:24
funginot to my knowledge. didn't know whether the author and approvers from that change had a simple fix in mind17:24
fungibut go for it17:24
*** tphummel has quit IRC17:25
*** tphummel has joined #openstack-infra17:26
*** cloudtrainme has joined #openstack-infra17:26
clarkbhttps://review.openstack.org/29359817:27
fungiis openstackgerrit on vacation again?17:27
*** tzn has quit IRC17:27
clarkbapparently17:27
*** schipiga has quit IRC17:28
*** zeih has quit IRC17:30
*** e0ne has quit IRC17:30
*** IlyaG has quit IRC17:30
*** IlyaG has joined #openstack-infra17:31
pabelangerclarkb: +217:32
*** lucasagomes is now known as lucas-afk17:33
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Restore ability to run nodepoold with fakes  https://review.openstack.org/23112017:34
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Make stopping more reliable  https://review.openstack.org/29360517:34
fungiand now it's back17:34
*** IlyaG has quit IRC17:34
fungimaybe it just doesn't like you clarkb17:34
*** SumitNaiksatam has quit IRC17:35
fungiconfirmed, rolling jjb back to the commit immediately prior to that one and it doesn't throw a traceback17:35
pabelangerfungi: clarkb: so, are we looking at standing up a 2nd nodepool-builder instance?17:36
fungipabelanger: possibly, though i don't know that it's in the immediate future17:36
clarkbpabelanger: I haven't been. A bigger benefit may be to have more upload threads based on jeblairs testing during the sprint17:37
*** armax has quit IRC17:37
jeblairyeah, i think we found we could support ~16?17:37
clarkbjeblair: ya was 4 total workes iirc which is 4x417:37
*** sfinucan has quit IRC17:38
*** johnthetubaguy_ has joined #openstack-infra17:39
jeblaironce shade lands, i'd like to push hard on landing those 2 patches above.  we've really let some things drift to the point where you can't really even reliably run nodepool manually for testing (fakes or no fakes in some cases).17:39
pabelangerAh ya. I remember that17:39
jeblair(they are built on the shade patch)17:39
jeblair(actually, we might want to land them concurrently with the shade patch and save us some potential headache)17:39
ddieterly    #TODO(jeblair): remove when this is no longer created by the image17:40
ddieterly    rm -fr ~/workspace-cache/17:40
jeblairddieterly: could have been removed years ago :)17:40
ddieterlyjeblair why is that in functions.sh? it is causing the instructions at https://github.com/openstack-infra/devstack-gate to fail17:40
ddieterlycan we safely remove it?17:41
*** johnthetubaguy has quit IRC17:41
*** johnthetubaguy_ is now known as johnthetubaguy17:41
jeblairddieterly: yes.  however, i'm not sure how well those instructions will work even after doing so.17:42
ddieterlyomg, that is very disheartening17:42
jeblairddieterly: are you trying to hack on devstack-gate itself, or reproduce a test run?17:42
ddieterlyhow are people testing gate jobs and tempest tests before submitting reviews then?17:42
ddieterlyjeblair produce a test run17:43
kro_snifflesjeblair: ianw has a patch up for puppet-openafs that uses our own RPM's for Centos7 instead of relying on the official ones (which don't exist). Do you have the time to look? https://review.openstack.org/#/c/290266/17:43
*** roxanagh_ has quit IRC17:43
pabelangerclarkb: fungi I think well need a nodepool-builder restart to pickup the debian-jessie dib.17:43
jeblairddieterly: if zuul has already run a job for a change, there is a script called "reproduce.sh" in the logs that you can use.  it's a lot easier than the process in the readme.17:43
ddieterlyi'd like to make changes to the repo under tests and the gate job to see if things are working17:44
*** _nadya_ has joined #openstack-infra17:45
jeblairkro_sniffles: (re)reviewed!17:45
kro_snifflesjeblair: Thanks :D17:45
*** dizquierdo has quit IRC17:45
* kro_sniffles waits for that to land before testing to see if we can do centos7 thing with wheel mirrors.17:45
*** jpr has quit IRC17:45
jeblairddieterly: i'd try reproduce.sh first17:46
*** sdague has quit IRC17:46
ddieterlyjeblair ok, thanks17:46
*** ddieterly is now known as ddieterly[away]17:47
pabelangeractually, if some infra-root can help land 293396 we can have updated logging on nodepool builders17:49
*** fawadkhaliq has quit IRC17:49
pabelangerbefore we restart17:49
*** derekh has quit IRC17:50
*** fawadkhaliq has joined #openstack-infra17:50
jeblairpabelanger: approved (i assume you ran the script?)17:51
pabelangerjeblair: yup17:52
openstackgerritMerged openstack-infra/puppet-openafs: Use CI built RPMs for CentOS7  https://review.openstack.org/29026617:52
*** abregman has quit IRC17:52
*** sdague has joined #openstack-infra17:52
*** rcrit has joined #openstack-infra17:52
kro_sniffleswoot17:53
clarkbwith ^ merged is the last piece booting a new centos7 builder host?17:53
*** rcernin has quit IRC17:53
clarkbs/last/next/17:53
fungipabelanger: yeah, in my review comment on that change i mentioned the need for a nodepool builder process restart17:53
*** salv-orl_ has joined #openstack-infra17:54
*** rossella_s has joined #openstack-infra17:54
pabelangerfungi: okay, once puppet deploys the new logging config, I'll restart17:54
pabelangerfungi: also, down to 39 devstack-trusty nodes now17:54
clarkbpabelanger: just check that there aren't any running builds17:54
pabelangerclarkb: Yup, they all finished for the day17:54
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests  https://review.openstack.org/28884017:54
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Reconcile compute and controller classes  https://review.openstack.org/29362017:54
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Parameterize libvirt virt type  https://review.openstack.org/29362117:54
rcritI'm trying to run tox in project-config and it fails because it can't install pbr. I also see: Download error on http://pypi.dfw.openstack.org/simple/pbr/: [Errno 101] Network is unreachable -- Some packages may not be found!17:54
rcritI don't know where it is getting this hostname17:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially  https://review.openstack.org/29247217:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs  https://review.openstack.org/29352617:56
*** salv-orlando has quit IRC17:56
*** flepied has joined #openstack-infra17:57
*** tphummel has quit IRC17:58
fungircrit: presumably you have a ~/.pip.conf or something setting that17:58
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the all CI jobs  https://review.openstack.org/29348117:58
openstackgerritPaul Belanger proposed openstack-infra/project-config: Convert last devstack-trusty to ubuntu-trusty  https://review.openstack.org/29362417:58
pabelangerAJaeger: fungi: looks like we missed one ^17:58
fungircrit: that server definitely doesn't exist any longer17:58
*** tphummel has joined #openstack-infra17:59
rcritI have a .pip directory but it is empty17:59
fungipabelanger: right, AJaeger mentioned there was a straggler which merged after clarkb wrote the big change17:59
openstackgerritMerged openstack-infra/system-config: Regenerate nodepool.logging config  https://review.openstack.org/29339617:59
fungircrit: /etc/pip maybe?17:59
rcritor ~/.pydistutils.cfg maybe?17:59
rcritit is in there17:59
fungircrit: that may also do it since you're seeing it complain about pbr and so it may be easy_install coming into the picture18:00
rcritI see it listed in /etc/pip.conf as a trusted host too18:00
rcritok, let me smack both of those and see where that gets me, thanks fungi18:00
openstackgerritClark Boylan proposed openstack-infra/project-config: Cleanup a last devstack-trusty straggler  https://review.openstack.org/29362718:01
clarkbfungi: pabelanger ^18:01
fungiyeah, you don't need any of that. we set that up on our workers but the mirrors are just for use within our ci and we reserve the right to rename/replace/remove them without notice18:01
clarkboh wait pabelanger wrote that change ugh18:01
* clarkb abandons18:01
pabelangerha18:01
pabelangerI just noticed it since nodepool launched a new devstack-trusty18:01
*** _nadya_ has quit IRC18:02
pabelangerdown to 31 used ATM18:02
*** smarcet has joined #openstack-infra18:02
openstackgerritsebastian marcet proposed openstack-infra/system-config: Openstackid Production Release 1.0.14  https://review.openstack.org/29357318:02
electrofelixfungi: quick fix would be to set 'self.lockfile = None' after the lockfile is closed similarly set self.data = None before self._unlock() gets called and we'll think about how to ensure it's only called once better subsequently18:03
*** baoli has quit IRC18:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Publishing barbican API Guide to OpenStack site  https://review.openstack.org/28551918:04
clarkbelectrofelix: I really think we want to avoid a quick fix and use pylockfile18:04
clarkbelectrofelix: since that fixes all of these problems18:04
*** ihrachys has quit IRC18:04
*** tqtran-afk has joined #openstack-infra18:05
fungilockfiles are easy to get wrong, and when you try to write lockfile implementations from scratch you get to relearn all the lessons learned by those who wrote them before you18:06
*** SumitNaiksatam has joined #openstack-infra18:06
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Added mising link to config file  https://review.openstack.org/29358618:06
clarkbhttp://docs.openstack.org/developer/pylockfile/18:06
electrofelixclarkb: that's assuming pylockfile doesn't cause problems when you call unlock on it twice either.18:06
clarkbelectrofelix: well you don't call unlock on it twice18:06
clarkbor rather don't have unmatched unlocks18:07
electrofelixactually, that's exactly what the error from above indicates, unlock followed by close, and subsequently calling unlock against, except this time the file was already closed.18:07
clarkbelectrofelix: which means that something that doesn't hold the lock is unlocking18:07
*** sigmavirus24_awa is now known as sigmavirus2418:07
*** tphummel has quit IRC18:08
clarkbthere should always be matched locks and unlocks18:08
electrofelixit means the destructor is triggering a second unlock because save is getting called a second time by accident.18:08
electrofelixon the unlock should be moved to the destructor18:09
*** tphummel has joined #openstack-infra18:09
electrofelixso we can call save multiple times within the same process without unlocking the file until we're completely finished with it18:09
*** tqtran-afk has quit IRC18:10
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Openstackid Production Release 1.0.14  https://review.openstack.org/29357318:10
*** kzaitsev_mb has joined #openstack-infra18:10
fungipabelanger: smarcet provided a link to the change notes (from the tag's message) in the commit message ^18:11
*** rbrndt has quit IRC18:11
*** _nadya_ has joined #openstack-infra18:13
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box'  https://review.openstack.org/28613718:15
pabelangerfungi: smarcet: many thanks18:15
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Unlock when finished not when saving current cache state  https://review.openstack.org/29363118:16
clarkbbecause I have now discovered that openstack has deprecated pylockfile in favor of oslo.concurrency I think http://docs.openstack.org/developer/oslo.concurrency/usage.html could really use some love18:18
*** tesseract- has quit IRC18:19
clarkbcompare to http://docs.openstack.org/developer/pylockfile/#examples18:19
clarkbharlowja_at_home: dims ^18:19
harlowja_at_homesup18:19
dimsclarkb : i was just about to point harlowja_at_home to your comment :)18:19
harlowja_at_homewellll pylockfile isn't replaced by oslo.concurrency18:19
harlowja_at_homeimho18:19
clarkbharlowja_at_home: ok then we should update the warning on pylockfile :)18:20
harlowja_at_homeah, perhaps18:20
clarkbhttp://docs.openstack.org/developer/pylockfile/#module-lockfile18:20
harlowja_at_homehttps://fasteners.readthedocs.org/en/latest/18:20
harlowja_at_homemore of its direct replacement, or at least a piece of it18:20
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Unlock when finished not when saving current cache state  https://review.openstack.org/29363118:21
harlowja_at_home'It is highly preferred that instead of using this code base that instead fasteners or oslo.concurrency is used instead.'18:21
harlowja_at_homeseems like both already referenced18:21
clarkbyes so delete oslo.concurrency from that line?18:21
electrofelixfungi: https://review.openstack.org/293631 will take care of the immediate issue, we can discuss which locking library to use subsequently, but this will make sure the unlock is called at the correct point which was the problem before.18:21
openstackgerritSean M. Collins proposed openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes  https://review.openstack.org/29363718:21
harlowja_at_homeclarkb, i'd be ok with that18:21
*** ddieterly[away] is now known as ddieterly18:21
fungielectrofelix: thanks for the quick fix18:22
*** tqtran-afk has joined #openstack-infra18:22
*** sputnik13 has joined #openstack-infra18:22
*** mrmartin has joined #openstack-infra18:22
*** degorenko is now known as _degorenko|afk18:23
pvaneckHey puppet/infra-core. If available, this could use a review: https://review.openstack.org/#/c/281737 - This is to make refstack update via releases on pypi. Thanks!18:24
*** akshai has quit IRC18:25
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box'  https://review.openstack.org/28613718:25
*** nwkarsten has quit IRC18:26
openstackgerritSean M. Collins proposed openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes  https://review.openstack.org/29363718:26
*** nwkarsten has joined #openstack-infra18:27
pabelangerI guess we need more HDD for nodepool.o.o18:27
pabelanger2016-03-16 18:26:13,021 INFO nodepool.image.build.debian-jessie: fatal: failed to copy file to '/opt/dib_tmp/image.WtfS1aED/mnt/opt/git/openstack/heat-specs/.git/objects/e8/593c8c47d67a31106f6d1a6187068b723aa3b5': No space left on device18:27
clarkbpabelanger: there are some thing sI can clean out of /opt. I was testing the image compression which is a few gigabytes of unnecessary disk usage18:27
clarkba little annoying that we are that close though we can delete devstack-trusty soon and free that up too18:28
pabelangerclarkb: ya, devstack-trusty will help for sure18:28
pabelangerdown to 25 used nodes now18:28
clarkbpabelanger: /dev/mapper/main-opt  504G  352G  153G  70% /opt I didn't free 153GB of disk fwiw18:29
clarkbI got ~30GB18:29
openstackgerritSean M. Collins proposed openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes  https://review.openstack.org/29363718:29
clarkbonly 4% inodes used so not sure what is going on there18:30
pabelangerYa, I noticed that too18:30
pabelangerlet me kick off debian-jessie again18:30
*** nwkarsten has quit IRC18:31
AJaegerfungi, infra-root, do you think we can remove bare-trusty now from nodepool? Or should we wait some more days?18:33
*** akshai has joined #openstack-infra18:33
openstackgerritMerged openstack-infra/system-config: Openstackid Production Release 1.0.14  https://review.openstack.org/29357318:33
fungiAJaeger: i think it's probably fine at this point18:33
pabelangerAJaeger: I haven't heard any failures with ubuntu-trusty recently18:33
openstackgerritMerged openstack-infra/project-config: Convert last devstack-trusty to ubuntu-trusty  https://review.openstack.org/29362418:33
*** jpr has joined #openstack-infra18:34
AJaegerso, I removed WIP and -2 from https://review.openstack.org/#/c/29123818:34
pabelangerclarkb: looks like we are only using 3.9GB for tmpfs (/opt/dib_tmp)18:34
pabelangerdidn't think we limited it18:34
*** ldnunes has quit IRC18:35
*** mriedem has quit IRC18:35
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add normalize stack function for heat stack_list  https://review.openstack.org/25433718:36
pabelangerAJaeger: fungi if we are ready, I can +A it18:36
fungigo for it18:36
clarkbpabelanger: I think it makes a dir for the build in there too, but that shouldnt grow to 153GB18:36
clarkband towards the end of the build mv's it all into a loopback mounted raw device18:37
pabelangerclarkb: Ya, monitoring right now. Up to 38% use18:37
*** ldnunes has joined #openstack-infra18:37
pabelangerI thought --notmpfs removed that18:38
*** mriedem has joined #openstack-infra18:38
clarkboh we shouldn't be tmpfsing at all18:38
*** pvaneck has quit IRC18:38
clarkb/opt/dib_tmp is on disk and we use that instead18:38
pabelangersomething is causing tmpfs to be used18:38
openstackgerritDeklan Dieterly proposed openstack-infra/devstack-gate: Remove deletion of ~/workspace-cache  https://review.openstack.org/29364318:39
pabelangertmpfs                 3.9G  2.2G  1.8G  56% /opt/dib_tmp/image.YefSYwgq18:39
clarkbpabelanger: maybe something in the debian element?18:39
pabelangerclarkb: that is what I am checking18:39
zxiirofungi: electrofelix: just catching up do we need to revert a patch?18:39
ddieterlyjeblair https://review.openstack.org/29364318:39
*** cdent has quit IRC18:40
*** jordanP has quit IRC18:40
*** vgridnev has quit IRC18:40
*** vgridnev has joined #openstack-infra18:41
pabelangerexport DIB_NO_TMPFS=0 is what I am seeing in the logs18:41
electrofelixzxiiro: either https://review.openstack.org/#/c/293631/ or https://review.openstack.org/#/c/293598/1 should resolve the problem18:41
electrofelixso add additional fix or revert existing18:41
zxiiropersonally I'd prefer to revert and have it rewritten usign pylock18:42
fungizxiiro: apparently "fasteners" is the new pylockfile18:42
clarkbpabelanger: and --no-tmpfs is on the dib command line so it is trying18:42
pabelangerclarkb: ya18:42
pabelangerI've had this problem before locally18:43
*** MarkAtwood has quit IRC18:43
pabelangertrying to remember the issue18:43
zxiiroI +2'd both patches, I'll let you folks decide which route is better.18:43
anteayamordred: thanks for the history of infra in your candidate statement18:43
zxiiroI'm not immediately impacted by this so18:43
anteayamordred: also I think you set a reasonable goal for your platform18:43
*** ddieterly is now known as ddieterly[away]18:43
fungizxiiro: ultimately it seems to be benign, aside from masking real failures by causing the exit code to be nonzero all the time18:44
*** _nadya_ has quit IRC18:45
electrofelixclarkb fungi: how impacting were the previous problems with JJB stomping over cache files? can you live with the revert for a while or is that just adding back an issue.18:45
*** rossella_s has quit IRC18:45
clarkbI think either way JJB still stomps over the cache file18:46
clarkbthis doesn't solve that18:46
*** mrmartin has quit IRC18:46
clarkbwhich is why i have suggested the tmpfile + move18:46
*** rbrndt has joined #openstack-infra18:46
clarkbthe problem with the cache files is puppet runs jjb up to a timeout then jjb dies halfway through a long run. This leaves the cache file half written18:46
clarkbinteprocess locks only help the case of two jjbs writing to the same file which I don't think is a common one18:47
*** BobBall is now known as BobBall_AWOL18:47
electrofelixah, well then I think revert and rewrite is probably better. thought it was added to prevent somethnig18:47
fungiwell, cache file writing happens at the end. the reason we see it semi-regularly is when the timeout hits near the end _during_ cache writing18:47
*** sfinucan has joined #openstack-infra18:47
fungiand it's leaving a half-written/truncated cache18:48
electrofelixjust approved the revert, we can come back to how it should be written later18:48
*** ddieterly[away] is now known as ddieterly18:48
electrofelixsounds like the write tmp + move should be done first anyway18:48
*** electrofelix has quit IRC18:49
pabelangerclarkb: I know the issue -t is being passed empty18:49
pabelanger2016-03-16 18:18:20,480 INFO nodepool.builder: Running disk-image-create -x -t  --no-tmpfs  -o /opt/nodepool_dib/176615 debian-minimal vm simple-init openstack-repos nodepool-base node-devstack cache-bindep growroot18:49
clarkbpabelanger: oh!18:49
pabelangerI remember this18:49
clarkbpabelanger: is this image not a part of any providers?18:49
*** baoli has joined #openstack-infra18:49
pabelangerI think so18:49
clarkbbecause providers determine type18:49
pabelangerthat is the issue18:49
*** fawadkhaliq has quit IRC18:49
pabelangeryup18:49
fungiaha. i missed that in reviewing the change18:49
pabelangersame18:49
pabelangerlet me fix18:49
fungiwell shucks18:49
*** vgridnev has quit IRC18:50
pabelangerI actually just abandoned a patch in nodepool that exposed this18:50
openstackgerritSam Betts proposed openstack-infra/glean: Enable multiple VLANs per physical interface  https://review.openstack.org/28372618:51
openstackgerritSam Betts proposed openstack-infra/glean: [WIP] Enable support for configuring bonded interfaces  https://review.openstack.org/29364818:51
igorbelikovis there some other place where providers need to be set besides here? https://github.com/openstack-infra/project-config/blob/master/nodepool/nodepool.yaml#L35-L4018:51
openstackgerritMerged openstack/requirements: bump uc for congressclient to 1.2.3  https://review.openstack.org/29342518:51
clarkbigorbelikov: yes in each provider you have to tell it what flavor to use and where to get the image18:52
openstackgerritMerged openstack-infra/jenkins-job-builder: Revert "Allow using lockfile per jenkins master"  https://review.openstack.org/29359818:52
clarkbif you tell it to usr dib then the provider formats specified are used in the dib build18:52
*** sambetts is now known as sambetts|afk18:52
igorbelikovclarkb: ugh, yeah, see it now, thx18:53
*** mrmartin has joined #openstack-infra18:53
*** vgridnev has joined #openstack-infra18:53
*** esikachev has joined #openstack-infra18:54
*** amit2131 has quit IRC18:55
igorbelikovpabelanger: I can make a fix if you have something better to do:)18:55
*** amit213 has joined #openstack-infra18:55
openstackgerritMerged openstack-infra/project-config: Remove bare-trusty from nodepool  https://review.openstack.org/29123818:56
pabelangerigorbelikov: nope, almost done18:56
openstackgerritDeklan Dieterly proposed openstack-infra/project-config: Add pre_test_hook to freezer api gate job  https://review.openstack.org/29143018:58
igorbelikovpabelanger: ok, thank you18:58
*** Daisy has joined #openstack-infra18:58
igorbelikovwhen I glanced over providers section I though it was for non-dib stuff18:58
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add debian-jessie to providers  https://review.openstack.org/29365118:59
pabelangerigorbelikov: clarkb: fungi: I added debian-jessie to all providers, even tho we are only launching in rax and ovh ATM18:59
pabelanger^18:59
*** nwkarsten has joined #openstack-infra19:01
*** Swami has quit IRC19:02
*** claudiub has quit IRC19:03
*** Swami has joined #openstack-infra19:03
*** Daisy has quit IRC19:03
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka  https://review.openstack.org/29365519:03
*** maishsk has joined #openstack-infra19:03
fungipabelanger: does it not need the meta subsection like ubuntu-trusty has?19:04
*** harlowja has joined #openstack-infra19:04
fungiseems like that might be needed in rackspace19:04
fungiand also name-filter19:05
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka  https://review.openstack.org/29365519:05
fungipabelanger: you may want to git blame those lines and see why the commit message(s) say they were added19:05
pabelangerfungi: right, let me check that19:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka  https://review.openstack.org/29365519:06
clarkbname filter is to get erformance flavors and the metadata is so that rax knows the images are pvhvm iirc19:06
clarkbso not strictly required but should be used19:06
fungiand to not install the xen agent19:07
fungior, rather, to not expect it to be installed19:08
fungithose seem odd to define on the diskimage rather than on the provider, but whatever19:08
pabelangerYa, I used the fedora-23 syntax, switching to ubuntu-trusty now19:08
*** tosky has quit IRC19:09
*** sfinucan has quit IRC19:10
kro_snifflesianw: I'm working on making thick-slaves work on centos - do you know of any patches that address the lack of mysql-devel (as opposed to mariadb)19:10
*** acruz has quit IRC19:10
*** ayoung has quit IRC19:10
clarkbkro_sniffles: we are quickly moving away from the thick vs thin provisioned slaves and have just one setup per distro release19:10
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add debian-jessie to providers  https://review.openstack.org/29365119:10
*** vgridnev has quit IRC19:11
AJaegerdo we need to restart/cleanup nodepool since bare-trusty is gone now?19:11
clarkbkro_sniffles: for that I think you just want the db setup macro and use mariadb19:11
clarkbAJaeger: devstack-trusty and yes19:11
pabelangerkro_sniffles: ianw: which is this for? we shouldn't be using thick slave if possible19:11
*** vgridnev has joined #openstack-infra19:11
kro_snifflespabelanger: openstack_project::wheel_mirror_slave uses openstack_project::slave19:11
kro_snifflesIt seems to be coming in from there.19:11
openstackgerritMerged openstack/requirements: prevent installing old releases of congressclient  https://review.openstack.org/29344119:12
AJaegerclarkb: devstack-trusty from jobs - and bare-trusty completely from setup19:12
AJaegerclarkb, I was talking about 29123819:12
* kro_sniffles thinks that's the case anyway19:12
clarkbAJaeger: gotcha, I don't think we need a nodepool restart19:12
clarkbAJaeger: it should reread its config and figure it out19:12
pabelangerkro_sniffles: Ah, okay. So I haven't looked at that stuff yet.  Hopefully tomorrow.  We might want to wait until centos-7 dibs are online, then we can use bindep to control some of the dependencies19:13
clarkbkro_sniffles: its possible the puppet mysql module is smart19:13
openstackgerritMerged openstack-infra/storyboard: Add task notes to task email  https://review.openstack.org/29246219:13
pabelangerAJaeger: we'll likely need nodepool-builder restart once debian-jessie changes land too19:13
clarkbkro_sniffles: it may already know how to switch based on distro?19:13
pabelangerI would expect it to be too19:13
kro_snifflespabelanger: Coolio, can you ping me once that happens so I can get back into doing wheel things?19:14
kro_snifflesclarkb: Its switching is not perfect.19:14
pabelangerkro_sniffles: I hope to look into it later tonight and tomorrow. Otherwise, will be full steam into it next week19:14
pabelangerPTO on friday19:14
kro_snifflespabelanger: Gotcha19:15
pabelangerAJaeger: currently 19 devstack-trusty images used :)19:15
*** salv-orl_ has quit IRC19:15
AJaegerpabelanger: Yeah!19:16
*** vgridnev has quit IRC19:17
*** mariojv has left #openstack-infra19:17
pabelangerAJaeger: I guess the next step is to remove dsvm from all the job names?19:17
AJaegerpabelanger: that wasn't on my radar. Fungi, clarkb, what do you think?19:19
clarkbits mostly redundant at this point but probably not a priority19:19
clarkbmaybe we can do then when jjb updates are reliable :)19:19
fungiAJaeger: pabelanger: it's probably useful for now as an indicator that those are (mostly) devstack-gate based jobs19:19
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka  https://review.openstack.org/29365519:20
*** fverbs has quit IRC19:20
*** mrmartin has quit IRC19:20
fungiit was just convenient, for a time, to also use that to determine a non-default node label mapping19:20
* AJaeger would love to see some review on the change above ^, please19:21
*** vgridnev has joined #openstack-infra19:21
mgagnefungi or clarkb: do you know the TTL of tokens generated at other providers?19:22
clarkbmgagne: rax is 24 hours19:22
*** mrmartin has joined #openstack-infra19:22
clarkbI only know that one due to the swift upload issues19:22
AJaegerfungi, mriedem gave a +1 on the nova db patches. But I had to fix a problem... https://review.openstack.org/292472 is the next one to merge there19:23
fungimgagne: yeah, no clue really. i've heard that some providers dial it down as low as 1hr for database performance reasons or something19:23
kro_snifflesAJaeger, pabelanger: Waaaaait a sec. Are we setting up persistent devstack vm's? [/hopeful look]19:23
fungiAJaeger: yep, saw the update, just going back through it19:23
mgagnefungi: how long do you think it should be? 24h looks a bit too much IMO19:23
*** sridhar_ram1 has quit IRC19:24
fungimgagne: no clue. i think 24 hours is the default but maybe the security guide has good guidance on non-default values?19:24
pabelangerclarkb: fungi: I'll be stepping out again in 45mins. So, if 293651 lands while I am out, do you mind doing a debian-jessie image-build?19:24
pabelangerwill be back tonight after food19:24
pabelangerkro_sniffles: no, we are replacing devstack-trusty with ubuntu-trusty19:24
mgagnefungi: puppet defaults to 1h: https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L70-L7219:25
pabelangerkro_sniffles: that will reduce the amount of images we maintain in nodepool19:25
fungikro_sniffles: no, we've finally got all our other jobs working on similar base images to what devstack jobs use, so we're finally shuffling all those into one common node label19:25
pabelangerkro_sniffles: thanks to AJaeger efforts, we'll be down 2 images in the last 7 days19:25
*** roxanagh_ has joined #openstack-infra19:25
fungipabelanger: i believe debian-jessie images will automatically start building unless we've set min-ready to -119:26
mgagnefungi: same for keystone default: https://github.com/openstack/keystone/blob/99c52f9fbe56f5de01afa0c5fd09864c727b832d/keystone/common/config.py#L291-L29319:26
fungipabelanger: but we'll need a builder restart first19:26
kro_snifflesOh. Well, that's close anyway :)19:26
pabelangerfungi: ya, restarting builder is what I should have asked for19:26
clarkbmgagne: fungi this seems like a failing in our services if we fail this way, I don't think a longer token should be necessary19:27
fungimgagne: oh! right, 1hr default. some providers have been known to increase it so as to _reduce_ load from refreshes impacting performance19:27
*** kro_sniffles is now known as kro_mnckn_doc19:27
mgagnefungi: I think it's more related to image snapshot and such. Often can't snapshot in less than 1h19:27
*** piet has joined #openstack-infra19:28
*** daemontool has quit IRC19:28
*** vgridnev has quit IRC19:28
mgagnefungi: increasing it could be some issues we also see on our side but I feel there is room for improvement on nodepool side too, like renewing token right before starting an upload19:28
mgagnefungi: are there metrics related to upload time/duration in Nodepool?19:29
*** eliqiao has quit IRC19:29
mgagneI feel this could be very useful19:29
*** eliqiao has joined #openstack-infra19:29
*** akshai has quit IRC19:30
pabelangermgagne: I agree. I think we could expose more data into statsd19:30
fungimgagne: maybe, though i agree with clarkb that it's still a concern if a token expiring in the middle of an already running file upload causes that upload to break19:30
pabelangermgagne: also success / failures for dib builds19:30
fungimgagne: and probably needs to be properly solved in keystone/glance design19:31
mgagnefungi: a fix went in Glance to use trust in december, we don't use latest trunk so yea, can't use that yet =)19:31
fungiassuming that's what's actually happening19:31
*** tsg has joined #openstack-infra19:31
mgagnerenewing token would give more chance of successfully uploading image19:32
fungifor internap, this is probably going to start failing less often anyway once we get our qcow2 sizes down with a new diskimage-builder release19:32
*** tsg_ has joined #openstack-infra19:33
mgagnefungi: it think there are more than one step to take to improve change of success =)19:33
mgagnebbl, meeting19:33
*** yamahata has quit IRC19:34
openstackgerritRob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack  https://review.openstack.org/29309019:34
*** ociuhandu has quit IRC19:35
*** tsg has quit IRC19:36
*** vgridnev has joined #openstack-infra19:37
*** nwkarsten has quit IRC19:38
AJaegerpleia2: it would be great if translation check site would be up for translators to check. Do you have a chance to help give it the final kick, please? https://review.openstack.org/#/q/topic:i18n-checksite shows the current changes19:38
*** nwkarsten has joined #openstack-infra19:38
*** akshai has joined #openstack-infra19:39
anteayaAJaeger: she is in singapore at the moment19:40
anteayaunless, has she been online today?19:40
doug-fishof course it would be fine for anyone else to kick it too19:40
AJaegeranteaya: ok.19:41
*** _nadya_ has joined #openstack-infra19:41
*** cloudtrainme has quit IRC19:41
AJaegerdoug-fish: if you speak about translation check site, yes! I pinged pleia2 since she volunteered herself, her name is on the spec...19:41
doug-fishoh - I was actually having trouble with the translation site itself19:41
doug-fishnot the check site19:41
anteayaI'm not saying not to ask, just there might be a pause before she responds19:41
AJaegerunterstood, anteaya19:42
anteayathanks19:42
doug-fishIf she's in Singapore I think it would be traditional to be sleeping now19:42
AJaegerdoug-fish: zanata? what's the problem?19:42
doug-fishcan't log in - after providing credentials it still seems to think I'm not logged in19:42
eil397hi19:42
anteayayeah 3:45am in singapore atm19:43
AJaegerdoug-fish: same problem here.19:43
*** nwkarsten has quit IRC19:43
eil397anteaya: excuse me. yesterday I've added description in https://etherpad.openstack.org/p/infra-newton-summit-planning19:43
AJaegerStevenK: could you check translate.openstack.org, please? doug-fish and myself cannot log in ;(19:43
eil397I will remove it today.19:43
*** MarkAtwood has joined #openstack-infra19:44
clarkbAJaeger: openstackid has bene having trouble today aiui19:44
anteayaeil397: no no no, you don't have to remove it19:44
clarkbbut testing if I can login19:44
anteayaeil397: thanks for answering my question19:44
anteayaeil397: I don't know what you want to talk about is what I'm saying19:44
AJaegerclarkb: that could explain it as well...19:44
anteayaeil397: you are welcome to put items on the etherpad, thanks for participating19:44
anteayaeil397: but the sentence you wrote doen't help me understand what you want to discuss19:45
clarkbAJaeger: you do login but it doesn't actually log you in? this is what appears to have hapepend to me19:45
anteayaeil397: do you have a minute to help me understand what you want to talk about?19:45
eil397anteaya: thank you for understanding. I should propose it on irc before19:45
anteayaeil397: sure19:45
doug-fishclarkb: yep that's the problem I see19:45
anteayaeil397: have you time now to say what it is you want to discuss?19:45
clarkbsmarcet: is the above loging issue expected with the current state of openstackid?19:46
*** e0ne has joined #openstack-infra19:46
eil397anteaya: I just have idea. but not sure that it is right place and right way. briefly : it is about how community infra project can be integrated in internal CICD for Openstack19:46
*** harlowja has quit IRC19:46
anteayagreat19:46
AJaegerclarkb: yeah ;(19:46
anteayaright now I guess I need some help understanding how you define your terms19:47
openstackgerritMerged openstack-infra/shade: Add normalize stack function for heat stack_list  https://review.openstack.org/25433719:47
*** maishsk has quit IRC19:47
anteayaas my perception of those terms is we are doing that alreaady19:47
openstackgerritMerged openstack-infra/project-config: Setup translations for stable/mitaka  https://review.openstack.org/29365519:47
openstackgerritMerged openstack-infra/tripleo-ci: Use m1.small for ping test  https://review.openstack.org/29351119:47
anteayaeil397: but you don't think we are, otherwise you wouldn't have suggested the topic19:47
anteayaeil397: can you help me understand how you are defining your terms?19:48
anteayawhat do you mean by community infra project?19:48
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box'  https://review.openstack.org/28613719:48
thomasemSo, looking at different node types (fedora-23 specifically) for the LXC experimental gate job atm. I'm wondering what the usual way to run Nova gate tests using a WIP patch for openstack-infra/project-config repo? I tried Depends-On, but I didn't see it using a fedora-23 node instead of ubuntu-trusty.19:48
thomasemIf there is one.19:48
clarkbthomasem: there isn't a way to do that19:48
thomasem:(19:48
clarkbthomasem: we can't let unreviewed jenkins jobs through for testing because they can potentially do nefarious things19:49
thomasemAh, right. Good point.19:49
eil397anteaya: right now I'm busy little bit. it is nice that I've caught you.19:49
anteayaeil397: okay that's fine19:49
clarkbjenkins doesn't have an unprivileged mode for some jobs19:49
anteayaeil397: I'm interested in understanding what you mean19:49
anteayaeil397: take your time and we can work together to understand one another19:50
pabelangerclarkb: where should I be looking on puppetmaster.o.o to check JJB runs?19:50
*** kzaitsev_mb has quit IRC19:50
thomasemUnderstood. So, then, how about I throw up a patch ADDING another experimental job using fedora-23 in the devstack-gate's experimental pipeline?19:50
pabelangerclarkb: assuming they are still broken19:50
clarkbpabelanger: JJB runs on each jenkins master locally19:50
thomasemWould that be merge-worthy to y'all and thusly let me test that?19:50
clarkbthomasem: thats probably fine though I don't have all the context in my head19:50
eil397anteaya: thank you for your time. I'm working in walmartlabs. and I've spent few month to work with internal CICD for OpenStack. and I've started from clonning Comunity CICD. I should say it is awesome. and deployment automation works great. and it is pretty clear orginized19:50
pabelangerclarkb: right, but does ansible log that some place on puppetmaster? or just locally on the master19:51
anteayaeil397: I'm glad you are having a good experience19:51
clarkbpabelanger: puppet is only logged on the machine it runs on and the puppetboard report19:51
anteayaeil397: thanks for that feedback19:51
clarkbpabelanger: I do not know if we write the jjb output anywhere19:51
clarkbmy guess is we dont19:51
thomasemAh, for context, I'm having a horrible time with the current libvirt and kernel versions on the ubuntu-trusty node when trying to get the libvirt/LXC nova experimental job to pass consistently. I am, however, after several suggestions from the Nova folks, seeing much better behavior with Fedora 23 in my local simulations. So, I wanted to get the test moved over to using fedora-23 and see how that goes in the _actual_ pipeline as opposed to my19:52
thomasemlocal simulations.19:52
thomasemclarkb:  ^^19:52
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Bug Fixing  https://review.openstack.org/29368319:52
pabelangerclarkb: okay, so from what I see jenkins04,05 and 06 are running old JJB.  Was trying to see how you know the flock stuff was broken19:52
eil397anteaya: as a result of some work. I have made internal proposal about who can we be integrated with community cicd (infra) and what are critical components we should add . also I've proposed topic for summit. and I'm glad it was not voted. because it is little bit specific for company. and maybe not very good example for community.19:53
eil397anteaya: s/who/how/g19:53
thomasemThe goal right now is to at least get the code tested in the pipeline, but i can't even get it to non-voting status without some amount of stability exhibited.19:53
anteayaeil397: okay so you have some modifications to our tooling that you are using in your work19:54
anteayaeil397: and you would like to offer those modifications to the upstream tools?19:54
*** maishsk has joined #openstack-infra19:54
clarkbthomasem: can you just change the slave type out from under that job and call it good?19:54
*** adrian_otto has joined #openstack-infra19:54
clarkbthomasem: I am not sure how far along the ubuntu job got if it doesn't have reliable containerthings19:54
thomasemYou mean like this: https://review.openstack.org/#/c/293585/1/jenkins/jobs/devstack-gate.yaml19:54
eil397anteaya: it is not modification. not fork. because it will make very expensive syncing. but it is about what are critical components for internal cicd for openstack.19:55
*** ig0r_ has quit IRC19:55
clarkbthomasem: ya19:55
thomasemIt's still experimental. I'm hoping to get at least something non-voting so we can pursue it further.19:55
anteayaeil397: hmmmm19:55
thomasemThe idea is to get it working on one platform, then move to others as we find problems.19:55
clarkbthomasem: gotcha, so ya start with that change, run it in experimental a few times, show nova it works, then move it to an appropriate location19:55
eil397anteaya: for example deployment automation on hardware mulitinode cluster. and support of internal rm processes.19:55
eil397anteaya: rm ( release management)19:55
thomasemOkay19:56
*** rockyg has joined #openstack-infra19:56
*** ociuhandu has joined #openstack-infra19:56
anteayaeil397: okay so you are thinking you would like to have a discussion with people consuming our tooling and hear from the group deploying infra tools internally?19:56
anteayalike the infra tools operators group19:56
thomasemSo, let me get a commit and related bug tagged on that commit... I guess the reason I wanted to leave the ubuntu-trusty one around was because I'd like for us to get it working on that platform too at some point. Might have to wait for a newer Ubuntu, though.19:56
clarkbthomasem: ya I wouldn't worry about that until you have something working19:56
thomasemOkay, that's fair.19:56
*** rockyg has quit IRC19:57
*** rockyg has joined #openstack-infra19:57
*** tonytan_brb is now known as tonytan4ever19:58
thomasemThanks, clarkb19:58
thomasem:)19:58
*** sfinucan has joined #openstack-infra19:58
*** ericksonsantos has joined #openstack-infra19:58
thomasemI'll throw up a new review once I add the related bug(s) and context to the commit msg.19:58
pabelangeranteaya: eil397: The updates you are referring to, is anything public atm? I always fine it easier to look at code to see how people are using infra CI tooling19:58
anteayapabelanger: I wonder if eil397 is proposing a meeting of the minds more than "we need this fix"19:59
anteayathat is my current belief19:59
openstackgerritMerged openstack-infra/openstackid: Bug Fixing  https://review.openstack.org/29368319:59
anteayapabelanger: but I agree, code is helpful20:00
pabelangeranteaya: right. I was mostly interested to see if anything existed today and read up on. Almost like home work before said meeting20:00
eil397anteaya: my first intention when I've proposed as a talk on summit was to present some point of view. maybe something in my view can be corrected and some specifics are really do not exist.20:00
anteayapabelanger: yup, good thoughts20:00
eil397anteaya: that is right.20:00
anteayaeil397: okay, it is an interesting thought, I'm glad you raised it20:01
anteayaI don't think we have had this idea before20:01
anteayaeil397: can I make a suggestion for what to put on the etherpad so perhaps others can understand it?20:01
*** icey has quit IRC20:01
*** salv-orlando has joined #openstack-infra20:01
anteayaeil397: I can't guarentee they also won't have questions20:01
anteayabut I can suggest wording that would make sense to me20:02
*** icey has joined #openstack-infra20:02
*** tonytan4ever has quit IRC20:02
eil397anteaya: thank you.20:02
*** flwang has joined #openstack-infra20:02
anteayaokay here is my suggestion20:02
*** cloudtrainme has joined #openstack-infra20:02
*** tonytan4ever has joined #openstack-infra20:02
anteayaLeader: <your name>20:02
eil397anteaya: yes. of course.20:02
flwanghi there, anybody can help add me as the 'ptl' of zaqar-ui-core? so that i can add those UI experts, thanks a lot20:03
anteayaDescription: meetup for ci operators consuming tools internally for their own work20:03
eil397anteaya: I'm not sure about <leader>: <my name>  because right now I 'm not planning to attend summit20:03
*** jcoufal has quit IRC20:03
flwanghere is the related patch https://review.openstack.org/29316620:03
anteayaSpec: not applicable20:03
anteayaeil397: oh20:03
anteayawell that makes it tough20:03
eil397anteaya: but I'm very interesting to have such kind of discussion for such kind of topic20:03
anteayayes I can understand20:03
*** claudiub has joined #openstack-infra20:03
SamYapleso returning a bit to the kolla automated pushing of docker images, this can be done but there is a secret file that is needed to be kept secure. Can anyone tell me if we have a way in infra to protect a file like this? like a private-key type file?20:04
anteayawe usually lean toward folks proposing the sessions leading them20:04
pabelangerclarkb: okay, I can tell JJB is running on each system by SSH into them (pretty obvious) but still having trouble understanding how run_jjb playbook is called.20:04
pabelangerclarkb: unless that is manually20:04
eil397anteaya: maybe some one would like to talk about it. I can provide deatails about some xperience20:04
anteayaeil397: hmmmmm20:04
anteayaokay yeah, put your name on the etherpad and say you aren't able to attend summit20:04
flwangor i should sent a mail to openstack-dev as a request? thanks20:04
eil397anteaya: that is why maybe we should start from irc.20:04
clarkbpabelanger: puppet runs jjb not ansible20:05
anteayaand provide details either in the etherpad for the summit or a separate etherpad if there are a lot20:05
anteayaeil397: I would like to hear your experience20:05
eil397anteaya: I've added it to etherpad to show interest to it : - )20:05
pabelangerclarkb: Ah, so thats what I was confused about.  I thought we moved it into ansible and out of puppet20:05
anteayaeil397: and you spoke up early so we have some time to figure out what kind of interest there is20:05
anteayaeil397: thank you20:05
pabelangerso, people use run_jjb when they manually sync it20:05
*** claudiub|2 has joined #openstack-infra20:05
clarkbpabelanger: I don't20:05
clarkbbut others do20:06
*** sfinucan has quit IRC20:06
anteayaeil397: might I ask where you are located geographically? in case there might be an event closer to where you are in future that you could attend?20:06
clarkb(I like to be able to actually watch the progress and see how far ahead/behind it is20:06
pabelangerSamYaple: Is it a onetime password? Or same password all the time?20:06
pabelangerclarkb: right20:06
eil397anteaya: I'm in bay area - NorCal. sunnyvale20:07
anteayaoh wow20:07
anteayahard to think of summits that are going to be much closer to you20:07
pabelangerSamYaple: currently, the only place ATM would be in hiera and system-config20:07
anteayaunless they are in the bay area20:07
pabelangerSamYaple: but, if you are using ansible, why not setup ansible-vault for your project20:08
pabelangerguess you still need password to open it20:08
*** claudiub has quit IRC20:08
*** infra-red has quit IRC20:08
openstackgerritMerged openstack-infra/project-config: Fix two grafana entries to point at the right zuul pipeline  https://review.openstack.org/29354420:09
*** _nadya_ has quit IRC20:09
pabelangerokay, stepping away for food. If any infra-root would like to help 293651 along, I can continue work on debian-jessie later tonight20:09
anteayaenjoy food20:10
pabelangerAJaeger: we are also still at 16 devstack-trusty images. But JJB still running on some jenkins masters20:10
*** ayoung has joined #openstack-infra20:11
*** IlyaG has joined #openstack-infra20:12
*** bpokorny_ has joined #openstack-infra20:12
openstackgerritMerged openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue  https://review.openstack.org/29317820:13
AJaegerpabelanger: and I see some jobs that use devstack-trusty, investigating...20:13
*** bpokorny has quit IRC20:13
AJaegerexample gate-scalpels-dsvm-neutron-devstack-trusty20:13
AJaegerold cache, let me regenerate ;(20:13
clarkbAJaeger: that one should've been addressed in my change (I specifically rmember catching that one)20:13
fungiyeah, i still have 3 jenkins masters updating20:14
*** _nadya_ has joined #openstack-infra20:14
*** cloudtrainme has quit IRC20:14
fungipartly my fault since i initially ctrl-c'd them after realizing i'd rerun on those with --workers 2, and the interrupt resulted in broken jjb caches which i had to delete20:15
*** bpokorny_ has quit IRC20:15
fungibecause i think i ctrl-c'd a second time reflexively while they were writing out their caches20:15
*** _nadya_ has quit IRC20:15
*** bpokorny has joined #openstack-infra20:15
AJaegerclarkb: but we have gate-puppet-zanata-puppet-beaker-rspec-dsvm-trusty-nv20:15
fungithey're getting close to complete now though20:15
AJaegerSee http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/puppet-module-jobs.yaml#n320:16
*** kzaitsev_mb has joined #openstack-infra20:16
AJaegernode: 'devstack-{ostype}'20:16
*** cloudtrainme has joined #openstack-infra20:16
*** IlyaG has quit IRC20:16
clarkbAJaeger: gah thats how it got past my grep20:16
*** IlyaG has joined #openstack-infra20:16
clarkbok I will write a patch for that one20:16
AJaegerthanks, clarkb !20:16
fungithat one will take a little extra parameter work i guess20:16
clarkbfungi: depends on how many devstacks it supports20:17
clarkbwill find out shortly20:17
fungitrue!20:17
AJaegerclarkb, I run: rm -rf .test/jenkins-job-builder/.test/new/out/;tox -e zuul;grep ubuntu-trusty .test/jenkins-job-builder/.test/new/out/*20:17
clarkbwow aarently we did that all over20:18
* clarkb hatchets it20:18
AJaegerclarkb: 116 jobs left with ubunut-trusty20:19
AJaegerclarkb: 116 jobs left with devstack-trusty I mean20:19
SamYaplepabelanger: its very much a private key, but i would need to store it soomewhere and it would need to be projected so, say, only the periodic gate jobs could access it20:19
SamYapleotherwise it would be a security hole (user could submit a patchset and extract the password)20:20
*** bpokorny has quit IRC20:20
*** bpokorny has joined #openstack-infra20:20
clarkbAJaeger: ya I think most of them are these puppet jobs20:20
flwangclarkb: ping re (09:03:02) flwang: hi there, anybody can help add me as the 'ptl' of zaqar-ui-core? so that i can add those UI experts, thanks a lot20:21
AJaegerclarkb: yes - plus gate-openstackci-beaker-trusty-dsvm20:21
flwangclarkb: i think you mentioned how to do that, but i lost the msg now20:21
ianwpabelanger: hey; so centos-7 seems to have built ok ... but still not registered.  either it hasn't uploaded or it isn't working.  i haven't read scrollback to see if this is expected or not20:21
*** IlyaG has quit IRC20:21
clarkbwow these puppet jobs are a mess20:23
clarkbthey are also completely divergent from ours from what I can tell20:23
*** esikachev has quit IRC20:23
clarkbYOLO!20:23
* Clint chuckles.20:24
clarkbapparently we run all of our lint checks and that on trusty and they run all of theirs on centos20:25
*** nwkarsten has joined #openstack-infra20:25
clarkb(I am just grumping I have to make the same change twice)20:25
AJaegerclarkb: just merge them ;)20:26
fungiwow, the stable branch team's electorate is second only in size to infra's20:26
fungiwith neutron in third place20:26
fungithen docs, then nova, then qa20:26
*** IlyaG has joined #openstack-infra20:27
flwangAJaeger: could you pls help?20:28
AJaegerflwang: I have no permissions to help you20:29
AJaegerclarkb: https://review.openstack.org/#/c/290726/20:29
flwangAJaeger: how should i do then? thanks20:29
*** pvaneck has joined #openstack-infra20:29
flwanglike who is the right guy i should talk/email?20:29
clarkbAJaeger: interesting, I am sure my change will merge conflict with that20:29
clarkbflwang: patience, as soon as I am done with the thing I prevkously committed to I can do your thing20:30
*** esikachev has joined #openstack-infra20:30
fungiSamYaple: pabelanger: yeah, we do uploads to external systems like that from durable non-nodepool-managed workers which run very limited jobs in post or release pipelines which don't involve running code from the repositories being processed20:30
flwangclarkb: since i already got my answer, now i will be quite, thanks20:30
fungiSamYaple: pabelanger: so that we can safely store keys/passwords on them without risk of exposure. it's a solved problem20:31
*** IlyaG has quit IRC20:32
*** baoli has quit IRC20:32
clarkbthere is really only one set of jobs that cause problems in this and that is the beaker jobs beacuse they build ostype into the job itself20:32
clarkbI basically end up with two groups instead of one I think20:32
jeblairSamYaple: thank you for realizing the problem though!  so many people come through here with "simple ideas" and don't appreciate that.20:32
anteayafungi: interesting20:33
fungipabelanger: any objection to becoming a gerrit admin, so that you can help with initial group membership setting, troubleshooting via javamelody, et cetera?20:33
*** ldnunes_ has joined #openstack-infra20:35
* AJaeger waves good bye20:36
*** MarkAtwood has quit IRC20:37
*** ldnunes has quit IRC20:38
fungihave a good evening AJaeger!20:39
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics  https://review.openstack.org/26397120:39
fungiclarkb: jeblair: jhesketh: mordred: SergeyLukjanov: nibalizer: yolanda: i'm seeking a gerrit admin volunteer who can run the script from https://review.openstack.org/263971 some time between midnight utc tonight and midnight utc tomorrow night and send a signed copy of the results to tristanC and tonyb. instructions are at the top of the script, but i'm happy to walk you through them and help you20:39
fungiconfirm the results look like they should if you want20:39
anteayagood night AJaeger20:39
SamYaplefungi: so on that note, the idea is to build kolla images and push them to the docker repository, that _would_ required code running from the repository20:41
fungiSamYaple: why? you build the docker images on one system, then use a different system to upload them20:41
*** andymaier has quit IRC20:41
fungias long as your build step and your upload step are decoupled, should be fine20:42
*** asettle has joined #openstack-infra20:42
fungiit does require shuffling the files around, but it's a pattern we already follow for, e.g., releases to pypi20:42
SamYaplefungi: but transfering the data from the build to upload system20:42
fungiSamYaple: i feel like i've already replied to a -dev ml thread on this topic with a variety of concerns/options a few weeks ago20:43
openstackgerritThomas Maddox proposed openstack-infra/project-config: Move to Fedora 23 for Libvirt/LXC Nova Test  https://review.openstack.org/29358520:43
*** IlyaG has joined #openstack-infra20:43
SamYaplefungi: ill dig through it20:43
fungipretty sure it was some team wanting to come up with a mechanism for auto-building and uploading some sort of container images. might have been a thread sdake started?20:44
*** sridhar_ram has joined #openstack-infra20:44
*** ianw has quit IRC20:46
*** ianw_ is now known as ianw20:46
*** dtardivel has quit IRC20:48
*** abregman has joined #openstack-infra20:50
SamYaplefungi: he might have20:51
fungiare our openstackci-beaker jobs broken?20:51
SamYaplesdake: ping20:51
*** IlyaG has quit IRC20:51
clarkbyes it was sdake tlaking about uploading to dockers image repothingy20:51
SamYaplejeblair: im all 'bout dat security20:51
sdakesup20:51
fungihttp://logs.openstack.org/71/263971/10/check/gate-openstackci-beaker-trusty-dsvm/ca4bdb1/console.html#_2016-03-16_20_40_08_60120:51
fungi"Could not find puppet-4.4.0 in any of the sources"20:52
SamYaplewe were discussing the private key for docker uploads sdake20:52
*** aysyd has quit IRC20:52
SamYaplefungi mentioned a ML thread that i may have missed20:52
sdakeya private key20:52
sdakewe need that20:52
fungii'm scouring my sent mail for the subject line now20:52
SamYaplesdake: do you remember the ML thread? thats what the ping was for20:53
fungiSubject: Re: [openstack-dev] [kolla][infra] Publishing kolla images to docker-registry.openstack.org20:53
fungilooks like it'll be in the february archive20:53
*** jpr has quit IRC20:53
SamYaplefungi: i have it in my inbox20:54
* fungi ceases digging up a pipermail link20:54
SamYapleyea i dont see any solutions there fungi, but i can continue the thread20:54
SamYapleto address your comment from before, this isnt a few files. its, conservatively, ~15GB20:55
SamYapleso how would that move from build to push host20:55
fungiSamYaple: my reply from 2016-02-24 17:43:02 utc20:55
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs  https://review.openstack.org/28944520:55
SamYaplefungi: i do not have a reply from you in my inbox20:55
SamYaplefungi: did you reply to the ML? or a person?20:55
fungiSamYaple: http://lists.openstack.org/pipermail/openstack-dev/2016-February/087453.html20:56
sdakei posted the size of my registry disk usage20:56
sdakeit was about 1.5gb per distro20:56
SamYaplesdake: thats not quite accurate though20:57
SamYapleregistry usage and image usage are a bit different20:57
SamYaplefungi: wierd, not in my email, reading your reply20:57
sdakei was talking about the size of the disk space used on the target registry server20:58
*** e0ne has quit IRC20:58
*** maishsk has quit IRC20:59
SamYapleright. so that is another question. pushing to an openstack controlled registry, or the dockerhub20:59
SamYaplemy vote would be dockerhub directly20:59
*** mattoliverau has quit IRC20:59
*** annegentle has joined #openstack-infra20:59
*** matt6434 has joined #openstack-infra20:59
*** woodster_ has joined #openstack-infra21:00
*** matt6434 is now known as mattoliverau21:00
fungianyway, i'm happy to go into more detail on any of this, though i need to disappear for a while as it's approaching dinnertime21:00
*** thorst has quit IRC21:01
fungiand there are other denizens of this channel with equal or greater knowledge of the options than i possess21:01
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a simple status webapp  https://review.openstack.org/29370621:01
SamYaplefungi: if you would that would be fantastic. i wish i could reply to you on that, but im missing the email in my inbox21:01
doug-fishajaeger21:01
*** ddieterly is now known as ddieterly[away]21:01
doug-fishoops! I was trying to search back and see what is up with the openstackid/zanata issue21:01
doug-fishIt has some urgency because it blocks translators from making progress21:02
*** bpokorny has quit IRC21:03
jeblairfungi, pabelanger, ianw: James E. Blair proposed openstack-infra/nodepool: Add a simple status webapp  https://review.openstack.org/29370621:03
*** thorst has joined #openstack-infra21:03
*** bpokorny has joined #openstack-infra21:03
*** ybathia has joined #openstack-infra21:04
bknudsonQuestion about the reqs update tool -- see https://review.openstack.org/#/c/293429/1/requirements.txt21:04
bknudsonshould the reqs update tool be enhanced to keep the [kerberos] ?21:04
*** achanda has joined #openstack-infra21:05
bknudsonor were we mistaken adding it in the first place?21:05
*** ddieterly[away] is now known as ddieterly21:05
*** esikachev has quit IRC21:06
*** thorst has quit IRC21:07
*** maishsk has joined #openstack-infra21:08
*** madhu_ak has joined #openstack-infra21:10
*** annegent_ has joined #openstack-infra21:10
*** annegentle has quit IRC21:11
madhu_akJust checking, has anyone got into this error or aware of it: http://logs.openstack.org/28/164828/34/check/gate-neutron-lbaasv2-dsvm-api-namespace-nv/6cda6a4/logs/devstacklog.txt.gz#_2016-03-16_20_13_14_499 ? sudo: add-apt-repository: command not found21:11
*** doug-fish has quit IRC21:11
madhu_akdevstack-trusty is removed?21:11
*** DericHorn-HP has joined #openstack-infra21:12
*** DericHorn-HP has quit IRC21:12
*** doug-fish has joined #openstack-infra21:12
*** Daisy has joined #openstack-infra21:14
*** rfolco has quit IRC21:14
*** ldnunes_ has quit IRC21:15
clarkbyes devstack trusty is replaced by ubuntu-trusty21:15
tonybfungi: it looks like you're still looking for a volenteer, is that correct?21:15
*** maishsk has quit IRC21:16
*** ddieterly is now known as ddieterly[away]21:17
*** Daisy has quit IRC21:18
*** kro_mnckn_doc has quit IRC21:19
*** anteaya has quit IRC21:19
*** zaro has quit IRC21:19
*** samueldmq has quit IRC21:19
madhu_akclarkb, thanks for the info21:19
*** amitgandhinz has quit IRC21:20
sdakecan someone help me sort out how to do what Tristan asks here: https://review.openstack.org/#/c/293687/21:21
madhu_akso does it mean, should the line 14 from https://github.com/openstack/neutron-lbaas/blob/master/devstack/plugin.sh#L11 changed to BACKPORT="deb deb http://archive.ubuntu.com/ubuntu/ trusty main restricted" ?21:21
madhu_akclarkb ^6 ?21:21
sdakei'm not sure how to update a review that is already merged21:21
mgagnejeblair: fyi, default timezone is now UTC in our images at internap21:21
sdakenm its not merged21:22
*** annegent_ has quit IRC21:23
*** achanda has quit IRC21:24
*** tonytan_brb has joined #openstack-infra21:25
jeblairmgagne: yay!21:26
mgagnejeblair: thanks for the suggestion =)21:27
*** achanda has joined #openstack-infra21:28
*** shardy has quit IRC21:28
*** tonytan4ever has quit IRC21:28
openstackgerritClark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty  https://review.openstack.org/29371921:30
clarkbI expect ^ to fail every job ever but it gives a sense of the scale of refacorting required here21:30
nibalizerfungi: tonyb im happy to do it21:30
clarkbalso my eyes could no longer keep up with what is what21:30
clarkbso wnated to push something before I went cross eyed21:30
*** eharney has quit IRC21:30
tonybnibalizer: cool.  Thanks.21:31
*** darrenc is now known as darren_afk21:31
clarkb789 line diff thats less than I was owrried21:31
*** dtantsur|afk has quit IRC21:31
*** pradk has quit IRC21:32
*** csaba has quit IRC21:33
madhu_akclarkb, Just checking, is there any link that states that devstack-trusty is replaced by ubuntu-trusty?21:33
*** pblaho has quit IRC21:33
clarkbmadhu_ak: the goal was to make it transparent but AJaeger also sent mail to the dev list earlier this week or was it last week about it21:33
*** dprince has quit IRC21:34
madhu_akoh. I am pretty sure I might have overlooked that email. let me check mailing list..Incase you have it handy, if you could send me the link to the email thread would be awesome..21:34
*** pblaho has joined #openstack-infra21:34
clarkbmadhu_ak: [openstack-dev] [all][infra] Switching images for testing is the subject21:35
*** achanda has quit IRC21:36
madhu_akGot it clarkb thanks.21:36
*** e0ne has joined #openstack-infra21:37
*** dtantsur has joined #openstack-infra21:37
nibalizertonyb: tristanC prefered emails?21:38
tonybnibalizer: listed here: https://wiki.openstack.org/wiki/PTL_Elections_March_2016#Officials21:38
openstackgerritClark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty  https://review.openstack.org/29371921:38
tonybnibalizer: that's eaiser than typing :)21:39
*** nwkarsten has quit IRC21:39
*** e0ne has quit IRC21:40
*** johnsom has joined #openstack-infra21:40
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add a simple status webapp  https://review.openstack.org/29370621:40
johnsomIs this the right place to request a gerrit permissions fix for a project?21:40
nibalizertonyb: great thanks21:41
johnsomopenstack/neutron-lbaas-dashboard is missing two permissions for the neutron-release group21:41
jheskethMorning21:43
*** annegentle has joined #openstack-infra21:43
gordchmm... random question but did uuidgen command disappear from ubuntu trusty recently?21:44
clarkbjohnsom: typically you can just push the change for the acls by editing openstack-infra/gerrit/acls/ and pushing to gerrit21:45
gordchttp://logs.openstack.org/63/293463/1/gate/gate-telemetry-dsvm-integration-aodh/1d6b6e6/logs/devstacklog.txt.gz#_2016-03-16_19_38_36_94621:45
clarkbgordc: we recently just switched the underlying node type, it is possible the uuidgen was present in the other images but not ubuntu-trusty21:45
clarkbgordc: ideally with devstack jobs you install any dep via devstack21:45
*** matrohon has quit IRC21:45
*** csaba has joined #openstack-infra21:46
gordcclarkb: i see. i thought devstack was installing it already but i guess not.21:46
*** darren_afk is now known as darrenc21:46
*** ddieterly[away] is now known as ddieterly21:47
*** achanda has joined #openstack-infra21:47
openstackgerritClark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty  https://review.openstack.org/29371921:48
*** sigmavirus24 is now known as sigmavirus24_awa21:48
clarkbgordc: we can in theory add things like that to the image but if it isn't needed to bootstrap jobs then it is probably best placed in devstack's deps list21:48
*** annegentle has quit IRC21:48
johnsomclarkb Got it.  Thanks!21:48
gordcclarkb: will do. taking a look now. thanks for pointer21:49
clarkbjohnsom: sorry that path isn't correct, openstack-infra/project-config/gerrit/acls is better21:49
johnsomYeah, I knew what you meant and found it21:49
clarkbjohnsom: but ya you can edit the acl file directly and when approved the project acls are updated21:49
clarkbkk21:49
*** automagically_ has joined #openstack-infra21:50
*** smarcet has quit IRC21:50
*** piet has quit IRC21:50
thiagopGuys, I'm using puppet-openstackci to build a third party ci environment here but everytime I "puppet apply", the zuul-merger forgets gerrit's fingerprint and cannot clone the repository21:50
thiagopam I missing something?21:51
openstackgerritMichael Johnson proposed openstack-infra/project-config: neutron-lbaas-dashboard project is missing perms  https://review.openstack.org/29373121:53
*** rockyg has quit IRC21:53
clarkbthiagop: if I had to guess puppet is trying to add a known hosts entry and doesn't have the appropriate data to do so causing it to clear out anything that was there previously21:53
asselinthiagop, you should be able to set it in the common.yaml heira keys21:53
thiagoprigh clarkb asselin, the default heira key isn't the same as mine21:54
asselinclarkb, that said, a lot of poeple are complaining about this. ^^ Can you look at  https://review.openstack.org/#/c/266041/21:54
*** abregman has quit IRC21:55
asselinthiagop, can you comment on ^^21:55
*** _ari_ has quit IRC21:55
clarkbasselin: it has a -1 from BobBall_AWOL21:55
*** alex_xu has quit IRC21:56
jeblaircan we put that in public hiera yet?21:56
asselinyeah...he needs an extra newline...but why is that change needed?21:57
*** mriedem has quit IRC21:57
*** alex_xu has joined #openstack-infra21:58
*** sambetts|afk has quit IRC21:59
*** mrmartin has quit IRC21:59
*** sridhar_ram1 has joined #openstack-infra22:00
*** ybathia has quit IRC22:00
openstackgerritClark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty  https://review.openstack.org/29371922:01
*** sridhar_ram has quit IRC22:01
jeblairclarkb: in that change, why are you making 2 job groups instead of sticking with 'puppet-beaker-jobs'?22:01
clarkbjeblair: because we have to keep ostype for the jobs beaker config stuff and it does a full join on two parameters22:02
*** sambetts has joined #openstack-infra22:02
clarkbjeblair: if the beaker config is cleaned up we can go back to one job group but that requires updating every puppet project :/22:02
jeblairclarkb: gotcha, thanks22:02
*** tonytan_brb has quit IRC22:03
*** annegentle has joined #openstack-infra22:03
jeblairclarkb: 'dsvm' being special in job names was in openstack_functions.py, right?22:03
clarkbjeblair: correct, it is no longer required22:03
jeblaircool22:03
clarkbjeblair: pabelanger and AJaeger mentioned that as a cleanup for the future on the other jobs so I was just getting ahead of it here22:03
jeblairthought so, just wanted to double check22:04
jeblairclarkb: what's a future cleanup?22:04
clarkbjeblair: on the regular dsvm jobs removing dsvm from all the job names22:04
jeblairyeah ++22:04
craigeo/22:05
*** alex_xu has quit IRC22:06
jeblairclarkb: i'm +2 on your change.  further review from me is unlikely to produce anything that the tests won't catch unless you do more major refactoring, so feel free to carry that forward as you see fit.  :)22:06
clarkbjeblair: roger, I am leaning on the tests quite a bit myself22:07
jeblairwe have a lot of ready nodes; did we merge more job node changes?22:07
*** piet has joined #openstack-infra22:07
jeblair(i thought this was the only one in flight, but maybe i missed something)22:08
*** rfolco has joined #openstack-infra22:08
*** davideagnello has quit IRC22:08
*** davideagnello has joined #openstack-infra22:09
*** flepied has quit IRC22:09
NepocJust for sanity sake, is there an updated version of pypi-mirror that will work with pip 7.1.2 and also handle requirements-constraints.txt22:09
*** odyssey4me has quit IRC22:09
*** alex_xu has joined #openstack-infra22:09
*** odyssey4me has joined #openstack-infra22:09
openstackgerritClark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty  https://review.openstack.org/29371922:10
clarkbjeblair: ^ two more missed job name edits22:10
clarkbjeblair: its the devstack-trusty to ubuntu-trusty switch22:10
*** rfolco has quit IRC22:11
jeblairoh, we merged that earlier and this is a followup22:11
jeblairgot it22:12
*** roxanagh_ has quit IRC22:12
jeblairand yeah, they are largely devstack-trusty nodes22:12
clarkbNepoc: we no longer use pypi-mirror and its mostly unmaintained at this point. I don't know if anyone has it working22:12
Nepocclarkb: okay well I've got it working with 7.1.2... just didn't really want to deal with the second part22:12
johnsomAnother question, the changes to the image today, is it expected that the "software-properties-common" package is no longer installed on the systems?22:12
*** MarkAtwood has joined #openstack-infra22:12
*** MarkAtwood has quit IRC22:13
*** vgridnev has quit IRC22:13
johnsomThis seems to have broken the neutron-lbaas gates22:13
clarkbNepoc: we use bandersnatch to mirror pypi and `pip wheel` to make wheels22:14
Nepocclarkb: Can bandersnatch do a partial mirror, when I looked at it I had to make a 500GB mirror22:14
*** pcm_ has joined #openstack-infra22:14
*** mattymo_ has joined #openstack-infra22:15
*** SergeyLukjanov2 has joined #openstack-infra22:15
*** roxanagh_ has joined #openstack-infra22:15
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again  https://review.openstack.org/28891522:16
*** SergeyLukjanov has quit IRC22:16
*** mattymo has quit IRC22:16
*** pc_m has quit IRC22:16
*** SergeyLukjanov2 is now known as SergeyLukjanov22:16
*** dims_ has joined #openstack-infra22:16
*** acruz has joined #openstack-infra22:16
ianwjohnsom: i don't think it's expected, but we should investigate to see where the right place to put it back in is ...22:17
clarkbNepoc: I don't think it will make a partial mirror22:17
*** tpsilva has quit IRC22:17
*** erlon has quit IRC22:17
johnsomFYI, it is present in this older build: http://logs.openstack.org/62/282062/3/check/gate-neutron-lbaasv2-dsvm-healthmonitor/5d75ee4/logs/dpkg-l.txt.gz22:17
johnsomAnd missing now: http://logs.openstack.org/28/164828/34/check/gate-neutron-lbaasv2-dsvm-healthmonitor/56e68ad/logs/dpkg-l.txt.gz22:18
clarkbjohnsom: they are starting from two different bases so it is entirely possible. We are trying to push towards being explicit about deps rather than pulling things in by happenstance22:18
Nepocclarkb: Yeah I had a feeling. I guess I'll be updating pypi-mirror.22:18
*** gordc has quit IRC22:18
*** dims has quit IRC22:18
ianwjohnsom: what exactly is it?  i'm wondering if it's a red-herring when just comparing the package lists.  what's the failure?22:19
ianwahh -> sudo: add-apt-repository: command not found  i bet22:20
johnsomBingo22:20
johnsomYeah, neutron-lbaas is adding the trusty-backports repo22:20
*** yamahata has joined #openstack-infra22:21
ianwjohnsom: well, probably your plugin should hvae that in it's package list22:21
zigofungi: The image got built, I believe without an error, could you make it downloadable for me please?22:21
*** pvaneck has quit IRC22:21
johnsomianw I am interested.  Where does that live and does it support backports?22:21
*** ybathia has joined #openstack-infra22:22
clarkbzigo: I don't see a debian image built yet22:22
zigoclarkb: There's the log file at http://nodepool.openstack.org/dib.debian-jessie.log22:22
ianwjohnsom: as part of your devstack plugin22:22
zigoclarkb: Where did it go then?22:22
clarkbzigo: that build failed22:23
zigoclarkb: Oh !22:23
clarkbdue to the lack of iamge types I think there is a config update we need to restart the build service to pick it up22:23
zigoclarkb: How do you know it failed?22:23
zigoclarkb: Did I miss an error in the log?22:23
zigoAh yeah...22:24
clarkbzigo: exit 122:24
zigofatal: failed to copy file to '/opt/dib_tmp/image.YefSYwgq/mnt/opt/git/openstack/heat-specs/.git/objects/2a/5b47527d017fb4aee060c82a9c37695ff90efd': No space left on device22:24
zigoclarkb: Is there somewhere we can increase the space?22:24
clarkbzigo: thats a side effect https://review.openstack.org/#/c/293651/ is the fix I will review that now22:25
*** annegentle has quit IRC22:26
ianwjohnsom: http://docs.openstack.org/developer/devstack/plugins.html#system-packages in particular22:26
clarkbzigo: basically dib was using a tmpfs in memory that wasn't big enough and it did that due to arg parsing due to us not providing the image type which requires provider info22:26
zigoclarkb: So, basically, not giving it enough RAM?22:27
clarkbzigo: no, its due to us not passing the correct args because we didn't have enough configuration info22:27
johnsomianw Oh cool.  I will fix that now.  Too bad it doesn't seem to account for the backport repos, but this will work to get the tool back.  Thanks!22:27
clarkbthe change I linked adds the necessary info22:27
zigoOk.22:28
clarkbjohnsom: what packages do you need out of the backports?22:28
johnsomclarkb haproxy22:28
zigoThanks for the help.22:28
clarkbjohnsom: you need a newer haproxy?22:29
clarkbI didn't realize haproxy changed all that much22:29
johnsomSadly yes22:29
johnsomHA, outside of ubuntu they have done two major releases since the version in trusty.22:29
johnsomYeah, it gets updated.  We are good at finding the bugs too.  grin.22:30
*** Thelo has quit IRC22:30
clarkbI dunno we use it too and from release to realse we never change anything and it continue sto work22:30
clarkb*distro release to distro release22:30
*** ddieterly has quit IRC22:31
pabelangerfungi: re: gerrit admin. Sure, if people are willing to mentor me22:31
*** chlong has joined #openstack-infra22:32
pabelangerianw: sure, I know the image build fine, I can dig more into why it is not registered if you don't see why22:33
ianwpabelanger: well either it's not uploaded, or it's uploaded and doesn't work.  i'm hoping the first22:33
pabelangerjeblair: nice, looking22:33
openstackgerritMerged openstack-infra/project-config: Add debian-jessie to providers  https://review.openstack.org/29365122:33
pabelangerianw: I don't have access to rax yet, but I can check nodepool debug logs22:34
pabelangerianw: or even retry the upload22:34
clarkbpabelanger: you can find the rax cloud stuff in /etc/openstack/clouds.yaml22:35
pabelangerzigo: clarkb: review 293651 should address the build failure22:35
clarkbpabelanger: actually you might need all-clouds.yaml but ya22:35
pabelangerclarkb: okay, let me try that22:35
pabelangerclarkb: do people source it or just use the webui?22:35
clarkbpabelanger: depends on the operation. I do my best to use openstackclient out of a venv for most things, but some stuff like trove i end up webui ing it22:36
*** dizquierdo has joined #openstack-infra22:36
pabelangerclarkb: right, I'll start with that22:36
clarkbpabelanger: the *ci account is for long lived servers and *jenkins is what nodepool uses22:37
pabelangerclarkb: thanks22:37
openstackgerritIan Wienand proposed openstack-infra/system-config: Cleanup of nodepool builder logging  https://review.openstack.org/27598222:38
*** dalgaaf has quit IRC22:38
clarkbI can restart the nodepool builder to pick up that config change in a bit22:39
ianw^ get in quick before the bitrot22:39
pabelangerclarkb: sure works for me22:39
clarkband maybe get that new log config22:39
pabelangerianw: Ya, I want to gate on that some how. Need to think about it22:39
pabelangerianw: we'd have to move nodepool-builder.logging into project-config I think22:40
ianwpabelanger: it wouldn't be too hard to make the proposal bot do updates, i just wasn't sure if the rate of change justifies it22:40
clarkbonce we get away from bare and devstack the rate of update should be once every new distro release ~6 months?22:41
pabelangerianw: ya, thought about that too22:41
pabelangerclarkb: hopefullly22:41
*** sridhar_ram1 has quit IRC22:41
clarkbfedora is 6 months, centos and ubuntu every 2 years or so and debian is also ~2 years iirc22:41
*** edmondsw has quit IRC22:43
*** rcrit has quit IRC22:43
ianwpabelanger: it's still in system-config because it's an erb and gets the paths put into it.  we seem to have this conversation about moving it periodically22:44
pabelangerianw: so, according to nodepool logs, centos-7 should be ready in rax-dfw. Seems to fail in other zones22:44
pabelangerianw: confirming with rax now22:44
clarkbthere are 3 uploads going on right now, can restart builder when they are done22:46
pabelangerclarkb: where did you see that?22:46
pabelangerhaven't figured out the new nodepool-builders yet22:46
*** doug-fis_ has joined #openstack-infra22:47
clarkbpabelanger: nodepool image-list | grep building22:48
*** yamahata has quit IRC22:48
openstackgerritBen Nemec proposed openstack-infra/project-config: Add puppet-pacemaker to tripleo project list  https://review.openstack.org/29374922:48
pabelangerclarkb: Ah, right22:48
clarkbnow 222:48
*** IlyaG has joined #openstack-infra22:48
pabelangeryup22:48
ianwpabelanger: clarkb has managed to share the uploaded image with me before.  if you want to do that, i can boot it and make sure it's ok.  i don't see why it would be different to my test builds, but something always manages to be different22:48
clarkbianw: pabelanger oh for that I had to use the web ui22:49
clarkbyou log in and find the image in the image list in the ui and there is a share option that takes a tenant id iirc22:49
pabelangerokay, I can try that22:49
clarkband its region specific22:49
*** Jeffrey4l has joined #openstack-infra22:49
*** automagically_ has quit IRC22:50
*** doug-fish has quit IRC22:50
clarkbI am not sure if that is something glance does or if it is a rax thign22:51
*** doug-fis_ has quit IRC22:51
*** xyang1 has quit IRC22:52
ianwpretty sure it's a rax thing, i think glance just has a public/private flag22:54
*** asettle has quit IRC22:54
clarkbapparnetly none of the lcouds let us set the public flag :)22:55
clarkbotherwise we could probably just set that22:55
pabelangerokay, rax-iad just finished upload for centos-722:55
pabelangerlet me see if I can share that one22:55
*** mriedem has joined #openstack-infra22:57
*** yamahata has joined #openstack-infra22:57
ianwi'll also kick of a local build, i haven't tested in a few days22:57
*** rbrndt has quit IRC22:58
clarkbstill one image uploading then I can restart the builder22:59
ianwclarkb: yeah, you have to go poking around in glance policy.json to enable it (as wfoster figured out) so probably not common22:59
*** rbrndt has joined #openstack-infra22:59
*** tphummel has quit IRC22:59
*** dingyichen has joined #openstack-infra22:59
*** tphummel has joined #openstack-infra22:59
*** rbrndt has quit IRC23:00
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add octavia experimental tempest job  https://review.openstack.org/28487523:00
*** karlamrhein has joined #openstack-infra23:02
*** automagically_ has joined #openstack-infra23:02
*** davideagnello has quit IRC23:02
*** automagically_ has left #openstack-infra23:03
pabelangerstill getting some ubuntu-trusty images being launched, JJB must be out of sync still23:03
clarkbyou mean devstack-trusty?23:04
*** markmcd1 has quit IRC23:04
clarkbhttps://review.openstack.org/#/c/293719/ should help fix that23:04
*** davideagnello has joined #openstack-infra23:04
clarkbbut its a big change and lots of reorg :/23:04
SamYapleasking for an ack on my new regex for the kolla gates https://review.openstack.org/#/c/293021/23:05
SamYapleshould be less infra-taking-down'ish now23:06
pabelangerclarkb: ya, devstack-trusty23:06
*** sridhar_ram has joined #openstack-infra23:09
*** adrian_otto has quit IRC23:10
clarkbok builder is restarted and the -t flag to dib is filled23:10
pabelangerclarkb: okay, if we want to land puppet change, I can approve now.  Otherwise, we can push to tomorrow23:11
*** automagically has left #openstack-infra23:11
*** krotscheck has joined #openstack-infra23:11
clarkbuh its up to you I guess, there may be fallout just because its complicated enough to maybe miss things23:12
*** sdague has quit IRC23:14
*** annegentle has joined #openstack-infra23:14
*** dimtruck is now known as zz_dimtruck23:15
thiagopthe problem now is that nodepool is creating the nodes but jenkins are not using them to run the jobs23:15
pabelangerthiagop: sound like a label issue23:16
pabelangerclarkb: sure, let me ping EmilienM23:16
thiagoppabelanger: is it normal that nodepool keep building and deleting instances all the time?23:17
thiagoppabelanger: even when they're not consumed by jenkins?23:17
pabelangerthiagop: so, in that case, it might be because nodepool cannot SSH into the instance23:18
*** adrian_otto has joined #openstack-infra23:18
pabelangercheck the logs23:18
*** yamahata has quit IRC23:18
EmilienMI'm on my phone, what's up?23:18
*** piet has quit IRC23:18
*** bryan_att has joined #openstack-infra23:18
*** fawadkhaliq has joined #openstack-infra23:19
pabelangerEmilienM: when you get back to a computer, review 293719 is a reorg of JJB template for puppet23:19
*** annegentle has quit IRC23:19
pabelangerEmilienM: have a look over before I approve23:19
thiagoppabelanger: keeps repeating that: http://paste.openstack.org/show/490818/23:19
EmilienMOK23:19
*** adrian_otto has quit IRC23:20
pabelangerthiagop: what does debug.log show?23:20
*** samueldmq has joined #openstack-infra23:20
thiagoppabelanger: the paste was of debug.log from nodepool23:20
*** anteaya has joined #openstack-infra23:21
ianwpabelanger: ok, so it boots, but has no network23:21
*** adrian_otto has joined #openstack-infra23:21
*** zaro has joined #openstack-infra23:22
pabelangerianw: eep23:22
fungiokay, back now, lots of highlights. will address them in chronological order23:23
pabelangerthiagop: if nodepool build an image, it will then launch it which, your output reports23:23
EmilienMPaul, i checked at the patch, too big for my phone. Will look later tonight23:23
pabelangerthiagop: then, nodepool with SSh into the instance before jenkins, if that failed, it gets deleted and restart the process23:23
pabelangerEmilienM: Ya, big churn23:23
ianwpabelanger: alright, i'm going to try some mad hacking skills to see if i can get in23:24
*** akshai has quit IRC23:24
pabelangerianw: hack the gibson23:24
bkeroCareful in front of greghaynes23:25
*** karlamrhein has quit IRC23:25
pabelangerokay, back to debian jessie to see the progress23:26
fungitonyb: nibalizer: awesome. as the commit message (and a number of code comments) implies, the goal is for anyone to be able to run that eventually, but for now because we're not yet running with a particular gerrit patch backported that script only works for gerrit admins. in just over 30 minutes it should be safe to fire it up and take it for a spin. i've done rigorous testing over recent months (as23:27
fungirecently as several times today too) and am quite cnofident it's in good working order23:27
*** adrian_otto has quit IRC23:29
*** adrian_otto has joined #openstack-infra23:29
jheskethjeblair: ping23:29
tonybfungi: I've only skimmed it but I don't see handling for the "Stable Branch Maintenance" project.23:30
ianwpoll : what is worse -> the stupid f-ing rax java console app or systemd?23:31
clarkbheh I have actually gotten systemd to work23:33
*** adrian_otto has quit IRC23:33
*** markmcd has joined #openstack-infra23:34
tonybfungi: which is probbaly moot given that there are 0 candidates23:35
fungitonyb: see the comment block near the top23:35
fungitonyb: it's handled "specially" for now23:35
*** cloudtrainme has quit IRC23:35
*** mriedem1 has joined #openstack-infra23:35
fungiby rerunning with a different output directory, turning off extra-atcs and adding a branch regex23:35
*** SumitNaiksatam has quit IRC23:35
fungicomplicated, but i tried to document it as thoroughly as possible23:36
*** achanda has quit IRC23:36
tonybfungi: Thanks!  I'm not sure how I missed that.23:36
*** tsg_ has quit IRC23:36
fungiwhich is how i handled it with the predecessor script as well, but it could stand some better encoding23:36
fungifor next cycle i'd like to integrate that and also zanata querying, if possible by then23:36
*** dizquierdo has quit IRC23:37
*** mriedem has quit IRC23:37
*** asettle has joined #openstack-infra23:37
fungibecause the stable branch electorate is so "special" i tested that again as recently as today23:37
tonybfungi: so querying zanata for APC status rather then relying on extra-atcs?23:37
fungiyep23:38
*** denisra has quit IRC23:38
fungias for stable branch, they are the second largest electorate (right after infra)23:38
tonybfungi: yeah it's very special ;P  I knew you had a script from the first special election23:38
fungii was surprised initially, but then realized i shouldn't be23:38
fungilots of people do backports23:38
fungineutron has the third largest, then docs, then nova, then qa23:38
tonybfungi: Yeah23:38
fungitc electorate is well over 3k now23:39
*** smarcet has joined #openstack-infra23:40
mgagnefungi: I think the issue is really related to expiration of token. I see 4 images uploaded today. 1) 00:05:02 (success) 2) 00:32:03 (failed) 3) 00:32:54 (success) 4) 00:33:58 (failed). Failure occurred as followed 2) 01:11:55 4) 01:12:49. I suspect token used for 1) was also used for 2-3-4.23:40
fungiother things which we could improve with the script but which i'd want to put before the tc are: scraping co-authored-by and using git author (or committer?) instead of gerrit change owner23:40
tonybfungi: wow23:40
fungimgagne: makes sense. i wonder if any of this changes with the pending patches to switch to using more shade in nodepool23:41
mgagnewhat I don't fully understand is *when* is the token created. Obviously not before an operation. If we bump to 24h, we guaranty use that it won't happen again if image is uploaded at 23:4523:41
mgagnewhat*23:41
*** chlong has quit IRC23:41
mgagnewhat guaranty us*23:41
tonybfungi: mikal had a script to do the co-authored-by thing and output  an extra-atcs hunk, which we failed to run23:41
*** asettle has quit IRC23:42
tonybfungi: I'd think both of those changes would be good, but agree need TC buy-in23:42
fungitonyb: yep, it's not hard, and the gerrit api makes the commit message available so the script could be tweaked to do that pretty easily23:42
*** doug-fish has joined #openstack-infra23:43
fungion the other hand we've seen particularly shady stuff with one-line changes that have several co-authored-by lines, so i worry that makes this even easier to game23:43
*** adrian_otto has joined #openstack-infra23:43
thiagoppabelanger: I think the problem is with the image23:43
fungimostly it seems to be around people trying to get free summit passes though, not extra votes in elections (yet anyway)23:43
tonybfungi: I assume that the hardest part is mapping the email addresses when people change23:43
* tonyb is surprised that people do that23:44
thiagoppabelanger: It's kinda getting stuck at some point... I launched an instance with the same image by myself and it's stuck at the same point23:44
fungitonyb: yes, spotting and merging duplicate accounts is complicated. but for summit passes because we do several batches being able to track what people's addresses change to so that we don't send them additional passes is hard23:44
*** annegentle has joined #openstack-infra23:45
pabelangerthiagop: could be, in that cause nodepool cannot ssh, and it likely restarts23:46
fungifor author and co-authored-by there is also the added complication of them not requiring a gerrit account, so it'll probably need an account lookup pass for those23:46
pabelangerthiagop: which cloud are you using?23:46
thiagoppabelanger: is there a timeout for this process?23:46
thiagoppabelanger: one of the clouds in our lab at UFCG, Brazil23:47
tonybfungi: yeah I was thinkign about that side of things as well23:47
pabelangerthiagop: yes, there are some timeouts for nodepool.  eg: boot-timeout and launch-timeout23:47
pabelangerboot it 1hr and launch is 1min23:48
pabelangerIIRC23:48
*** eil397 has quit IRC23:49
thiagoppabelanger: so I think the only alternative here is update the image with nodepool image-update and wait that the image turns out to be good?23:49
*** acruz has quit IRC23:51
pabelangerthiagop: well, if the image is uploaded to the cloud, you can try and manually launch it and debug23:51
*** annegentle has quit IRC23:52
pabelangerthiagop: usually issues are networking if the instance doesn't boot properly23:52
pabelangerthiagop: but I think I seen an IP address on your debug.log23:52
*** gildub has joined #openstack-infra23:52
fungimgagne: so the timing would imply that we get the token immediately prior to the first upload and reuse it for the remaining ones23:53
mgagnefungi: that's my hypothesis23:53
clarkbwe get a new token when we need a new one23:53
clarkbkeystoneauth does it all for us23:53
*** piet has joined #openstack-infra23:53
mgagnefungi: I'm trying to see if something logs the actual token so I can get the creation or expiration time23:53
*** gyee has quit IRC23:53
thiagoppabelanger: I think I'm too tired to make sense of these logs today... :/23:53
*** salv-orl_ has joined #openstack-infra23:54
mgagneclarkb: what if it expires 30s later? glance needs the token to still be valid after the upload is completed23:54
pabelangerthiagop: fresh eyes always help23:55
thiagoppabelanger: so I'll call it a day, thank you for the help23:56
clarkbmgagne: I am not sure, I know it renews the token if it expires soon but don't know what soon is23:56
mgagnewe managed to replicate a similar situation by uploading an image that would take more than 1h to complete. result: upload fails with 500 error, image stays in saving state23:56
clarkbmgagne: this sounds like a flaw in openstack23:56
clarkbif the token is valid on the initial request it should continue to be valid23:56
clarkbthrough the entirety of htat request23:56
mgagneclarkb: it has already been taken care of in mitaka by using trusts. we don't run mitaka23:56
mgagneclarkb: not if the service uses the user token to contact another service after the upload23:56
clarkbright that sounds like a glance bug23:57
*** salv-orlando has quit IRC23:57
clarkbI mena we can work around it if we need to, but this is something that the apis should handle imo23:57
mgagneclarkb: but there is little we can do on our side if caller still uses a token expiring in 30m when upload takes 40m to complete23:58
*** ybathia has quit IRC23:58
clarkbmgagne: correct but glance shouldn't fail in that case because the token was valid when the request started23:58
clarkbthe idea that the token has to valid after is silly beucase we can't change anything at that point23:58
mgagneI don't think we disagree here23:58
mgagnethere is little I can do to fix that except telling consumers to renew their token if they are about to perform a long request/task23:59
*** roxanagh_ has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!