| *** mriedem has joined #openstack-infra | 00:01 | |
| *** sdake has quit IRC | 00:01 | |
| *** dchen has joined #openstack-infra | 00:03 | |
| *** Swami has quit IRC | 00:03 | |
| *** dchen has quit IRC | 00:03 | |
| *** pahuang has quit IRC | 00:04 | |
| *** camunoz has quit IRC | 00:04 | |
| *** aeng has quit IRC | 00:04 | |
| *** sflanigan has quit IRC | 00:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add zookeeper test infrastructure https://review.openstack.org/293176 | 00:05 |
|---|---|---|
| openstackgerrit | Merged openstack/diskimage-builder: Don't cache debootstrap rootfs by default https://review.openstack.org/280536 | 00:06 |
| fungi | ianw: clarkb: if we merge 264726 we're going to need another manual nodepool image builder restart for now, yeah? | 00:06 |
| zigo | fungi: clarkb: Is it possible for me to actually download an image built by infra? | 00:07 |
| jeblair | clarkb, greghaynes: i realize i'm getting ahead of the spec, but i've been exploring zookeeper to validate my understanding before formally proposing the spec, and i wanted to make sure it was testable. so i went ahead and wrote 293176. | 00:07 |
| clarkb | fungi: yes | 00:07 |
| fungi | zigo: not without help at the moment, and also these images bake specific access keys into them so you'd need to modify it before boot to be able to log in anyway | 00:07 |
| jeblair | clarkb, greghaynes: i have also put the zk spec on the next infra meeting agenda | 00:07 |
| greghaynes | jeblair: nice! | 00:07 |
| *** dims has joined #openstack-infra | 00:08 | |
| *** sflanigan has joined #openstack-infra | 00:08 | |
| clarkb | fungi: zigo or use metadata service to slip your key in | 00:08 |
| fungi | oh, true | 00:08 |
| *** sdake_ has quit IRC | 00:08 | |
| fungi | i keep forgetting about the metadata service since we shun it like a plague | 00:08 |
| SergeyLukjanov | jeblair hey, zk == zookeeper? | 00:08 |
| zigo | Ok. | 00:08 |
| clarkb | well config drive works too but that general thing | 00:08 |
| openstackgerrit | Merged openstack-infra/system-config: Add #osic to meetbot https://review.openstack.org/291798 | 00:08 |
| fungi | zigo: on the other hand, once we get one built and booted i can hold it and add your key so you can log in and inspect how it comes up | 00:09 |
| jeblair | greghaynes: i still want to sync up with you on the last little bit about whether the builder should have an internal scheduler; i know you're busy, so we can do that whenever. i figure we're largely in agreement so we probably don't have to block on that. | 00:09 |
| jeblair | SergeyLukjanov: yes | 00:09 |
| fungi | zigo: or rather, inspect the result once it's up | 00:09 |
| prometheanfire | greghaynes: ianw: how to non-voting job? | 00:09 |
| SergeyLukjanov | jeblair oh, I see the change, actually if you're using kazoo there is a fake zk impl | 00:09 |
| zigo | fungi: I was thinking of getting the image, so I could re-design the build tool I have to adapt it to the image. | 00:09 |
| jeblair | SergeyLukjanov: oh neat! | 00:09 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix issue with non-ascii characters in get_job_md5 https://review.openstack.org/283050 | 00:09 |
| zigo | Definitively, having a copy of the image would help me. | 00:09 |
| SergeyLukjanov | jeblair lemme find | 00:10 |
| *** ianw has quit IRC | 00:10 | |
| *** MarkAtwood has joined #openstack-infra | 00:10 | |
| prometheanfire | zigo: I'm actually going to need to do the same thing eventually (infra built images, etc) | 00:10 |
| greghaynes | jeblair: Ya, so what I am trying to ague against really is trying to make the scheduler be shared-nothing. It can still be internal to the builder | 00:10 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Allow using lockfile per jenkins master https://review.openstack.org/292405 | 00:11 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix ircbot's any-failure parameter documentation https://review.openstack.org/271707 | 00:11 |
| greghaynes | jeblair: I worry that doing a shared-nothing scheduler is harder with the upside that itll scale a ton better, but we dont actually need that here | 00:11 |
| *** sdake has joined #openstack-infra | 00:11 | |
| jeblair | greghaynes: by scheduler, do you mean the nodepool daemon? | 00:11 |
| jeblair | greghaynes: the thing that decides how many nodes to launch? | 00:11 |
| greghaynes | jeblair: nope, the builder daemon | 00:12 |
| SergeyLukjanov | jeblair there are helpers available for zk testing - https://kazoo.readthedocs.org/en/latest/testing.html and zake is the lib providing mocks for kazoo, it actually have the same API as kazoo https://pypi.python.org/pypi/zake/ and it's supporting mostly all usually needed stuff | 00:12 |
| *** Jeffrey4l has joined #openstack-infra | 00:12 | |
| *** yamamoto_ has quit IRC | 00:12 | |
| SergeyLukjanov | (btw zake was written by harlowja AFAIK) | 00:13 |
| jeblair | SergeyLukjanov: awesome, i missed that; i should definitely look into that... | 00:13 |
| *** adrian_otto has quit IRC | 00:13 | |
| SergeyLukjanov | jeblair it's used in ceilo and tooz | 00:13 |
| *** gildub has joined #openstack-infra | 00:13 | |
| jeblair | SergeyLukjanov: the main things i want to be sure of are: 1) we get the same behavior (eg, watches on nodes, etc). 2) parallel tests can run without interfering with each other. | 00:13 |
| jeblair | SergeyLukjanov: i looked at tooz and saw that it set up a zookeeper before running tox. so i based that change on that idea, except that i did it inside of the unit test framework so we get zookeeper-per-test rather than a single one | 00:14 |
| *** armax has joined #openstack-infra | 00:14 | |
| *** armax has quit IRC | 00:14 | |
| jeblair | greghaynes: hrm, i'm not seeing how it's really that much harder... | 00:15 |
| SergeyLukjanov | jeblair I think harlowja could comment on it better, but AFAIK for unit tests it's more than enough and for the functional tests you're usually don't want to mock the db, so, it'll not replace the real zk for the functional tests | 00:15 |
| *** SumitNaiksatam has quit IRC | 00:15 | |
| jeblair | SergeyLukjanov: yeah, all of our testing here will basically be functional testing | 00:16 |
| *** Qiming has quit IRC | 00:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue https://review.openstack.org/293178 | 00:16 |
| SergeyLukjanov | jeblair it's absolutely safe to dedicated the single prefix path for each unit test, not necessarily to run zk per unit test | 00:16 |
| EmilienM | pabelanger, _ody, crinkle: when you have time, feel free to review: https://review.openstack.org/293178 | 00:16 |
| jeblair | greghaynes: if the scheduler is external, it's basically just cron. it doesn't seem very hard to do the same thing internally... | 00:17 |
| *** pahuang has joined #openstack-infra | 00:17 | |
| jeblair | SergeyLukjanov: i thought about that, but then we would have to have an option to nodepool to specify the zookeeper root... | 00:17 |
| SergeyLukjanov | jeblair when I was doing the ZK stuff, I was using some prefix like /tests/<uuid>/<unit test full name> as a root for each unit test with create/cleanup in setup/teardown | 00:18 |
| jeblair | SergeyLukjanov: maybe we should... is that a best practice? | 00:18 |
| SergeyLukjanov | jeblair you probably anyway want to have one | 00:18 |
| SergeyLukjanov | jeblair I would say yes | 00:18 |
| fungi | zigo: well, if nothing else, it's probably easy enough to mount the image on a loop block device | 00:18 |
| SergeyLukjanov | jeblair you need at least 3 nodes Zk cluster for HA, so, you'll be end up with sharing it between different services | 00:18 |
| zigo | fungi: Yup ! :) | 00:18 |
| jeblair | SergeyLukjanov: yeah, that makes sense. | 00:19 |
| SergeyLukjanov | jeblair is it the mentioned spec https://review.openstack.org/#/c/278777/3/specs/nodepool-zookeeper.rst ? | 00:19 |
| fungi | zigo: i can totally manually put the image in a downloadable location temporarily for you to grab once it's building successfully on nodepool.o.o | 00:19 |
| zigo | fungi: kpartx is my friend for such things ! :) | 00:19 |
| jeblair | SergeyLukjanov: that's the one. sounds like i should add a note about the root being configurable. | 00:20 |
| zigo | fungi: Thanks, that will be helpful indeed. | 00:20 |
| greghaynes | jeblair: Internal vs external isnt really the concern I have, its whether the per image scheduling algorithm has to be written to be fully distributed/shared-nothing or whether it can be written under the assumption that it is only running in one place at a given point in time. The former is always a lot harder to reason about IME | 00:20 |
| zigo | fungi: Should it wait for another core reviewer to approve? | 00:20 |
| zigo | (ie: workflow...) | 00:20 |
| greghaynes | jeblair: And since the only benefit we want from making a distributed scheduler is fault tolerance (not crazy scsalability) we can get away with not doing a fully distributed scheduler | 00:21 |
| jeblair | greghaynes: i think the algo is basically "wake up at 14:00 utc, for each image type i can build, if there is not a build in progress, try to start one" | 00:21 |
| fungi | zigo: i'm trying to figure out whether i have time to restart the image builder process (it doesn't have on the fly reconfigure support yet), and make sure i'm not stepping on the toes of anyone else troubleshooting our existing image builds | 00:21 |
| *** aeng has joined #openstack-infra | 00:21 | |
| *** camunoz has joined #openstack-infra | 00:21 | |
| zigo | fungi: Ok. I guess I'll go to sleep then. Thanks, much appreciated. | 00:22 |
| jeblair | greghaynes: the reason i keep pushing on shared-nothing for this is that the next step (alluded to in the spec, but i'm planning on writing a spec for) is to remove the rest of the sharing in nodepool. so then "where to run the build scheduler" actually becomes a difficult question :) | 00:22 |
| *** openstack has joined #openstack-infra | 00:24 | |
| jeblair | SergeyLukjanov: oh that's good to know | 00:24 |
| *** ddieterly has joined #openstack-infra | 00:24 | |
| SergeyLukjanov | clarkb yeah, I've been testing on many hundreds per seconds :) | 00:24 |
| greghaynes | jeblair: I do like the nodepoold removal btw | 00:25 |
| fungi | SergeyLukjanov: since you're retiring from being a ptl, will we be seeing more of you on infra work again? | 00:25 |
| * fungi asks hopefully | 00:26 | |
| SergeyLukjanov | fungi honestly I'm not sure, but I'm planning to work more on infra then on sahara on a free time | 00:26 |
| fungi | SergeyLukjanov: yay! ;) | 00:26 |
| jeblair | greghaynes: oh, you're suggesting any builder could run the scheduling algorithm, but only one does it one at a time, by way of a "global build scheduling lock"? | 00:26 |
| SergeyLukjanov | I'm glad to help with some stuff, but very unreliably... I can say for sure that I can help with projects renames :) | 00:26 |
| greghaynes | jeblair: exactly | 00:26 |
| greghaynes | jeblair: but not global - per image | 00:26 |
| fungi | SergeyLukjanov: well, i really hope those are increasingly rare now | 00:27 |
| greghaynes | jeblair: since some builders might not have every image | 00:27 |
| *** baoli has joined #openstack-infra | 00:27 | |
| *** fverbs has quit IRC | 00:27 | |
| jeblair | greghaynes: now i'm not seeing the difference between what you are suggesting and what i am. :) | 00:27 |
| *** mtanino has quit IRC | 00:28 | |
| greghaynes | jeblair: :) | 00:28 |
| *** Hal has joined #openstack-infra | 00:28 | |
| *** Hal is now known as Guest22575 | 00:29 | |
| *** ianw has joined #openstack-infra | 00:29 | |
| prometheanfire | greghaynes: should I add my test to https://github.com/openstack/diskimage-builder/blob/master/tests/run_functests.sh#L16 ? | 00:29 |
| *** IlyaG has quit IRC | 00:29 | |
| *** fawadkhaliq has quit IRC | 00:30 | |
| greghaynes | prometheanfire: Yes, and then add a nonvoting job which runs it | 00:30 |
| SergeyLukjanov | jeblair I have one more nit tip - if you'll be using the "lock" or "party" primitives - make sure that you're not creating any subnodes manually - it'll make acquire operations sometimes in a dead lock, I mean if you have kazoo.lock('/a/b/c'), don't create '/a/b/foobar', so, the good approach is to have lock under separated znode like '/a/b/c/.lock' | 00:30 |
| *** sheeprine has quit IRC | 00:30 | |
| ianw | freaking linode, every time they do network maintenance my host seems to disappear (and my irc bouncers with it) | 00:30 |
| greghaynes | ianw: I heard about this neat cloud software you should check out ;) | 00:30 |
| prometheanfire | greghaynes: ok, I'll start there (ammend that review) | 00:31 |
| jeblair | SergeyLukjanov: did you mean "/a/b/c/foobar" in that example? | 00:31 |
| fungi | clarkb: looks like your ansible pr is passing travesty jobs. was the plan to wait for that to trickle down from upstream before we can puppet review.o.o/git.o.o (and safely build images again), or were you thinking there's some way to run a forked version locally while we wait? | 00:32 |
| fungi | or option #3 (unspecified) | 00:32 |
| SergeyLukjanov | jeblair I mean that lock works as creating znode per each requester, so, you shouldn't mix it with other znodes | 00:32 |
| clarkb | fungi: well I have no idea how to test that properly so depends on how adventurous we want to be | 00:32 |
| fungi | heh | 00:32 |
| clarkb | fungi: in theory we can stop puppet on the puppetmaster, apply that delta to /etc/ansible/hosts/openstack | 00:32 |
| fungi | it looks like the upstream testsuite likes it, fwiw | 00:32 |
| clarkb | and see if that fixes us | 00:32 |
| *** fawadkhaliq has joined #openstack-infra | 00:33 | |
| clarkb | fungi: I don't know that upstream testsuite is testing it at all though :) its in the contrib dir | 00:33 |
| *** MarkAtwood has quit IRC | 00:33 | |
| jeblair | greghaynes: here's what's in my head: https://etherpad.openstack.org/p/ePcdXiqHOb | 00:33 |
| fungi | but i'm on the fence as to whether faster fixing is preferable to possibly breaking worse | 00:33 |
| *** pvaneck has quit IRC | 00:33 | |
| fungi | clarkb: fair point | 00:33 |
| fungi | coverage of that file... ?% | 00:33 |
| clarkb | fungi: in theory if it breaks we just remove it and rerun puppet without it | 00:34 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add testing for the Gentoo element https://review.openstack.org/290894 | 00:34 |
| fungi | if only we had a bat signal but for mordred | 00:34 |
| clarkb | fungi: so the rollbcak and go back to status quo is simple | 00:34 |
| *** sheeprine has joined #openstack-infra | 00:34 | |
| prometheanfire | greghaynes: ianw: ^ I think that's what you want | 00:34 |
| fungi | clarkb: yeah, as long as it only breaks itself and not... you know... everything | 00:35 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue https://review.openstack.org/293178 | 00:35 |
| prometheanfire | so, in project-config, I see a check named gate-dib-dsvm-functests-devstack-centos7 | 00:35 |
| clarkb | fungi: right, I think worst case we would still run puppet on the wrong host but for those few cases it shouldn't be worse than now | 00:35 |
| jeblair | SergeyLukjanov: if i understand you, i think i designed it that way in the spec; if you have time to double check that, that would be great :) | 00:35 |
| prometheanfire | where is whatever that calls so I know what I need to edit for a non-voting version for gentoo? | 00:35 |
| *** MarkAtwood has joined #openstack-infra | 00:36 | |
| clarkb | fungi: but thats based on my limited understanding of this. Which is far too complicated imo | 00:36 |
| ianw | prometheanfire: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/diskimage-builder.yaml | 00:36 |
| clarkb | I almost thing we should go to a purely uuid based system and just have one list of disables | 00:36 |
| ianw | prometheanfire: let me start something there and you can finish it, it will be easier | 00:36 |
| clarkb | input from nova less disables then run | 00:36 |
| clarkb | right now we have like 5 files that all interact on this its pretty crazy | 00:37 |
| *** erlon has quit IRC | 00:37 | |
| prometheanfire | ianw: thanks | 00:37 |
| *** eharney has quit IRC | 00:37 | |
| prometheanfire | atm I'd just copy lines 1-27 and make it more gentoo specific | 00:38 |
| *** MarkAtwood has quit IRC | 00:38 | |
| prometheanfire | I take it the name is a match statement? | 00:38 |
| clarkb | btw I couldn't find any glaring issues with the initial devstack-trusty to ubuntu-trusty switch | 00:40 |
| clarkb | https://review.openstack.org/#/c/292573/ should be a complete siwtch for the remainder | 00:40 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: [WIP] Add non-voting diskimage-builder jobs https://review.openstack.org/293183 | 00:40 |
| clarkb | I can chop tat up into many changes but it seems easier to deal with jenkins just once so I crammed it together | 00:40 |
| ianw | prometheanfire: we should be able to modify the template to have two jobs with suffix "-nv" (that automatically makes them non-voting) | 00:41 |
| ianw | and run whatever tests we specify | 00:41 |
| prometheanfire | right | 00:41 |
| prometheanfire | ./diskimage-builder/tests/run_functests.sh gentoo/build-succeeds so at least that's a thing | 00:41 |
| prometheanfire | didn't actually test locally with just gentoo :P | 00:42 |
| ianw | prometheanfire: that's exactly why i reworked run_functests.sh to do that :) | 00:42 |
| prometheanfire | :D | 00:42 |
| *** baoli has quit IRC | 00:42 | |
| fungi | clarkb: the only reason we went with lots of changes on the bare-trusty cutover is that it was more nuanced and different jobs had different reconfiguration needs. for devstack-trusty it's pretty much just a straight up node name replacement so one big change is probably easier. either we press forward or we revert all at once | 00:42 |
| ianw | we could move logic about what to run where into dib, rather than project-config ... depends what looks neater | 00:43 |
| *** angdraug has quit IRC | 00:43 | |
| ianw | functionally, all the same | 00:43 |
| prometheanfire | I kinda like test run logic for infra existing in project-config | 00:43 |
| prometheanfire | rather then the endpoints (in this case dib) | 00:44 |
| eliqiao | hi AJaeger there? | 00:44 |
| *** moravec has quit IRC | 00:45 | |
| clarkb | eliqiao: I doubt it, he is in europe so this is very late for him | 00:45 |
| *** dims has joined #openstack-infra | 00:45 | |
| clarkb | eliqiao: if you have a question you can probably ask the channel | 00:45 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 00:45 |
| eliqiao | clarkb: oh, thanks. | 00:45 |
| eliqiao | I have a question about, https://review.openstack.org/#/c/292687/ AJaeger post comments, just want to make sure I understand him correctly before I do respin again. | 00:46 |
| prometheanfire | more reviews on bumping libvirt-python in liberty would be appreciated https://review.openstack.org/#/c/291859/ | 00:47 |
| clarkb | eliqiao: basically keep the python3-jobs from https://review.openstack.org/#/c/292687/2/zuul/layout.yaml | 00:47 |
| *** tphummel has quit IRC | 00:47 | |
| *** zz_dimtruck is now known as dimtruck | 00:49 | |
| greghaynes | jeblair: similarly - http://paste.ubuntu.com/15398740/ | 00:50 |
| greghaynes | jeblair: I actually like your psuedocode more | 00:51 |
| greghaynes | jeblair: so lets do yours | 00:51 |
| jeblair | greghaynes: okay, cool. if it starts to go south in implementation, i think it will be simple to change course. :) | 00:52 |
| greghaynes | jeblair: ya | 00:52 |
| *** afazekas has quit IRC | 00:53 | |
| *** Jeffrey4l has quit IRC | 00:53 | |
| *** gyee has quit IRC | 00:54 | |
| ianw | 12.158g | 526:08.80 | python -m subunit.run discover -t ./ nodepool/tests/ --load-list /tmp/tmperG9So <- so that's why i've been wondering why my vm was running a bit slow. stray subunit.run from nodepool testing last week! | 00:54 |
| *** docaedo has quit IRC | 00:55 | |
| prometheanfire | lol | 00:55 |
| jeblair | isn't there a timeout? | 00:55 |
| ianw | jeblair: i would have ctrl-c'd it when we were looking at that mysql user stuff last week. clearly it doesn't get everything, and gets in some sort of terrible loop | 00:56 |
| greghaynes | ianw: I wonder if its a dead thread issue | 00:57 |
| greghaynes | where we have a non-daemon thread hanging around even though the interpreter was killed by timeout | 00:58 |
| *** Kennan has quit IRC | 00:58 | |
| *** fawadkhaliq has quit IRC | 00:58 | |
| greghaynes | or ctrlc | 00:58 |
| *** sputnik13 has quit IRC | 00:58 | |
| *** afazekas has joined #openstack-infra | 00:59 | |
| *** docaedo has joined #openstack-infra | 01:01 | |
| *** fawadkhaliq has joined #openstack-infra | 01:02 | |
| *** fawadkhaliq has quit IRC | 01:02 | |
| *** dchen has joined #openstack-infra | 01:03 | |
| prometheanfire | ianw: let me know when you submit that project-config review | 01:03 |
| *** zhurong has joined #openstack-infra | 01:03 | |
| *** ZZelle has quit IRC | 01:03 | |
| ianw | prometheanfire: oh, i thought you were going to finish it? that's just the gist | 01:03 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Revert "Tolerate both new and old setuptools" https://review.openstack.org/293187 | 01:04 |
| prometheanfire | oh, I thought you were going to do a review and I would make a gentoo version or something | 01:04 |
| SergeyLukjanov | jeblair yeah, it seems correct in a spec, if you want - ping me when you'll have code and I'll double check the zk usage | 01:04 |
| prometheanfire | k, I can do it | 01:04 |
| *** david-lyle has joined #openstack-infra | 01:04 | |
| SergeyLukjanov | is there somewhere the description of the specs concept? | 01:06 |
| *** fawadkhaliq has joined #openstack-infra | 01:06 | |
| *** fawadkhaliq has quit IRC | 01:06 | |
| *** dims has quit IRC | 01:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Update to oslo.messaging 4.5.1 https://review.openstack.org/292567 | 01:08 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Revert "Tolerate both new and old setuptools" https://review.openstack.org/293187 | 01:08 |
| EmilienM | fungi: did we figure why centos7 images failed to upload on internap cloud? | 01:08 |
| *** SumitNaiksatam has joined #openstack-infra | 01:09 | |
| *** Kennan has joined #openstack-infra | 01:10 | |
| prometheanfire | ianw: this is what I have https://gist.github.com/8b5b3dc5ec42fce7a31e | 01:11 |
| prometheanfire | I prepended devstack because {node} included it (based on the match at least) | 01:11 |
| fungi | EmilienM: i didn't hear back from mgagne but his suspicion was that the token was expiring before the upload completed | 01:11 |
| prometheanfire | this allows {node} to be used in the script | 01:12 |
| EmilienM | fungi: ok no prob | 01:12 |
| fungi | EmilienM: was that yesterday | 01:12 |
| EmilienM | fungi: I guess image upload still fails | 01:12 |
| fungi | ? | 01:12 |
| *** dims has joined #openstack-infra | 01:12 | |
| EmilienM | yeah, I believe | 01:12 |
| fungi | today has been so crazy it seems like it was last week | 01:12 |
| EmilienM | yeah :) Tuesday is busy | 01:12 |
| fungi | well, today's images didn't upload anywhere due to other issues we're tracking down | 01:12 |
| EmilienM | all the meetings, etc | 01:12 |
| prometheanfire | atm that's all I think is needed | 01:13 |
| ianw | prometheanfire: you should be able to just add two new jobs to https://review.openstack.org/#/c/293183/1/jenkins/jobs/projects.yaml using the existing template | 01:13 |
| fungi | didn't even build actually | 01:13 |
| *** yamamoto has joined #openstack-infra | 01:13 | |
| ianw | prometheanfire: except the new jobs specify -nv as the job-suffix, and pass in an arg for "tests" | 01:13 |
| *** tphummel has joined #openstack-infra | 01:14 | |
| prometheanfire | ok | 01:15 |
| prometheanfire | why two jobs? | 01:16 |
| *** Qiming has joined #openstack-infra | 01:16 | |
| prometheanfire | one for cent one for trusty? | 01:16 |
| *** Guest22575 has quit IRC | 01:17 | |
| prometheanfire | will I need to depend on your review or should I just replace it? | 01:18 |
| ianw | prometheanfire: yep, although honestly i'm not sure that the centos job is really testing that much, realistically whatever nodepool is doing is what's important | 01:18 |
| *** fawadkhaliq has joined #openstack-infra | 01:18 | |
| ianw | prometheanfire: just replace it | 01:18 |
| prometheanfire | k | 01:18 |
| *** fawadkhaliq has quit IRC | 01:19 | |
| *** yamamoto has quit IRC | 01:20 | |
| clarkb | EmilienM: I did some basic triage on https://bugs.launchpad.net/glance/+bug/1557814 it is a tempest bug not a glance bug | 01:22 |
| openstack | Launchpad bug 1557814 in tempest "Tempest tests fail on Glance: "Endpoint not found" when IPv6" [Undecided,New] | 01:22 |
| EmilienM | right | 01:22 |
| EmilienM | clarkb: mtreinish just told me | 01:22 |
| mtreinish | clarkb: yeah it's also likely a bug in glanceclient (the code in tempest that's failing there was originally based on glanceclient) | 01:23 |
| *** boris-42 has quit IRC | 01:24 | |
| prometheanfire | ianw: separate reviews for adding nv and adding gentoo jobs based on it I assume? | 01:24 |
| clarkb | tl;dr is socket.socket(AF_INET,...) means ipv4 only | 01:24 |
| *** asselin_ has joined #openstack-infra | 01:24 | |
| clarkb | but I updated the bug and all that | 01:24 |
| *** docaedo has quit IRC | 01:24 | |
| fungi | clarkb: one other workaround option for the ansible openstack inventory bug... we could nova rename the replacement review.o.o temporarily? | 01:25 |
| EmilienM | clarkb: thanks for looking! | 01:25 |
| clarkb | I guess getaddrinfo should also give you the args | 01:25 |
| clarkb | fungi: is that a thign? | 01:25 |
| * clarkb checks nova client help | 01:25 | |
| *** afazekas has quit IRC | 01:25 | |
| fungi | er, it's called nova update now | 01:25 |
| ianw | prometheanfire: no we can just do it together; the point of the nv job is to add gentoo | 01:25 |
| prometheanfire | ok | 01:26 |
| clarkb | fungi: ya I never knew that was a thing, seems like something we could try | 01:26 |
| clarkb | fungi: I worry it won't let us rename it back due to the name collision :) | 01:26 |
| mtreinish | clarkb: oh, I didn't realize it was opening a raw socket with AF.INET | 01:26 |
| mtreinish | I thought that code path was all just httplib | 01:26 |
| clarkb | mtreinish: yup | 01:26 |
| mtreinish | oh, it's in the https code | 01:27 |
| mtreinish | hmm | 01:27 |
| clarkb | mtreinish: EmilienM so I think the fix is to socket.socket(socket.getaddrinfo(addr)) | 01:27 |
| *** kingia_ has joined #openstack-infra | 01:27 | |
| clarkb | mtreinish: EmilienM that should select the correct family based on the name/addr | 01:27 |
| clarkb | maybe | 01:28 |
| clarkb | oh that always does a name lookup hrm | 01:28 |
| *** ianw_ has joined #openstack-infra | 01:28 | |
| *** asselin_ has quit IRC | 01:28 | |
| *** afazekas has joined #openstack-infra | 01:28 | |
| *** dims has quit IRC | 01:29 | |
| clarkb | I was wrong the []s confused it | 01:29 |
| clarkb | mtreinish: EmilienM socket.getaddrinfo('::1', 443) | 01:29 |
| clarkb | that works so thats the real fix | 01:29 |
| *** dims has joined #openstack-infra | 01:29 | |
| *** fawadkhaliq has joined #openstack-infra | 01:30 | |
| *** kingia has quit IRC | 01:30 | |
| clarkb | commented | 01:30 |
| *** docaedo has joined #openstack-infra | 01:30 | |
| *** Daisy has joined #openstack-infra | 01:31 | |
| *** fawadkhaliq has quit IRC | 01:31 | |
| *** akshai has quit IRC | 01:33 | |
| prometheanfire | ianw: ok, https://gist.github.com/9a8e4caa58febdbad9d1 | 01:33 |
| *** jiangliucheng has joined #openstack-infra | 01:33 | |
| prometheanfire | I think that's it | 01:34 |
| clarkb | you know we have a service to review these changes ;) | 01:35 |
| *** esikachev has joined #openstack-infra | 01:35 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 01:35 |
| prometheanfire | clarkb: you don't say :P | 01:35 |
| prometheanfire | I just didn't want to waste cycles :P | 01:35 |
| jiangliucheng | Does anyone know what's going on with zuul? My patch in workflow for one day, but no info be found in status.openstack.org/zuul | 01:35 |
| prometheanfire | but look at that, I'm wasting cycles :P | 01:35 |
| clarkb | jiangliucheng: what change | 01:35 |
| clarkb | jiangliucheng: always provide the change :) | 01:35 |
| jiangliucheng | review : 292037 | 01:36 |
| jiangliucheng | a bug | 01:36 |
| jiangliucheng | aboute replication v2.1 | 01:36 |
| Clint | hmm | 01:36 |
| *** tsg has quit IRC | 01:36 | |
| EmilienM | mtreinish, clarkb: thx a lot, I'm working on a patch | 01:36 |
| jiangliucheng | https://review.openstack.org/#/c/292037/ | 01:37 |
| *** esker has joined #openstack-infra | 01:37 | |
| *** Daisy has quit IRC | 01:37 | |
| *** kingia_ has quit IRC | 01:37 | |
| *** kingia has joined #openstack-infra | 01:37 | |
| *** Daisy has joined #openstack-infra | 01:37 | |
| clarkb | jiangliucheng: it might have gotten lost when gerrit was restarted earlier today | 01:37 |
| clarkb | er zuul | 01:37 |
| clarkb | its actually being tested now, looks like it has failed. | 01:38 |
| clarkb | you should be able to recheck it again once that reports back | 01:38 |
| prometheanfire | ianw: just realized, I still depend on https://review.openstack.org/#/c/290894/ | 01:38 |
| prometheanfire | not sure how to do cross project deps | 01:38 |
| *** esikachev has quit IRC | 01:39 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 01:39 |
| SamYaple | prometheanfire: depends-on: <changeid> | 01:39 |
| prometheanfire | k | 01:39 |
| jiangliucheng | Now i find it in check list, will it be move to gate list? | 01:39 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 01:40 |
| EmilienM | clarkb, mtreinish: https://review.openstack.org/293195 | 01:41 |
| *** IlyaG has joined #openstack-infra | 01:41 | |
| *** fawadkhaliq has joined #openstack-infra | 01:42 | |
| *** Daisy has quit IRC | 01:42 | |
| clarkb | EmilienM: I think you don't need an if else there | 01:42 |
| *** mriedem has quit IRC | 01:42 | |
| EmilienM | clarkb: I re-used glanceclient code | 01:42 |
| clarkb | EmilienM: just pass getadderinfo()[0] details striaght through | 01:42 |
| clarkb | heh glanceclient code is still broken then :) | 01:42 |
| clarkb | but ok | 01:42 |
| *** kaisers1 has joined #openstack-infra | 01:44 | |
| *** kaisers has quit IRC | 01:45 | |
| *** asselin_ has joined #openstack-infra | 01:50 | |
| *** fawadkhaliq has quit IRC | 01:51 | |
| *** fawadkhaliq has joined #openstack-infra | 01:51 | |
| *** dims has quit IRC | 01:52 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 01:52 |
| *** yamahata has joined #openstack-infra | 01:53 | |
| *** baoli has joined #openstack-infra | 01:55 | |
| *** dims has joined #openstack-infra | 01:56 | |
| *** IlyaG has quit IRC | 01:57 | |
| Nakato | lifeless: AJaeger: This is my initial take on what needs to be supported for post jobs. https://review.openstack.org/#/c/293194/ | 01:57 |
| Nakato | It's quite a different take than the one listed in the meeting logs, which I don't think is going to make post jobs works. | 01:57 |
| *** ddieterly is now known as ddieterly[away] | 01:59 | |
| *** aeng has quit IRC | 01:59 | |
| *** adrian_otto has joined #openstack-infra | 02:01 | |
| *** dims has quit IRC | 02:03 | |
| *** apoorvad has quit IRC | 02:03 | |
| *** dims has joined #openstack-infra | 02:05 | |
| *** akshai has joined #openstack-infra | 02:05 | |
| *** fawadkhaliq has quit IRC | 02:06 | |
| *** woodster_ has quit IRC | 02:07 | |
| *** akshai_ has joined #openstack-infra | 02:07 | |
| *** akshai has quit IRC | 02:10 | |
| fungi | jesusaur: ^ | 02:10 |
| *** ddieterly[away] is now known as ddieterly | 02:11 | |
| prometheanfire | ianw: jjb? | 02:11 |
| *** baoli_ has joined #openstack-infra | 02:11 | |
| prometheanfire | ianw: gentoo/build-succeeds is the name that run_functests.sh -l returns | 02:11 |
| *** fitoduarte has quit IRC | 02:12 | |
| *** aeng has joined #openstack-infra | 02:12 | |
| *** baoli has quit IRC | 02:14 | |
| *** ddieterly is now known as ddieterly[away] | 02:16 | |
| ianw | prometheanfire: i mean jenkins job builder, the thing that, well, builds the jobs | 02:17 |
| ianw | the template is in diskimage-builder.yaml, and then that is instantiated into jobs in projects.yaml | 02:17 |
| prometheanfire | ok, so I need to add another entry to that | 02:18 |
| *** thorst_ has quit IRC | 02:18 | |
| *** yamamoto_ has joined #openstack-infra | 02:19 | |
| *** nwkarsten has joined #openstack-infra | 02:20 | |
| *** yamahata has quit IRC | 02:21 | |
| prometheanfire | ianw: isn't it already instantiated by the review I'm doing? | 02:21 |
| prometheanfire | the job template was already edited | 02:22 |
| EmilienM | clarkb, mtreinish: puppet CI now works with ipv6 \o/ https://review.openstack.org/#/c/287521/ | 02:22 |
| EmilienM | clarkb: ty again for your help on tempest | 02:23 |
| *** baoli_ has quit IRC | 02:23 | |
| *** baoli has joined #openstack-infra | 02:23 | |
| *** pahuang has quit IRC | 02:24 | |
| *** amrith is now known as _amrith_ | 02:24 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services https://review.openstack.org/271128 | 02:24 |
| prometheanfire | ianw: I know I'm missing something, but I don't see it | 02:25 |
| jamielennox | AJaeger: around? what's my chance of getting ^ merged without a +1 from every project? | 02:25 |
| *** fawadkhaliq has joined #openstack-infra | 02:25 | |
| prometheanfire | ah, see your comment | 02:26 |
| jamielennox | if i need it then i'm going to split it up into multiple patches, the base then one per project | 02:26 |
| jamielennox | i can't get that many approvals between a merge conflict :p | 02:26 |
| *** ddieterly[away] has quit IRC | 02:26 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 02:28 |
| *** Daisy has joined #openstack-infra | 02:28 | |
| jamielennox | bah, going to do it preemptively | 02:29 |
| *** Daisy_ has joined #openstack-infra | 02:29 | |
| *** baoli has quit IRC | 02:29 | |
| *** yamahata has joined #openstack-infra | 02:30 | |
| *** Daisy__ has joined #openstack-infra | 02:31 | |
| *** sridhar_ram has joined #openstack-infra | 02:31 | |
| *** fawadkhaliq has quit IRC | 02:32 | |
| *** katyafervent_awa has quit IRC | 02:32 | |
| *** tqtran-afk has quit IRC | 02:32 | |
| *** Daisy has quit IRC | 02:33 | |
| *** Daisy has joined #openstack-infra | 02:33 | |
| *** Daisy_ has quit IRC | 02:34 | |
| *** baoli has joined #openstack-infra | 02:35 | |
| *** Daisy__ has quit IRC | 02:37 | |
| *** pahuang has joined #openstack-infra | 02:37 | |
| *** katyafervent_awa has joined #openstack-infra | 02:37 | |
| *** Daisy has quit IRC | 02:37 | |
| *** Daisy has joined #openstack-infra | 02:39 | |
| *** yamahata has quit IRC | 02:39 | |
| *** sridhar_ram has quit IRC | 02:41 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Refactor the v3 only job https://review.openstack.org/271128 | 02:43 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the v3 only job in glance https://review.openstack.org/293199 | 02:43 |
| *** rhallisey has quit IRC | 02:45 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 02:45 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the v3 only job in neutron https://review.openstack.org/293200 | 02:46 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 02:46 |
| *** dchen has quit IRC | 02:52 | |
| *** fitoduarte has joined #openstack-infra | 02:55 | |
| *** Jeffrey4l has joined #openstack-infra | 02:57 | |
| *** krtaylor has quit IRC | 02:58 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 02:58 |
| *** aeng has quit IRC | 03:01 | |
| *** pahuang has quit IRC | 03:06 | |
| *** hichihara has joined #openstack-infra | 03:06 | |
| *** krtaylor has joined #openstack-infra | 03:10 | |
| *** esker has quit IRC | 03:11 | |
| *** aeng has joined #openstack-infra | 03:12 | |
| prometheanfire | ianw: wooo it passed | 03:15 |
| *** binbincong has quit IRC | 03:15 | |
| *** fitoduarte has quit IRC | 03:15 | |
| prometheanfire | ianw: thanks for all the hand holding | 03:16 |
| *** thorst has joined #openstack-infra | 03:19 | |
| *** Daisy has quit IRC | 03:19 | |
| *** yuanying has quit IRC | 03:20 | |
| *** pahuang has joined #openstack-infra | 03:23 | |
| *** yamamoto_ has quit IRC | 03:25 | |
| *** dimtruck is now known as zz_dimtruck | 03:25 | |
| *** thorst has quit IRC | 03:27 | |
| *** binbincong has joined #openstack-infra | 03:27 | |
| ianw | cool | 03:27 |
| prometheanfire | guess I still need +2s to be readded to https://review.openstack.org/#/c/290894/ | 03:30 |
| prometheanfire | greghaynes: ^ | 03:30 |
| prometheanfire | greghaynes: mind re-reviewing that? | 03:30 |
| *** gongysh has joined #openstack-infra | 03:34 | |
| *** esikachev has joined #openstack-infra | 03:35 | |
| *** dims has quit IRC | 03:37 | |
| *** adrian_otto has quit IRC | 03:37 | |
| *** baoli has quit IRC | 03:39 | |
| *** esikachev has quit IRC | 03:40 | |
| *** cody-somerville has quit IRC | 03:46 | |
| *** rguillebert has quit IRC | 03:47 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fixed doble form submision issue https://review.openstack.org/293241 | 03:51 |
| *** akshai_ has quit IRC | 03:52 | |
| *** adrian_otto has joined #openstack-infra | 03:54 | |
| *** Daisy has joined #openstack-infra | 03:54 | |
| *** kingia_ has joined #openstack-infra | 03:55 | |
| *** watanabe_isao has joined #openstack-infra | 03:57 | |
| *** kingia has quit IRC | 03:58 | |
| *** cody-somerville has joined #openstack-infra | 04:01 | |
| *** esker has joined #openstack-infra | 04:01 | |
| *** ayoung is now known as ayoung_Zzz__ | 04:01 | |
| *** gongysh has quit IRC | 04:04 | |
| *** kingia has joined #openstack-infra | 04:05 | |
| *** kingia_ has quit IRC | 04:05 | |
| *** yamamoto_ has joined #openstack-infra | 04:06 | |
| *** esker has quit IRC | 04:06 | |
| *** fedexo has joined #openstack-infra | 04:07 | |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** links has joined #openstack-infra | 04:10 | |
| *** ayoung_Zzz__ has quit IRC | 04:12 | |
| prometheanfire | ianw: thanks and nn | 04:13 |
| *** [HeOS] has joined #openstack-infra | 04:13 | |
| *** fawadkhaliq has joined #openstack-infra | 04:14 | |
| *** HeOS has quit IRC | 04:15 | |
| *** flwang1 has quit IRC | 04:16 | |
| *** nwkarsten has quit IRC | 04:18 | |
| *** Daisy has quit IRC | 04:20 | |
| *** kingia_ has joined #openstack-infra | 04:21 | |
| *** kingia has quit IRC | 04:22 | |
| *** Daisy has joined #openstack-infra | 04:24 | |
| *** thorst has joined #openstack-infra | 04:25 | |
| *** kingia has joined #openstack-infra | 04:26 | |
| *** kingia_ has quit IRC | 04:26 | |
| *** rkukura has quit IRC | 04:28 | |
| *** tqtran-afk has joined #openstack-infra | 04:29 | |
| *** rkukura has joined #openstack-infra | 04:29 | |
| *** Daisy has quit IRC | 04:29 | |
| *** treaki__ has joined #openstack-infra | 04:30 | |
| *** thorst has quit IRC | 04:32 | |
| *** tqtran-afk has quit IRC | 04:33 | |
| *** treaki_ has quit IRC | 04:34 | |
| *** [1]Thelo has joined #openstack-infra | 04:34 | |
| *** pahuang has quit IRC | 04:35 | |
| *** akshai has joined #openstack-infra | 04:36 | |
| *** Thelo has quit IRC | 04:37 | |
| *** [1]Thelo is now known as Thelo | 04:37 | |
| *** roxanagh_ has joined #openstack-infra | 04:38 | |
| *** asettle has quit IRC | 04:50 | |
| *** sdake has quit IRC | 04:50 | |
| *** roxanagh_ has quit IRC | 04:58 | |
| *** pahuang has joined #openstack-infra | 05:03 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes https://review.openstack.org/270273 | 05:03 |
| *** akshai_ has joined #openstack-infra | 05:04 | |
| *** akshai has quit IRC | 05:07 | |
| *** yamahata has joined #openstack-infra | 05:08 | |
| tonyb | jhesketh: Can you help me understand why https://review.openstack.org/292657 isn't progressing to the gate pipeline? COUld it have been missed with the zuul/gerrit restart last night? | 05:08 |
| *** adrian_otto1 has joined #openstack-infra | 05:08 | |
| *** maishsk has joined #openstack-infra | 05:09 | |
| *** adrian_otto has quit IRC | 05:10 | |
| jhesketh | tonyb: yeah, looking at the time it's possibly one of the one results were lost from | 05:11 |
| jhesketh | tonyb: shouldn't be any harm in rechecking | 05:11 |
| *** asettle has joined #openstack-infra | 05:11 | |
| tonyb | jhesketh: Thanks. done. | 05:12 |
| *** boris-42 has joined #openstack-infra | 05:14 | |
| *** Daisy has joined #openstack-infra | 05:15 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Nova API sub-team meeting time https://review.openstack.org/292222 | 05:16 |
| *** FallenPegasus has joined #openstack-infra | 05:18 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Reverting the openstack-salt chair back to Jay Clark. https://review.openstack.org/292129 | 05:19 |
| openstackgerrit | Merged openstack-infra/irc-meetings: As per discussion, extend Quota WG time to 60 mins https://review.openstack.org/292625 | 05:20 |
| tonyb | jhesketh: recheck came back with a +1 from jenkins but that review still isn't prgressing to the gate pipeline | 05:23 |
| jhesketh | tonyb: hmm right you, give me a sec and I'll take a look | 05:23 |
| tonyb | jhesketh: Thanks. | 05:24 |
| *** alex_xu has quit IRC | 05:25 | |
| *** sc68cal has joined #openstack-infra | 05:26 | |
| *** hichihara has quit IRC | 05:27 | |
| *** tqtran-afk has joined #openstack-infra | 05:30 | |
| *** thorst has joined #openstack-infra | 05:30 | |
| *** aeng has quit IRC | 05:31 | |
| *** alex_xu has joined #openstack-infra | 05:31 | |
| *** sridhar_ram has joined #openstack-infra | 05:31 | |
| *** Jeffrey4l has quit IRC | 05:32 | |
| *** esker has joined #openstack-infra | 05:33 | |
| *** tqtran-afk has quit IRC | 05:35 | |
| jhesketh | tonyb: so in the zuul logs I can see where jenkins reports back the +1 from the check, but it doesn't appear to trigger it into the gate pipeline | 05:36 |
| jhesketh | and it's not clear why | 05:36 |
| *** esikachev has joined #openstack-infra | 05:36 | |
| jhesketh | I reviewed it myself and now it's in the gate pipeline | 05:36 |
| *** adrian_otto1 has quit IRC | 05:36 | |
| jhesketh | lets see if that posts a result back | 05:36 |
| *** thorst has quit IRC | 05:37 | |
| *** infra-red has joined #openstack-infra | 05:38 | |
| *** esker has quit IRC | 05:38 | |
| openstackgerrit | Merged openstack-infra/release-tools: Actually use the URL specific in --project-list https://review.openstack.org/292657 | 05:39 |
| jhesketh | tonyb: ^ seemed to work.. not sure why recheck didn't put it back in the gate though :-( | 05:40 |
| *** esikachev has quit IRC | 05:41 | |
| *** infra-red has quit IRC | 05:41 | |
| *** HeOS_ has joined #openstack-infra | 05:41 | |
| *** fawadkhaliq has quit IRC | 05:42 | |
| *** infra-red has joined #openstack-infra | 05:42 | |
| *** sridhar_ram has quit IRC | 05:42 | |
| *** fawadkhaliq has joined #openstack-infra | 05:43 | |
| *** aeng has joined #openstack-infra | 05:43 | |
| *** [HeOS] has quit IRC | 05:43 | |
| *** akshai_ has quit IRC | 05:44 | |
| *** david-lyle has quit IRC | 05:45 | |
| *** vgridnev has joined #openstack-infra | 05:45 | |
| *** david-lyle has joined #openstack-infra | 05:45 | |
| clarkb | if the vote doesnt change gerrit doesnt emit an event for it | 05:46 |
| clarkb | because an event isnt really an event its a delta | 05:46 |
| *** maishsk has quit IRC | 05:48 | |
| *** Daisy has quit IRC | 05:48 | |
| *** Daisy has joined #openstack-infra | 05:48 | |
| jhesketh | clarkb: oh that's right, thanks! | 05:51 |
| *** mrmartin has joined #openstack-infra | 05:52 | |
| jhesketh | clarkb: maybe we need to add the reverify comment to trigger the gate queue specifically.. it should be protected by the require approvals to not queue without an existing jenkins +1 and a +w | 05:52 |
| *** fawadkhaliq has quit IRC | 05:52 | |
| *** fawadkhaliq has joined #openstack-infra | 05:53 | |
| clarkb | jhesketh: I think zaro has a fix to make events events we might be able to upgrade to as well | 05:53 |
| jhesketh | against gerrit or zuul? | 05:53 |
| clarkb | gerrit | 05:54 |
| *** camunoz has quit IRC | 05:56 | |
| *** camunoz has joined #openstack-infra | 05:56 | |
| *** _nadya_ has joined #openstack-infra | 06:00 | |
| *** flwang1 has joined #openstack-infra | 06:01 | |
| *** _nadya_ has quit IRC | 06:01 | |
| *** mrmartin has quit IRC | 06:03 | |
| *** FallenPe_ has joined #openstack-infra | 06:09 | |
| *** flwang1 has quit IRC | 06:09 | |
| AJaeger | Nakato: Thanks for working on this! Let's discuss with jesusaur and jeblair about these | 06:09 |
| *** FallenPegasus has quit IRC | 06:10 | |
| *** infra-re_ has joined #openstack-infra | 06:10 | |
| *** Daisy_ has joined #openstack-infra | 06:10 | |
| AJaeger | eliqiao: I'm here now - are you still around? | 06:12 |
| *** infra-red has quit IRC | 06:12 | |
| eliqiao | hi AJaeger ,ya | 06:12 |
| *** Daisy has quit IRC | 06:13 | |
| *** infra-red has joined #openstack-infra | 06:14 | |
| Nakato | AJaeger: Sounds good. Though I don't plan to be around much longer tonight. | 06:14 |
| *** infra-re_ has quit IRC | 06:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/293268 | 06:17 |
| *** gongysh has joined #openstack-infra | 06:19 | |
| *** HeOS_ is now known as HeOS | 06:21 | |
| AJaeger | Nakato: Try to catch them tomorrow morning your time, they are gone already I guess. Thanks | 06:24 |
| AJaeger | eliqiao: Your previous revision used the python3-jobs template, use that one again and your change is fine. | 06:25 |
| *** camunoz has quit IRC | 06:25 | |
| *** Daisy_ has quit IRC | 06:26 | |
| *** Daisy has joined #openstack-infra | 06:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:31 |
| *** Daisy has quit IRC | 06:31 | |
| *** infra-red has quit IRC | 06:32 | |
| tonyb | jhesketh: Thanks. | 06:32 |
| *** thorst has joined #openstack-infra | 06:35 | |
| *** sc68cal has quit IRC | 06:36 | |
| *** jtomasek has joined #openstack-infra | 06:36 | |
| *** vgridnev has quit IRC | 06:39 | |
| *** camunoz has joined #openstack-infra | 06:39 | |
| *** harlowja_at_home has quit IRC | 06:41 | |
| *** pahuang has quit IRC | 06:41 | |
| *** thorst has quit IRC | 06:42 | |
| *** FallenPe_ has quit IRC | 06:42 | |
| *** vgridnev has joined #openstack-infra | 06:45 | |
| *** kushal has joined #openstack-infra | 06:45 | |
| *** abregman has joined #openstack-infra | 06:46 | |
| AJaeger | ianw: please review 293193 again and reconsider your +2. | 06:49 |
| *** asettle has quit IRC | 06:49 | |
| *** esker has joined #openstack-infra | 06:49 | |
| ianw | AJaeger: added some comments | 06:54 |
| *** esker has quit IRC | 06:54 | |
| *** gildub has quit IRC | 06:54 | |
| *** Daisy has joined #openstack-infra | 06:55 | |
| prometheanfire | AJaeger: I'm in irc as well so we can all talk about https://review.openstack.org/#/c/293193/ | 06:55 |
| tobiash | jhesketh: This patch got lost in zuul during connection refactoring which breaks our gate with new zuul versions: https://review.openstack.org/#/c/138132/1 | 06:56 |
| AJaeger | prometheanfire: Ah, that's you - didn't know ;) | 06:56 |
| prometheanfire | :P | 06:56 |
| AJaeger | ianw: I'm flagging the ubuntu-trusty already since we will do the switch today/tomorrow and not have several iterations on them... | 06:56 |
| prometheanfire | I really need to add my nick to my account | 06:56 |
| AJaeger | Using -extra or -gentoo works for me. | 06:57 |
| *** asselin_ has quit IRC | 06:57 | |
| ianw | AJaeger; yeah that's cool. i think we can switch the name if you feel strongly. i'd prefer "-extra", unless we can think of something else even more descriptive but not too verbose | 06:57 |
| prometheanfire | there a way to edit your 'full name'? | 06:57 |
| AJaeger | I know the job is not gentoo specific, but you parametize it in two different ways with the same name. | 06:58 |
| *** vgridnev has quit IRC | 06:58 | |
| AJaeger | Ones as X and then as X-nv and change paremeters to use gentoo. And that's my major complaint | 06:58 |
| prometheanfire | I can probably add a comment to that section | 06:59 |
| prometheanfire | I know thierry wanted to do debian stuff that could fit there as well (at first) | 07:00 |
| prometheanfire | as far as adding the -extras stuff, does that designation already exist as a non-voting job? | 07:00 |
| *** tqtran-afk has joined #openstack-infra | 07:01 | |
| prometheanfire | doesn't look like it | 07:01 |
| ianw | prometheanfire: no, if we switch it to "-extras" then we'll have to add a regex to make it non-voting. lots of examples at the top of the zuul file | 07:01 |
| AJaeger | prometheanfire: if debian is another job, then let's use -debian, -gentoo... | 07:02 |
| AJaeger | unless we can differentiate that in the node. | 07:02 |
| ianw | AJaeger: they will all be under the same job. it's testing we can *build* gentoo, debian images, etc, on a trusty/centos node | 07:02 |
| prometheanfire | debian would go in the same -extras job | 07:02 |
| prometheanfire | ya, the regex for -nv is simple, I'll copy that | 07:03 |
| *** fawadkhaliq has quit IRC | 07:03 | |
| AJaeger | ianw: Ah... right now it has only "gentoo/build-succeeds" and that's why I thought it would be separate ones. | 07:03 |
| *** ihrachys has joined #openstack-infra | 07:03 | |
| AJaeger | Then you might need to change the parameter and make it more general. | 07:04 |
| *** fawadkhaliq has joined #openstack-infra | 07:04 | |
| prometheanfire | ya, it's simply the first | 07:04 |
| prometheanfire | the parameter is a list of tests, gentoo being the first added | 07:04 |
| ianw | AJaeger: yeah, we might move the logic into dib if it becomes impractical | 07:04 |
| AJaeger | prometheanfire: better give one parameter "extra" and pass that in - then there's no need to touch the job everytime | 07:04 |
| AJaeger | ianw: agreed | 07:04 |
| *** tqtran-afk has quit IRC | 07:05 | |
| ianw | ok, i gotta disappear, will check later | 07:05 |
| prometheanfire | I'll do the work to make it just renamed to -extras for now | 07:05 |
| prometheanfire | we'll see about extra dib work in the morning | 07:06 |
| AJaeger | ianw: but I'm fine with starting the parameter in for now and refine later - but let's keep that in mind | 07:06 |
| *** ihrachys has quit IRC | 07:06 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: enable py34 voting https://review.openstack.org/292687 | 07:07 |
| prometheanfire | I don't think it'd be a hard thing | 07:07 |
| prometheanfire | there's already a list of ignored tests, making another list and specifying a parameter to trigger that list shouldn't be too bad | 07:07 |
| *** exploreshaifali has joined #openstack-infra | 07:12 | |
| *** vgridnev has joined #openstack-infra | 07:12 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 07:13 |
| *** flwang1 has joined #openstack-infra | 07:13 | |
| prometheanfire | ianw: AJaeger ^ | 07:13 |
| *** HeOS_ has joined #openstack-infra | 07:15 | |
| *** infra-red has joined #openstack-infra | 07:15 | |
| *** infra-red has quit IRC | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:16 | |
| prometheanfire | needs to pass of course | 07:17 |
| *** tesseract is now known as Guest64967 | 07:17 | |
| AJaeger | prometheanfire: commented - better ;) | 07:17 |
| AJaeger | Note, you can also name it: 'gate-dib-dsvm-functests-{node}{sth}{job-suffix}' and pass an empty something in | 07:18 |
| *** jaosorior has joined #openstack-infra | 07:18 | |
| *** _nadya_ has joined #openstack-infra | 07:19 | |
| prometheanfire | sth? | 07:20 |
| *** flepied1 has joined #openstack-infra | 07:20 | |
| prometheanfire | extras-nv does make sense though | 07:20 |
| AJaeger | sth=something. I didn't come up with a good name | 07:21 |
| *** _nadya_ has quit IRC | 07:21 | |
| AJaeger | I'm fine with trying extras-nv ;) | 07:21 |
| *** Guest64967 has quit IRC | 07:21 | |
| prometheanfire | ya, one change at a time :P | 07:22 |
| *** flepied has quit IRC | 07:22 | |
| *** esker has joined #openstack-infra | 07:23 | |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Add a core group for zaqar-ui https://review.openstack.org/293166 | 07:25 |
| prometheanfire | so slow | 07:25 |
| *** fawadkhaliq has quit IRC | 07:25 | |
| *** fawadkhaliq has joined #openstack-infra | 07:26 | |
| *** fawadkhaliq has quit IRC | 07:26 | |
| openstackgerrit | kevin proposed openstack-infra/project-config: New Project - jacket https://review.openstack.org/263500 | 07:26 |
| *** fawadkhaliq has joined #openstack-infra | 07:27 | |
| *** esker has quit IRC | 07:28 | |
| *** eric_lopez has quit IRC | 07:28 | |
| *** flwang1 has quit IRC | 07:30 | |
| *** elo has joined #openstack-infra | 07:30 | |
| prometheanfire | gerrit is timing out for git reviews | 07:30 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 07:30 |
| prometheanfire | and there it goes | 07:30 |
| prometheanfire | just had to complain about it | 07:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add a core group for zaqar-ui https://review.openstack.org/293166 | 07:30 |
| *** akaszuba has joined #openstack-infra | 07:32 | |
| AJaeger | ianw, could you review https://review.openstack.org/292135, please? | 07:32 |
| *** fedexo has quit IRC | 07:32 | |
| *** FallenPegasus has joined #openstack-infra | 07:33 | |
| *** tesseract- has joined #openstack-infra | 07:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Magnum: enable py34 voting https://review.openstack.org/292687 | 07:37 |
| *** FallenPegasus has quit IRC | 07:38 | |
| *** thorst has joined #openstack-infra | 07:40 | |
| openstackgerrit | Alexey Ovchinnikov proposed openstack-infra/project-config: Add publishing capability to buildimage-lxd job https://review.openstack.org/290418 | 07:41 |
| *** chlong has quit IRC | 07:42 | |
| *** maishsk has joined #openstack-infra | 07:45 | |
| *** amotoki has joined #openstack-infra | 07:46 | |
| yolanda | good morning | 07:47 |
| *** llu has joined #openstack-infra | 07:47 | |
| *** thorst has quit IRC | 07:48 | |
| prometheanfire | nn :P | 07:51 |
| *** nmagnezi has joined #openstack-infra | 07:52 | |
| *** dizquierdo has joined #openstack-infra | 07:53 | |
| *** llu has left #openstack-infra | 07:53 | |
| *** vgridnev has quit IRC | 07:54 | |
| penguinolog | Good morning! | 07:55 |
| penguinolog | Review, please: https://review.openstack.org/#/c/292746/ | 07:55 |
| *** scheuran has joined #openstack-infra | 07:56 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 07:58 |
| *** xiangxinyong has joined #openstack-infra | 07:58 | |
| *** maishsk_ has joined #openstack-infra | 07:59 | |
| jhesketh | tonyb: oh sorry about that.. there was so much rebasing and refactoring going on it must have gotten lost. Do you want to re-propose it? | 07:59 |
| *** maishsk has quit IRC | 08:00 | |
| *** maishsk_ is now known as maishsk | 08:00 | |
| yolanda | AJaeger, so no resolution yet on projects not being created on gerrit? | 08:00 |
| *** kingia_ has joined #openstack-infra | 08:01 | |
| yolanda | seems that they are not even present on review.openstack.org list of projects :( | 08:01 |
| jhesketh | tonyb: woops, wrong tab completion | 08:03 |
| jhesketh | tobiash: oh sorry about that.. there was so much rebasing and refactoring going on it must have gotten lost. Do you want to re-propose it? | 08:03 |
| *** kingia has quit IRC | 08:04 | |
| *** ihrachys has joined #openstack-infra | 08:04 | |
| tobiash | jhesketh: no problem, a collegue of mine will do this today or tomorrow | 08:10 |
| *** matrohon has joined #openstack-infra | 08:12 | |
| *** dizquierdo has quit IRC | 08:12 | |
| *** rossella_s has joined #openstack-infra | 08:13 | |
| *** flepied1 has quit IRC | 08:18 | |
| *** exploreshaifali has quit IRC | 08:18 | |
| *** MCoLo has joined #openstack-infra | 08:19 | |
| *** mikelk has joined #openstack-infra | 08:21 | |
| *** FallenPegasus has joined #openstack-infra | 08:28 | |
| jklare | morning | 08:28 |
| *** fhubik has joined #openstack-infra | 08:29 | |
| jklare | it might be too early for me, but i can not figure out why gerrit keeps mapping my changes to the wrong branch | 08:30 |
| jklare | somebody got a minute to help me here? | 08:30 |
| jklare | https://review.openstack.org/#/c/293308/ | 08:31 |
| *** mrmartin has joined #openstack-infra | 08:32 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce https://review.openstack.org/292792 | 08:32 |
| *** FallenPegasus has quit IRC | 08:32 | |
| *** yamahata has quit IRC | 08:33 | |
| *** vincentll has joined #openstack-infra | 08:33 | |
| *** dingyichen has quit IRC | 08:36 | |
| *** oanson has joined #openstack-infra | 08:37 | |
| *** abregman has quit IRC | 08:37 | |
| *** vincentll has quit IRC | 08:39 | |
| *** ig0r_ has quit IRC | 08:39 | |
| *** acruz has joined #openstack-infra | 08:43 | |
| *** vincentll has joined #openstack-infra | 08:43 | |
| *** esikachev has joined #openstack-infra | 08:45 | |
| *** andymaier has joined #openstack-infra | 08:45 | |
| *** thorst has joined #openstack-infra | 08:46 | |
| AJaeger | yolanda: please read backscroll, I didn't follow that part ;( | 08:47 |
| AJaeger | yolanda, jhesketh, ianw : A review of https://review.openstack.org/#/c/292135/ would be appreciated to move us forward wiht api jobs, please | 08:47 |
| * AJaeger sends coffee to jklare | 08:47 | |
| AJaeger | jklare: not sure what you do there - but we really suggest to use git.openstack.org since that'S our master server, github is just a mirror... | 08:48 |
| *** flepied has joined #openstack-infra | 08:49 | |
| yolanda | AJaeger i don't see that repo being created. Going to check if that's present on manage_projects log | 08:50 |
| AJaeger | yolanda: problem seems to be the non-running puppet on review.o.o | 08:50 |
| yolanda | i ran puppet manually there and worked | 08:50 |
| AJaeger | according to backlog - but I have no clue how that all works ;( | 08:50 |
| yolanda | but i still don't see that created, even not on review.openstack.org | 08:51 |
| AJaeger | :( | 08:51 |
| *** abregman has joined #openstack-infra | 08:51 | |
| *** watanabe_isao has quit IRC | 08:52 | |
| *** vgridnev has joined #openstack-infra | 08:52 | |
| jklare | AJaeger thanks for the coffee :D i solved i by specifiying the branch in the git review command, not sure why this is needed while it works for most other patches without | 08:52 |
| yolanda | AJaeger, do you have the link for that change handy? i don't even see that project on projects.yaml | 08:52 |
| *** thorst has quit IRC | 08:52 | |
| *** schipiga has joined #openstack-infra | 08:53 | |
| *** zeih has joined #openstack-infra | 08:54 | |
| yolanda | AJaeger, i'd say that even if that ansible-puppet reports as run, something is wrong. I don't see content in review.openstack.org updated since 10 Mar :( | 08:55 |
| AJaeger | new projects are: Ie0c7440e88c918045f1c5ca8288d826397599099 (tripleo-ui) and Icd3810574b72f98da8f51c26e46c527bdd89680b (openstack-ansible...designate) | 08:55 |
| AJaeger | that's from git log, so no direct review links | 08:55 |
| *** dalgaaf has joined #openstack-infra | 08:55 | |
| yolanda | so they were copied to git, but not to gerrit | 08:55 |
| *** ianychoi has quit IRC | 08:55 | |
| yolanda | so... more troubleshooting fun today | 08:56 |
| yolanda | so... looks to me that rsync is trying to copy content to the wrong review.openstack.org :( | 08:59 |
| AJaeger | Argh ;( | 09:00 |
| yolanda | the other parts of the ansible roles seem to be applying correctly but for rsync i see address 2001:4800:7819:103:be76:4eff:fe05:8525 | 09:00 |
| yolanda | that's new gerrit | 09:00 |
| yolanda | :( | 09:01 |
| yolanda | i also don't know so much about how this works, but going to try to understand | 09:01 |
| *** HeOS has quit IRC | 09:01 | |
| *** HeOS has joined #openstack-infra | 09:02 | |
| *** HeOS has quit IRC | 09:03 | |
| *** HeOS has joined #openstack-infra | 09:03 | |
| *** kingia has joined #openstack-infra | 09:03 | |
| *** nijaba has quit IRC | 09:05 | |
| *** kingia_ has quit IRC | 09:06 | |
| *** hashar has joined #openstack-infra | 09:06 | |
| *** jlanoux has joined #openstack-infra | 09:07 | |
| *** takashi has quit IRC | 09:07 | |
| *** ifarkas has joined #openstack-infra | 09:07 | |
| yolanda | something is totally broken in ansible inventory cache. It has the id of the current review.openstack.org, but with ip addresses of the new one | 09:08 |
| yolanda | mordred ` | 09:08 |
| *** _david_ has joined #openstack-infra | 09:09 | |
| *** nijaba has joined #openstack-infra | 09:11 | |
| rcarrillocruz | do you use a dynamic inventory? | 09:11 |
| rcarrillocruz | what's the ansible command you run | 09:11 |
| rcarrillocruz | ? | 09:11 |
| *** exploreshaifali has joined #openstack-infra | 09:11 | |
| *** ianychoi has joined #openstack-infra | 09:11 | |
| rcarrillocruz | if it's dynamic, the dynamic inventory scripts typically have a flag for caches | 09:11 |
| yolanda | ansible-playbook ç-vvv -f 10 /opt/system-config/production/playbooks/remote_puppet_git.yaml --limit localhost,review.openstack.org | 09:11 |
| *** salv-orlando has joined #openstack-infra | 09:11 | |
| *** esker has joined #openstack-infra | 09:12 | |
| yolanda | i'm seeing the entries totally wrong on /var/cache/ansible-inventory/ansible-inventory.cache | 09:12 |
| yolanda | so they have the right id of the vm in rax, but the ips are pointing to the new vm | 09:12 |
| *** kushal has quit IRC | 09:12 | |
| *** ianychoi has quit IRC | 09:12 | |
| *** ianychoi has joined #openstack-infra | 09:12 | |
| rcarrillocruz | if you are not passing explicitly the inventory from command line, that must be put on the ansible configuration... | 09:13 |
| rcarrillocruz | do you have an /etc/ansible/ansible.cfg | 09:13 |
| rcarrillocruz | or your own ansible.cfg at ~ | 09:13 |
| rcarrillocruz | ? | 09:13 |
| *** ianychoi2 has joined #openstack-infra | 09:14 | |
| yolanda | there is an ansible.cfg, with hostsfile just pointing at /etc/ansible/hosts | 09:14 |
| rcarrillocruz | k | 09:14 |
| rcarrillocruz | what you have on /etc/ansible/hosts | 09:14 |
| rcarrillocruz | is that a folder | 09:14 |
| rcarrillocruz | or a plain file | 09:14 |
| *** sbelous_ has joined #openstack-infra | 09:14 | |
| rcarrillocruz | ? | 09:14 |
| yolanda | ok inside /etc/ansible/hosts there is an openstack python file | 09:15 |
| yolanda | i'd say that this is buggy | 09:15 |
| rcarrillocruz | vi the openstack python file | 09:15 |
| rcarrillocruz | there must be a flag for cache | 09:15 |
| yolanda | i can see the entries for both review.openstack.org, different ids, and both point to the new ips | 09:15 |
| rcarrillocruz | at least the upstream openstack.py has it | 09:15 |
| yolanda | it's shade openstack inventory | 09:16 |
| *** esker has quit IRC | 09:16 | |
| rcarrillocruz | that one has a --refresh | 09:17 |
| rcarrillocruz | but not sure if that will fix it | 09:17 |
| rcarrillocruz | issue is you have two servers with same name | 09:17 |
| yolanda | yes but that was supposed to work, according to documentation and instructions on how to setup the replacement server... | 09:17 |
| rcarrillocruz | you can do this: | 09:17 |
| *** ZZelle has joined #openstack-infra | 09:17 | |
| yolanda | in theory the ansible inventory shall manage it properly | 09:17 |
| rcarrillocruz | python /etc/ansible/hosts/<whatever shade name>.py --list | 09:18 |
| *** ianychoi has quit IRC | 09:18 | |
| rcarrillocruz | that will show the hostvars | 09:18 |
| rcarrillocruz | which can help you troubleshoot the problem | 09:18 |
| rcarrillocruz | sorry i can't look further, without getting my hands on it i'm a bit limited | 09:18 |
| yolanda | yep, that's buggy | 09:18 |
| yolanda | it reports the id of the current review.openstack.org vm, but with the ips of the new one | 09:19 |
| rcarrillocruz | the upstream openstack.py inventory switched to uuids to circumvent the issue of servers with duplicated names | 09:19 |
| rcarrillocruz | mordred did | 09:19 |
| rcarrillocruz | not sure if that's on shade inventory | 09:19 |
| yolanda | there is something wrong there, with that uuids | 09:19 |
| yolanda | i'm double checking ids and ips, with the entries on rax, and is totally mismatching | 09:19 |
| yolanda | going to try the refresh | 09:20 |
| rcarrillocruz | look | 09:20 |
| rcarrillocruz | https://github.com/ansible/ansible/commit/c737bd48bc4ee246c378898abb80bacdd80c0e2f | 09:20 |
| rcarrillocruz | https://github.com/ansible/ansible/commit/9f61144401a16c9d610193522c71e8852addf63e | 09:20 |
| rcarrillocruz | i'm not sure if there's that in shade inventory | 09:21 |
| rcarrillocruz | but you get the idea, there's logic needed to handle duplicate hostnames | 09:21 |
| *** kingia_ has joined #openstack-infra | 09:21 | |
| yolanda | going to check in shade, it should be, but i'm worried about the logic not being the right one | 09:21 |
| yolanda | it seems to always pick the first server for the two ids | 09:22 |
| yolanda | both vms are configured in the inventory with same ips | 09:22 |
| yolanda | we are on shade 1.5.1 on puppetmaster | 09:23 |
| *** kingia has quit IRC | 09:23 | |
| yolanda | that looks as the latest anway | 09:23 |
| *** kzaitsev_mb has joined #openstack-infra | 09:24 | |
| *** jistr has joined #openstack-infra | 09:24 | |
| *** Daisy has quit IRC | 09:25 | |
| *** jordanP has joined #openstack-infra | 09:25 | |
| *** Daisy has joined #openstack-infra | 09:26 | |
| yolanda | the code in puppetmaster looks similar but it may be doing something nasty | 09:27 |
| yolanda | going to debug on my environment.. | 09:28 |
| odyssey4me | morning everyone - do we have any progress on resolving the empty repo http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate , the fact that it hasn't mirrored to github and the fact that it's empty even though https://review.openstack.org/292326 specified that it should be seeded? | 09:29 |
| rcarrillocruz | yolanda: if you want to quickly fix it | 09:29 |
| rcarrillocruz | what you can do | 09:29 |
| rcarrillocruz | is create an inventory file | 09:29 |
| rcarrillocruz | that points review.openstack.org | 09:29 |
| rcarrillocruz | to the IP you need | 09:29 |
| rcarrillocruz | and pass that to ansible-playbook -i <my new inventory> | 09:29 |
| yolanda | odyssey4me, we are having problems with puppet in review.openstack.org | 09:29 |
| odyssey4me | oh dear :/ I'll leave you to it then yolanda - hope it's resolved ASAP | 09:30 |
| rcarrillocruz | you'd need to put something like | 09:30 |
| yolanda | rcarrillocruz, i thought about that yes... | 09:30 |
| rcarrillocruz | review.openstack.org ansible_host=<the ip i need> | 09:30 |
| *** Daisy has quit IRC | 09:30 | |
| yolanda | however the inventory i see in ansible cache has some other facts, i was doubting because of it | 09:31 |
| *** BobBall has joined #openstack-infra | 09:31 | |
| yolanda | i'll check if there is an obvious bug and if that's more complicated, we can try that way | 09:31 |
| *** tzn has joined #openstack-infra | 09:31 | |
| yolanda | mm | 09:32 |
| yolanda | for server in servers: | 09:32 |
| yolanda | append_hostvars( | 09:32 |
| yolanda | hostvars, groups, server['id'], servers[0], | 09:32 |
| yolanda | namegroup=True) | 09:32 |
| yolanda | that always picks the first entry | 09:32 |
| yolanda | and in our case, the first entry is the new vm :( | 09:32 |
| *** esker has joined #openstack-infra | 09:32 | |
| yolanda | going to debug to be sure | 09:32 |
| rcarrillocruz | yep | 09:32 |
| yolanda | !!! | 09:33 |
| openstack | yolanda: Error: "!!" is not a valid command. | 09:33 |
| yolanda | it is :) | 09:33 |
| *** kushal has joined #openstack-infra | 09:33 | |
| *** derekh has joined #openstack-infra | 09:34 | |
| *** zhurong has quit IRC | 09:35 | |
| *** _nadya_ has joined #openstack-infra | 09:37 | |
| *** lucas-dinner is now known as lucasagomes | 09:38 | |
| *** esker has quit IRC | 09:39 | |
| yolanda | rcarrillocruz yes, the issue is confirmed. But i'm looking at how can we enforce that match properly, not relying on picking the first entry | 09:40 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu https://review.openstack.org/293012 | 09:42 |
| *** vincentll has quit IRC | 09:42 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** amotoki has quit IRC | 09:43 | |
| *** rguillebert has joined #openstack-infra | 09:44 | |
| *** abregman has quit IRC | 09:45 | |
| yolanda | ok i have something.. | 09:45 |
| *** vincentll has joined #openstack-infra | 09:46 | |
| yolanda | so... this needs to be fixed on ansible repo | 09:49 |
| *** esikachev has quit IRC | 09:50 | |
| *** thorst has joined #openstack-infra | 09:50 | |
| *** dims has joined #openstack-infra | 09:51 | |
| *** abregman has joined #openstack-infra | 09:53 | |
| yolanda | i fixed on puppetmaster by moving the fixed file to the ansible cache, but i'll send a PR to ansible | 09:53 |
| *** openstackgerrit has quit IRC | 09:53 | |
| *** openstackgerrit_ is now known as openstackgerrit | 09:53 | |
| *** openstackgerrit has quit IRC | 09:53 | |
| *** openstackgerrit_ has joined #openstack-infra | 09:53 | |
| openstackgerrit_ | Sebastian Schuberth proposed openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory https://review.openstack.org/237694 | 09:54 |
| *** openstackgerrit_ is now known as openstackgerrit | 09:54 | |
| *** openstackgerrit_ has joined #openstack-infra | 09:55 | |
| *** openstackgerrit has quit IRC | 09:55 | |
| *** openstackgerrit has joined #openstack-infra | 09:56 | |
| *** thorst has quit IRC | 09:57 | |
| yolanda | AJaeger, puppet is running on the right review.openstack.org now... i hope that this fixes the problems | 09:59 |
| *** esikachev has joined #openstack-infra | 09:59 | |
| *** placko_ has joined #openstack-infra | 10:01 | |
| *** placko_ has quit IRC | 10:01 | |
| *** tqtran-afk has joined #openstack-infra | 10:01 | |
| *** Qiming has quit IRC | 10:02 | |
| *** placko_ has joined #openstack-infra | 10:02 | |
| *** _david_ has quit IRC | 10:03 | |
| AJaeger | yolanda: woot! | 10:03 |
| *** sdague has joined #openstack-infra | 10:03 | |
| openstackgerrit | Timur Sufiev proposed openstack-infra/project-config: Fix the location of devstack_gate_rc script at horizon job https://review.openstack.org/293077 | 10:04 |
| *** placko_ has quit IRC | 10:04 | |
| yolanda | thx rcarrillocruz for the pointers | 10:05 |
| *** tqtran-afk has quit IRC | 10:06 | |
| *** tpsilva has joined #openstack-infra | 10:07 | |
| *** sdague has quit IRC | 10:11 | |
| *** sdague has joined #openstack-infra | 10:12 | |
| sbelous_ | AJaeger: ping | 10:14 |
| Zara | hi, re: storyboard email notifications, in light of the comments on https://review.openstack.org/#/c/293080/ , where else would we be able to set the sender address? I thought this manifest was consumed directly? | 10:15 |
| *** electrofelix has joined #openstack-infra | 10:15 | |
| AJaeger | yolanda: http://git.openstack.org/cgit/openstack/tripleo-ui/ has content! | 10:15 |
| yolanda | woot! | 10:15 |
| yolanda | and i see projeccts created on gerrit now | 10:16 |
| AJaeger | and also http://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/ is imported! | 10:16 |
| AJaeger | woot | 10:16 |
| yolanda | that bug has been challenging :) | 10:16 |
| AJaeger | sbelous_: please speak up, I'll read backscroll if necessary and others can chime in as well... | 10:16 |
| AJaeger | yolanda: thanks for fixing it! | 10:16 |
| rcarrillocruz | yolanda: np | 10:17 |
| sbelous_ | AJaeger: ok :) | 10:17 |
| * AJaeger sends a virtual ansible-mastering-pledge to rcarrillocruz and yolanda ;) | 10:17 | |
| sbelous_ | AJaeger: can you please take a look at my patch to infra when you have a time? https://review.openstack.org/#/c/286085/ | 10:17 |
| yolanda | i'll check with mordred before asking a pull request for that case, it may have some reason for that logic but i don't understand | 10:17 |
| AJaeger | sbelous_: that's a nova change, best work with sdague further on it... | 10:18 |
| rcarrillocruz | heh | 10:18 |
| sbelous_ | AJaeger: okay, will ask | 10:18 |
| sdague | AJaeger / sbelous_ +2 | 10:18 |
| sbelous_ | sdague: thanks! | 10:19 |
| AJaeger | sbelous_: just commented on a few issues ;) | 10:20 |
| AJaeger | but not so critical to revoke sdague 's +A ;) | 10:20 |
| sdague | yeh, we can do as a follow up | 10:20 |
| AJaeger | yolanda: github syncing worked as well ;) | 10:21 |
| AJaeger | https://github.com/openstack/openstack-ansible-os_designate/ | 10:21 |
| yolanda | nice.. i feel happy now | 10:21 |
| sbelous_ | AJaeger: thanks for comments, yeah, I'll consider it in further reworks | 10:22 |
| *** dizquierdo has joined #openstack-infra | 10:22 | |
| zaro | clarkb, jhesketh :at gerrit hackathon this week. i'm hopig to get the changes to stream events merged this week. | 10:22 |
| *** sambetts|afk is now known as sambetts | 10:22 | |
| AJaeger | yolanda, sdague : A review of https://review.openstack.org/#/c/292135/ would be appreciated to move us forward wiht api jobs, please | 10:23 |
| * yolanda is going with the review queue after the daily troubleshooting session :) | 10:23 | |
| AJaeger | sdague: could I trouble you with a nova review as well - https://review.openstack.org/292137 , please? | 10:23 |
| AJaeger | sdague: and then there's another compute api review: https://review.openstack.org/292888 | 10:24 |
| openstackgerrit | Christoph Gysin proposed openstack/gertty: Add support for auth-type=form https://review.openstack.org/169467 | 10:24 |
| yolanda | ah sdague, care reviewing https://review.openstack.org/292792 again? the devstack-gate fix | 10:25 |
| openstackgerrit | Merged openstack-infra/project-config: Initial job for Nova with os-vif https://review.openstack.org/286085 | 10:25 |
| *** claudiub has joined #openstack-infra | 10:27 | |
| *** daemontool has joined #openstack-infra | 10:28 | |
| yolanda | AJaeger, another thing that merged today is the changes in jjb to use lockfiles. So if you see any problems with jenkins jobs in masters let me know... | 10:28 |
| sdague | yolanda: that's complicated enough, could you put a test together for it? | 10:29 |
| AJaeger | yolanda: will do - thanks for the heads up | 10:29 |
| yolanda | oh tests :) | 10:29 |
| yolanda | ok | 10:29 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/293268 | 10:31 |
| *** rhallisey has joined #openstack-infra | 10:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix the location of devstack_gate_rc script at horizon job https://review.openstack.org/293077 | 10:40 |
| odyssey4me | yolanda AJaeger aweomse, great to see the openstack-ansible-os_designate repo is properly imported and mirrored to github :) | 10:49 |
| odyssey4me | thank you for persisting! | 10:49 |
| yolanda | glad to help | 10:50 |
| AJaeger | odyssey4me: the cookies and coffee you send to Spain have helped ;) | 10:52 |
| openstackgerrit | Zara proposed openstack-infra/system-config: Add sender email for storyboard.openstack.org https://review.openstack.org/293367 | 10:52 |
| yolanda | :) | 10:53 |
| openstackgerrit | Merged openstack-infra/project-config: Add a core group for zaqar-ui https://review.openstack.org/293166 | 10:53 |
| *** dims has quit IRC | 10:53 | |
| kozhukalov | guys, could you please review https://review.openstack.org/#/c/292760/ ? | 10:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add Zaqar Role to OpenStack-Ansible https://review.openstack.org/292771 | 10:55 |
| *** thorst has joined #openstack-infra | 10:55 | |
| *** ociuhandu has quit IRC | 10:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Update Zuul Layout for Puppet OpenStack https://review.openstack.org/292555 | 10:56 |
| *** salv-orlando has quit IRC | 10:56 | |
| *** exploreshaifali has quit IRC | 10:57 | |
| odyssey4me | AJaeger yolanda I'd like to make arrangements for a repository rename - how do I go about doing that? | 10:57 |
| rcarrillocruz | AJaeger, odyssey4me: i love cookies :-) | 10:57 |
| yolanda | odyssey4me, check in http://docs.openstack.org/infra/manual/creators.html, Project Renames section | 10:58 |
| openstackgerrit | Merged openstack-infra/project-config: Pre-release stable/mitaka ACLs https://review.openstack.org/292781 | 10:58 |
| *** flwang1 has joined #openstack-infra | 10:59 | |
| AJaeger | odyssey4me: renames are really expensive, it needs complete gerrit downtime and manual steps. Please reconsider | 10:59 |
| *** dims has joined #openstack-infra | 10:59 | |
| *** kzaitsev_mb has quit IRC | 10:59 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Update Email Configuration file https://review.openstack.org/292605 | 10:59 |
| AJaeger | odyssey4me: but if it's needed, follow the steps and expect several weeks until this gets done. | 10:59 |
| *** gongysh has quit IRC | 11:01 | |
| odyssey4me | AJaeger will see what we can do | 11:02 |
| *** FallenPegasus has joined #openstack-infra | 11:02 | |
| *** thorst has quit IRC | 11:02 | |
| openstackgerrit | Merged openstack-infra/project-config: fuel-qa: move python 2.7 checks to community https://review.openstack.org/292746 | 11:03 |
| *** esker has joined #openstack-infra | 11:04 | |
| *** yamamoto_ has quit IRC | 11:05 | |
| *** Jeffrey4l has joined #openstack-infra | 11:06 | |
| *** FallenPegasus has quit IRC | 11:07 | |
| rcarrillocruz | Zara: sorry, i just realized https://review.openstack.org/#/c/293367/1 will fail | 11:07 |
| rcarrillocruz | the sender_email_address param is not exposed in openstack_project::storyboard :/ | 11:07 |
| *** esker has quit IRC | 11:08 | |
| rcarrillocruz | pedroalvarez: ^ | 11:08 |
| pedroalvarez | ouch | 11:09 |
| rcarrillocruz | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/storyboard.pp | 11:09 |
| rcarrillocruz | put sender_email_address as undef to ^ class signature | 11:10 |
| rcarrillocruz | then pass in to ::storyboard::app | 11:10 |
| pedroalvarez | right, sounds easy. Thanks for spotting that! | 11:11 |
| rcarrillocruz | np | 11:13 |
| *** _amrith_ is now known as amrith | 11:16 | |
| *** openstackgerrit has quit IRC | 11:18 | |
| *** openstackgerrit has joined #openstack-infra | 11:18 | |
| jhesketh | jeblair: more feedback/discuss on the logs in afs (https://review.openstack.org/#/c/269928/1).. I wonder if we should move it to the mailing list. Anyway, talk to you about it later. | 11:19 |
| * penguinolog happy, now waiting for the first start of newly enabled jobs | 11:20 | |
| penguinolog | Thanks to igorbelikov , yolanda and AJaeger | 11:20 |
| openstackgerrit | Merged openstack-infra/openstackid: Fixed doble form submision issue https://review.openstack.org/293241 | 11:21 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 11:22 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenStackId: Update Email configuration https://review.openstack.org/293379 | 11:23 |
| *** kushal has quit IRC | 11:25 | |
| *** Daisy has joined #openstack-infra | 11:26 | |
| *** smarcet has joined #openstack-infra | 11:27 | |
| openstackgerrit | Zara proposed openstack-infra/system-config: Add sender email for storyboard.openstack.org https://review.openstack.org/293367 | 11:27 |
| smarcet | fungi: morning ! :) its seems that puppet deployment for openstackid-dev is failing, take a look at http://puppetboard.openstack.org/report/openstackid-dev.openstack.org/773192c582fcceaec2482ccd2359f1b41df0b4dd | 11:27 |
| Zara | rcarrillocruz: thanks for your help! hopefully I did it right this time... | 11:28 |
| *** hieulq has joined #openstack-infra | 11:29 | |
| *** hieulq has quit IRC | 11:31 | |
| openstackgerrit | Sebastian Schuberth proposed openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory https://review.openstack.org/237694 | 11:31 |
| *** Daisy has quit IRC | 11:31 | |
| openstackgerrit | Zara proposed openstack-infra/system-config: Add sender email for storyboard.openstack.org https://review.openstack.org/293367 | 11:32 |
| *** weshay has joined #openstack-infra | 11:34 | |
| *** vgridnev has quit IRC | 11:34 | |
| *** dtantsur is now known as dtantsur|brb | 11:34 | |
| *** thorst has joined #openstack-infra | 11:35 | |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| *** rfolco has joined #openstack-infra | 11:39 | |
| *** exploreshaifali has joined #openstack-infra | 11:41 | |
| *** ihrachys has quit IRC | 11:42 | |
| *** placko_ has joined #openstack-infra | 11:42 | |
| *** ihrachys has joined #openstack-infra | 11:42 | |
| *** ihrachys has quit IRC | 11:42 | |
| *** gordc has joined #openstack-infra | 11:44 | |
| *** erlon has joined #openstack-infra | 11:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:44 | |
| *** fverbs has joined #openstack-infra | 11:45 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 11:48 |
| *** zeih has quit IRC | 11:49 | |
| *** zeih has joined #openstack-infra | 11:49 | |
| fungi | smarcet: looking in the logs, i have Mar 16 09:54:55 openstackid-dev puppet-user[25843]: (/Stage[main]/Openstackid/Exec[update-site]/returns) executed successfully | 11:50 |
| openstackgerrit | Srushti Gadadare proposed openstack-dev/hacking: Add check for mutable objects as method arguments https://review.openstack.org/293386 | 11:50 |
| smarcet | fungi: tnx | 11:51 |
| fungi | smarcet: if you suspect the update isn't actually working but is still returning a success exit code, i can try running it by hand to get more detailed output | 11:51 |
| fungi | i should look to see if we're logging the output for that somewhere else too | 11:52 |
| smarcet | fungi: well not sure | 11:53 |
| smarcet | yesterday i was waiting for an update that never ran | 11:53 |
| smarcet | well nows works | 11:53 |
| smarcet | also | 11:53 |
| fungi | root@openstackid-dev:~# /opt/deploy/deploy.sh status openstackid | 11:54 |
| fungi | INSTALLED [16-Mar-2016 11:31] | 11:54 |
| smarcet | fungi: i noticed that openstackid.org dns server does not has a AAAA record | 11:54 |
| *** zeih has quit IRC | 11:54 | |
| smarcet | fungi: ok, i will take note of that | 11:54 |
| *** zeih has joined #openstack-infra | 11:54 | |
| fungi | fungi@dhole:~$ host -t aaaa openstackid.org | 11:54 |
| fungi | openstackid.org has IPv6 address 2001:4800:7818:103:53f7:aea1:b964:d61c | 11:54 |
| smarcet | fungi: we have some dns issues yesterday when one tester wanted to access from a cell phone | 11:55 |
| smarcet | weird | 11:55 |
| smarcet | ok will dig on that | 11:55 |
| smarcet | tnx for all :) | 11:55 |
| smarcet | have a good day :) | 11:55 |
| fungi | you too! | 11:55 |
| fungi | i'm happy to dig deeper on ipv6 related stuff too if you need help | 11:56 |
| smarcet | ok cool thx fungi for your offer :) | 11:57 |
| *** salv-orlando has joined #openstack-infra | 11:57 | |
| smarcet | will let u know | 11:57 |
| fungi | smarcet: i know rackspace said they were having some issues with their dns servers yesterday that broke reverse dns, so maybe something related to that | 11:57 |
| smarcet | oh rite, we have some issues with db host yesterday too | 11:59 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: builders: update cmake to match CMake plugin versions 2.x https://review.openstack.org/279754 | 11:59 |
| *** dtantsur|brb is now known as dtantsur | 12:01 | |
| *** _degorenko|afk is now known as degorenko | 12:05 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce https://review.openstack.org/292792 | 12:06 |
| *** baoli has joined #openstack-infra | 12:07 | |
| *** jaosorior has quit IRC | 12:08 | |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| *** jaosorior has joined #openstack-infra | 12:09 | |
| *** jaosorior has quit IRC | 12:09 | |
| *** zeih has quit IRC | 12:09 | |
| *** aysyd has joined #openstack-infra | 12:09 | |
| *** kgiusti has joined #openstack-infra | 12:10 | |
| *** vgridnev has joined #openstack-infra | 12:10 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Set default locale to image in ubuntu-minimal https://review.openstack.org/290789 | 12:10 |
| *** jaosorior has joined #openstack-infra | 12:11 | |
| *** zeih has joined #openstack-infra | 12:11 | |
| smarcet | fungi: from JP : "It is not openstackid that doesn't have, | 12:13 |
| smarcet | It is the DNS servers I think, Try host -t aaaa dns1.stabletransit.com" | 12:13 |
| fungi | smarcet: right, those are rackspace's dns servers | 12:13 |
| fungi | smarcet: jpmaxman: those are the same nameservers as for openstack.org | 12:14 |
| fungi | kinda sad state of affairs, honestly. i get around it for my personal domains by running my own bind 10 on instances in different rackspace regions (iad and syd) | 12:15 |
| smarcet | fungi: ok, on another subject, there is any change to get ssh access to openstackid.org, we really wanna get access to production log file live due summit app upcomming | 12:15 |
| *** ihrachys has joined #openstack-infra | 12:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Regenerate nodepool.logging config https://review.openstack.org/293396 | 12:16 |
| jpmaxman | It was a weird situation. | 12:18 |
| jpmaxman | Openstackid won't work on the T-Mobille LTE network in Austin | 12:18 |
| fungi | smarcet: that runs counter to the separation of concerns we'd established for servers where we're providing system support. we can hand control over them back to tipit, but we give up hope of sharing an authentication system in the future with our infrastructure | 12:18 |
| jpmaxman | Fungi we were trying to follow the procedure described for gaining root access to an infra server | 12:19 |
| fungi | though that's worth discussing. i think now that we have gotten agreement from the board that we can ignore cla enforcement, we no longer need to plan for a strong tie between the community infrastructure resources and whatever the foundation and its contractors are developing | 12:19 |
| jpmaxman | We were going to have smarcet submit a patch similar to | 12:20 |
| jpmaxman | https://review.openstack.org/#/c/285641/1 | 12:20 |
| jpmaxman | On the ipv6 thing I am not sure why T-Mobile is running in pure ipv6 mode and why even its own nameservers would query for host names in ipv4 | 12:21 |
| jpmaxman | https://usercontent.irccloud-cdn.com/file/2uxySLPr/Screenshot_20160315-222520.png | 12:21 |
| fungi | yep, we've done that in the past for some outlying systems (wiki, pbx, lists), and given that we don't currently rely on openstackid.org for authenticating anything besides community.o.o and the www site right now it's worth discussing with the team | 12:22 |
| fungi | we mostly want to be sure, in cases like this, that anyone with direct access to the servers will still work through configuration management to implement fixes for problems rather than patching around them locally | 12:23 |
| jpmaxman | Anyway not sure what we can do about it but definitely would not be good at the summit in Austin if people roaming in T-Mobile could access the app | 12:23 |
| fungi | jpmaxman: i agree, though as i pointed out we'd also need to fix it for www.openstack.org since its nameservers are also not ipv6-connected | 12:23 |
| vkramskikh | hey folks, could you please review https://review.openstack.org/#/c/292760/ ? this is javascript jobs enablement for fuel-ui project | 12:24 |
| *** kzaitsev_mb has quit IRC | 12:24 | |
| pabelanger | need a better way to keep nodepool-builder.logging.conf in sync | 12:25 |
| pabelanger | maybe we should move that into project-config | 12:25 |
| smarcet | fungi: so i am good to go , and submit a patch like this one https://review.openstack.org/#/c/285641/1 to get ssh to production server ? | 12:25 |
| *** ociuhandu has joined #openstack-infra | 12:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:25 | |
| jpmaxman | Right. Any rackspace site. Though I would equally blame T-Mobile. Anyway like I said may not be anything we can do... | 12:25 |
| fungi | jpmaxman: moving all of our dns would be a significant undertaking on relatively short notice, though perhaps a support request to rackspace letting them know their nameservers are no longer serving anything t-mobile can resolve would be helpful | 12:26 |
| fungi | smarcet: that's how access would be implemented, but would still be pending some majority of the existing root sysadmins agreeing to approve that access | 12:27 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add docs publishing jobs for fuel-docs https://review.openstack.org/293399 | 12:27 |
| *** salv-orlando has quit IRC | 12:28 | |
| fungi | anyway, i need to disappear for a bit. need to take my car to the mechanic this morning | 12:28 |
| *** pcaruana has quit IRC | 12:28 | |
| jpmaxman | If is working on T-Mobile now | 12:29 |
| *** akshai has joined #openstack-infra | 12:29 | |
| jpmaxman | I am chalking it up to a tmobile glitch of not having their nameservers support ipv4 queries | 12:30 |
| jpmaxman | But will keep an eye on it | 12:30 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add docs publishing jobs for fuel-docs https://review.openstack.org/293399 | 12:30 |
| jpmaxman | Rackspace is aware | 12:31 |
| jpmaxman | https://feedback.rackspace.com/forums/298158-network/suggestions/5977739-ipv6-addresses-for-the-nameservers-cloud-dns | 12:31 |
| *** fawadkhaliq has quit IRC | 12:31 | |
| fungi | i hope there's a similar one for their lack of dnssec support | 12:32 |
| fungi | which is the other main reason i run my own nameservers instead of using theirs | 12:33 |
| *** boris-42 has quit IRC | 12:34 | |
| *** thiagop has joined #openstack-infra | 12:34 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jeepyb: Allow create/update projects with description into Gerrit https://review.openstack.org/110009 | 12:34 |
| *** esikachev has quit IRC | 12:35 | |
| *** lucasagomes is now known as lucas-hungry | 12:35 | |
| pabelanger | Where did the fix to ansible inventory go that clarkb did lastnight? | 12:38 |
| pabelanger | AJaeger: yolanda: Ah, looks like github sync is working again | 12:39 |
| jhesketh | speaking of ansible inventory fixes, if anybody knows how/who to get eyes on https://github.com/ansible/ansible/pull/14699 that'd be handy | 12:40 |
| openstackgerrit | Merged openstack-infra/project-config: Templatize api guide jobs https://review.openstack.org/292135 | 12:40 |
| yolanda | pabelanger there is a problem on ansible inventory, when there are duplicated hosts with same name | 12:40 |
| yolanda | it always assumes that the first server is the one on the list | 12:40 |
| yolanda | and for review.openstack.org that was the new vm created, so puppet was applied there | 12:40 |
| yolanda | i did a fix on puppetmaster, but i want to check with mordred, before sending a pull request | 12:41 |
| openstackgerrit | Merged openstack-infra/project-config: Only publish API guides from master https://review.openstack.org/292888 | 12:42 |
| *** ramishra has quit IRC | 12:43 | |
| *** rabi has joined #openstack-infra | 12:44 | |
| *** esikachev has joined #openstack-infra | 12:44 | |
| *** yamamoto has quit IRC | 12:45 | |
| pabelanger | yolanda: actually, clarkb did a fix last night | 12:45 |
| pabelanger | yolanda: and pull request | 12:45 |
| pabelanger | https://github.com/ansible/ansible/pull/14984 | 12:46 |
| yolanda | pabelanger i saw the inventory in a bad state this morning on review.openstack.org, and puppet had not run since 10 mar | 12:46 |
| pabelanger | yolanda: right, see above | 12:46 |
| *** rabi is now known as ramishra | 12:46 | |
| yolanda | i did the same fix :) | 12:46 |
| pabelanger | and check eavesdrop.o.o from yesterday, we had a discussion abou tit | 12:46 |
| yolanda | so maybe he ran it, but not flushed cache? | 12:46 |
| pabelanger | about it* | 12:46 |
| pabelanger | I don't actually think he ran it | 12:47 |
| pabelanger | but will have to wait for clarkb to confirm | 12:47 |
| yolanda | ah ok, so i didn't see that conversation, but i debugged and arrived the same conclusion | 12:47 |
| yolanda | what i did is to test locally and once i had the inventory working properly, applied it with refreshing cache on puppetmaster, to fix review.openstack.org | 12:47 |
| pabelanger | okay, not sure the path moving forward. I suspect mordred can confirm the fix in PR and merge | 12:48 |
| pabelanger | I think we run devel branch too | 12:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add experimental centos-7 puppet-apply https://review.openstack.org/293409 | 12:49 |
| *** pradk has joined #openstack-infra | 12:51 | |
| *** FallenPegasus has joined #openstack-infra | 12:51 | |
| fungi | yolanda: right, we decided not to fix it in production until the ansible reviewers bless the solution | 12:52 |
| fungi | in case our understanding was flawed | 12:52 |
| *** csaba has quit IRC | 12:52 | |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| *** jlvillal has quit IRC | 12:52 | |
| *** jlvillal has joined #openstack-infra | 12:52 | |
| *** smarcet has quit IRC | 12:53 | |
| pabelanger | yolanda: AJaeger: when you have a moment, ^ brings online a centos7-minimal dib for experimental testing | 12:53 |
| fungi | didn't want to risk breaking everything everywhere, when the current impact is really just new project creation remains in limbo and we can't update nodepool images at the moment | 12:53 |
| pabelanger | fungi: Do you want to hold off on debian-jessie DIB until we fix inventory 100%? | 12:54 |
| fungi | fixing minor inconveniences like that at the risk of much broader breakage is more reckless than we like to be in our systems administration practices | 12:55 |
| *** ddieterly has joined #openstack-infra | 12:56 | |
| *** FallenPegasus has quit IRC | 12:56 | |
| fungi | pabelanger: well, it sounds like maybe yolanda went ahead and applied an unsanctioned patch to ansible on the puppetmaster to work around the situation, so there's presumably nothing stopping us from approving the debian nodepool images now | 12:56 |
| *** csaba has joined #openstack-infra | 12:56 | |
| anteaya | I'm away this morning running errands | 12:57 |
| openstackgerrit | Thiago Paiva Brito proposed openstack-infra/puppet-openstackci: Initiate nodepool-builder on third_party_ci.rst https://review.openstack.org/293087 | 12:57 |
| pabelanger | fungi: Okay, I'll bring debian online and monitor | 12:57 |
| *** eharney has joined #openstack-infra | 12:57 | |
| pabelanger | will wait for nodepool to autobuild at 14hrs | 12:58 |
| fungi | anyway, dropping back offline again to deal with car stuff | 12:59 |
| anteaya | happy car stuff | 13:00 |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite https://review.openstack.org/274701 | 13:00 |
| *** Jeffrey4l has quit IRC | 13:02 | |
| *** dtardivel has joined #openstack-infra | 13:02 | |
| *** dprince has joined #openstack-infra | 13:02 | |
| *** rcernin has joined #openstack-infra | 13:03 | |
| *** Hal has joined #openstack-infra | 13:04 | |
| *** Hal is now known as Guest51184 | 13:05 | |
| yolanda | ah fungi i lost that conversation... i checked that the results were the expected before doing that, and i also did not update the code in puppetmaster, i was running a local copy on my directory | 13:06 |
| *** kaisers1 has quit IRC | 13:09 | |
| pabelanger | okay, relocating to the coffee shop in town. should be back online in a hour | 13:09 |
| pabelanger | will continue with debian jessie then | 13:09 |
| *** e0ne has joined #openstack-infra | 13:10 | |
| *** ddieterly has quit IRC | 13:10 | |
| *** bdemers has joined #openstack-infra | 13:11 | |
| yolanda | fungi, i'm thinking on a better way to pass information between infra-roots on different shifts. In Gozer, for some time, we were putting notes on a wiki page, about the different pending tasks, important information... that needed to be passed across shifts | 13:11 |
| yolanda | doing that, it was easier that the important information was well transmitted, rather than having to read the whole backlog | 13:11 |
| yolanda | what do you think? | 13:12 |
| *** zeih has quit IRC | 13:12 | |
| *** placko_ has quit IRC | 13:13 | |
| AJaeger | yolanda: yes, having something like that would be great. Whether it's wiki page, etherpad,... | 13:13 |
| *** Jeffrey4l has joined #openstack-infra | 13:14 | |
| *** kro_sinus_hell is now known as krot_sinus_purga | 13:15 | |
| *** krot_sinus_purga is now known as kro_sniffles | 13:15 | |
| *** nmagnezi has quit IRC | 13:15 | |
| *** smcginnis_away is now known as smcginnis | 13:15 | |
| *** moravec has joined #openstack-infra | 13:16 | |
| openstackgerrit | Merged openstack-infra/release-tools: Add update_reviews tool https://review.openstack.org/290853 | 13:16 |
| *** kaisers has joined #openstack-infra | 13:19 | |
| *** smarcet has joined #openstack-infra | 13:19 | |
| *** xyang1 has joined #openstack-infra | 13:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Add debian-jessie image to nodepool https://review.openstack.org/264726 | 13:20 |
| *** zeih has joined #openstack-infra | 13:20 | |
| stevemar | AJaeger: ughhhh why wasn't UC updated! https://review.openstack.org/#/c/293042/ | 13:21 |
| stevemar | AJaeger: thanks for the detective work | 13:21 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add curl package to puppet nodepool element https://review.openstack.org/293417 | 13:22 |
| stevemar | AJaeger: it was updated at one point: https://review.openstack.org/#/c/288124/1/upper-constraints.txt | 13:23 |
| stevemar | looks like https://github.com/openstack/requirements/commit/7953162c5b297f326a975a2520afb05f605ba67b reverted it | 13:24 |
| stevemar | thats unsettling | 13:24 |
| *** ig0r_ has joined #openstack-infra | 13:24 | |
| *** _ari_ has joined #openstack-infra | 13:24 | |
| AJaeger | stevemar: ask dims, maybe he made an error... | 13:25 |
| openstackgerrit | Merged openstack/requirements: Revert "Tolerate both new and old setuptools" https://review.openstack.org/293187 | 13:25 |
| stevemar | AJaeger: yeah, i will post a change updating upper-constraints and explain it all, seems like it was a mistake | 13:25 |
| openstackgerrit | Merged openstack/requirements: Update to oslo.messaging 4.5.1 https://review.openstack.org/292567 | 13:25 |
| *** _nadya_ has quit IRC | 13:25 | |
| *** lucas-hungry is now known as lucasagomes | 13:26 | |
| smarcet | fungi:yolanda: please when u get a chance review https://review.openstack.org/#/c/293379/ thx | 13:26 |
| dims | stevemar : what did i do? :) | 13:26 |
| *** kaisers has quit IRC | 13:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/293423 | 13:26 |
| *** Jeffrey4l has quit IRC | 13:27 | |
| yolanda | fungi, AJaeger, and easy thing for that... why not send status log messages on the things we need to transmit on shifts? by that way , it will be logged on the wiki page, and when we come at EMEA any problems, status updates, etc... will be easily reachable | 13:27 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test https://review.openstack.org/289445 | 13:27 |
| *** e0ne has quit IRC | 13:27 | |
| *** placko_ has joined #openstack-infra | 13:28 | |
| yolanda | smarcet, what about hiera values? are they set on puppetmaster? | 13:28 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Do not Gnocchi coordinator backend https://review.openstack.org/293424 | 13:28 |
| *** kaisers has joined #openstack-infra | 13:28 | |
| *** Guest51184 has quit IRC | 13:28 | |
| *** mriedem has joined #openstack-infra | 13:28 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump uc for congressclient to 1.2.3 https://review.openstack.org/293425 | 13:29 |
| *** fawadkhaliq has joined #openstack-infra | 13:29 | |
| stevemar | dims: ^ | 13:29 |
| *** automagically_ has joined #openstack-infra | 13:29 | |
| *** salv-orlando has joined #openstack-infra | 13:29 | |
| *** _nadya_ has joined #openstack-infra | 13:29 | |
| *** nmagnezi has joined #openstack-infra | 13:30 | |
| stevemar | dhellmann: ^ | 13:31 |
| *** automagically_ has left #openstack-infra | 13:31 | |
| igorbelikov | pabelanger: just caught an issue with trying to generate debian image with all the infra elements - missing curl/wget:( hotfix: https://review.openstack.org/#/c/293417/ could you take a look? | 13:32 |
| AJaeger | dims, you downgraded congressclient by accident (I guess) | 13:32 |
| dims | stevemar : hmm, i watch the proposal bot daily as well and it did not catch this either :( https://review.openstack.org/#/c/288849/ | 13:32 |
| dims | stevemar : apologies, bad merge on my part | 13:32 |
| dhellmann | stevemar : +2a | 13:33 |
| dims | AJaeger : ^^ | 13:33 |
| *** sdake has joined #openstack-infra | 13:33 | |
| *** jswarren has quit IRC | 13:34 | |
| AJaeger | dims, dhellmann: patch 2 comes from the proposal bot - is there a bug in our scripts? see https://review.openstack.org/#/c/287613/2/upper-constraints.txt | 13:34 |
| dims | dhellmann : looks like proposal bot thinks 2015 is newer than 1.x so it may re-propose | 13:34 |
| clarkb | stevemar: it is going to keep doing that because 2015.1.0 is "newer" | 13:34 |
| AJaeger | dims: I agree | 13:35 |
| dhellmann | ugh | 13:35 |
| dims | whew!!! | 13:35 |
| dims | not entirely my fault :) | 13:35 |
| *** d0ugal has quit IRC | 13:35 | |
| AJaeger | dims: ;) | 13:35 |
| dhellmann | well, we won't be approving any proposal bot changes until the freeze lifts so we have some time to fix it | 13:35 |
| *** d0ugal has joined #openstack-infra | 13:35 | |
| dims | right dhellmann | 13:35 |
| * dims adds a post it on my wall | 13:36 | |
| dhellmann | I wonder if hiding the 2015 release would help? I'm not sure how the bot finds new releases | 13:36 |
| *** vgridnev has quit IRC | 13:36 | |
| ttx | or special case it in the bot | 13:36 |
| *** jpr has joined #openstack-infra | 13:36 | |
| dhellmann | yeah, that would be fairly simple, too | 13:37 |
| ttx | so taht it knows 2015 is actually < 1 | 13:37 |
| *** vgridnev has joined #openstack-infra | 13:37 | |
| ttx | at least in python-congressclient | 13:37 |
| clarkb | preferably we dont need to special case just do <2000 in the reqs | 13:37 |
| stevemar | that should work for most cases | 13:37 |
| clarkb | or something like that and let existing dep resolution filter it out | 13:37 |
| *** ddieterly has joined #openstack-infra | 13:38 | |
| dhellmann | noted in https://etherpad.openstack.org/p/newton-relmgt-plan | 13:38 |
| *** vgridnev has quit IRC | 13:38 | |
| ttx | right is there is a 20xx but no 19xx version | 13:38 |
| dhellmann | clarkb : oh, that's a good idea, too | 13:38 |
| clarkb | I want to say hiding a release doesnt affect pip only web ui browsing | 13:38 |
| ttx | humans are good at 2015 < 1. Computers aren't | 13:39 |
| dhellmann | I think you're right, but capping the requirement spec would fix it | 13:39 |
| clarkb | not sure what we did for the other clients | 13:39 |
| *** doug-fish has quit IRC | 13:39 | |
| dhellmann | stevemar : do you want to add a <2000 to global requirements in that patch? | 13:39 |
| clarkb | there were long email threads about it to go back and remember | 13:39 |
| dims | so python-congressclient<=2015 ? | 13:39 |
| *** salv-orlando has quit IRC | 13:40 | |
| dims | ah <2000 | 13:40 |
| dims | +1 | 13:40 |
| *** edmondsw has joined #openstack-infra | 13:41 | |
| stevemar | dhellmann: it's +a'ed still want to change g-r? | 13:42 |
| *** doug-fish has joined #openstack-infra | 13:42 | |
| fungi | yolanda: AJaeger: we do have "#status log" which will update https://wiki.openstack.org/wiki/Infrastructure_Status without doing noisier things in irc channels. we could start relying on that more for cross-timezone coordination (that's what it was mostly intended for, if memory serves) | 13:42 |
| dhellmann | stevemar : a follow up? | 13:42 |
| stevemar | dhellmann: rgr | 13:42 |
| dhellmann | I thought if it hadn't even passed check it wouldn't matter, but whatever you like | 13:42 |
| *** shardy has joined #openstack-infra | 13:42 | |
| yolanda | fungi, that's what i said as well, i think it's a good idea | 13:42 |
| *** tonytan4ever has joined #openstack-infra | 13:42 | |
| *** yamamoto has quit IRC | 13:43 | |
| yolanda | fungi it will be great for us in EMEA if you start applying on the US timezone, because it's hard to know about the status sometimes | 13:43 |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: prevent installing old releases of congressclient https://review.openstack.org/293441 | 13:44 |
| stevemar | dhellmann: ^ just in case that one is more contentious :) | 13:45 |
| dhellmann | stevemar : +2, maybe ttx will +2a | 13:45 |
| *** tzn has quit IRC | 13:45 | |
| *** ddieterly is now known as ddieterly[away] | 13:45 | |
| *** doug-fish has quit IRC | 13:46 | |
| *** zeih has quit IRC | 13:47 | |
| fungi | yolanda: i already added the hiera keys for 293379 yesterday in the corresponding fqdn files, yes | 13:47 |
| *** nwkarsten has joined #openstack-infra | 13:48 | |
| ttx | dims did +2a | 13:48 |
| dims | ttx : oops, did not see dhellmann 's response to wait for yours | 13:48 |
| yolanda | cool | 13:48 |
| dhellmann | dims, ttx: either way, a second person to share the blame ;-) | 13:49 |
| *** yuval has joined #openstack-infra | 13:49 | |
| dims | haha | 13:49 |
| ttx | dims: yep, I prefer people to blame you so awesome | 13:49 |
| dims | lol | 13:49 |
| stevemar | i blame dims for everything :] | 13:50 |
| yuval | Hey, who should I contact to add my details to Stackalytics default_data.json? | 13:50 |
| dims | single point of failure :) | 13:50 |
| dims | yuval : just log a review just like any other project | 13:50 |
| stevemar | yuval: you can post a change | 13:50 |
| yuval | dims: thanks | 13:50 |
| stevemar | AJaeger: got time for another question? :) | 13:50 |
| AJaeger | stevemar: a quick one - more details in a few hours... | 13:51 |
| stevemar | AJaeger: a few days ago, this change merged: https://review.openstack.org/#/c/290351/ | 13:52 |
| fungi | ttx: dims: dhellmann: clarkb: if we start adding <2000 in our requirements lists, the people who insisted on us adding python version epochs will be coming back out of the woodwork in very short order | 13:52 |
| clarkb | fungi: do you remember how we worked around that? | 13:52 |
| stevemar | but i'm not seeing the change take effect for all jobs, like this one: https://review.openstack.org/#/c/293042/ (the osc-check-plugins) output didn't change | 13:53 |
| clarkb | as far as I know this isnt a problem for the other clients so we did something | 13:53 |
| stevemar | AJaeger: do the servers need to refresh their script or something like that? | 13:53 |
| *** amitgandhinz has joined #openstack-infra | 13:53 | |
| clarkb | and I am sure it is in one of those long epoch threads | 13:53 |
| fungi | clarkb: novaclient was the only other library to suffer this fate, and it was pre-pypi | 13:53 |
| clarkb | ah | 13:53 |
| *** zeih has joined #openstack-infra | 13:54 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes https://review.openstack.org/270273 | 13:54 |
| dhellmann | do setuptools, pbr, and pip all handle epochs? | 13:54 |
| fungi | our argument for _not_ epoching things is that it was only affecting servers which are not published on pypi | 13:54 |
| *** mtanino has joined #openstack-infra | 13:54 | |
| *** mriedem is now known as mriedem_meeting | 13:54 | |
| *** ddieterly[away] is now known as ddieterly | 13:54 | |
| openstackgerrit | Merged openstack-infra/system-config: OpenStackId: Update Email configuration https://review.openstack.org/293379 | 13:54 |
| *** armax has joined #openstack-infra | 13:54 | |
| AJaeger | stevemar: let's ask fungi and clarkb - this is ubuntu-trusty-ovh-bhs1 image. I think it needs to be build with the content of jenkins/scripts. And if that image wasn't updated since then ;( | 13:55 |
| fungi | so maybe an epock for congressclient is actually in order | 13:55 |
| ttx | dhellmann: humans don't handle epochs | 13:55 |
| *** vgridnev has joined #openstack-infra | 13:55 | |
| fungi | er, epoch | 13:55 |
| ttx | spok | 13:55 |
| ttx | spock | 13:55 |
| * AJaeger will be back later | 13:55 | |
| stevemar | AJaeger: take care sir | 13:55 |
| *** jswarren has joined #openstack-infra | 13:55 | |
| fungi | live long and version | 13:55 |
| mordred | clarkb: I merged your inventory patch in ansible - thanks! | 13:56 |
| mordred | clarkb: also, I have core on ansible/ansible now | 13:56 |
| ttx | those Ansible people clearly don't know what they opening themselves to | 13:57 |
| *** exploreshaifali has quit IRC | 13:57 | |
| ttx | they *are* opening | 13:57 |
| fungi | stevemar: AJaeger: yeah, it looks like the last time we got a successful ubuntu-trusty upload to ovh-bhs1 was nearly 9 days ago | 13:57 |
| *** rossella_s has quit IRC | 13:58 | |
| mordred | fungi: that's no good | 13:58 |
| stevemar | fungi: what does one need to do to trigger a newer version? | 13:58 |
| *** zeih has quit IRC | 13:58 | |
| fungi | stevemar: figure out why glance in ovh hates us? or determine whether we've got a bug with our uploading | 13:58 |
| mordred | fungi: I wonder if maybe we should add a thing to nodepool to not boot nodes on providers whose image is older than X days | 13:58 |
| mordred | fungi: there is an age point where continuing to use the image is likely worse overall for the gate | 13:59 |
| stevemar | fungi: should it happen daily? | 13:59 |
| fungi | mordred: it's worth considering. maybe set it to 4 days or a week or something | 13:59 |
| fungi | stevemar: yep, daily like clockwork when things are working correctly | 13:59 |
| *** tonytan4ever has quit IRC | 14:00 | |
| stevemar | fungi: i see. sounds like this one is beyond capabilities of a mere mortal like myself, and needs an infra demi-god to fix >.< | 14:00 |
| *** doug-fish has joined #openstack-infra | 14:00 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Enable ceph and netiso for the nonha job https://review.openstack.org/293446 | 14:00 |
| *** tonytan4ever has joined #openstack-infra | 14:00 | |
| *** doug-fish has quit IRC | 14:01 | |
| fungi | stevemar: what's odd is that we have no trouble in ovh's gra1 region, it's just uploads to the bhs1 region that seem to be impacted by this | 14:01 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Do not set Gnocchi coordinator backend https://review.openstack.org/293424 | 14:01 |
| *** HeOS has quit IRC | 14:01 | |
| *** doug-fish has joined #openstack-infra | 14:01 | |
| stevemar | fungi: that is odd, i wonder whats different with that region | 14:01 |
| fungi | stevemar: sad considering the bits don't have nearly as far to go to reach bhs1 | 14:01 |
| fungi | while they cross the atlantic to reach gra1 | 14:02 |
| *** dmellado is now known as dmellado|lunch | 14:02 | |
| clarkb | I think ovh has trouble with those 17GB uploads... | 14:03 |
| clarkb | not a bug on our side as far as I know | 14:03 |
| *** tqtran-afk has joined #openstack-infra | 14:03 | |
| *** jpr1 has joined #openstack-infra | 14:03 | |
| fungi | clarkb: sounded like we're very close on having a dib with SamYaple's fstrim patch? | 14:03 |
| SamYaple | im here | 14:03 |
| fungi | maybe that will improve dramatically once we're using it | 14:04 |
| SamYaple | it was merged into dib, i dont think dibs been tagged | 14:04 |
| clarkb | these are raw images so wont be affected that much | 14:04 |
| fungi | oh | 14:04 |
| clarkb | also this is why using raw/vhd as transport format is silly | 14:04 |
| fungi | ovh, y u no qcow2? | 14:04 |
| SamYaple | clarkb: do you have access to glancev2 stuff with ovh? you can upload qcow2 and convert on thier backend if you do | 14:05 |
| *** jpr has quit IRC | 14:06 | |
| clarkb | uh I am not sure that properly exists yet | 14:06 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:06 | |
| *** nwkarste_ has joined #openstack-infra | 14:06 | |
| clarkb | aiui the only way to support that is via the broken tasks interface whoch is deploymwnt specific | 14:06 |
| *** roaet- has quit IRC | 14:06 | |
| *** HeOS has joined #openstack-infra | 14:06 | |
| *** nwkarsten has quit IRC | 14:06 | |
| *** mgagne has quit IRC | 14:07 | |
| *** hughsaunders has quit IRC | 14:07 | |
| *** mgagne has joined #openstack-infra | 14:07 | |
| *** jbryce has quit IRC | 14:07 | |
| SamYaple | clarkb: in ovh or in general? because thats waht i do for my cluster and ceph | 14:07 |
| *** tqtran-afk has quit IRC | 14:08 | |
| *** hughsaunders has joined #openstack-infra | 14:08 | |
| *** jbryce has joined #openstack-infra | 14:08 | |
| *** roaet_ has joined #openstack-infra | 14:09 | |
| SamYaple | it requires glance tasks, but maybe we are crossing wires? | 14:09 |
| clarkb | in general | 14:09 |
| clarkb | its not a glance feature | 14:09 |
| clarkb | it is a thing you can hide behind glance tasks ina non portable or discoverable manner that may or may not exost depending on deployment | 14:09 |
| SamYaple | but we are talking about ovh specifically requiring raw, so if that can be abstracted.... | 14:10 |
| clarkb | because tasks as an api dont do these things | 14:10 |
| clarkb | SamYaple: I am not going to use tasks for more stuff | 14:10 |
| SamYaple | fair enough, but this _is_ a feature of glance | 14:11 |
| *** nwkarste_ has quit IRC | 14:11 | |
| clarkb | it is probably the worst part of using opensta k for the infra team right now | 14:11 |
| SamYaple | that you can use in vanilla glance | 14:11 |
| SamYaple | hey you dont have to tell me. im just offering suggestions | 14:11 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Deploy ceph in the nonha job https://review.openstack.org/293051 | 14:11 |
| clarkb | SamYaple: sort of, tasks are part of glance but there is no standard set of tasks aiui | 14:11 |
| clarkb | SamYaple: so image convert isnt really part of glance | 14:12 |
| *** yamamoto has joined #openstack-infra | 14:12 | |
| mordred | we've talked about adding an image convert option to glance server | 14:12 |
| clarkb | and you cant really find out if it is available as a user, or know what data to pass a task or what the return data means or any of it | 14:12 |
| mordred | but that won't happen soon | 14:12 |
| mordred | also - the spec for fixing image upload is pretty much done now, so I expect that to get good implementation in newton | 14:13 |
| SamYaple | clarkb: glance task to import from a url. the url is hosted here with infra and its a qcow2 file, it spits out a raw image | 14:13 |
| SamYaple | that should be completely possible as a user | 14:14 |
| clarkb | SamYaple: where is it documented? that isnt in the api docs | 14:14 |
| clarkb | because tasks are not standard only the minimal api to start tadks is | 14:14 |
| mordred | almost no clouds have tasks | 14:15 |
| pabelanger | igorbelikov: do you have a log file of the problem? Curious when you get the error. We recently added curl into puppet manifests, but that could be too late in the build | 14:15 |
| mordred | and no clouds have tasks that do image type conversions | 14:15 |
| *** salv-orlando has joined #openstack-infra | 14:15 | |
| mordred | and to my knowledge there are no clouds that have a task for import-from-url | 14:15 |
| *** Hal has joined #openstack-infra | 14:17 | |
| igorbelikov | pabelanger: the interesting part is here http://paste.openstack.org/show/490714/, though I can provide a full log if you need it | 14:17 |
| SamYaple | clarkb: where is it documented, i dont know. but this is what i use glance --os-image-api-version 2 task-create --type import --input '{"import_from_format": "qcow2", "import_from": "http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img", "image_properties": {"name": "cirros-0.3.4-raw", "disk_format": "raw", "container_format": "bare"}}' | 14:17 |
| *** Hal is now known as Guest50051 | 14:17 | |
| *** rbrndt has joined #openstack-infra | 14:17 | |
| SamYaple | mordred: if its not available its not availble, this was just a suggestion | 14:17 |
| clarkb | SamYaple: right thats not part of glance | 14:17 |
| pabelanger | igorbelikov: Odd, did you have an issue before with url / wget on debian? | 14:18 |
| clarkb | thats part of a particular glance deployment that has chosen to support a task that takes this input | 14:18 |
| clarkb | this is the problem.with tasks | 14:18 |
| mordred | SamYaple: what cloud do you use that with? | 14:18 |
| SamYaple | mordred: vanilia deployed services in a cloud i manage (minimal configs, using defaults) | 14:19 |
| clarkb | but with ceph | 14:19 |
| SamYaple | it requires v2 glance obviously | 14:19 |
| igorbelikov | pabelanger: think I may have tested this with a set of patches which included curl at some point | 14:19 |
| SamYaple | clarkb: that doesnt matter here | 14:19 |
| clarkb | I am guessing the ceph glance bacjend currenetly supports that task | 14:19 |
| SamYaple | no i can do it without ceph too | 14:19 |
| SamYaple | with file backend | 14:19 |
| mordred | SamYaple: weird. is this mitaka? | 14:20 |
| mordred | maybe this got fixed/added recently | 14:20 |
| SamYaple | this would be mitaka yes, but its in liberty | 14:20 |
| mordred | k | 14:20 |
| SamYaple | im no glance expert here. more testing is needed | 14:20 |
| mordred | I will go investigate the clouds we have access to | 14:20 |
| SamYaple | i thought this was a known thing, i certainly use it | 14:20 |
| mordred | and see if that thing works | 14:20 |
| mordred | on them | 14:20 |
| *** jpr1 has quit IRC | 14:20 | |
| clarkb | SamYaple: it isnt documented in the api | 14:20 |
| sdague | some version of this has been around for a while | 14:20 |
| SamYaple | clarkb: well i found it by digging through code so that makes sense | 14:20 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Bug Fixing https://review.openstack.org/293459 | 14:21 |
| *** zeih has joined #openstack-infra | 14:21 | |
| pabelanger | igorbelikov: ok | 14:21 |
| clarkb | the only api you get is create list and show tasks | 14:21 |
| clarkb | but no info on what available tasks are | 14:21 |
| clarkb | and what inputs they accept and outputs they write | 14:21 |
| SamYaple | clarkb: technically this is just a create task | 14:22 |
| clarkb | yes but the import from task is part of the api | 14:22 |
| *** tonytan4ever has quit IRC | 14:22 | |
| clarkb | basically its how not to api | 14:22 |
| *** tonytan_brb has joined #openstack-infra | 14:22 | |
| SamYaple | is it? it looks like normal glance stuff to me (ignoring tasks all together import_from is a glance thing) | 14:22 |
| clarkb | we could rewrite all our apis to be "do random undocumented thing" | 14:23 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Bug Fixing https://review.openstack.org/293459 | 14:23 |
| SamYaple | clarkb: im not saying youre wrong, but the import_from_format and import_from are all valid glance things, without tasks | 14:24 |
| sdague | ah, we only test the v1 version of this upstream - http://logs.openstack.org/67/271467/2/check/gate-tempest-dsvm-full/a17f519/console.html.gz#_2016-02-25_02_17_31_794 | 14:24 |
| sdague | yes, glance v2 is definitely super undocumented | 14:24 |
| clarkb | SamYaple: but they arent formally part of the api | 14:24 |
| clarkb | SamYaple: instead its hidden behind tasks which completely breaks the whole.system for users | 14:24 |
| SamYaple | clarkb: im not arguing there :) | 14:24 |
| sdague | clarkb: payload is part of the API | 14:24 |
| sdague | allowed tasks should be documented | 14:25 |
| clarkb | sdague: yes thats the problem | 14:25 |
| clarkb | sdague: it is arguably part of the api but it isnt treated that eay | 14:25 |
| sdague | so, I agree, the API is not documented here | 14:25 |
| sdague | right, well that's garbage :) | 14:25 |
| clarkb | instead we have documentation for run a function | 14:25 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Enable voting for gate jobs in sahara-tests https://review.openstack.org/293464 | 14:25 |
| clarkb | the names and args to the function are neglected | 14:25 |
| fungi | turing-complete api | 14:25 |
| clarkb | POST dosoemthing | 14:25 |
| sdague | that being said, updating API docs is hard today | 14:25 |
| clarkb | thats ehat we get | 14:25 |
| SamYaple | are you guys telling me you dont dig through the projects looking for cool undocumented thigns? | 14:26 |
| SamYaple | cause thats like 40% of what i do | 14:26 |
| sdague | SamYaple: :) | 14:26 |
| yolanda | sdague, i fixed https://review.openstack.org/292792, added some test as well | 14:26 |
| *** Daisy_ has joined #openstack-infra | 14:26 | |
| sdague | yolanda: thanks | 14:27 |
| sdague | +2 | 14:27 |
| yolanda | great | 14:27 |
| *** sdake has quit IRC | 14:27 | |
| clarkb | then on top of post dosomething other deployments can add extra random undocumented functions | 14:27 |
| *** zeih has quit IRC | 14:27 | |
| clarkb | why bother having a rest api just give me a generic computing tpc interface | 14:27 |
| *** nwkarsten has joined #openstack-infra | 14:27 | |
| SamYaple | i think clarkb might have a heart attack over this | 14:27 |
| clarkb | SamYaple: its something we have struggled witha lot in infra | 14:28 |
| clarkb | SamYaple: rax has/had their own tasks that we cant easily test against because the code isnt upstream | 14:28 |
| clarkb | but these tasks are the only eay to upload to rax | 14:28 |
| *** Swami has joined #openstack-infra | 14:29 | |
| SamYaple | i worked at the RAX, i know about the special snowflake stuff | 14:29 |
| clarkb | and you get errors back in json like "error 396" | 14:29 |
| clarkb | in an http 200 | 14:29 |
| fungi | we get the very distinct impression that the number of openstack users who use more than 1 or 2 different openstack public clouds are rare enough not to provide feedback on the terribly inconsistencies we have in our ecosystem | 14:29 |
| SamYaple | its a bit annoying, i give you that | 14:29 |
| clarkb | so the api says "success! error" | 14:29 |
| *** zz_dimtruck is now known as dimtruck | 14:29 | |
| fungi | er, terrible | 14:29 |
| SamYaple | thats kinda like getting a pizza with a supoena inside | 14:30 |
| *** fawadkhaliq has quit IRC | 14:30 | |
| clarkb | as a resukt my opinion of the tasks api is very bad | 14:30 |
| fungi | SamYaple: subpoena pizza with anchovy is the best | 14:30 |
| *** fawadkhaliq has joined #openstack-infra | 14:30 | |
| *** Daisy_ has quit IRC | 14:31 | |
| SamYaple | i like bonchovy better | 14:31 |
| *** nwkarsten has quit IRC | 14:31 | |
| vkramskikh | hi, AJaeger, anteaya, could you please help with review of https://review.openstack.org/#/c/292760/ ? this patch will add linting jobs for fuel-ui project | 14:32 |
| clarkb | the best part is glance has an upload api too | 14:32 |
| fungi | for the record, the majority of images in ovh bhs1 are over a week old. i'm tempted to delete all images over a week old and get nodepool to aggressively reupload until new ones stick. any reasons i shouldn't? | 14:32 |
| clarkb | so making users suffer through an undocumented task to upload is pretty terrible | 14:32 |
| clarkb | fungi: if we need the capacity that region is one of our biggest | 14:33 |
| *** sc68cal has joined #openstack-infra | 14:33 | |
| *** ayoung has joined #openstack-infra | 14:33 | |
| fungi | yeah, we're maxxed out on our quotas for the past hour or two, so maybe now is not the best time | 14:34 |
| pabelanger | but if uploads haven't happened for a week, do we want jobs to run in that zone on stale images? | 14:34 |
| *** [1]Thelo has joined #openstack-infra | 14:35 | |
| fungi | "stale" is not necessarily bad. they will be lacking slave script updates or anything that we adjusted in puppet/elements but they still update their packages and git repos when they run | 14:35 |
| clarkb | right caches get updated and are generally fine | 14:36 |
| pabelanger | /me nods | 14:36 |
| fungi | also we're running a pretty large band of ready and building nodes so i'm going to make sure something's not hobbling us somwhere | 14:37 |
| pabelanger | I want to get a grafana dashboard to better track dib builds and image uploads | 14:37 |
| pabelanger | so we can easy see what each zone is running | 14:37 |
| *** Thelo has quit IRC | 14:37 | |
| *** [1]Thelo is now known as Thelo | 14:37 | |
| clarkb | fungi: jenkins07 needs to be restarted, the playbook died on it yssterday | 14:37 |
| fungi | i'll take care of that unless you're already on it | 14:37 |
| fungi | looks like it's to blame for a lot of these ready nodes actually | 14:38 |
| fungi | i'm clearing those away now | 14:38 |
| pabelanger | fungi: clarkb: when you have a second, could use some feedback on 293417 regarding curl on dibs | 14:39 |
| pabelanger | specific minimal images | 14:39 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-changes option https://review.openstack.org/293472 | 14:39 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 14:40 |
| *** _nadya_ has quit IRC | 14:41 | |
| clarkb | pabelanger: I almost wonder if we should have a make-minimal-useable element that installs curl and sshd and so on | 14:41 |
| clarkb | but since the puppet element needs curl thats a decent locatio for now | 14:42 |
| jeblair | pabelanger: i think just a simple nodepool status interface would be more useful for image status than a dashboard. i think those data are best in tabular form. | 14:42 |
| *** roxanagh_ has joined #openstack-infra | 14:42 | |
| *** yuval has left #openstack-infra | 14:43 | |
| clarkb | "make minimal images great again" element | 14:43 |
| fungi | would be nice to slowly move content from the puppet element into a shim element | 14:43 |
| openstackgerrit | Alexey Ovchinnikov proposed openstack-infra/project-config: Reverting Manila LXD driver changes https://review.openstack.org/293474 | 14:43 |
| fungi | as we un-puppet bits our job workers need | 14:43 |
| fungi | okay, jenkins07 is cleaned up and on its way back into service now | 14:44 |
| *** asselin_ has joined #openstack-infra | 14:44 | |
| *** _nadya_ has joined #openstack-infra | 14:45 | |
| *** zeih has joined #openstack-infra | 14:45 | |
| pabelanger | clarkb: Right, I would agree with that | 14:45 |
| pabelanger | jeblair: Ya, only issue with that, my html foo is weak. Grafana makes it easy for me to render stuff :) | 14:46 |
| jeblair | pabelanger: i don't think it needs to be html :) | 14:46 |
| pabelanger | Ahh | 14:46 |
| fungi | browsers are great at rendering preformatted text/plain | 14:46 |
| fungi | or it could be json and let someone else who cares write the renderer | 14:47 |
| *** asselin__ has joined #openstack-infra | 14:47 | |
| pabelanger | fungi: I was thinking using minimal2 as the element name :D | 14:48 |
| fungi | `wget | flamel` works for me as a renderer ;) | 14:48 |
| *** nwkarsten has joined #openstack-infra | 14:48 | |
| *** asselin_ has quit IRC | 14:48 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job https://review.openstack.org/260466 | 14:49 |
| *** roxanagh_ has quit IRC | 14:50 | |
| jeblair | fungi: i'm still a little bleary eyed from waking up, and was wondering why you're so positive about "wet flannel" | 14:50 |
| fungi | it's a seattle thing | 14:51 |
| *** akaszuba has quit IRC | 14:52 | |
| *** nwkarsten has quit IRC | 14:53 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue https://review.openstack.org/293178 | 14:54 |
| *** cloudtrainme has joined #openstack-infra | 14:55 | |
| *** tosky has joined #openstack-infra | 14:55 | |
| *** ogelbukh has quit IRC | 14:55 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the container CI job https://review.openstack.org/293481 | 14:56 |
| openstackgerrit | Merged openstack-infra/project-config: Add curl package to puppet nodepool element https://review.openstack.org/293417 | 14:56 |
| *** mriedem_meeting is now known as mriedem | 14:56 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fixed DH error https://review.openstack.org/293482 | 14:57 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 14:58 |
| EmilienM | jeblair: where can I report a but to Gertty? | 14:58 |
| *** zeih has quit IRC | 14:58 | |
| jeblair | EmilienM: here or storyboard | 14:58 |
| *** nwkarsten has joined #openstack-infra | 14:58 | |
| EmilienM | jeblair: since recently, when I open latest Gertty and go on a review and press on "Diff" to read code, Gertty crash with http://paste.openstack.org/show/nvFDq0eCgkeIYeiYDK2U/ | 14:59 |
| jeblair | EmilienM: if you update to latest master, it should be fixed | 15:00 |
| fungi | do we still need the LaunchpadSync account to be a member of the Administrators group in gerrit? i'm thinking the only thing it's doing now is bug/bp updates, and the reason we had it in administrators previously was for auto-abandon (and further back, also group creation/membership updates from lp) | 15:01 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-projects option https://review.openstack.org/293472 | 15:01 |
| EmilienM | jeblair: so 1.3.1 is "broken" ? | 15:01 |
| fungi | all the cool kids continuously deploy gertty from master ;) | 15:01 |
| EmilienM | lol | 15:02 |
| EmilienM | ok ok get it! | 15:02 |
| jeblair | EmilienM: gitpython shipped a breaking change :/ | 15:02 |
| fungi | which they also seem unwilling to acknowledge as a problem, much less fix | 15:02 |
| jeblair | well, i think they acknowledged the problem | 15:02 |
| fungi | EmilienM: you _can_ also downgrade gitpython to similar effect | 15:02 |
| jeblair | but are completely baffeled by the idea of fixing it | 15:02 |
| EmilienM | fungi: I want to be cool kid. | 15:03 |
| jeblair | i guess we could make a brnach and tag a 1.3.2, or i could see if master is ready to be 1.4.0 | 15:03 |
| fungi | jeblair: i may have not read closely enough, but it seemed like they acknowledged it was a behavior change, they just didn't acknowledge that behavior changes like that in a library are a problem from their perspective | 15:03 |
| EmilienM | fungi, jeblair: works fine on master, thanks guys | 15:04 |
| jeblair | fungi: he said something like "something is broken either way" | 15:04 |
| fungi | and apparently new broken is superior to old broken | 15:04 |
| jeblair | in a tone that made it seem like there was some vague sense of regret | 15:04 |
| jeblair | and that's about as far as it goes | 15:04 |
| *** dizquierdo has quit IRC | 15:05 | |
| *** vgridnev has quit IRC | 15:05 | |
| *** Guest50051 has quit IRC | 15:05 | |
| *** yamamoto has quit IRC | 15:06 | |
| jeblair | i'll work on 132 | 15:06 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fixed DH error https://review.openstack.org/293482 | 15:07 |
| *** sfinucan has joined #openstack-infra | 15:07 | |
| *** asselin__ has quit IRC | 15:08 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 15:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: WIP: Add python-db jobs - for nova initially https://review.openstack.org/292472 | 15:11 |
| jeblair | EmilienM: I tagged gertty 1.3.2 with the fix | 15:12 |
| EmilienM | jeblair++ | 15:12 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: Add user smarcet to openstackid production server https://review.openstack.org/293491 | 15:12 |
| EmilienM | jeblair: thx, I love this tool really | 15:12 |
| jeblair | EmilienM: thanks! :) | 15:12 |
| *** dizquierdo has joined #openstack-infra | 15:13 | |
| *** mrmartin has quit IRC | 15:13 | |
| pabelanger | okay, relocating again. Should be back in a hour | 15:13 |
| *** david-lyle has quit IRC | 15:15 | |
| *** david-lyle has joined #openstack-infra | 15:15 | |
| *** yamahata has joined #openstack-infra | 15:15 | |
| AJaeger | clarkb: how were today's ubuntu-trusty dsvm runs? Are we ready to merge https://review.openstack.org/#/c/292573/ ? | 15:15 |
| *** dmellado|lunch is now known as dmellado | 15:15 | |
| *** roxanagh_ has joined #openstack-infra | 15:16 | |
| *** ociuhandu has quit IRC | 15:16 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fixed DH error https://review.openstack.org/293482 | 15:16 |
| *** jpr has joined #openstack-infra | 15:16 | |
| clarkb | AJaeger: when I looked yesterday they seemed fine. I think we can merge that change | 15:17 |
| zaro | fungi: i think you were interested in something like this? https://gerrit.googlesource.com/plugins/review-strategy/+/master/src/main/resources/Documentation/about.md | 15:19 |
| AJaeger | clarkb: I suggest a followup to change nodepool min-ready for ubuntu-trusty and devstack-trusty | 15:19 |
| AJaeger | clarkb: ONe new devstack-trusty sneaked in - do you want to rebase - or fix afterwards? | 15:20 |
| AJaeger | clarkb: +2 - let's see whether somebody else wants to +2 and then we need a volunteer to mass update jenkins... | 15:21 |
| clarkb | AJaeger: we can fix after | 15:21 |
| clarkb | its also only 400 something job updates so may not need manual jjb | 15:21 |
| AJaeger | clarkb: I fear we need it on jenkins02 at least - but let's see | 15:22 |
| *** placko_ has quit IRC | 15:22 | |
| *** ociuhandu has joined #openstack-infra | 15:23 | |
| openstackgerrit | Merged openstack-infra/project-config: Reverting Manila LXD driver changes https://review.openstack.org/293474 | 15:23 |
| fungi | clarkb: AJaeger: i'll babysit some manual jjb updates once it merges | 15:24 |
| fungi | just to be sure | 15:24 |
| *** vgridnev has joined #openstack-infra | 15:25 | |
| fungi | once we're down to 0 references to devstack-trusty in jjb we'll also want another zuul restart so that we wipe out the old registrations | 15:25 |
| *** sdake has joined #openstack-infra | 15:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Change min-ready for ubuntu-trusty/devstack-trusty https://review.openstack.org/293506 | 15:26 |
| AJaeger | clarkb, fungi, here's the nodepool update ^ | 15:26 |
| AJaeger | fungi, we still need somebody to +2A https://review.openstack.org/#/c/292573 | 15:27 |
| jeblair | zaro: i can tell you are the author of the change in that screenshot, since you are the first person listed in the history. | 15:27 |
| AJaeger | thanks fungi for babysitting this! | 15:29 |
| SamYaple | mordred: any luck with the clouds? | 15:30 |
| SamYaple | and glance specifically | 15:30 |
| AJaeger | fungi, is https://review.openstack.org/292472 (the python-db change) still on your list of changes to review in a quiet minute? | 15:30 |
| *** pvaneck has joined #openstack-infra | 15:31 | |
| fungi | AJaeger: yep | 15:31 |
| fungi | adding that now | 15:31 |
| *** nmagnezi has quit IRC | 15:31 | |
| AJaeger | thanks | 15:31 |
| *** zeih has joined #openstack-infra | 15:32 | |
| tosky | question about code moving between repositories (for example, tests from or to tempest): usually people simply drop the last version of the code in a review | 15:32 |
| SamYaple | so dare i ask for reviews on this patch https://review.openstack.org/#/c/293021/ | 15:32 |
| tosky | and then remove the old code from the source repository | 15:32 |
| *** fawadkhaliq has quit IRC | 15:33 | |
| *** ianychoi3 has joined #openstack-infra | 15:33 | |
| tosky | would/did you allow or executed in the past code moving which kept the history? Like pushing a filtered subset of commits which affected the moved code | 15:33 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use m1.small for ping test https://review.openstack.org/293511 | 15:34 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Boot from volume in ping test https://review.openstack.org/293512 | 15:34 |
| *** fawadkhaliq has joined #openstack-infra | 15:34 | |
| *** david-lyle_ has joined #openstack-infra | 15:34 | |
| fungi | AJaeger: what do you think about splitting the jjb and zuul parts of 292472 into separate changes? i'm worried that we'll have a bunch of nova devs coming in here asking why all their jobs are coming back NOT_REGISTERED | 15:34 |
| AJaeger | SamYaple: ?>! - really strange... | 15:34 |
| AJaeger | fungi: sure, let me do that... | 15:34 |
| *** david-lyle has quit IRC | 15:34 | |
| fungi | AJaeger: though i guess that means double-instantiating some jobs if you keep the job groups so that may not be possible | 15:35 |
| SamYaple | AJaeger: whats strange? | 15:35 |
| SamYaple | you mean ?<! | 15:35 |
| AJaeger | SamYaple: I learned something new ;) | 15:35 |
| fungi | AJaeger: we could instead shoot for applying it at a low-impact time | 15:35 |
| SamYaple | thats teh look behind operator | 15:35 |
| SamYaple | well thats the dont match if exists behind operator (thanks to !) | 15:36 |
| fungi | AJaeger: also, should probably get a +1 from a nova liaison or ptl since their job names are changing (they might have tools tracking/trending some results by job name or something) | 15:36 |
| clarkb | tosky you can always push and approve a stack of changes. If your goal is to push a git filtered tree into the repo bypassing review we tend to avoid that | 15:36 |
| *** ianychoi2 has quit IRC | 15:36 | |
| *** jaosorior has quit IRC | 15:36 | |
| prometheanfire | AJaeger: I think ubuntu-trusty isn't so happy here https://review.openstack.org/#/c/293193/ | 15:37 |
| prometheanfire | AJaeger: it was passing before that patchest | 15:37 |
| tosky | clarkb: I understand the reason for that (not bypassing review); the idea would be to have the review containing the last version of the filtered git tree, test it, but then push the set | 15:37 |
| tosky | clarkb: but I understand that it would be non-standard | 15:37 |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Add javascript jobs to fuel-ui project https://review.openstack.org/292760 | 15:38 |
| *** fawadkhaliq has quit IRC | 15:38 | |
| prometheanfire | AJaeger: it was patchset 9->10 that started failing, the only change in that interdiff was ubuntu-trusty | 15:39 |
| fungi | tosky: also, splitting repo histories is fairly trivial, but grafting different repos together when they lack a common history is not | 15:39 |
| prometheanfire | patchest 11 was a commit-message change | 15:39 |
| openstackgerrit | Merged openstack-infra/project-config: Switch rest of devstack-trusty to ubuntu-trusty https://review.openstack.org/292573 | 15:39 |
| tosky | fungi: I know, we did it sometime in another project (kde.org) | 15:39 |
| tosky | there are (very) few repos with two "heads" joining at some point | 15:40 |
| *** jroll has quit IRC | 15:41 | |
| *** jroll has joined #openstack-infra | 15:41 | |
| *** jroll has quit IRC | 15:41 | |
| *** jroll has joined #openstack-infra | 15:41 | |
| AJaeger | prometheanfire: red the log file, it says " gate-dib-dsvm-functests-devstack-trusty not defined" | 15:42 |
| AJaeger | You renamed the job to gate-dib-dsvm-functests-ubuntu-trusty, so fix it in layout.yaml | 15:42 |
| *** bpokorny has joined #openstack-infra | 15:42 | |
| prometheanfire | ah | 15:42 |
| prometheanfire | thanks | 15:43 |
| *** adrian_otto has joined #openstack-infra | 15:45 | |
| clarkb | in theory you can push all the changes to a brach with a parent of 0 then merge them into master | 15:45 |
| mordred | SamYaple: I have not tried them yet | 15:45 |
| clarkb | and thats doable without infra doing anything I think | 15:45 |
| clarkb | well maybe an acle update for the merge commit push | 15:46 |
| tosky | clarkb: I was going to ask about that point, yes :) | 15:48 |
| tosky | clarkb: so, right now, no one can do merge commits? | 15:48 |
| clarkb | tosky: no, many people can just not by default | 15:49 |
| clarkb | it requires a special permission be added to the project acl | 15:49 |
| fungi | mainly because the number of times people accidentally create a merge commit and then try to push that in gerrit far outnumbers the number of times people legitimately need to push a merge commit into gerrit | 15:50 |
| tosky | thinking about it, putting the changes in a separate branch means no use of gates, as the code will be different before the merging operation | 15:50 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 15:50 |
| fungi | much like how we restrict the ability of most users to push tags into gerrit | 15:51 |
| fungi | tosky: well, jobs still run on the merge commit and it gets reviewed like any other commit, though gerrit's display of merge commits is a bit... strange | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially https://review.openstack.org/292472 | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs https://review.openstack.org/293526 | 15:51 |
| AJaeger | fungi, updated ^ | 15:51 |
| AJaeger | johnthetubaguy, sdague, could you review the two changes above, please? ^ | 15:52 |
| fungi | AJaeger: awesome. and we should get nova feedback on the second one | 15:52 |
| fungi | AJaeger: hah, you just asked for it--thanks! | 15:52 |
| tosky | fungi: oh, I see, so the ACL for merge should set up before pushing the review for the merge commit - got it | 15:52 |
| AJaeger | johnthetubaguy, sdague: This renames gate-nova-python27 to gate-nova-python27-db etc. So, that in a followup we can remove database setup from the normal python27 jobs (similar for python34 etc) | 15:52 |
| *** apoorvad has joined #openstack-infra | 15:52 | |
| tosky | thanks for the hints - I'm going to see if I can make it working, and in case I would kindly ping you for the ACL | 15:53 |
| fungi | tosky: yep, it's generally restricted to just the people who will regularly need to push merge commits for a particular project, or even just on a temporary basis | 15:53 |
| AJaeger | fungi, clarkb, could you review the nodepool update, please? https://review.openstack.org/293506 | 15:53 |
| fungi | AJaeger: i will once i get these jjb updates running | 15:54 |
| AJaeger | thanks, fungi | 15:54 |
| *** gyee has joined #openstack-infra | 15:55 | |
| AJaeger | sdague, johnthetubaguy I'll be offline for the next hour or two. If you have any questions about 293526 and 292472, please ask and I'll answer later - or discuss with fungi. | 15:55 |
| *** tosky has quit IRC | 15:57 | |
| AJaeger | fungi, pinged them on #openstack-nova as well... | 15:58 |
| fungi | yep, totally happy to discuss that change with them as needed | 15:58 |
| *** maishsk has quit IRC | 15:59 | |
| *** cdent has joined #openstack-infra | 15:59 | |
| *** tosky has joined #openstack-infra | 16:00 | |
| cdent | per request, I'm showing up to say "if there's desire to get purplerbot on infrastructure, I'm happy to push" but also to say "I'm fine leaving it run on my own stuff" | 16:00 |
| cdent | It will need some work to be correctly deployable (some static paths at the moment) but that's easy enough | 16:01 |
| *** e0ne has joined #openstack-infra | 16:01 | |
| *** jpr has quit IRC | 16:01 | |
| mriedem | AJaeger: why do we need a database setup for pep8/docs jobs? | 16:01 |
| AJaeger | mriedem: only for python27 jobs, not for pep8/docs | 16:02 |
| mriedem | AJaeger: well, on https://review.openstack.org/#/c/292472/6/zuul/layout.yaml | 16:02 |
| mriedem | pep8 and docs are under python-db-jobs | 16:02 |
| AJaeger | mriedem: we change the template from python-jobs which includes docs and pep to python-db-jobs | 16:02 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Create a test flavor for the pingtest VM https://review.openstack.org/293533 | 16:02 |
| AJaeger | mriedem: scroll up, the few liens above | 16:03 |
| mriedem | AJaeger: but they don't use a db? | 16:03 |
| *** sridhar_ram has joined #openstack-infra | 16:03 | |
| ddieterly | need some help with https://github.com/openstack-infra/devstack-gate | 16:03 |
| AJaeger | mriedem: The job names are not changed, so those will not use a database | 16:03 |
| AJaeger | but I don't see a better way of doing it. | 16:03 |
| mriedem | oh i see | 16:03 |
| ddieterly | when i checkout my changes to workspace-cache, the changes don't ever get copied into /opt/stack/new | 16:03 |
| ddieterly | the changes just get deleted | 16:03 |
| AJaeger | mriedem: but if you have ideas for better naming to not confuse people - and you have been confused - please bring them up. | 16:04 |
| *** fawadkhaliq has joined #openstack-infra | 16:04 | |
| *** matrohon has quit IRC | 16:05 | |
| cdent | biab | 16:05 |
| *** cdent has quit IRC | 16:05 | |
| AJaeger | mriedem: let me update the comments in zuul/layout | 16:05 |
| Zara | corrected storyboard sender email patch in review: https://review.openstack.org/#/c/293367/ ; *this* one should be the last one, argh. | 16:06 |
| *** vgridnev has quit IRC | 16:06 | |
| *** yamamoto has joined #openstack-infra | 16:07 | |
| mriedem | AJaeger: clarification in the comments would be helpful | 16:07 |
| mriedem | i added some more | 16:07 |
| jeblair | mordred: in the _create method in https://review.openstack.org/#/c/226751/60/nodepool/fakeprovider.py why are you setting public_v6 but not public_v4? | 16:07 |
| *** harlowja_at_home has joined #openstack-infra | 16:07 | |
| mordred | jeblair: looking | 16:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially https://review.openstack.org/292472 | 16:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs https://review.openstack.org/293526 | 16:09 |
| fungi | zaro: does that plugin also mask comment authors on inline comments? it's an interesting feature i might use if we had it (though i'd be more likely to benefit from an implementation in gertty than the webui) | 16:10 |
| *** IlyaG has joined #openstack-infra | 16:10 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 16:10 |
| AJaeger | mriedem: I added comments and TODOs, better now? | 16:10 |
| openstackgerrit | Igor Belikov proposed openstack-infra/system-config: Regenerate nodepool.logging config https://review.openstack.org/293396 | 16:10 |
| mordred | jeblair: I cannot come up with a good reason - it seems like a mistake to do so | 16:10 |
| jeblair | mordred: those used to be something set later by get_openstack_vars(), but that seems to have been dropped... | 16:10 |
| mordred | yah. because you get them filled out by default now | 16:11 |
| jeblair | mordred: does shade ...ok yeah, so shade does that for all created servers? | 16:11 |
| jeblair | and all servers you "get'? | 16:11 |
| AJaeger | will be back online later... | 16:11 |
| mordred | yah. it does all the things it can do without additional API calls all the time | 16:11 |
| jeblair | cool | 16:11 |
| *** tzn has joined #openstack-infra | 16:11 | |
| mugsie | hey - is there a way for me to take ownership of a group in gerrit? I cannot add people to designate-stable-maint | 16:12 |
| *** cloudtrainme has quit IRC | 16:12 | |
| fungi | mugsie: i believe the *-stable-maint groups in projects under official release management are owned by the stable-maint-core team | 16:12 |
| mugsie | the stable-maint team don't look after designate | 16:13 |
| fungi | mugsie: the idea was that changes to those stable subteams would be vetted by the main stable team for openstack | 16:13 |
| *** nwkarsten has quit IRC | 16:13 | |
| fungi | mugsie: so that they could confirm the reviewers being added were familiar with stable maintenance guidelines | 16:13 |
| mriedem | AJaeger: yeah | 16:13 |
| *** nwkarsten has joined #openstack-infra | 16:14 | |
| mugsie | ok. so #openstack-stable is my best bet then? | 16:14 |
| fungi | mugsie: it's worth discussing with the current and/or future stable maint ptl on whether they want to change teh process too, yes | 16:14 |
| *** harlowja_at_home has quit IRC | 16:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Barbican & Magnum roles to OpenStack-Ansible https://review.openstack.org/293540 | 16:15 |
| *** yamamoto has quit IRC | 16:16 | |
| *** nwkarsten has quit IRC | 16:18 | |
| *** Swami has quit IRC | 16:19 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Barbican & Magnum roles to OpenStack-Ansible https://review.openstack.org/293540 | 16:19 |
| pabelanger | and back online | 16:19 |
| *** tphummel has quit IRC | 16:20 | |
| *** david-lyle_ is now known as david-lyle | 16:20 | |
| *** esikachev has quit IRC | 16:21 | |
| pabelanger | AJaeger: are we ready for 293506 to land? Didn't know we did the devstack-trusty migration already | 16:21 |
| *** cloudtrainme has joined #openstack-infra | 16:21 | |
| *** jpr has joined #openstack-infra | 16:22 | |
| pabelanger | just found the commit :) | 16:22 |
| *** IlyaG has quit IRC | 16:22 | |
| *** harlowja_at_home has joined #openstack-infra | 16:22 | |
| *** IlyaG has joined #openstack-infra | 16:23 | |
| pabelanger | fungi: actually if you are good with 293506 I can help monitor it landing | 16:24 |
| pabelanger | then see what is needed to clean up devstack-trusty images | 16:24 |
| fungi | pabelanger: well, i wanted the job updates to finish first. i'm running/monitoring those still | 16:25 |
| pabelanger | fungi: Right, that's what I am looking at too | 16:26 |
| fungi | the telltale sign will be once we see the devstack-trusty used count fall to 0 | 16:26 |
| pabelanger | and where do you monitor that? | 16:26 |
| fungi | at that point, switching the min-ready to 0 seems fine to me | 16:26 |
| pabelanger | graphite.o.o? | 16:26 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Fix two grafana entries to point at the right zuul pipeline https://review.openstack.org/293544 | 16:26 |
| fungi | pabelanger: that's one place to do it. or spot checks of nodepool list | 16:27 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the all CI jobs https://review.openstack.org/293481 | 16:27 |
| pabelanger | fungi: great | 16:27 |
| *** sbelous_ has quit IRC | 16:27 | |
| pabelanger | we should setup grafana too :) | 16:27 |
| fungi | yeah, node states by label would be an interesting set of graphs | 16:28 |
| *** nwkarsten has joined #openstack-infra | 16:28 | |
| *** yuval has joined #openstack-infra | 16:28 | |
| fungi | i wouldn't put all the node labels on one graph because ubuntu-trusty is going to dominate by orders of magnitude | 16:29 |
| fungi | so it would be pretty hard to read | 16:29 |
| pabelanger | ya, we could add a template var to the nodepool dashboard, allowing you to limit the singleton panel by node label | 16:30 |
| pabelanger | could be the easiest | 16:30 |
| fungi | but a graph per node label, trending each of the states for that label, would be useful | 16:30 |
| clarkb | oh good east is in houston now, I suppose its just a matter of time for them to give it power and networking | 16:30 |
| *** ramishra has quit IRC | 16:30 | |
| pabelanger | nice | 16:30 |
| fungi | clarkb: except we wanted west because the hardware was better | 16:30 |
| *** cdent has joined #openstack-infra | 16:30 | |
| clarkb | fungi: ya.... | 16:30 |
| fungi | and that still hasn't left the hangar | 16:30 |
| clarkb | but at least we have some progress | 16:30 |
| clarkb | I really want to be able to track down the io performance issues | 16:31 |
| clarkb | and that requires having clouds of some sort | 16:31 |
| *** ramishra has joined #openstack-infra | 16:32 | |
| fungi | i thought the i/o performance issues boiled down to the lack of bbwc cards for the raid controllers | 16:32 |
| *** fawadkhaliq has quit IRC | 16:32 | |
| *** fawadkhaliq has joined #openstack-infra | 16:33 | |
| *** fawadkhaliq has quit IRC | 16:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Apt mirror for ceph hammer release https://review.openstack.org/293155 | 16:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Make reprepro useable on multiple mirrors https://review.openstack.org/293154 | 16:34 |
| clarkb | fungi: aiui not really. bbwc cards allow you to more safely reduce the caching in kvm which is conservative by default | 16:34 |
| *** yuval has quit IRC | 16:35 | |
| clarkb | fungi: the theory was making kvm pretend writes were complete before they were would give us the erformance we wanted but we never got to test that | 16:35 |
| clarkb | and that bbwc cards would protect our VMs with that setting in place | 16:35 |
| clarkb | mordred: ^ the above reprepro changes are a thing you may be interested in | 16:35 |
| mordred | looking | 16:36 |
| smarcet | fungi: i am waiting for a new deployment on openstackid-dev, but its been a while that i am looking PENDING STATUS | 16:36 |
| smarcet | i am not seeing any error on puppet board | 16:36 |
| *** Swami has joined #openstack-infra | 16:37 | |
| fungi | smarcet: deploy status says "INSTALLED [16-Mar-2016 15:24]" | 16:38 |
| *** links has quit IRC | 16:38 | |
| fungi | in the log i find: Mar 16 15:54:55 openstackid-dev puppet-user[22721]: (/Stage[main]/Openstackid/Exec[update-site]/returns) executed successfully | 16:39 |
| *** tphummel has joined #openstack-infra | 16:39 | |
| fungi | smarcet: you were waiting for https://review.openstack.org/293482 to deploy presumably? | 16:40 |
| clarkb | that was long enough ago that puppetboard should know about it | 16:40 |
| fungi | merged at 15:16 utc | 16:41 |
| clarkb | http://puppetboard.openstack.org/report/openstackid-dev.openstack.org/c66b8649704781c0466dc390adfb11e7fe302cd3 appers to be the run that ran that exec | 16:41 |
| clarkb | just doesn't tell you it did so which may be a bug in the ansible-puppet puppetdb support | 16:41 |
| smarcet | fungi: yeah i am waiting for it | 16:41 |
| *** matrohon has joined #openstack-infra | 16:42 | |
| *** kzaitsev_mb has quit IRC | 16:42 | |
| fungi | smarcet: is there an easy way for me to confirm which version of it is deployed? | 16:42 |
| pabelanger | clarkb: so now that we have nodepool-builders in place, I actually thing nodepool dib-image-list should have a queued state that is displayed. Since building isn't technically correct with 1 builder | 16:42 |
| *** Daisy has joined #openstack-infra | 16:42 | |
| clarkb | pabelanger: its the same with image-list too | 16:43 |
| *** eil397 has joined #openstack-infra | 16:43 | |
| *** hashar has quit IRC | 16:43 | |
| clarkb | pabelanger: nodepools state machine considers them all the same once submitted to gearman though | 16:43 |
| clarkb | anything sent to gearman is "building" | 16:43 |
| mgagne | fungi: so latest centos7 uploaded fine to internap :-/ | 16:44 |
| *** jlanoux has quit IRC | 16:44 | |
| smarcet | fungi: not really, i am just checking with the status script | 16:44 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add non-voting diskimage-builder jobs https://review.openstack.org/293193 | 16:44 |
| smarcet | version label is only available for production | 16:44 |
| fungi | mgagne: looks like it. most recent two uploads were ~12 days ago and ~16 hours ago | 16:45 |
| mgagne | fungi: looks like trusty failed | 16:45 |
| *** cloudtrainme has quit IRC | 16:46 | |
| *** tphummel has quit IRC | 16:46 | |
| mgagne | fungi: I have much more logs now. checking | 16:46 |
| fungi | smarcet: for me the status script is saying it installed something from 15:24 utc, which is about 8 minutes after your change merged | 16:46 |
| *** e0ne has quit IRC | 16:47 | |
| *** e0ne_ has joined #openstack-infra | 16:47 | |
| smarcet | fungi : ok | 16:47 |
| *** Daisy has quit IRC | 16:47 | |
| fungi | smarcet: INSTALLED [16-Mar-2016 15:24] | 16:48 |
| pabelanger | zigo: I started https://review.openstack.org/#/c/185680/ a while back. He was my first attempt at creating some chroots. | 16:48 |
| pabelanger | zigo: it was using puppet to do it, which happened during dib build | 16:48 |
| *** scheuran has quit IRC | 16:49 | |
| zigo | pabelanger: Thanks, I'm bookmarking it for later tonight ! :) | 16:49 |
| *** vgridnev has joined #openstack-infra | 16:49 | |
| zigo | pabelanger: Is using puppet mandatory? I'm dumb with puppet ... | 16:49 |
| pabelanger | zigo: for you, I would update your JJB template to call your tooling on job launch, get everything working as expected, then deal with optimizing it maybe as a DIB element | 16:49 |
| *** flwang1 has quit IRC | 16:49 | |
| pabelanger | zigo: As long as it is something infra can read, you'll be fine. | 16:49 |
| zigo | pabelanger: Then only thing I can write is a shell script to setup sbuild properly, Monty said it was ok to do that. | 16:50 |
| *** e0ne has joined #openstack-infra | 16:50 | |
| zigo | pabelanger: What I don't know is how the script will get access to the git repo + the reviewed patch. | 16:50 |
| fungi | smarcet: though i agree, for example /opt/openstackid/app/start/global.php does not match the new version in change 293482 | 16:50 |
| zigo | pabelanger: How does that work? | 16:50 |
| pabelanger | zigo: Yup, should be fine. Add it as a JJB template to start, then move it to the optimized place | 16:50 |
| *** fawadkhaliq has joined #openstack-infra | 16:50 | |
| *** e0ne_ has quit IRC | 16:50 | |
| fungi | smarcet: the last time that file was updated seems to be 5 weeks ago | 16:51 |
| *** adrian_otto has quit IRC | 16:51 | |
| *** fawadkhaliq has quit IRC | 16:51 | |
| zigo | pabelanger: What do you mean a JJB template? | 16:51 |
| zigo | (note: I know JJB, I use it and even maintain the package in Debian) | 16:51 |
| smarcet | fungi: yeah looking at server code, disregard sorry its already deployed :) | 16:51 |
| mgagne | fungi: is a new token used for each upload? | 16:51 |
| *** fawadkhaliq has joined #openstack-infra | 16:51 | |
| fungi | smarcet: oh, right i keep forgetting /opt/openstackid isn't actually used for deploying | 16:51 |
| *** vgridnev has quit IRC | 16:52 | |
| mgagne | fungi: or is a single token fetched and used to upload all images? | 16:52 |
| fungi | smarcet: i was looking for the version in /srv/openstackid/slot0 | 16:52 |
| zigo | pabelanger: What I wont know is where to hook in project-config to get my script started... | 16:52 |
| *** tonytan_brb has quit IRC | 16:52 | |
| clarkb | fungi: those ubuntu-trusty updates are happening? | 16:52 |
| *** vincentll has quit IRC | 16:52 | |
| zigo | pabelanger: The script, I know how to write it (I already have it... just need to adapt it a bit) | 16:52 |
| clarkb | starting to see some of those jobs run on ubuntu-trusty now fwiw | 16:52 |
| fungi | mgagne: i don't recall whether we refresh the token between uploads but clarkb or greghaynes may | 16:52 |
| fungi | clarkb: still underway | 16:52 |
| *** tonytan4ever has joined #openstack-infra | 16:53 | |
| pabelanger | zigo: you'll need to create a new jenkins job, so jenkins/jobs/debian-packaging.yaml (for example) | 16:53 |
| clarkb | fungi: mgagne: I believe that shade + glance should use keystoneauth sessions and refresh the token if necessary | 16:53 |
| pabelanger | zigo: you likely have a noop job for your project ATM | 16:53 |
| clarkb | fungi: mgagne but that is something that should be double checked | 16:53 |
| mgagne | clarkb: when? | 16:53 |
| *** e0ne has quit IRC | 16:53 | |
| mgagne | clarkb: before I suspect the token expires before the upload completes | 16:53 |
| mgagne | because* | 16:53 |
| clarkb | mgagne: prior to ever request keystoneauth should check the token expiry and if it is soon it is supposed to get a new one first | 16:53 |
| clarkb | I do not know what happens if the token expires during a very long request | 16:54 |
| mgagne | clarkb: if an image upload operation is done, I suggest to always renew it | 16:54 |
| mgagne | clarkb: glance crashes because it tries to contact registry and token is expired | 16:54 |
| *** tphummel has joined #openstack-infra | 16:54 | |
| clarkb | mgagne: is it erroring on the next request? | 16:54 |
| clarkb | because that should get a new token with keystoneauth if that is all working | 16:55 |
| mgagne | clarkb: yes | 16:55 |
| *** nwkarsten has quit IRC | 16:55 | |
| mgagne | clarkb: it's not glean that crashes | 16:55 |
| mgagne | clarkb: it's glance-api itself | 16:55 |
| *** FallenPegasus has joined #openstack-infra | 16:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:55 | |
| *** nwkarsten has joined #openstack-infra | 16:55 | |
| clarkb | mgagne: so it is happening server side? | 16:56 |
| mgagne | yes | 16:56 |
| clarkb | hrm ok | 16:56 |
| fungi | ugh. jjb on our jenkins masters is no longer able to update its cache | 16:56 |
| fungi | no, wait, that's a lockfile | 16:56 |
| fungi | did jjb recently grow a (broken) lockfile feature? | 16:57 |
| clarkb | I think yolanda proposed a thing not sure if it merged | 16:57 |
| fungi | i need to go digging in the source, but getting "ValueError: I/O operation on closed file" | 16:57 |
| *** vgridnev has joined #openstack-infra | 16:58 | |
| *** tonytan_brb has joined #openstack-infra | 16:58 | |
| fungi | yep, "Allow using lockfile per jenkins master" merged today (utc) | 16:58 |
| *** tonytan4ever has quit IRC | 16:58 | |
| fungi | https://review.openstack.org/292405 | 16:59 |
| fungi | yolanda: zxiiro: electrofelix: ^ | 16:59 |
| fungi | i'll get a full traceback up momentarily | 16:59 |
| mgagne | https://bugs.launchpad.net/glance/+bug/1182536 exists but there is an OSSN associated to it https://wiki.openstack.org/wiki/OSSN/OSSN-0060 | 17:00 |
| *** maishsk has joined #openstack-infra | 17:00 | |
| *** maishsk has quit IRC | 17:00 | |
| openstack | Launchpad bug 1182536 in Glance "Glance api fails to set image to active after upload when user token expires or is revoked." [Medium,Fix released] - Assigned to Alex Meade (alex-meade) | 17:00 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: Openstackid Production Release 1.0.14 https://review.openstack.org/293573 | 17:00 |
| *** IlyaG has quit IRC | 17:00 | |
| *** e0ne has joined #openstack-infra | 17:00 | |
| *** FallenPegasus is now known as MarkAtwood | 17:00 | |
| *** nwkarsten has quit IRC | 17:00 | |
| *** piet has joined #openstack-infra | 17:01 | |
| *** fhubik has quit IRC | 17:01 | |
| *** IlyaG has joined #openstack-infra | 17:01 | |
| *** HeOS has quit IRC | 17:01 | |
| *** HeOS has joined #openstack-infra | 17:02 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:02 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add NodeJS and NPM to single-use-slave https://review.openstack.org/242161 | 17:02 |
| smarcet | fungi: yolanda: when u get a chance please review https://review.openstack.org/#/c/293573 :) | 17:03 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add NodeJS and NPM to single-use-slave https://review.openstack.org/242161 | 17:03 |
| *** ifarkas has quit IRC | 17:03 | |
| openstackgerrit | Merged openstack-infra/shade: Add wait_for_server API call. https://review.openstack.org/292631 | 17:03 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Fix setting new task status and priority https://review.openstack.org/293577 | 17:05 |
| clarkb | fungi: that change should probably use pylockfile | 17:05 |
| fungi | yolanda: zxiiro: electrofelix: does http://paste.openstack.org/show/490758/ look like a reentrancy issue? are there multiple threads that might be calling the _unlock() method? | 17:06 |
| *** ddieterly is now known as ddieterly[away] | 17:06 | |
| fungi | clarkb: agreed, it does some wheel reinventing | 17:07 |
| *** _nadya_ has quit IRC | 17:07 | |
| clarkb | it also reopens the file over and over every second without properly closing it | 17:07 |
| *** Nepoc has joined #openstack-infra | 17:07 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Removed npm-install macro https://review.openstack.org/242193 | 17:08 |
| mgagne | fungi: looks like multiple images are also uploaded at the same time. will check on that side | 17:08 |
| clarkb | honeslty I think a better way of handling that entire thing is to write the cache to a tmp file then mv it in place when done | 17:08 |
| fungi | immediately before the traceback, there is another "INFO:jenkins_jobs.builder:Cache saved" message so the jjb cache is being written more than once i think | 17:08 |
| clarkb | the races between processes will still exist but be tiny | 17:08 |
| kro_sniffles | Well, _that_ was just plain wrong | 17:09 |
| pabelanger | I thought ansible was just doing to use flock for JJB | 17:09 |
| *** infra-red has joined #openstack-infra | 17:09 | |
| Nepoc | Hello, I'm setting up a mirror of git.openstack.org for openstack-ansible and I was hoping someone could help me replicate the server settings you have so git clone/checkout work as expected with the version/commit hash in the url | 17:09 |
| * kro_sniffles is talking to himself. | 17:09 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu https://review.openstack.org/293012 | 17:09 |
| pabelanger | Nepoc: we use puppet-cgit have you looked at it? | 17:09 |
| Nepoc | pabelanger: I have not | 17:10 |
| fungi | Nepoc: our git.openstack.org mirror servers are passive mirrors, not active mirrors. our gerrit server pushes into them continually | 17:10 |
| pabelanger | Nepoc: so, you'll be able to look at system-config, which will then call puppet-cgit | 17:11 |
| pabelanger | Nepoc: all under the openstack-infra namespace | 17:11 |
| fungi | kro_sniffles: i hope you get over your sniffles (and now whatever psychopathy has you muttering to yourself) soon! | 17:11 |
| Nepoc | pabelanger: I don't need the web front end that cgit provides. I only need the git interactions to work. | 17:12 |
| *** aspiers has quit IRC | 17:12 | |
| kro_sniffles | fungi: THanks :) | 17:12 |
| kro_sniffles | fungi: The latter ls less likely | 17:12 |
| *** mikelk has quit IRC | 17:12 | |
| *** jistr has quit IRC | 17:12 | |
| pabelanger | Nepoc: like fungi says, we use gerrit as our upstream git repos, which are replicated to git.o.o and github.com | 17:13 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Removed npm-install macro https://review.openstack.org/242193 | 17:13 |
| pabelanger | Nepoc: but all the configuration will be in system-config, if you are up for digging into it | 17:13 |
| fungi | Nepoc: pabelanger: so using our puppet-cgit module alone won't actually get you complete mirrors of our repositories. you'll still need to design something to do the active mirror updating piece | 17:13 |
| pabelanger | Ya, that is true | 17:13 |
| kro_sniffles | There, that seems a little more sane. | 17:13 |
| fungi | and puppet-cgit is mostly just going to get you a web frontend to browsing git repositories | 17:14 |
| *** kzaitsev_mb has quit IRC | 17:14 | |
| openstackgerrit | Thomas Maddox proposed openstack-infra/project-config: testing with fedora-23 https://review.openstack.org/293585 | 17:14 |
| Nepoc | pabelanger: fungi: I have the mirroring "done" just need to make sure I'm serving it the same way. | 17:14 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Added mising link to config file https://review.openstack.org/293586 | 17:14 |
| Nepoc | without the web frontend | 17:14 |
| fungi | though i think we do have the vhost stuff on there set up for doing smart git http backend cgi as well | 17:14 |
| fungi | Nepoc: so, yeah, maybe you just want to crib our apache configuration for the serving part (and ignore the parts of that vhost config for cgit's webui) | 17:15 |
| Nepoc | sounds good, that's under system-config? | 17:16 |
| fungi | Nepoc: https://git.openstack.org/cgit/openstack-infra/puppet-cgit/tree/ | 17:16 |
| Nepoc | fungi: thank you | 17:16 |
| fungi | Nepoc: specifically https://git.openstack.org/cgit/openstack-infra/puppet-cgit/tree/templates/git.vhost.erb | 17:16 |
| Nepoc | fungi: excellent already looking at it | 17:17 |
| *** aspiers has joined #openstack-infra | 17:17 | |
| *** ddieterly[away] is now known as ddieterly | 17:18 | |
| *** flepied has quit IRC | 17:18 | |
| fungi | we've also had some requests for setting up grokmirror support on our mirrors, to make it easier for others to mirror from them | 17:18 |
| fungi | though i don't recall if anyone has started working on that | 17:19 |
| *** dtantsur is now known as dtantsur|afk | 17:20 | |
| *** sridhar_ram1 has joined #openstack-infra | 17:21 | |
| *** sridhar_ram has quit IRC | 17:22 | |
| *** piet has quit IRC | 17:22 | |
| *** nwkarsten has joined #openstack-infra | 17:23 | |
| clarkb | as an alternative you could watch the event stream for replication events and do pulls that way | 17:23 |
| clarkb | probably more work than just adding grokmirror to puppet | 17:23 |
| fungi | for now i'm just going to roll back jjb to pre-292405 and rerun to make sure everything's succeeding | 17:23 |
| *** smarcet has quit IRC | 17:24 | |
| clarkb | fungi: sounds good, is there a revert change yet? | 17:24 |
| clarkb | I can push taht through but didn't catch it going by in here if so | 17:24 |
| fungi | not to my knowledge. didn't know whether the author and approvers from that change had a simple fix in mind | 17:24 |
| fungi | but go for it | 17:24 |
| *** tphummel has quit IRC | 17:25 | |
| *** tphummel has joined #openstack-infra | 17:26 | |
| *** cloudtrainme has joined #openstack-infra | 17:26 | |
| clarkb | https://review.openstack.org/293598 | 17:27 |
| fungi | is openstackgerrit on vacation again? | 17:27 |
| *** tzn has quit IRC | 17:27 | |
| clarkb | apparently | 17:27 |
| *** schipiga has quit IRC | 17:28 | |
| *** zeih has quit IRC | 17:30 | |
| *** e0ne has quit IRC | 17:30 | |
| *** IlyaG has quit IRC | 17:30 | |
| *** IlyaG has joined #openstack-infra | 17:31 | |
| pabelanger | clarkb: +2 | 17:32 |
| *** lucasagomes is now known as lucas-afk | 17:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Restore ability to run nodepoold with fakes https://review.openstack.org/231120 | 17:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Make stopping more reliable https://review.openstack.org/293605 | 17:34 |
| fungi | and now it's back | 17:34 |
| *** IlyaG has quit IRC | 17:34 | |
| fungi | maybe it just doesn't like you clarkb | 17:34 |
| *** SumitNaiksatam has quit IRC | 17:35 | |
| fungi | confirmed, rolling jjb back to the commit immediately prior to that one and it doesn't throw a traceback | 17:35 |
| pabelanger | fungi: clarkb: so, are we looking at standing up a 2nd nodepool-builder instance? | 17:36 |
| fungi | pabelanger: possibly, though i don't know that it's in the immediate future | 17:36 |
| clarkb | pabelanger: I haven't been. A bigger benefit may be to have more upload threads based on jeblairs testing during the sprint | 17:37 |
| *** armax has quit IRC | 17:37 | |
| jeblair | yeah, i think we found we could support ~16? | 17:37 |
| clarkb | jeblair: ya was 4 total workes iirc which is 4x4 | 17:37 |
| *** sfinucan has quit IRC | 17:38 | |
| *** johnthetubaguy_ has joined #openstack-infra | 17:39 | |
| jeblair | once shade lands, i'd like to push hard on landing those 2 patches above. we've really let some things drift to the point where you can't really even reliably run nodepool manually for testing (fakes or no fakes in some cases). | 17:39 |
| pabelanger | Ah ya. I remember that | 17:39 |
| jeblair | (they are built on the shade patch) | 17:39 |
| jeblair | (actually, we might want to land them concurrently with the shade patch and save us some potential headache) | 17:39 |
| ddieterly | #TODO(jeblair): remove when this is no longer created by the image | 17:40 |
| ddieterly | rm -fr ~/workspace-cache/ | 17:40 |
| jeblair | ddieterly: could have been removed years ago :) | 17:40 |
| ddieterly | jeblair why is that in functions.sh? it is causing the instructions at https://github.com/openstack-infra/devstack-gate to fail | 17:40 |
| ddieterly | can we safely remove it? | 17:41 |
| *** johnthetubaguy has quit IRC | 17:41 | |
| *** johnthetubaguy_ is now known as johnthetubaguy | 17:41 | |
| jeblair | ddieterly: yes. however, i'm not sure how well those instructions will work even after doing so. | 17:42 |
| ddieterly | omg, that is very disheartening | 17:42 |
| jeblair | ddieterly: are you trying to hack on devstack-gate itself, or reproduce a test run? | 17:42 |
| ddieterly | how are people testing gate jobs and tempest tests before submitting reviews then? | 17:42 |
| ddieterly | jeblair produce a test run | 17:43 |
| kro_sniffles | jeblair: ianw has a patch up for puppet-openafs that uses our own RPM's for Centos7 instead of relying on the official ones (which don't exist). Do you have the time to look? https://review.openstack.org/#/c/290266/ | 17:43 |
| *** roxanagh_ has quit IRC | 17:43 | |
| pabelanger | clarkb: fungi I think well need a nodepool-builder restart to pickup the debian-jessie dib. | 17:43 |
| jeblair | ddieterly: if zuul has already run a job for a change, there is a script called "reproduce.sh" in the logs that you can use. it's a lot easier than the process in the readme. | 17:43 |
| ddieterly | i'd like to make changes to the repo under tests and the gate job to see if things are working | 17:44 |
| *** _nadya_ has joined #openstack-infra | 17:45 | |
| jeblair | kro_sniffles: (re)reviewed! | 17:45 |
| kro_sniffles | jeblair: Thanks :D | 17:45 |
| *** dizquierdo has quit IRC | 17:45 | |
| * kro_sniffles waits for that to land before testing to see if we can do centos7 thing with wheel mirrors. | 17:45 | |
| *** jpr has quit IRC | 17:45 | |
| jeblair | ddieterly: i'd try reproduce.sh first | 17:46 |
| *** sdague has quit IRC | 17:46 | |
| ddieterly | jeblair ok, thanks | 17:46 |
| *** ddieterly is now known as ddieterly[away] | 17:47 | |
| pabelanger | actually, if some infra-root can help land 293396 we can have updated logging on nodepool builders | 17:49 |
| *** fawadkhaliq has quit IRC | 17:49 | |
| pabelanger | before we restart | 17:49 |
| *** derekh has quit IRC | 17:50 | |
| *** fawadkhaliq has joined #openstack-infra | 17:50 | |
| jeblair | pabelanger: approved (i assume you ran the script?) | 17:51 |
| pabelanger | jeblair: yup | 17:52 |
| openstackgerrit | Merged openstack-infra/puppet-openafs: Use CI built RPMs for CentOS7 https://review.openstack.org/290266 | 17:52 |
| *** abregman has quit IRC | 17:52 | |
| *** sdague has joined #openstack-infra | 17:52 | |
| *** rcrit has joined #openstack-infra | 17:52 | |
| kro_sniffles | woot | 17:53 |
| clarkb | with ^ merged is the last piece booting a new centos7 builder host? | 17:53 |
| *** rcernin has quit IRC | 17:53 | |
| clarkb | s/last/next/ | 17:53 |
| fungi | pabelanger: yeah, in my review comment on that change i mentioned the need for a nodepool builder process restart | 17:53 |
| *** salv-orl_ has joined #openstack-infra | 17:54 | |
| *** rossella_s has joined #openstack-infra | 17:54 | |
| pabelanger | fungi: okay, once puppet deploys the new logging config, I'll restart | 17:54 |
| pabelanger | fungi: also, down to 39 devstack-trusty nodes now | 17:54 |
| clarkb | pabelanger: just check that there aren't any running builds | 17:54 |
| pabelanger | clarkb: Yup, they all finished for the day | 17:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests https://review.openstack.org/288840 | 17:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Reconcile compute and controller classes https://review.openstack.org/293620 | 17:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Parameterize libvirt virt type https://review.openstack.org/293621 | 17:54 |
| rcrit | I'm trying to run tox in project-config and it fails because it can't install pbr. I also see: Download error on http://pypi.dfw.openstack.org/simple/pbr/: [Errno 101] Network is unreachable -- Some packages may not be found! | 17:54 |
| rcrit | I don't know where it is getting this hostname | 17:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially https://review.openstack.org/292472 | 17:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs https://review.openstack.org/293526 | 17:56 |
| *** salv-orlando has quit IRC | 17:56 | |
| *** flepied has joined #openstack-infra | 17:57 | |
| *** tphummel has quit IRC | 17:58 | |
| fungi | rcrit: presumably you have a ~/.pip.conf or something setting that | 17:58 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Properly setup DNS for the all CI jobs https://review.openstack.org/293481 | 17:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Convert last devstack-trusty to ubuntu-trusty https://review.openstack.org/293624 | 17:58 |
| pabelanger | AJaeger: fungi: looks like we missed one ^ | 17:58 |
| fungi | rcrit: that server definitely doesn't exist any longer | 17:58 |
| *** tphummel has joined #openstack-infra | 17:59 | |
| rcrit | I have a .pip directory but it is empty | 17:59 |
| fungi | pabelanger: right, AJaeger mentioned there was a straggler which merged after clarkb wrote the big change | 17:59 |
| openstackgerrit | Merged openstack-infra/system-config: Regenerate nodepool.logging config https://review.openstack.org/293396 | 17:59 |
| fungi | rcrit: /etc/pip maybe? | 17:59 |
| rcrit | or ~/.pydistutils.cfg maybe? | 17:59 |
| rcrit | it is in there | 17:59 |
| fungi | rcrit: that may also do it since you're seeing it complain about pbr and so it may be easy_install coming into the picture | 18:00 |
| rcrit | I see it listed in /etc/pip.conf as a trusted host too | 18:00 |
| rcrit | ok, let me smack both of those and see where that gets me, thanks fungi | 18:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Cleanup a last devstack-trusty straggler https://review.openstack.org/293627 | 18:01 |
| clarkb | fungi: pabelanger ^ | 18:01 |
| fungi | yeah, you don't need any of that. we set that up on our workers but the mirrors are just for use within our ci and we reserve the right to rename/replace/remove them without notice | 18:01 |
| clarkb | oh wait pabelanger wrote that change ugh | 18:01 |
| * clarkb abandons | 18:01 | |
| pabelanger | ha | 18:01 |
| pabelanger | I just noticed it since nodepool launched a new devstack-trusty | 18:01 |
| *** _nadya_ has quit IRC | 18:02 | |
| pabelanger | down to 31 used ATM | 18:02 |
| *** smarcet has joined #openstack-infra | 18:02 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: Openstackid Production Release 1.0.14 https://review.openstack.org/293573 | 18:02 |
| electrofelix | fungi: quick fix would be to set 'self.lockfile = None' after the lockfile is closed similarly set self.data = None before self._unlock() gets called and we'll think about how to ensure it's only called once better subsequently | 18:03 |
| *** baoli has quit IRC | 18:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publishing barbican API Guide to OpenStack site https://review.openstack.org/285519 | 18:04 |
| clarkb | electrofelix: I really think we want to avoid a quick fix and use pylockfile | 18:04 |
| clarkb | electrofelix: since that fixes all of these problems | 18:04 |
| *** ihrachys has quit IRC | 18:04 | |
| *** tqtran-afk has joined #openstack-infra | 18:05 | |
| fungi | lockfiles are easy to get wrong, and when you try to write lockfile implementations from scratch you get to relearn all the lessons learned by those who wrote them before you | 18:06 |
| *** SumitNaiksatam has joined #openstack-infra | 18:06 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Added mising link to config file https://review.openstack.org/293586 | 18:06 |
| clarkb | http://docs.openstack.org/developer/pylockfile/ | 18:06 |
| electrofelix | clarkb: that's assuming pylockfile doesn't cause problems when you call unlock on it twice either. | 18:06 |
| clarkb | electrofelix: well you don't call unlock on it twice | 18:06 |
| clarkb | or rather don't have unmatched unlocks | 18:07 |
| electrofelix | actually, that's exactly what the error from above indicates, unlock followed by close, and subsequently calling unlock against, except this time the file was already closed. | 18:07 |
| clarkb | electrofelix: which means that something that doesn't hold the lock is unlocking | 18:07 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:07 | |
| *** tphummel has quit IRC | 18:08 | |
| clarkb | there should always be matched locks and unlocks | 18:08 |
| electrofelix | it means the destructor is triggering a second unlock because save is getting called a second time by accident. | 18:08 |
| electrofelix | on the unlock should be moved to the destructor | 18:09 |
| *** tphummel has joined #openstack-infra | 18:09 | |
| electrofelix | so we can call save multiple times within the same process without unlocking the file until we're completely finished with it | 18:09 |
| *** tqtran-afk has quit IRC | 18:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Openstackid Production Release 1.0.14 https://review.openstack.org/293573 | 18:10 |
| *** kzaitsev_mb has joined #openstack-infra | 18:10 | |
| fungi | pabelanger: smarcet provided a link to the change notes (from the tag's message) in the commit message ^ | 18:11 |
| *** rbrndt has quit IRC | 18:11 | |
| *** _nadya_ has joined #openstack-infra | 18:13 | |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box' https://review.openstack.org/286137 | 18:15 |
| pabelanger | fungi: smarcet: many thanks | 18:15 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Unlock when finished not when saving current cache state https://review.openstack.org/293631 | 18:16 |
| clarkb | because I have now discovered that openstack has deprecated pylockfile in favor of oslo.concurrency I think http://docs.openstack.org/developer/oslo.concurrency/usage.html could really use some love | 18:18 |
| *** tesseract- has quit IRC | 18:19 | |
| clarkb | compare to http://docs.openstack.org/developer/pylockfile/#examples | 18:19 |
| clarkb | harlowja_at_home: dims ^ | 18:19 |
| harlowja_at_home | sup | 18:19 |
| dims | clarkb : i was just about to point harlowja_at_home to your comment :) | 18:19 |
| harlowja_at_home | wellll pylockfile isn't replaced by oslo.concurrency | 18:19 |
| harlowja_at_home | imho | 18:19 |
| clarkb | harlowja_at_home: ok then we should update the warning on pylockfile :) | 18:20 |
| harlowja_at_home | ah, perhaps | 18:20 |
| clarkb | http://docs.openstack.org/developer/pylockfile/#module-lockfile | 18:20 |
| harlowja_at_home | https://fasteners.readthedocs.org/en/latest/ | 18:20 |
| harlowja_at_home | more of its direct replacement, or at least a piece of it | 18:20 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Unlock when finished not when saving current cache state https://review.openstack.org/293631 | 18:21 |
| harlowja_at_home | 'It is highly preferred that instead of using this code base that instead fasteners or oslo.concurrency is used instead.' | 18:21 |
| harlowja_at_home | seems like both already referenced | 18:21 |
| clarkb | yes so delete oslo.concurrency from that line? | 18:21 |
| electrofelix | fungi: https://review.openstack.org/293631 will take care of the immediate issue, we can discuss which locking library to use subsequently, but this will make sure the unlock is called at the correct point which was the problem before. | 18:21 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes https://review.openstack.org/293637 | 18:21 |
| harlowja_at_home | clarkb, i'd be ok with that | 18:21 |
| *** ddieterly[away] is now known as ddieterly | 18:21 | |
| fungi | electrofelix: thanks for the quick fix | 18:22 |
| *** tqtran-afk has joined #openstack-infra | 18:22 | |
| *** sputnik13 has joined #openstack-infra | 18:22 | |
| *** mrmartin has joined #openstack-infra | 18:22 | |
| *** degorenko is now known as _degorenko|afk | 18:23 | |
| pvaneck | Hey puppet/infra-core. If available, this could use a review: https://review.openstack.org/#/c/281737 - This is to make refstack update via releases on pypi. Thanks! | 18:24 |
| *** akshai has quit IRC | 18:25 | |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box' https://review.openstack.org/286137 | 18:25 |
| *** nwkarsten has quit IRC | 18:26 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes https://review.openstack.org/293637 | 18:26 |
| *** nwkarsten has joined #openstack-infra | 18:27 | |
| pabelanger | I guess we need more HDD for nodepool.o.o | 18:27 |
| pabelanger | 2016-03-16 18:26:13,021 INFO nodepool.image.build.debian-jessie: fatal: failed to copy file to '/opt/dib_tmp/image.WtfS1aED/mnt/opt/git/openstack/heat-specs/.git/objects/e8/593c8c47d67a31106f6d1a6187068b723aa3b5': No space left on device | 18:27 |
| clarkb | pabelanger: there are some thing sI can clean out of /opt. I was testing the image compression which is a few gigabytes of unnecessary disk usage | 18:27 |
| clarkb | a little annoying that we are that close though we can delete devstack-trusty soon and free that up too | 18:28 |
| pabelanger | clarkb: ya, devstack-trusty will help for sure | 18:28 |
| pabelanger | down to 25 used nodes now | 18:28 |
| clarkb | pabelanger: /dev/mapper/main-opt 504G 352G 153G 70% /opt I didn't free 153GB of disk fwiw | 18:29 |
| clarkb | I got ~30GB | 18:29 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes https://review.openstack.org/293637 | 18:29 |
| clarkb | only 4% inodes used so not sure what is going on there | 18:30 |
| pabelanger | Ya, I noticed that too | 18:30 |
| pabelanger | let me kick off debian-jessie again | 18:30 |
| *** nwkarsten has quit IRC | 18:31 | |
| AJaeger | fungi, infra-root, do you think we can remove bare-trusty now from nodepool? Or should we wait some more days? | 18:33 |
| *** akshai has joined #openstack-infra | 18:33 | |
| openstackgerrit | Merged openstack-infra/system-config: Openstackid Production Release 1.0.14 https://review.openstack.org/293573 | 18:33 |
| fungi | AJaeger: i think it's probably fine at this point | 18:33 |
| pabelanger | AJaeger: I haven't heard any failures with ubuntu-trusty recently | 18:33 |
| openstackgerrit | Merged openstack-infra/project-config: Convert last devstack-trusty to ubuntu-trusty https://review.openstack.org/293624 | 18:33 |
| *** jpr has joined #openstack-infra | 18:34 | |
| AJaeger | so, I removed WIP and -2 from https://review.openstack.org/#/c/291238 | 18:34 |
| pabelanger | clarkb: looks like we are only using 3.9GB for tmpfs (/opt/dib_tmp) | 18:34 |
| pabelanger | didn't think we limited it | 18:34 |
| *** ldnunes has quit IRC | 18:35 | |
| *** mriedem has quit IRC | 18:35 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 18:36 |
| pabelanger | AJaeger: fungi if we are ready, I can +A it | 18:36 |
| fungi | go for it | 18:36 |
| clarkb | pabelanger: I think it makes a dir for the build in there too, but that shouldnt grow to 153GB | 18:36 |
| clarkb | and towards the end of the build mv's it all into a loopback mounted raw device | 18:37 |
| pabelanger | clarkb: Ya, monitoring right now. Up to 38% use | 18:37 |
| *** ldnunes has joined #openstack-infra | 18:37 | |
| pabelanger | I thought --notmpfs removed that | 18:38 |
| *** mriedem has joined #openstack-infra | 18:38 | |
| clarkb | oh we shouldn't be tmpfsing at all | 18:38 |
| *** pvaneck has quit IRC | 18:38 | |
| clarkb | /opt/dib_tmp is on disk and we use that instead | 18:38 |
| pabelanger | something is causing tmpfs to be used | 18:38 |
| openstackgerrit | Deklan Dieterly proposed openstack-infra/devstack-gate: Remove deletion of ~/workspace-cache https://review.openstack.org/293643 | 18:39 |
| pabelanger | tmpfs 3.9G 2.2G 1.8G 56% /opt/dib_tmp/image.YefSYwgq | 18:39 |
| clarkb | pabelanger: maybe something in the debian element? | 18:39 |
| pabelanger | clarkb: that is what I am checking | 18:39 |
| zxiiro | fungi: electrofelix: just catching up do we need to revert a patch? | 18:39 |
| ddieterly | jeblair https://review.openstack.org/293643 | 18:39 |
| *** cdent has quit IRC | 18:40 | |
| *** jordanP has quit IRC | 18:40 | |
| *** vgridnev has quit IRC | 18:40 | |
| *** vgridnev has joined #openstack-infra | 18:41 | |
| pabelanger | export DIB_NO_TMPFS=0 is what I am seeing in the logs | 18:41 |
| electrofelix | zxiiro: either https://review.openstack.org/#/c/293631/ or https://review.openstack.org/#/c/293598/1 should resolve the problem | 18:41 |
| electrofelix | so add additional fix or revert existing | 18:41 |
| zxiiro | personally I'd prefer to revert and have it rewritten usign pylock | 18:42 |
| fungi | zxiiro: apparently "fasteners" is the new pylockfile | 18:42 |
| clarkb | pabelanger: and --no-tmpfs is on the dib command line so it is trying | 18:42 |
| pabelanger | clarkb: ya | 18:42 |
| pabelanger | I've had this problem before locally | 18:43 |
| *** MarkAtwood has quit IRC | 18:43 | |
| pabelanger | trying to remember the issue | 18:43 |
| zxiiro | I +2'd both patches, I'll let you folks decide which route is better. | 18:43 |
| anteaya | mordred: thanks for the history of infra in your candidate statement | 18:43 |
| zxiiro | I'm not immediately impacted by this so | 18:43 |
| anteaya | mordred: also I think you set a reasonable goal for your platform | 18:43 |
| *** ddieterly is now known as ddieterly[away] | 18:43 | |
| fungi | zxiiro: ultimately it seems to be benign, aside from masking real failures by causing the exit code to be nonzero all the time | 18:44 |
| *** _nadya_ has quit IRC | 18:45 | |
| electrofelix | clarkb fungi: how impacting were the previous problems with JJB stomping over cache files? can you live with the revert for a while or is that just adding back an issue. | 18:45 |
| *** rossella_s has quit IRC | 18:45 | |
| clarkb | I think either way JJB still stomps over the cache file | 18:46 |
| clarkb | this doesn't solve that | 18:46 |
| *** mrmartin has quit IRC | 18:46 | |
| clarkb | which is why i have suggested the tmpfile + move | 18:46 |
| *** rbrndt has joined #openstack-infra | 18:46 | |
| clarkb | the problem with the cache files is puppet runs jjb up to a timeout then jjb dies halfway through a long run. This leaves the cache file half written | 18:46 |
| clarkb | inteprocess locks only help the case of two jjbs writing to the same file which I don't think is a common one | 18:47 |
| *** BobBall is now known as BobBall_AWOL | 18:47 | |
| electrofelix | ah, well then I think revert and rewrite is probably better. thought it was added to prevent somethnig | 18:47 |
| fungi | well, cache file writing happens at the end. the reason we see it semi-regularly is when the timeout hits near the end _during_ cache writing | 18:47 |
| *** sfinucan has joined #openstack-infra | 18:47 | |
| fungi | and it's leaving a half-written/truncated cache | 18:48 |
| electrofelix | just approved the revert, we can come back to how it should be written later | 18:48 |
| *** ddieterly[away] is now known as ddieterly | 18:48 | |
| electrofelix | sounds like the write tmp + move should be done first anyway | 18:48 |
| *** electrofelix has quit IRC | 18:49 | |
| pabelanger | clarkb: I know the issue -t is being passed empty | 18:49 |
| pabelanger | 2016-03-16 18:18:20,480 INFO nodepool.builder: Running disk-image-create -x -t --no-tmpfs -o /opt/nodepool_dib/176615 debian-minimal vm simple-init openstack-repos nodepool-base node-devstack cache-bindep growroot | 18:49 |
| clarkb | pabelanger: oh! | 18:49 |
| pabelanger | I remember this | 18:49 |
| clarkb | pabelanger: is this image not a part of any providers? | 18:49 |
| *** baoli has joined #openstack-infra | 18:49 | |
| pabelanger | I think so | 18:49 |
| clarkb | because providers determine type | 18:49 |
| pabelanger | that is the issue | 18:49 |
| *** fawadkhaliq has quit IRC | 18:49 | |
| pabelanger | yup | 18:49 |
| fungi | aha. i missed that in reviewing the change | 18:49 |
| pabelanger | same | 18:49 |
| pabelanger | let me fix | 18:49 |
| fungi | well shucks | 18:49 |
| *** vgridnev has quit IRC | 18:50 | |
| pabelanger | I actually just abandoned a patch in nodepool that exposed this | 18:50 |
| openstackgerrit | Sam Betts proposed openstack-infra/glean: Enable multiple VLANs per physical interface https://review.openstack.org/283726 | 18:51 |
| openstackgerrit | Sam Betts proposed openstack-infra/glean: [WIP] Enable support for configuring bonded interfaces https://review.openstack.org/293648 | 18:51 |
| igorbelikov | is there some other place where providers need to be set besides here? https://github.com/openstack-infra/project-config/blob/master/nodepool/nodepool.yaml#L35-L40 | 18:51 |
| openstackgerrit | Merged openstack/requirements: bump uc for congressclient to 1.2.3 https://review.openstack.org/293425 | 18:51 |
| clarkb | igorbelikov: yes in each provider you have to tell it what flavor to use and where to get the image | 18:52 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Revert "Allow using lockfile per jenkins master" https://review.openstack.org/293598 | 18:52 |
| clarkb | if you tell it to usr dib then the provider formats specified are used in the dib build | 18:52 |
| *** sambetts is now known as sambetts|afk | 18:52 | |
| igorbelikov | clarkb: ugh, yeah, see it now, thx | 18:53 |
| *** mrmartin has joined #openstack-infra | 18:53 | |
| *** vgridnev has joined #openstack-infra | 18:53 | |
| *** esikachev has joined #openstack-infra | 18:54 | |
| *** amit2131 has quit IRC | 18:55 | |
| igorbelikov | pabelanger: I can make a fix if you have something better to do:) | 18:55 |
| *** amit213 has joined #openstack-infra | 18:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove bare-trusty from nodepool https://review.openstack.org/291238 | 18:56 |
| pabelanger | igorbelikov: nope, almost done | 18:56 |
| openstackgerrit | Deklan Dieterly proposed openstack-infra/project-config: Add pre_test_hook to freezer api gate job https://review.openstack.org/291430 | 18:58 |
| igorbelikov | pabelanger: ok, thank you | 18:58 |
| *** Daisy has joined #openstack-infra | 18:58 | |
| igorbelikov | when I glanced over providers section I though it was for non-dib stuff | 18:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add debian-jessie to providers https://review.openstack.org/293651 | 18:59 |
| pabelanger | igorbelikov: clarkb: fungi: I added debian-jessie to all providers, even tho we are only launching in rax and ovh ATM | 18:59 |
| pabelanger | ^ | 18:59 |
| *** nwkarsten has joined #openstack-infra | 19:01 | |
| *** Swami has quit IRC | 19:02 | |
| *** claudiub has quit IRC | 19:03 | |
| *** Swami has joined #openstack-infra | 19:03 | |
| *** Daisy has quit IRC | 19:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka https://review.openstack.org/293655 | 19:03 |
| *** maishsk has joined #openstack-infra | 19:03 | |
| fungi | pabelanger: does it not need the meta subsection like ubuntu-trusty has? | 19:04 |
| *** harlowja has joined #openstack-infra | 19:04 | |
| fungi | seems like that might be needed in rackspace | 19:04 |
| fungi | and also name-filter | 19:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka https://review.openstack.org/293655 | 19:05 |
| fungi | pabelanger: you may want to git blame those lines and see why the commit message(s) say they were added | 19:05 |
| pabelanger | fungi: right, let me check that | 19:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka https://review.openstack.org/293655 | 19:06 |
| clarkb | name filter is to get erformance flavors and the metadata is so that rax knows the images are pvhvm iirc | 19:06 |
| clarkb | so not strictly required but should be used | 19:06 |
| fungi | and to not install the xen agent | 19:07 |
| fungi | or, rather, to not expect it to be installed | 19:08 |
| fungi | those seem odd to define on the diskimage rather than on the provider, but whatever | 19:08 |
| pabelanger | Ya, I used the fedora-23 syntax, switching to ubuntu-trusty now | 19:08 |
| *** tosky has quit IRC | 19:09 | |
| *** sfinucan has quit IRC | 19:10 | |
| kro_sniffles | ianw: I'm working on making thick-slaves work on centos - do you know of any patches that address the lack of mysql-devel (as opposed to mariadb) | 19:10 |
| *** acruz has quit IRC | 19:10 | |
| *** ayoung has quit IRC | 19:10 | |
| clarkb | kro_sniffles: we are quickly moving away from the thick vs thin provisioned slaves and have just one setup per distro release | 19:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add debian-jessie to providers https://review.openstack.org/293651 | 19:10 |
| *** vgridnev has quit IRC | 19:11 | |
| AJaeger | do we need to restart/cleanup nodepool since bare-trusty is gone now? | 19:11 |
| clarkb | kro_sniffles: for that I think you just want the db setup macro and use mariadb | 19:11 |
| clarkb | AJaeger: devstack-trusty and yes | 19:11 |
| pabelanger | kro_sniffles: ianw: which is this for? we shouldn't be using thick slave if possible | 19:11 |
| *** vgridnev has joined #openstack-infra | 19:11 | |
| kro_sniffles | pabelanger: openstack_project::wheel_mirror_slave uses openstack_project::slave | 19:11 |
| kro_sniffles | It seems to be coming in from there. | 19:11 |
| openstackgerrit | Merged openstack/requirements: prevent installing old releases of congressclient https://review.openstack.org/293441 | 19:12 |
| AJaeger | clarkb: devstack-trusty from jobs - and bare-trusty completely from setup | 19:12 |
| AJaeger | clarkb, I was talking about 291238 | 19:12 |
| * kro_sniffles thinks that's the case anyway | 19:12 | |
| clarkb | AJaeger: gotcha, I don't think we need a nodepool restart | 19:12 |
| clarkb | AJaeger: it should reread its config and figure it out | 19:12 |
| pabelanger | kro_sniffles: Ah, okay. So I haven't looked at that stuff yet. Hopefully tomorrow. We might want to wait until centos-7 dibs are online, then we can use bindep to control some of the dependencies | 19:13 |
| clarkb | kro_sniffles: its possible the puppet mysql module is smart | 19:13 |
| openstackgerrit | Merged openstack-infra/storyboard: Add task notes to task email https://review.openstack.org/292462 | 19:13 |
| pabelanger | AJaeger: we'll likely need nodepool-builder restart once debian-jessie changes land too | 19:13 |
| clarkb | kro_sniffles: it may already know how to switch based on distro? | 19:13 |
| pabelanger | I would expect it to be too | 19:13 |
| kro_sniffles | pabelanger: Coolio, can you ping me once that happens so I can get back into doing wheel things? | 19:14 |
| kro_sniffles | clarkb: Its switching is not perfect. | 19:14 |
| pabelanger | kro_sniffles: I hope to look into it later tonight and tomorrow. Otherwise, will be full steam into it next week | 19:14 |
| pabelanger | PTO on friday | 19:14 |
| kro_sniffles | pabelanger: Gotcha | 19:15 |
| pabelanger | AJaeger: currently 19 devstack-trusty images used :) | 19:15 |
| *** salv-orl_ has quit IRC | 19:15 | |
| AJaeger | pabelanger: Yeah! | 19:16 |
| *** vgridnev has quit IRC | 19:17 | |
| *** mariojv has left #openstack-infra | 19:17 | |
| pabelanger | AJaeger: I guess the next step is to remove dsvm from all the job names? | 19:17 |
| AJaeger | pabelanger: that wasn't on my radar. Fungi, clarkb, what do you think? | 19:19 |
| clarkb | its mostly redundant at this point but probably not a priority | 19:19 |
| clarkb | maybe we can do then when jjb updates are reliable :) | 19:19 |
| fungi | AJaeger: pabelanger: it's probably useful for now as an indicator that those are (mostly) devstack-gate based jobs | 19:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Setup translations for stable/mitaka https://review.openstack.org/293655 | 19:20 |
| *** fverbs has quit IRC | 19:20 | |
| *** mrmartin has quit IRC | 19:20 | |
| fungi | it was just convenient, for a time, to also use that to determine a non-default node label mapping | 19:20 |
| * AJaeger would love to see some review on the change above ^, please | 19:21 | |
| *** vgridnev has joined #openstack-infra | 19:21 | |
| mgagne | fungi or clarkb: do you know the TTL of tokens generated at other providers? | 19:22 |
| clarkb | mgagne: rax is 24 hours | 19:22 |
| *** mrmartin has joined #openstack-infra | 19:22 | |
| clarkb | I only know that one due to the swift upload issues | 19:22 |
| AJaeger | fungi, mriedem gave a +1 on the nova db patches. But I had to fix a problem... https://review.openstack.org/292472 is the next one to merge there | 19:23 |
| fungi | mgagne: yeah, no clue really. i've heard that some providers dial it down as low as 1hr for database performance reasons or something | 19:23 |
| kro_sniffles | AJaeger, pabelanger: Waaaaait a sec. Are we setting up persistent devstack vm's? [/hopeful look] | 19:23 |
| fungi | AJaeger: yep, saw the update, just going back through it | 19:23 |
| mgagne | fungi: how long do you think it should be? 24h looks a bit too much IMO | 19:23 |
| *** sridhar_ram1 has quit IRC | 19:24 | |
| fungi | mgagne: no clue. i think 24 hours is the default but maybe the security guide has good guidance on non-default values? | 19:24 |
| pabelanger | clarkb: fungi: I'll be stepping out again in 45mins. So, if 293651 lands while I am out, do you mind doing a debian-jessie image-build? | 19:24 |
| pabelanger | will be back tonight after food | 19:24 |
| pabelanger | kro_sniffles: no, we are replacing devstack-trusty with ubuntu-trusty | 19:24 |
| mgagne | fungi: puppet defaults to 1h: https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L70-L72 | 19:25 |
| pabelanger | kro_sniffles: that will reduce the amount of images we maintain in nodepool | 19:25 |
| fungi | kro_sniffles: no, we've finally got all our other jobs working on similar base images to what devstack jobs use, so we're finally shuffling all those into one common node label | 19:25 |
| pabelanger | kro_sniffles: thanks to AJaeger efforts, we'll be down 2 images in the last 7 days | 19:25 |
| *** roxanagh_ has joined #openstack-infra | 19:25 | |
| fungi | pabelanger: i believe debian-jessie images will automatically start building unless we've set min-ready to -1 | 19:26 |
| mgagne | fungi: same for keystone default: https://github.com/openstack/keystone/blob/99c52f9fbe56f5de01afa0c5fd09864c727b832d/keystone/common/config.py#L291-L293 | 19:26 |
| fungi | pabelanger: but we'll need a builder restart first | 19:26 |
| kro_sniffles | Oh. Well, that's close anyway :) | 19:26 |
| pabelanger | fungi: ya, restarting builder is what I should have asked for | 19:26 |
| clarkb | mgagne: fungi this seems like a failing in our services if we fail this way, I don't think a longer token should be necessary | 19:27 |
| fungi | mgagne: oh! right, 1hr default. some providers have been known to increase it so as to _reduce_ load from refreshes impacting performance | 19:27 |
| *** kro_sniffles is now known as kro_mnckn_doc | 19:27 | |
| mgagne | fungi: I think it's more related to image snapshot and such. Often can't snapshot in less than 1h | 19:27 |
| *** piet has joined #openstack-infra | 19:28 | |
| *** daemontool has quit IRC | 19:28 | |
| *** vgridnev has quit IRC | 19:28 | |
| mgagne | fungi: increasing it could be some issues we also see on our side but I feel there is room for improvement on nodepool side too, like renewing token right before starting an upload | 19:28 |
| mgagne | fungi: are there metrics related to upload time/duration in Nodepool? | 19:29 |
| *** eliqiao has quit IRC | 19:29 | |
| mgagne | I feel this could be very useful | 19:29 |
| *** eliqiao has joined #openstack-infra | 19:29 | |
| *** akshai has quit IRC | 19:30 | |
| pabelanger | mgagne: I agree. I think we could expose more data into statsd | 19:30 |
| fungi | mgagne: maybe, though i agree with clarkb that it's still a concern if a token expiring in the middle of an already running file upload causes that upload to break | 19:30 |
| pabelanger | mgagne: also success / failures for dib builds | 19:30 |
| fungi | mgagne: and probably needs to be properly solved in keystone/glance design | 19:31 |
| mgagne | fungi: a fix went in Glance to use trust in december, we don't use latest trunk so yea, can't use that yet =) | 19:31 |
| fungi | assuming that's what's actually happening | 19:31 |
| *** tsg has joined #openstack-infra | 19:31 | |
| mgagne | renewing token would give more chance of successfully uploading image | 19:32 |
| fungi | for internap, this is probably going to start failing less often anyway once we get our qcow2 sizes down with a new diskimage-builder release | 19:32 |
| *** tsg_ has joined #openstack-infra | 19:33 | |
| mgagne | fungi: it think there are more than one step to take to improve change of success =) | 19:33 |
| mgagne | bbl, meeting | 19:33 |
| *** yamahata has quit IRC | 19:34 | |
| openstackgerrit | Rob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack https://review.openstack.org/293090 | 19:34 |
| *** ociuhandu has quit IRC | 19:35 | |
| *** tsg has quit IRC | 19:36 | |
| *** vgridnev has joined #openstack-infra | 19:37 | |
| *** nwkarsten has quit IRC | 19:38 | |
| AJaeger | pleia2: it would be great if translation check site would be up for translators to check. Do you have a chance to help give it the final kick, please? https://review.openstack.org/#/q/topic:i18n-checksite shows the current changes | 19:38 |
| *** nwkarsten has joined #openstack-infra | 19:38 | |
| *** akshai has joined #openstack-infra | 19:39 | |
| anteaya | AJaeger: she is in singapore at the moment | 19:40 |
| anteaya | unless, has she been online today? | 19:40 |
| doug-fish | of course it would be fine for anyone else to kick it too | 19:40 |
| AJaeger | anteaya: ok. | 19:41 |
| *** _nadya_ has joined #openstack-infra | 19:41 | |
| *** cloudtrainme has quit IRC | 19:41 | |
| AJaeger | doug-fish: if you speak about translation check site, yes! I pinged pleia2 since she volunteered herself, her name is on the spec... | 19:41 |
| doug-fish | oh - I was actually having trouble with the translation site itself | 19:41 |
| doug-fish | not the check site | 19:41 |
| anteaya | I'm not saying not to ask, just there might be a pause before she responds | 19:41 |
| AJaeger | unterstood, anteaya | 19:42 |
| anteaya | thanks | 19:42 |
| doug-fish | If she's in Singapore I think it would be traditional to be sleeping now | 19:42 |
| AJaeger | doug-fish: zanata? what's the problem? | 19:42 |
| doug-fish | can't log in - after providing credentials it still seems to think I'm not logged in | 19:42 |
| eil397 | hi | 19:42 |
| anteaya | yeah 3:45am in singapore atm | 19:43 |
| AJaeger | doug-fish: same problem here. | 19:43 |
| *** nwkarsten has quit IRC | 19:43 | |
| eil397 | anteaya: excuse me. yesterday I've added description in https://etherpad.openstack.org/p/infra-newton-summit-planning | 19:43 |
| AJaeger | StevenK: could you check translate.openstack.org, please? doug-fish and myself cannot log in ;( | 19:43 |
| eil397 | I will remove it today. | 19:43 |
| *** MarkAtwood has joined #openstack-infra | 19:44 | |
| clarkb | AJaeger: openstackid has bene having trouble today aiui | 19:44 |
| anteaya | eil397: no no no, you don't have to remove it | 19:44 |
| clarkb | but testing if I can login | 19:44 |
| anteaya | eil397: thanks for answering my question | 19:44 |
| anteaya | eil397: I don't know what you want to talk about is what I'm saying | 19:44 |
| AJaeger | clarkb: that could explain it as well... | 19:44 |
| anteaya | eil397: you are welcome to put items on the etherpad, thanks for participating | 19:44 |
| anteaya | eil397: but the sentence you wrote doen't help me understand what you want to discuss | 19:45 |
| clarkb | AJaeger: you do login but it doesn't actually log you in? this is what appears to have hapepend to me | 19:45 |
| anteaya | eil397: do you have a minute to help me understand what you want to talk about? | 19:45 |
| eil397 | anteaya: thank you for understanding. I should propose it on irc before | 19:45 |
| anteaya | eil397: sure | 19:45 |
| doug-fish | clarkb: yep that's the problem I see | 19:45 |
| anteaya | eil397: have you time now to say what it is you want to discuss? | 19:45 |
| clarkb | smarcet: is the above loging issue expected with the current state of openstackid? | 19:46 |
| *** e0ne has joined #openstack-infra | 19:46 | |
| eil397 | anteaya: I just have idea. but not sure that it is right place and right way. briefly : it is about how community infra project can be integrated in internal CICD for Openstack | 19:46 |
| *** harlowja has quit IRC | 19:46 | |
| anteaya | great | 19:46 |
| AJaeger | clarkb: yeah ;( | 19:46 |
| anteaya | right now I guess I need some help understanding how you define your terms | 19:47 |
| openstackgerrit | Merged openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 19:47 |
| *** maishsk has quit IRC | 19:47 | |
| anteaya | as my perception of those terms is we are doing that alreaady | 19:47 |
| openstackgerrit | Merged openstack-infra/project-config: Setup translations for stable/mitaka https://review.openstack.org/293655 | 19:47 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use m1.small for ping test https://review.openstack.org/293511 | 19:47 |
| anteaya | eil397: but you don't think we are, otherwise you wouldn't have suggested the topic | 19:47 |
| anteaya | eil397: can you help me understand how you are defining your terms? | 19:48 |
| anteaya | what do you mean by community infra project? | 19:48 |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'tuning-box' https://review.openstack.org/286137 | 19:48 |
| thomasem | So, looking at different node types (fedora-23 specifically) for the LXC experimental gate job atm. I'm wondering what the usual way to run Nova gate tests using a WIP patch for openstack-infra/project-config repo? I tried Depends-On, but I didn't see it using a fedora-23 node instead of ubuntu-trusty. | 19:48 |
| thomasem | If there is one. | 19:48 |
| clarkb | thomasem: there isn't a way to do that | 19:48 |
| thomasem | :( | 19:48 |
| clarkb | thomasem: we can't let unreviewed jenkins jobs through for testing because they can potentially do nefarious things | 19:49 |
| thomasem | Ah, right. Good point. | 19:49 |
| eil397 | anteaya: right now I'm busy little bit. it is nice that I've caught you. | 19:49 |
| anteaya | eil397: okay that's fine | 19:49 |
| clarkb | jenkins doesn't have an unprivileged mode for some jobs | 19:49 |
| anteaya | eil397: I'm interested in understanding what you mean | 19:49 |
| anteaya | eil397: take your time and we can work together to understand one another | 19:50 |
| pabelanger | clarkb: where should I be looking on puppetmaster.o.o to check JJB runs? | 19:50 |
| *** kzaitsev_mb has quit IRC | 19:50 | |
| thomasem | Understood. So, then, how about I throw up a patch ADDING another experimental job using fedora-23 in the devstack-gate's experimental pipeline? | 19:50 |
| pabelanger | clarkb: assuming they are still broken | 19:50 |
| clarkb | pabelanger: JJB runs on each jenkins master locally | 19:50 |
| thomasem | Would that be merge-worthy to y'all and thusly let me test that? | 19:50 |
| clarkb | thomasem: thats probably fine though I don't have all the context in my head | 19:50 |
| eil397 | anteaya: thank you for your time. I'm working in walmartlabs. and I've spent few month to work with internal CICD for OpenStack. and I've started from clonning Comunity CICD. I should say it is awesome. and deployment automation works great. and it is pretty clear orginized | 19:50 |
| pabelanger | clarkb: right, but does ansible log that some place on puppetmaster? or just locally on the master | 19:51 |
| anteaya | eil397: I'm glad you are having a good experience | 19:51 |
| clarkb | pabelanger: puppet is only logged on the machine it runs on and the puppetboard report | 19:51 |
| anteaya | eil397: thanks for that feedback | 19:51 |
| clarkb | pabelanger: I do not know if we write the jjb output anywhere | 19:51 |
| clarkb | my guess is we dont | 19:51 |
| thomasem | Ah, for context, I'm having a horrible time with the current libvirt and kernel versions on the ubuntu-trusty node when trying to get the libvirt/LXC nova experimental job to pass consistently. I am, however, after several suggestions from the Nova folks, seeing much better behavior with Fedora 23 in my local simulations. So, I wanted to get the test moved over to using fedora-23 and see how that goes in the _actual_ pipeline as opposed to my | 19:52 |
| thomasem | local simulations. | 19:52 |
| thomasem | clarkb: ^^ | 19:52 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Bug Fixing https://review.openstack.org/293683 | 19:52 |
| pabelanger | clarkb: okay, so from what I see jenkins04,05 and 06 are running old JJB. Was trying to see how you know the flock stuff was broken | 19:52 |
| eil397 | anteaya: as a result of some work. I have made internal proposal about who can we be integrated with community cicd (infra) and what are critical components we should add . also I've proposed topic for summit. and I'm glad it was not voted. because it is little bit specific for company. and maybe not very good example for community. | 19:53 |
| eil397 | anteaya: s/who/how/g | 19:53 |
| thomasem | The goal right now is to at least get the code tested in the pipeline, but i can't even get it to non-voting status without some amount of stability exhibited. | 19:53 |
| anteaya | eil397: okay so you have some modifications to our tooling that you are using in your work | 19:54 |
| anteaya | eil397: and you would like to offer those modifications to the upstream tools? | 19:54 |
| *** maishsk has joined #openstack-infra | 19:54 | |
| clarkb | thomasem: can you just change the slave type out from under that job and call it good? | 19:54 |
| *** adrian_otto has joined #openstack-infra | 19:54 | |
| clarkb | thomasem: I am not sure how far along the ubuntu job got if it doesn't have reliable containerthings | 19:54 |
| thomasem | You mean like this: https://review.openstack.org/#/c/293585/1/jenkins/jobs/devstack-gate.yaml | 19:54 |
| eil397 | anteaya: it is not modification. not fork. because it will make very expensive syncing. but it is about what are critical components for internal cicd for openstack. | 19:55 |
| *** ig0r_ has quit IRC | 19:55 | |
| clarkb | thomasem: ya | 19:55 |
| thomasem | It's still experimental. I'm hoping to get at least something non-voting so we can pursue it further. | 19:55 |
| anteaya | eil397: hmmmm | 19:55 |
| thomasem | The idea is to get it working on one platform, then move to others as we find problems. | 19:55 |
| clarkb | thomasem: gotcha, so ya start with that change, run it in experimental a few times, show nova it works, then move it to an appropriate location | 19:55 |
| eil397 | anteaya: for example deployment automation on hardware mulitinode cluster. and support of internal rm processes. | 19:55 |
| eil397 | anteaya: rm ( release management) | 19:55 |
| thomasem | Okay | 19:56 |
| *** rockyg has joined #openstack-infra | 19:56 | |
| *** ociuhandu has joined #openstack-infra | 19:56 | |
| anteaya | eil397: okay so you are thinking you would like to have a discussion with people consuming our tooling and hear from the group deploying infra tools internally? | 19:56 |
| anteaya | like the infra tools operators group | 19:56 |
| thomasem | So, let me get a commit and related bug tagged on that commit... I guess the reason I wanted to leave the ubuntu-trusty one around was because I'd like for us to get it working on that platform too at some point. Might have to wait for a newer Ubuntu, though. | 19:56 |
| clarkb | thomasem: ya I wouldn't worry about that until you have something working | 19:56 |
| thomasem | Okay, that's fair. | 19:56 |
| *** rockyg has quit IRC | 19:57 | |
| *** rockyg has joined #openstack-infra | 19:57 | |
| *** tonytan_brb is now known as tonytan4ever | 19:58 | |
| thomasem | Thanks, clarkb | 19:58 |
| thomasem | :) | 19:58 |
| *** sfinucan has joined #openstack-infra | 19:58 | |
| *** ericksonsantos has joined #openstack-infra | 19:58 | |
| thomasem | I'll throw up a new review once I add the related bug(s) and context to the commit msg. | 19:58 |
| pabelanger | anteaya: eil397: The updates you are referring to, is anything public atm? I always fine it easier to look at code to see how people are using infra CI tooling | 19:58 |
| anteaya | pabelanger: I wonder if eil397 is proposing a meeting of the minds more than "we need this fix" | 19:59 |
| anteaya | that is my current belief | 19:59 |
| openstackgerrit | Merged openstack-infra/openstackid: Bug Fixing https://review.openstack.org/293683 | 19:59 |
| anteaya | pabelanger: but I agree, code is helpful | 20:00 |
| pabelanger | anteaya: right. I was mostly interested to see if anything existed today and read up on. Almost like home work before said meeting | 20:00 |
| eil397 | anteaya: my first intention when I've proposed as a talk on summit was to present some point of view. maybe something in my view can be corrected and some specifics are really do not exist. | 20:00 |
| anteaya | pabelanger: yup, good thoughts | 20:00 |
| eil397 | anteaya: that is right. | 20:00 |
| anteaya | eil397: okay, it is an interesting thought, I'm glad you raised it | 20:01 |
| anteaya | I don't think we have had this idea before | 20:01 |
| anteaya | eil397: can I make a suggestion for what to put on the etherpad so perhaps others can understand it? | 20:01 |
| *** icey has quit IRC | 20:01 | |
| *** salv-orlando has joined #openstack-infra | 20:01 | |
| anteaya | eil397: I can't guarentee they also won't have questions | 20:01 |
| anteaya | but I can suggest wording that would make sense to me | 20:02 |
| *** icey has joined #openstack-infra | 20:02 | |
| *** tonytan4ever has quit IRC | 20:02 | |
| eil397 | anteaya: thank you. | 20:02 |
| *** flwang has joined #openstack-infra | 20:02 | |
| anteaya | okay here is my suggestion | 20:02 |
| *** cloudtrainme has joined #openstack-infra | 20:02 | |
| *** tonytan4ever has joined #openstack-infra | 20:02 | |
| anteaya | Leader: <your name> | 20:02 |
| eil397 | anteaya: yes. of course. | 20:02 |
| flwang | hi there, anybody can help add me as the 'ptl' of zaqar-ui-core? so that i can add those UI experts, thanks a lot | 20:03 |
| anteaya | Description: meetup for ci operators consuming tools internally for their own work | 20:03 |
| eil397 | anteaya: I'm not sure about <leader>: <my name> because right now I 'm not planning to attend summit | 20:03 |
| *** jcoufal has quit IRC | 20:03 | |
| flwang | here is the related patch https://review.openstack.org/293166 | 20:03 |
| anteaya | Spec: not applicable | 20:03 |
| anteaya | eil397: oh | 20:03 |
| anteaya | well that makes it tough | 20:03 |
| eil397 | anteaya: but I'm very interesting to have such kind of discussion for such kind of topic | 20:03 |
| anteaya | yes I can understand | 20:03 |
| *** claudiub has joined #openstack-infra | 20:03 | |
| SamYaple | so returning a bit to the kolla automated pushing of docker images, this can be done but there is a secret file that is needed to be kept secure. Can anyone tell me if we have a way in infra to protect a file like this? like a private-key type file? | 20:04 |
| anteaya | we usually lean toward folks proposing the sessions leading them | 20:04 |
| pabelanger | clarkb: okay, I can tell JJB is running on each system by SSH into them (pretty obvious) but still having trouble understanding how run_jjb playbook is called. | 20:04 |
| pabelanger | clarkb: unless that is manually | 20:04 |
| eil397 | anteaya: maybe some one would like to talk about it. I can provide deatails about some xperience | 20:04 |
| anteaya | eil397: hmmmmm | 20:04 |
| anteaya | okay yeah, put your name on the etherpad and say you aren't able to attend summit | 20:04 |
| flwang | or i should sent a mail to openstack-dev as a request? thanks | 20:04 |
| eil397 | anteaya: that is why maybe we should start from irc. | 20:04 |
| clarkb | pabelanger: puppet runs jjb not ansible | 20:05 |
| anteaya | and provide details either in the etherpad for the summit or a separate etherpad if there are a lot | 20:05 |
| anteaya | eil397: I would like to hear your experience | 20:05 |
| eil397 | anteaya: I've added it to etherpad to show interest to it : - ) | 20:05 |
| pabelanger | clarkb: Ah, so thats what I was confused about. I thought we moved it into ansible and out of puppet | 20:05 |
| anteaya | eil397: and you spoke up early so we have some time to figure out what kind of interest there is | 20:05 |
| anteaya | eil397: thank you | 20:05 |
| pabelanger | so, people use run_jjb when they manually sync it | 20:05 |
| *** claudiub|2 has joined #openstack-infra | 20:05 | |
| clarkb | pabelanger: I don't | 20:05 |
| clarkb | but others do | 20:06 |
| *** sfinucan has quit IRC | 20:06 | |
| anteaya | eil397: might I ask where you are located geographically? in case there might be an event closer to where you are in future that you could attend? | 20:06 |
| clarkb | (I like to be able to actually watch the progress and see how far ahead/behind it is | 20:06 |
| pabelanger | SamYaple: Is it a onetime password? Or same password all the time? | 20:06 |
| pabelanger | clarkb: right | 20:06 |
| eil397 | anteaya: I'm in bay area - NorCal. sunnyvale | 20:07 |
| anteaya | oh wow | 20:07 |
| anteaya | hard to think of summits that are going to be much closer to you | 20:07 |
| pabelanger | SamYaple: currently, the only place ATM would be in hiera and system-config | 20:07 |
| anteaya | unless they are in the bay area | 20:07 |
| pabelanger | SamYaple: but, if you are using ansible, why not setup ansible-vault for your project | 20:08 |
| pabelanger | guess you still need password to open it | 20:08 |
| *** claudiub has quit IRC | 20:08 | |
| *** infra-red has quit IRC | 20:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix two grafana entries to point at the right zuul pipeline https://review.openstack.org/293544 | 20:09 |
| *** _nadya_ has quit IRC | 20:09 | |
| pabelanger | okay, stepping away for food. If any infra-root would like to help 293651 along, I can continue work on debian-jessie later tonight | 20:09 |
| anteaya | enjoy food | 20:10 |
| pabelanger | AJaeger: we are also still at 16 devstack-trusty images. But JJB still running on some jenkins masters | 20:10 |
| *** ayoung has joined #openstack-infra | 20:11 | |
| *** IlyaG has joined #openstack-infra | 20:12 | |
| *** bpokorny_ has joined #openstack-infra | 20:12 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: add puppet4 jobs in experimental queue https://review.openstack.org/293178 | 20:13 |
| AJaeger | pabelanger: and I see some jobs that use devstack-trusty, investigating... | 20:13 |
| *** bpokorny has quit IRC | 20:13 | |
| AJaeger | example gate-scalpels-dsvm-neutron-devstack-trusty | 20:13 |
| AJaeger | old cache, let me regenerate ;( | 20:13 |
| clarkb | AJaeger: that one should've been addressed in my change (I specifically rmember catching that one) | 20:13 |
| fungi | yeah, i still have 3 jenkins masters updating | 20:14 |
| *** _nadya_ has joined #openstack-infra | 20:14 | |
| *** cloudtrainme has quit IRC | 20:14 | |
| fungi | partly my fault since i initially ctrl-c'd them after realizing i'd rerun on those with --workers 2, and the interrupt resulted in broken jjb caches which i had to delete | 20:15 |
| *** bpokorny_ has quit IRC | 20:15 | |
| fungi | because i think i ctrl-c'd a second time reflexively while they were writing out their caches | 20:15 |
| *** _nadya_ has quit IRC | 20:15 | |
| *** bpokorny has joined #openstack-infra | 20:15 | |
| AJaeger | clarkb: but we have gate-puppet-zanata-puppet-beaker-rspec-dsvm-trusty-nv | 20:15 |
| fungi | they're getting close to complete now though | 20:15 |
| AJaeger | See http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/puppet-module-jobs.yaml#n3 | 20:16 |
| *** kzaitsev_mb has joined #openstack-infra | 20:16 | |
| AJaeger | node: 'devstack-{ostype}' | 20:16 |
| *** cloudtrainme has joined #openstack-infra | 20:16 | |
| *** IlyaG has quit IRC | 20:16 | |
| clarkb | AJaeger: gah thats how it got past my grep | 20:16 |
| *** IlyaG has joined #openstack-infra | 20:16 | |
| clarkb | ok I will write a patch for that one | 20:16 |
| AJaeger | thanks, clarkb ! | 20:16 |
| fungi | that one will take a little extra parameter work i guess | 20:16 |
| clarkb | fungi: depends on how many devstacks it supports | 20:17 |
| clarkb | will find out shortly | 20:17 |
| fungi | true! | 20:17 |
| AJaeger | clarkb, I run: rm -rf .test/jenkins-job-builder/.test/new/out/;tox -e zuul;grep ubuntu-trusty .test/jenkins-job-builder/.test/new/out/* | 20:17 |
| clarkb | wow aarently we did that all over | 20:18 |
| * clarkb hatchets it | 20:18 | |
| AJaeger | clarkb: 116 jobs left with ubunut-trusty | 20:19 |
| AJaeger | clarkb: 116 jobs left with devstack-trusty I mean | 20:19 |
| SamYaple | pabelanger: its very much a private key, but i would need to store it soomewhere and it would need to be projected so, say, only the periodic gate jobs could access it | 20:19 |
| SamYaple | otherwise it would be a security hole (user could submit a patchset and extract the password) | 20:20 |
| *** bpokorny has quit IRC | 20:20 | |
| *** bpokorny has joined #openstack-infra | 20:20 | |
| clarkb | AJaeger: ya I think most of them are these puppet jobs | 20:20 |
| flwang | clarkb: ping re (09:03:02) flwang: hi there, anybody can help add me as the 'ptl' of zaqar-ui-core? so that i can add those UI experts, thanks a lot | 20:21 |
| AJaeger | clarkb: yes - plus gate-openstackci-beaker-trusty-dsvm | 20:21 |
| flwang | clarkb: i think you mentioned how to do that, but i lost the msg now | 20:21 |
| ianw | pabelanger: hey; so centos-7 seems to have built ok ... but still not registered. either it hasn't uploaded or it isn't working. i haven't read scrollback to see if this is expected or not | 20:21 |
| *** IlyaG has quit IRC | 20:21 | |
| clarkb | wow these puppet jobs are a mess | 20:23 |
| clarkb | they are also completely divergent from ours from what I can tell | 20:23 |
| *** esikachev has quit IRC | 20:23 | |
| clarkb | YOLO! | 20:23 |
| * Clint chuckles. | 20:24 | |
| clarkb | apparently we run all of our lint checks and that on trusty and they run all of theirs on centos | 20:25 |
| *** nwkarsten has joined #openstack-infra | 20:25 | |
| clarkb | (I am just grumping I have to make the same change twice) | 20:25 |
| AJaeger | clarkb: just merge them ;) | 20:26 |
| fungi | wow, the stable branch team's electorate is second only in size to infra's | 20:26 |
| fungi | with neutron in third place | 20:26 |
| fungi | then docs, then nova, then qa | 20:26 |
| *** IlyaG has joined #openstack-infra | 20:27 | |
| flwang | AJaeger: could you pls help? | 20:28 |
| AJaeger | flwang: I have no permissions to help you | 20:29 |
| AJaeger | clarkb: https://review.openstack.org/#/c/290726/ | 20:29 |
| flwang | AJaeger: how should i do then? thanks | 20:29 |
| *** pvaneck has joined #openstack-infra | 20:29 | |
| flwang | like who is the right guy i should talk/email? | 20:29 |
| clarkb | AJaeger: interesting, I am sure my change will merge conflict with that | 20:29 |
| clarkb | flwang: patience, as soon as I am done with the thing I prevkously committed to I can do your thing | 20:30 |
| *** esikachev has joined #openstack-infra | 20:30 | |
| fungi | SamYaple: pabelanger: yeah, we do uploads to external systems like that from durable non-nodepool-managed workers which run very limited jobs in post or release pipelines which don't involve running code from the repositories being processed | 20:30 |
| flwang | clarkb: since i already got my answer, now i will be quite, thanks | 20:30 |
| fungi | SamYaple: pabelanger: so that we can safely store keys/passwords on them without risk of exposure. it's a solved problem | 20:31 |
| *** IlyaG has quit IRC | 20:32 | |
| *** baoli has quit IRC | 20:32 | |
| clarkb | there is really only one set of jobs that cause problems in this and that is the beaker jobs beacuse they build ostype into the job itself | 20:32 |
| clarkb | I basically end up with two groups instead of one I think | 20:32 |
| jeblair | SamYaple: thank you for realizing the problem though! so many people come through here with "simple ideas" and don't appreciate that. | 20:32 |
| anteaya | fungi: interesting | 20:33 |
| fungi | pabelanger: any objection to becoming a gerrit admin, so that you can help with initial group membership setting, troubleshooting via javamelody, et cetera? | 20:33 |
| *** ldnunes_ has joined #openstack-infra | 20:35 | |
| * AJaeger waves good bye | 20:36 | |
| *** MarkAtwood has quit IRC | 20:37 | |
| *** ldnunes has quit IRC | 20:38 | |
| fungi | have a good evening AJaeger! | 20:39 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics https://review.openstack.org/263971 | 20:39 |
| fungi | clarkb: jeblair: jhesketh: mordred: SergeyLukjanov: nibalizer: yolanda: i'm seeking a gerrit admin volunteer who can run the script from https://review.openstack.org/263971 some time between midnight utc tonight and midnight utc tomorrow night and send a signed copy of the results to tristanC and tonyb. instructions are at the top of the script, but i'm happy to walk you through them and help you | 20:39 |
| fungi | confirm the results look like they should if you want | 20:39 |
| anteaya | good night AJaeger | 20:39 |
| SamYaple | fungi: so on that note, the idea is to build kolla images and push them to the docker repository, that _would_ required code running from the repository | 20:41 |
| fungi | SamYaple: why? you build the docker images on one system, then use a different system to upload them | 20:41 |
| *** andymaier has quit IRC | 20:41 | |
| fungi | as long as your build step and your upload step are decoupled, should be fine | 20:42 |
| *** asettle has joined #openstack-infra | 20:42 | |
| fungi | it does require shuffling the files around, but it's a pattern we already follow for, e.g., releases to pypi | 20:42 |
| SamYaple | fungi: but transfering the data from the build to upload system | 20:42 |
| fungi | SamYaple: i feel like i've already replied to a -dev ml thread on this topic with a variety of concerns/options a few weeks ago | 20:43 |
| openstackgerrit | Thomas Maddox proposed openstack-infra/project-config: Move to Fedora 23 for Libvirt/LXC Nova Test https://review.openstack.org/293585 | 20:43 |
| *** IlyaG has joined #openstack-infra | 20:43 | |
| SamYaple | fungi: ill dig through it | 20:43 |
| fungi | pretty sure it was some team wanting to come up with a mechanism for auto-building and uploading some sort of container images. might have been a thread sdake started? | 20:44 |
| *** sridhar_ram has joined #openstack-infra | 20:44 | |
| *** ianw has quit IRC | 20:46 | |
| *** ianw_ is now known as ianw | 20:46 | |
| *** dtardivel has quit IRC | 20:48 | |
| *** abregman has joined #openstack-infra | 20:50 | |
| SamYaple | fungi: he might have | 20:51 |
| fungi | are our openstackci-beaker jobs broken? | 20:51 |
| SamYaple | sdake: ping | 20:51 |
| *** IlyaG has quit IRC | 20:51 | |
| clarkb | yes it was sdake tlaking about uploading to dockers image repothingy | 20:51 |
| SamYaple | jeblair: im all 'bout dat security | 20:51 |
| sdake | sup | 20:51 |
| fungi | http://logs.openstack.org/71/263971/10/check/gate-openstackci-beaker-trusty-dsvm/ca4bdb1/console.html#_2016-03-16_20_40_08_601 | 20:51 |
| fungi | "Could not find puppet-4.4.0 in any of the sources" | 20:52 |
| SamYaple | we were discussing the private key for docker uploads sdake | 20:52 |
| *** aysyd has quit IRC | 20:52 | |
| SamYaple | fungi mentioned a ML thread that i may have missed | 20:52 |
| sdake | ya private key | 20:52 |
| sdake | we need that | 20:52 |
| fungi | i'm scouring my sent mail for the subject line now | 20:52 |
| SamYaple | sdake: do you remember the ML thread? thats what the ping was for | 20:53 |
| fungi | Subject: Re: [openstack-dev] [kolla][infra] Publishing kolla images to docker-registry.openstack.org | 20:53 |
| fungi | looks like it'll be in the february archive | 20:53 |
| *** jpr has quit IRC | 20:53 | |
| SamYaple | fungi: i have it in my inbox | 20:54 |
| * fungi ceases digging up a pipermail link | 20:54 | |
| SamYaple | yea i dont see any solutions there fungi, but i can continue the thread | 20:54 |
| SamYaple | to address your comment from before, this isnt a few files. its, conservatively, ~15GB | 20:55 |
| SamYaple | so how would that move from build to push host | 20:55 |
| fungi | SamYaple: my reply from 2016-02-24 17:43:02 utc | 20:55 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 20:55 |
| SamYaple | fungi: i do not have a reply from you in my inbox | 20:55 |
| SamYaple | fungi: did you reply to the ML? or a person? | 20:55 |
| fungi | SamYaple: http://lists.openstack.org/pipermail/openstack-dev/2016-February/087453.html | 20:56 |
| sdake | i posted the size of my registry disk usage | 20:56 |
| sdake | it was about 1.5gb per distro | 20:56 |
| SamYaple | sdake: thats not quite accurate though | 20:57 |
| SamYaple | registry usage and image usage are a bit different | 20:57 |
| SamYaple | fungi: wierd, not in my email, reading your reply | 20:57 |
| sdake | i was talking about the size of the disk space used on the target registry server | 20:58 |
| *** e0ne has quit IRC | 20:58 | |
| *** maishsk has quit IRC | 20:59 | |
| SamYaple | right. so that is another question. pushing to an openstack controlled registry, or the dockerhub | 20:59 |
| SamYaple | my vote would be dockerhub directly | 20:59 |
| *** mattoliverau has quit IRC | 20:59 | |
| *** annegentle has joined #openstack-infra | 20:59 | |
| *** matt6434 has joined #openstack-infra | 20:59 | |
| *** woodster_ has joined #openstack-infra | 21:00 | |
| *** matt6434 is now known as mattoliverau | 21:00 | |
| fungi | anyway, i'm happy to go into more detail on any of this, though i need to disappear for a while as it's approaching dinnertime | 21:00 |
| *** thorst has quit IRC | 21:01 | |
| fungi | and there are other denizens of this channel with equal or greater knowledge of the options than i possess | 21:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a simple status webapp https://review.openstack.org/293706 | 21:01 |
| SamYaple | fungi: if you would that would be fantastic. i wish i could reply to you on that, but im missing the email in my inbox | 21:01 |
| doug-fish | ajaeger | 21:01 |
| *** ddieterly is now known as ddieterly[away] | 21:01 | |
| doug-fish | oops! I was trying to search back and see what is up with the openstackid/zanata issue | 21:01 |
| doug-fish | It has some urgency because it blocks translators from making progress | 21:02 |
| *** bpokorny has quit IRC | 21:03 | |
| jeblair | fungi, pabelanger, ianw: James E. Blair proposed openstack-infra/nodepool: Add a simple status webapp https://review.openstack.org/293706 | 21:03 |
| *** thorst has joined #openstack-infra | 21:03 | |
| *** bpokorny has joined #openstack-infra | 21:03 | |
| *** ybathia has joined #openstack-infra | 21:04 | |
| bknudson | Question about the reqs update tool -- see https://review.openstack.org/#/c/293429/1/requirements.txt | 21:04 |
| bknudson | should the reqs update tool be enhanced to keep the [kerberos] ? | 21:04 |
| *** achanda has joined #openstack-infra | 21:05 | |
| bknudson | or were we mistaken adding it in the first place? | 21:05 |
| *** ddieterly[away] is now known as ddieterly | 21:05 | |
| *** esikachev has quit IRC | 21:06 | |
| *** thorst has quit IRC | 21:07 | |
| *** maishsk has joined #openstack-infra | 21:08 | |
| *** madhu_ak has joined #openstack-infra | 21:10 | |
| *** annegent_ has joined #openstack-infra | 21:10 | |
| *** annegentle has quit IRC | 21:11 | |
| madhu_ak | Just checking, has anyone got into this error or aware of it: http://logs.openstack.org/28/164828/34/check/gate-neutron-lbaasv2-dsvm-api-namespace-nv/6cda6a4/logs/devstacklog.txt.gz#_2016-03-16_20_13_14_499 ? sudo: add-apt-repository: command not found | 21:11 |
| *** doug-fish has quit IRC | 21:11 | |
| madhu_ak | devstack-trusty is removed? | 21:11 |
| *** DericHorn-HP has joined #openstack-infra | 21:12 | |
| *** DericHorn-HP has quit IRC | 21:12 | |
| *** doug-fish has joined #openstack-infra | 21:12 | |
| *** Daisy has joined #openstack-infra | 21:14 | |
| *** rfolco has quit IRC | 21:14 | |
| *** ldnunes_ has quit IRC | 21:15 | |
| clarkb | yes devstack trusty is replaced by ubuntu-trusty | 21:15 |
| tonyb | fungi: it looks like you're still looking for a volenteer, is that correct? | 21:15 |
| *** maishsk has quit IRC | 21:16 | |
| *** ddieterly is now known as ddieterly[away] | 21:17 | |
| *** Daisy has quit IRC | 21:18 | |
| *** kro_mnckn_doc has quit IRC | 21:19 | |
| *** anteaya has quit IRC | 21:19 | |
| *** zaro has quit IRC | 21:19 | |
| *** samueldmq has quit IRC | 21:19 | |
| madhu_ak | clarkb, thanks for the info | 21:19 |
| *** amitgandhinz has quit IRC | 21:20 | |
| sdake | can someone help me sort out how to do what Tristan asks here: https://review.openstack.org/#/c/293687/ | 21:21 |
| madhu_ak | so does it mean, should the line 14 from https://github.com/openstack/neutron-lbaas/blob/master/devstack/plugin.sh#L11 changed to BACKPORT="deb deb http://archive.ubuntu.com/ubuntu/ trusty main restricted" ? | 21:21 |
| madhu_ak | clarkb ^6 ? | 21:21 |
| sdake | i'm not sure how to update a review that is already merged | 21:21 |
| mgagne | jeblair: fyi, default timezone is now UTC in our images at internap | 21:21 |
| sdake | nm its not merged | 21:22 |
| *** annegent_ has quit IRC | 21:23 | |
| *** achanda has quit IRC | 21:24 | |
| *** tonytan_brb has joined #openstack-infra | 21:25 | |
| jeblair | mgagne: yay! | 21:26 |
| mgagne | jeblair: thanks for the suggestion =) | 21:27 |
| *** achanda has joined #openstack-infra | 21:28 | |
| *** shardy has quit IRC | 21:28 | |
| *** tonytan4ever has quit IRC | 21:28 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 21:30 |
| clarkb | I expect ^ to fail every job ever but it gives a sense of the scale of refacorting required here | 21:30 |
| nibalizer | fungi: tonyb im happy to do it | 21:30 |
| clarkb | also my eyes could no longer keep up with what is what | 21:30 |
| clarkb | so wnated to push something before I went cross eyed | 21:30 |
| *** eharney has quit IRC | 21:30 | |
| tonyb | nibalizer: cool. Thanks. | 21:31 |
| *** darrenc is now known as darren_afk | 21:31 | |
| clarkb | 789 line diff thats less than I was owrried | 21:31 |
| *** dtantsur|afk has quit IRC | 21:31 | |
| *** pradk has quit IRC | 21:32 | |
| *** csaba has quit IRC | 21:33 | |
| madhu_ak | clarkb, Just checking, is there any link that states that devstack-trusty is replaced by ubuntu-trusty? | 21:33 |
| *** pblaho has quit IRC | 21:33 | |
| clarkb | madhu_ak: the goal was to make it transparent but AJaeger also sent mail to the dev list earlier this week or was it last week about it | 21:33 |
| *** dprince has quit IRC | 21:34 | |
| madhu_ak | oh. I am pretty sure I might have overlooked that email. let me check mailing list..Incase you have it handy, if you could send me the link to the email thread would be awesome.. | 21:34 |
| *** pblaho has joined #openstack-infra | 21:34 | |
| clarkb | madhu_ak: [openstack-dev] [all][infra] Switching images for testing is the subject | 21:35 |
| *** achanda has quit IRC | 21:36 | |
| madhu_ak | Got it clarkb thanks. | 21:36 |
| *** e0ne has joined #openstack-infra | 21:37 | |
| *** dtantsur has joined #openstack-infra | 21:37 | |
| nibalizer | tonyb: tristanC prefered emails? | 21:38 |
| tonyb | nibalizer: listed here: https://wiki.openstack.org/wiki/PTL_Elections_March_2016#Officials | 21:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 21:38 |
| tonyb | nibalizer: that's eaiser than typing :) | 21:39 |
| *** nwkarsten has quit IRC | 21:39 | |
| *** e0ne has quit IRC | 21:40 | |
| *** johnsom has joined #openstack-infra | 21:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add a simple status webapp https://review.openstack.org/293706 | 21:40 |
| johnsom | Is this the right place to request a gerrit permissions fix for a project? | 21:40 |
| nibalizer | tonyb: great thanks | 21:41 |
| johnsom | openstack/neutron-lbaas-dashboard is missing two permissions for the neutron-release group | 21:41 |
| jhesketh | Morning | 21:43 |
| *** annegentle has joined #openstack-infra | 21:43 | |
| gordc | hmm... random question but did uuidgen command disappear from ubuntu trusty recently? | 21:44 |
| clarkb | johnsom: typically you can just push the change for the acls by editing openstack-infra/gerrit/acls/ and pushing to gerrit | 21:45 |
| gordc | http://logs.openstack.org/63/293463/1/gate/gate-telemetry-dsvm-integration-aodh/1d6b6e6/logs/devstacklog.txt.gz#_2016-03-16_19_38_36_946 | 21:45 |
| clarkb | gordc: we recently just switched the underlying node type, it is possible the uuidgen was present in the other images but not ubuntu-trusty | 21:45 |
| clarkb | gordc: ideally with devstack jobs you install any dep via devstack | 21:45 |
| *** matrohon has quit IRC | 21:45 | |
| *** csaba has joined #openstack-infra | 21:46 | |
| gordc | clarkb: i see. i thought devstack was installing it already but i guess not. | 21:46 |
| *** darren_afk is now known as darrenc | 21:46 | |
| *** ddieterly[away] is now known as ddieterly | 21:47 | |
| *** achanda has joined #openstack-infra | 21:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 21:48 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:48 | |
| clarkb | gordc: we can in theory add things like that to the image but if it isn't needed to bootstrap jobs then it is probably best placed in devstack's deps list | 21:48 |
| *** annegentle has quit IRC | 21:48 | |
| johnsom | clarkb Got it. Thanks! | 21:48 |
| gordc | clarkb: will do. taking a look now. thanks for pointer | 21:49 |
| clarkb | johnsom: sorry that path isn't correct, openstack-infra/project-config/gerrit/acls is better | 21:49 |
| johnsom | Yeah, I knew what you meant and found it | 21:49 |
| clarkb | johnsom: but ya you can edit the acl file directly and when approved the project acls are updated | 21:49 |
| clarkb | kk | 21:49 |
| *** automagically_ has joined #openstack-infra | 21:50 | |
| *** smarcet has quit IRC | 21:50 | |
| *** piet has quit IRC | 21:50 | |
| thiagop | Guys, I'm using puppet-openstackci to build a third party ci environment here but everytime I "puppet apply", the zuul-merger forgets gerrit's fingerprint and cannot clone the repository | 21:50 |
| thiagop | am I missing something? | 21:51 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config: neutron-lbaas-dashboard project is missing perms https://review.openstack.org/293731 | 21:53 |
| *** rockyg has quit IRC | 21:53 | |
| clarkb | thiagop: if I had to guess puppet is trying to add a known hosts entry and doesn't have the appropriate data to do so causing it to clear out anything that was there previously | 21:53 |
| asselin | thiagop, you should be able to set it in the common.yaml heira keys | 21:53 |
| thiagop | righ clarkb asselin, the default heira key isn't the same as mine | 21:54 |
| asselin | clarkb, that said, a lot of poeple are complaining about this. ^^ Can you look at https://review.openstack.org/#/c/266041/ | 21:54 |
| *** abregman has quit IRC | 21:55 | |
| asselin | thiagop, can you comment on ^^ | 21:55 |
| *** _ari_ has quit IRC | 21:55 | |
| clarkb | asselin: it has a -1 from BobBall_AWOL | 21:55 |
| *** alex_xu has quit IRC | 21:56 | |
| jeblair | can we put that in public hiera yet? | 21:56 |
| asselin | yeah...he needs an extra newline...but why is that change needed? | 21:57 |
| *** mriedem has quit IRC | 21:57 | |
| *** alex_xu has joined #openstack-infra | 21:58 | |
| *** sambetts|afk has quit IRC | 21:59 | |
| *** mrmartin has quit IRC | 21:59 | |
| *** sridhar_ram1 has joined #openstack-infra | 22:00 | |
| *** ybathia has quit IRC | 22:00 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 22:01 |
| *** sridhar_ram has quit IRC | 22:01 | |
| jeblair | clarkb: in that change, why are you making 2 job groups instead of sticking with 'puppet-beaker-jobs'? | 22:01 |
| clarkb | jeblair: because we have to keep ostype for the jobs beaker config stuff and it does a full join on two parameters | 22:02 |
| *** sambetts has joined #openstack-infra | 22:02 | |
| clarkb | jeblair: if the beaker config is cleaned up we can go back to one job group but that requires updating every puppet project :/ | 22:02 |
| jeblair | clarkb: gotcha, thanks | 22:02 |
| *** tonytan_brb has quit IRC | 22:03 | |
| *** annegentle has joined #openstack-infra | 22:03 | |
| jeblair | clarkb: 'dsvm' being special in job names was in openstack_functions.py, right? | 22:03 |
| clarkb | jeblair: correct, it is no longer required | 22:03 |
| jeblair | cool | 22:03 |
| clarkb | jeblair: pabelanger and AJaeger mentioned that as a cleanup for the future on the other jobs so I was just getting ahead of it here | 22:03 |
| jeblair | thought so, just wanted to double check | 22:04 |
| jeblair | clarkb: what's a future cleanup? | 22:04 |
| clarkb | jeblair: on the regular dsvm jobs removing dsvm from all the job names | 22:04 |
| jeblair | yeah ++ | 22:04 |
| craige | o/ | 22:05 |
| *** alex_xu has quit IRC | 22:06 | |
| jeblair | clarkb: i'm +2 on your change. further review from me is unlikely to produce anything that the tests won't catch unless you do more major refactoring, so feel free to carry that forward as you see fit. :) | 22:06 |
| clarkb | jeblair: roger, I am leaning on the tests quite a bit myself | 22:07 |
| jeblair | we have a lot of ready nodes; did we merge more job node changes? | 22:07 |
| *** piet has joined #openstack-infra | 22:07 | |
| jeblair | (i thought this was the only one in flight, but maybe i missed something) | 22:08 |
| *** rfolco has joined #openstack-infra | 22:08 | |
| *** davideagnello has quit IRC | 22:08 | |
| *** davideagnello has joined #openstack-infra | 22:09 | |
| *** flepied has quit IRC | 22:09 | |
| Nepoc | Just for sanity sake, is there an updated version of pypi-mirror that will work with pip 7.1.2 and also handle requirements-constraints.txt | 22:09 |
| *** odyssey4me has quit IRC | 22:09 | |
| *** alex_xu has joined #openstack-infra | 22:09 | |
| *** odyssey4me has joined #openstack-infra | 22:09 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 22:10 |
| clarkb | jeblair: ^ two more missed job name edits | 22:10 |
| clarkb | jeblair: its the devstack-trusty to ubuntu-trusty switch | 22:10 |
| *** rfolco has quit IRC | 22:11 | |
| jeblair | oh, we merged that earlier and this is a followup | 22:11 |
| jeblair | got it | 22:12 |
| *** roxanagh_ has quit IRC | 22:12 | |
| jeblair | and yeah, they are largely devstack-trusty nodes | 22:12 |
| clarkb | Nepoc: we no longer use pypi-mirror and its mostly unmaintained at this point. I don't know if anyone has it working | 22:12 |
| Nepoc | clarkb: okay well I've got it working with 7.1.2... just didn't really want to deal with the second part | 22:12 |
| johnsom | Another question, the changes to the image today, is it expected that the "software-properties-common" package is no longer installed on the systems? | 22:12 |
| *** MarkAtwood has joined #openstack-infra | 22:12 | |
| *** MarkAtwood has quit IRC | 22:13 | |
| *** vgridnev has quit IRC | 22:13 | |
| johnsom | This seems to have broken the neutron-lbaas gates | 22:13 |
| clarkb | Nepoc: we use bandersnatch to mirror pypi and `pip wheel` to make wheels | 22:14 |
| Nepoc | clarkb: Can bandersnatch do a partial mirror, when I looked at it I had to make a 500GB mirror | 22:14 |
| *** pcm_ has joined #openstack-infra | 22:14 | |
| *** mattymo_ has joined #openstack-infra | 22:15 | |
| *** SergeyLukjanov2 has joined #openstack-infra | 22:15 | |
| *** roxanagh_ has joined #openstack-infra | 22:15 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 22:16 |
| *** SergeyLukjanov has quit IRC | 22:16 | |
| *** mattymo has quit IRC | 22:16 | |
| *** pc_m has quit IRC | 22:16 | |
| *** SergeyLukjanov2 is now known as SergeyLukjanov | 22:16 | |
| *** dims_ has joined #openstack-infra | 22:16 | |
| *** acruz has joined #openstack-infra | 22:16 | |
| ianw | johnsom: i don't think it's expected, but we should investigate to see where the right place to put it back in is ... | 22:17 |
| clarkb | Nepoc: I don't think it will make a partial mirror | 22:17 |
| *** tpsilva has quit IRC | 22:17 | |
| *** erlon has quit IRC | 22:17 | |
| johnsom | FYI, it is present in this older build: http://logs.openstack.org/62/282062/3/check/gate-neutron-lbaasv2-dsvm-healthmonitor/5d75ee4/logs/dpkg-l.txt.gz | 22:17 |
| johnsom | And missing now: http://logs.openstack.org/28/164828/34/check/gate-neutron-lbaasv2-dsvm-healthmonitor/56e68ad/logs/dpkg-l.txt.gz | 22:18 |
| clarkb | johnsom: they are starting from two different bases so it is entirely possible. We are trying to push towards being explicit about deps rather than pulling things in by happenstance | 22:18 |
| Nepoc | clarkb: Yeah I had a feeling. I guess I'll be updating pypi-mirror. | 22:18 |
| *** gordc has quit IRC | 22:18 | |
| *** dims has quit IRC | 22:18 | |
| ianw | johnsom: what exactly is it? i'm wondering if it's a red-herring when just comparing the package lists. what's the failure? | 22:19 |
| ianw | ahh -> sudo: add-apt-repository: command not found i bet | 22:20 |
| johnsom | Bingo | 22:20 |
| johnsom | Yeah, neutron-lbaas is adding the trusty-backports repo | 22:20 |
| *** yamahata has joined #openstack-infra | 22:21 | |
| ianw | johnsom: well, probably your plugin should hvae that in it's package list | 22:21 |
| zigo | fungi: The image got built, I believe without an error, could you make it downloadable for me please? | 22:21 |
| *** pvaneck has quit IRC | 22:21 | |
| johnsom | ianw I am interested. Where does that live and does it support backports? | 22:21 |
| *** ybathia has joined #openstack-infra | 22:22 | |
| clarkb | zigo: I don't see a debian image built yet | 22:22 |
| zigo | clarkb: There's the log file at http://nodepool.openstack.org/dib.debian-jessie.log | 22:22 |
| ianw | johnsom: as part of your devstack plugin | 22:22 |
| zigo | clarkb: Where did it go then? | 22:22 |
| clarkb | zigo: that build failed | 22:23 |
| zigo | clarkb: Oh ! | 22:23 |
| clarkb | due to the lack of iamge types I think there is a config update we need to restart the build service to pick it up | 22:23 |
| zigo | clarkb: How do you know it failed? | 22:23 |
| zigo | clarkb: Did I miss an error in the log? | 22:23 |
| zigo | Ah yeah... | 22:24 |
| clarkb | zigo: exit 1 | 22:24 |
| zigo | fatal: failed to copy file to '/opt/dib_tmp/image.YefSYwgq/mnt/opt/git/openstack/heat-specs/.git/objects/2a/5b47527d017fb4aee060c82a9c37695ff90efd': No space left on device | 22:24 |
| zigo | clarkb: Is there somewhere we can increase the space? | 22:24 |
| clarkb | zigo: thats a side effect https://review.openstack.org/#/c/293651/ is the fix I will review that now | 22:25 |
| *** annegentle has quit IRC | 22:26 | |
| ianw | johnsom: http://docs.openstack.org/developer/devstack/plugins.html#system-packages in particular | 22:26 |
| clarkb | zigo: basically dib was using a tmpfs in memory that wasn't big enough and it did that due to arg parsing due to us not providing the image type which requires provider info | 22:26 |
| zigo | clarkb: So, basically, not giving it enough RAM? | 22:27 |
| clarkb | zigo: no, its due to us not passing the correct args because we didn't have enough configuration info | 22:27 |
| johnsom | ianw Oh cool. I will fix that now. Too bad it doesn't seem to account for the backport repos, but this will work to get the tool back. Thanks! | 22:27 |
| clarkb | the change I linked adds the necessary info | 22:27 |
| zigo | Ok. | 22:28 |
| clarkb | johnsom: what packages do you need out of the backports? | 22:28 |
| johnsom | clarkb haproxy | 22:28 |
| zigo | Thanks for the help. | 22:28 |
| clarkb | johnsom: you need a newer haproxy? | 22:29 |
| clarkb | I didn't realize haproxy changed all that much | 22:29 |
| johnsom | Sadly yes | 22:29 |
| johnsom | HA, outside of ubuntu they have done two major releases since the version in trusty. | 22:29 |
| johnsom | Yeah, it gets updated. We are good at finding the bugs too. grin. | 22:30 |
| *** Thelo has quit IRC | 22:30 | |
| clarkb | I dunno we use it too and from release to realse we never change anything and it continue sto work | 22:30 |
| clarkb | *distro release to distro release | 22:30 |
| *** ddieterly has quit IRC | 22:31 | |
| pabelanger | fungi: re: gerrit admin. Sure, if people are willing to mentor me | 22:31 |
| *** chlong has joined #openstack-infra | 22:32 | |
| pabelanger | ianw: sure, I know the image build fine, I can dig more into why it is not registered if you don't see why | 22:33 |
| ianw | pabelanger: well either it's not uploaded, or it's uploaded and doesn't work. i'm hoping the first | 22:33 |
| pabelanger | jeblair: nice, looking | 22:33 |
| openstackgerrit | Merged openstack-infra/project-config: Add debian-jessie to providers https://review.openstack.org/293651 | 22:33 |
| pabelanger | ianw: I don't have access to rax yet, but I can check nodepool debug logs | 22:34 |
| pabelanger | ianw: or even retry the upload | 22:34 |
| clarkb | pabelanger: you can find the rax cloud stuff in /etc/openstack/clouds.yaml | 22:35 |
| pabelanger | zigo: clarkb: review 293651 should address the build failure | 22:35 |
| clarkb | pabelanger: actually you might need all-clouds.yaml but ya | 22:35 |
| pabelanger | clarkb: okay, let me try that | 22:35 |
| pabelanger | clarkb: do people source it or just use the webui? | 22:35 |
| clarkb | pabelanger: depends on the operation. I do my best to use openstackclient out of a venv for most things, but some stuff like trove i end up webui ing it | 22:36 |
| *** dizquierdo has joined #openstack-infra | 22:36 | |
| pabelanger | clarkb: right, I'll start with that | 22:36 |
| clarkb | pabelanger: the *ci account is for long lived servers and *jenkins is what nodepool uses | 22:37 |
| pabelanger | clarkb: thanks | 22:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Cleanup of nodepool builder logging https://review.openstack.org/275982 | 22:38 |
| *** dalgaaf has quit IRC | 22:38 | |
| clarkb | I can restart the nodepool builder to pick up that config change in a bit | 22:39 |
| ianw | ^ get in quick before the bitrot | 22:39 |
| pabelanger | clarkb: sure works for me | 22:39 |
| clarkb | and maybe get that new log config | 22:39 |
| pabelanger | ianw: Ya, I want to gate on that some how. Need to think about it | 22:39 |
| pabelanger | ianw: we'd have to move nodepool-builder.logging into project-config I think | 22:40 |
| ianw | pabelanger: it wouldn't be too hard to make the proposal bot do updates, i just wasn't sure if the rate of change justifies it | 22:40 |
| clarkb | once we get away from bare and devstack the rate of update should be once every new distro release ~6 months? | 22:41 |
| pabelanger | ianw: ya, thought about that too | 22:41 |
| pabelanger | clarkb: hopefullly | 22:41 |
| *** sridhar_ram1 has quit IRC | 22:41 | |
| clarkb | fedora is 6 months, centos and ubuntu every 2 years or so and debian is also ~2 years iirc | 22:41 |
| *** edmondsw has quit IRC | 22:43 | |
| *** rcrit has quit IRC | 22:43 | |
| ianw | pabelanger: it's still in system-config because it's an erb and gets the paths put into it. we seem to have this conversation about moving it periodically | 22:44 |
| pabelanger | ianw: so, according to nodepool logs, centos-7 should be ready in rax-dfw. Seems to fail in other zones | 22:44 |
| pabelanger | ianw: confirming with rax now | 22:44 |
| clarkb | there are 3 uploads going on right now, can restart builder when they are done | 22:46 |
| pabelanger | clarkb: where did you see that? | 22:46 |
| pabelanger | haven't figured out the new nodepool-builders yet | 22:46 |
| *** doug-fis_ has joined #openstack-infra | 22:47 | |
| clarkb | pabelanger: nodepool image-list | grep building | 22:48 |
| *** yamahata has quit IRC | 22:48 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config: Add puppet-pacemaker to tripleo project list https://review.openstack.org/293749 | 22:48 |
| pabelanger | clarkb: Ah, right | 22:48 |
| clarkb | now 2 | 22:48 |
| *** IlyaG has joined #openstack-infra | 22:48 | |
| pabelanger | yup | 22:48 |
| ianw | pabelanger: clarkb has managed to share the uploaded image with me before. if you want to do that, i can boot it and make sure it's ok. i don't see why it would be different to my test builds, but something always manages to be different | 22:48 |
| clarkb | ianw: pabelanger oh for that I had to use the web ui | 22:49 |
| clarkb | you log in and find the image in the image list in the ui and there is a share option that takes a tenant id iirc | 22:49 |
| pabelanger | okay, I can try that | 22:49 |
| clarkb | and its region specific | 22:49 |
| *** Jeffrey4l has joined #openstack-infra | 22:49 | |
| *** automagically_ has quit IRC | 22:50 | |
| *** doug-fish has quit IRC | 22:50 | |
| clarkb | I am not sure if that is something glance does or if it is a rax thign | 22:51 |
| *** doug-fis_ has quit IRC | 22:51 | |
| *** xyang1 has quit IRC | 22:52 | |
| ianw | pretty sure it's a rax thing, i think glance just has a public/private flag | 22:54 |
| *** asettle has quit IRC | 22:54 | |
| clarkb | apparnetly none of the lcouds let us set the public flag :) | 22:55 |
| clarkb | otherwise we could probably just set that | 22:55 |
| pabelanger | okay, rax-iad just finished upload for centos-7 | 22:55 |
| pabelanger | let me see if I can share that one | 22:55 |
| *** mriedem has joined #openstack-infra | 22:57 | |
| *** yamahata has joined #openstack-infra | 22:57 | |
| ianw | i'll also kick of a local build, i haven't tested in a few days | 22:57 |
| *** rbrndt has quit IRC | 22:58 | |
| clarkb | still one image uploading then I can restart the builder | 22:59 |
| ianw | clarkb: yeah, you have to go poking around in glance policy.json to enable it (as wfoster figured out) so probably not common | 22:59 |
| *** rbrndt has joined #openstack-infra | 22:59 | |
| *** tphummel has quit IRC | 22:59 | |
| *** dingyichen has joined #openstack-infra | 22:59 | |
| *** tphummel has joined #openstack-infra | 22:59 | |
| *** rbrndt has quit IRC | 23:00 | |
| openstackgerrit | Madhusudhan Kandadai proposed openstack-infra/project-config: Add octavia experimental tempest job https://review.openstack.org/284875 | 23:00 |
| *** karlamrhein has joined #openstack-infra | 23:02 | |
| *** automagically_ has joined #openstack-infra | 23:02 | |
| *** davideagnello has quit IRC | 23:02 | |
| *** automagically_ has left #openstack-infra | 23:03 | |
| pabelanger | still getting some ubuntu-trusty images being launched, JJB must be out of sync still | 23:03 |
| clarkb | you mean devstack-trusty? | 23:04 |
| *** markmcd1 has quit IRC | 23:04 | |
| clarkb | https://review.openstack.org/#/c/293719/ should help fix that | 23:04 |
| *** davideagnello has joined #openstack-infra | 23:04 | |
| clarkb | but its a big change and lots of reorg :/ | 23:04 |
| SamYaple | asking for an ack on my new regex for the kolla gates https://review.openstack.org/#/c/293021/ | 23:05 |
| SamYaple | should be less infra-taking-down'ish now | 23:06 |
| pabelanger | clarkb: ya, devstack-trusty | 23:06 |
| *** sridhar_ram has joined #openstack-infra | 23:09 | |
| *** adrian_otto has quit IRC | 23:10 | |
| clarkb | ok builder is restarted and the -t flag to dib is filled | 23:10 |
| pabelanger | clarkb: okay, if we want to land puppet change, I can approve now. Otherwise, we can push to tomorrow | 23:11 |
| *** automagically has left #openstack-infra | 23:11 | |
| *** krotscheck has joined #openstack-infra | 23:11 | |
| clarkb | uh its up to you I guess, there may be fallout just because its complicated enough to maybe miss things | 23:12 |
| *** sdague has quit IRC | 23:14 | |
| *** annegentle has joined #openstack-infra | 23:14 | |
| *** dimtruck is now known as zz_dimtruck | 23:15 | |
| thiagop | the problem now is that nodepool is creating the nodes but jenkins are not using them to run the jobs | 23:15 |
| pabelanger | thiagop: sound like a label issue | 23:16 |
| pabelanger | clarkb: sure, let me ping EmilienM | 23:16 |
| thiagop | pabelanger: is it normal that nodepool keep building and deleting instances all the time? | 23:17 |
| thiagop | pabelanger: even when they're not consumed by jenkins? | 23:17 |
| pabelanger | thiagop: so, in that case, it might be because nodepool cannot SSH into the instance | 23:18 |
| *** adrian_otto has joined #openstack-infra | 23:18 | |
| pabelanger | check the logs | 23:18 |
| *** yamahata has quit IRC | 23:18 | |
| EmilienM | I'm on my phone, what's up? | 23:18 |
| *** piet has quit IRC | 23:18 | |
| *** bryan_att has joined #openstack-infra | 23:18 | |
| *** fawadkhaliq has joined #openstack-infra | 23:19 | |
| pabelanger | EmilienM: when you get back to a computer, review 293719 is a reorg of JJB template for puppet | 23:19 |
| *** annegentle has quit IRC | 23:19 | |
| pabelanger | EmilienM: have a look over before I approve | 23:19 |
| thiagop | pabelanger: keeps repeating that: http://paste.openstack.org/show/490818/ | 23:19 |
| EmilienM | OK | 23:19 |
| *** adrian_otto has quit IRC | 23:20 | |
| pabelanger | thiagop: what does debug.log show? | 23:20 |
| *** samueldmq has joined #openstack-infra | 23:20 | |
| thiagop | pabelanger: the paste was of debug.log from nodepool | 23:20 |
| *** anteaya has joined #openstack-infra | 23:21 | |
| ianw | pabelanger: ok, so it boots, but has no network | 23:21 |
| *** adrian_otto has joined #openstack-infra | 23:21 | |
| *** zaro has joined #openstack-infra | 23:22 | |
| pabelanger | ianw: eep | 23:22 |
| fungi | okay, back now, lots of highlights. will address them in chronological order | 23:23 |
| pabelanger | thiagop: if nodepool build an image, it will then launch it which, your output reports | 23:23 |
| EmilienM | Paul, i checked at the patch, too big for my phone. Will look later tonight | 23:23 |
| pabelanger | thiagop: then, nodepool with SSh into the instance before jenkins, if that failed, it gets deleted and restart the process | 23:23 |
| pabelanger | EmilienM: Ya, big churn | 23:23 |
| ianw | pabelanger: alright, i'm going to try some mad hacking skills to see if i can get in | 23:24 |
| *** akshai has quit IRC | 23:24 | |
| pabelanger | ianw: hack the gibson | 23:24 |
| bkero | Careful in front of greghaynes | 23:25 |
| *** karlamrhein has quit IRC | 23:25 | |
| pabelanger | okay, back to debian jessie to see the progress | 23:26 |
| fungi | tonyb: nibalizer: awesome. as the commit message (and a number of code comments) implies, the goal is for anyone to be able to run that eventually, but for now because we're not yet running with a particular gerrit patch backported that script only works for gerrit admins. in just over 30 minutes it should be safe to fire it up and take it for a spin. i've done rigorous testing over recent months (as | 23:27 |
| fungi | recently as several times today too) and am quite cnofident it's in good working order | 23:27 |
| *** adrian_otto has quit IRC | 23:29 | |
| *** adrian_otto has joined #openstack-infra | 23:29 | |
| jhesketh | jeblair: ping | 23:29 |
| tonyb | fungi: I've only skimmed it but I don't see handling for the "Stable Branch Maintenance" project. | 23:30 |
| ianw | poll : what is worse -> the stupid f-ing rax java console app or systemd? | 23:31 |
| clarkb | heh I have actually gotten systemd to work | 23:33 |
| *** adrian_otto has quit IRC | 23:33 | |
| *** markmcd has joined #openstack-infra | 23:34 | |
| tonyb | fungi: which is probbaly moot given that there are 0 candidates | 23:35 |
| fungi | tonyb: see the comment block near the top | 23:35 |
| fungi | tonyb: it's handled "specially" for now | 23:35 |
| *** cloudtrainme has quit IRC | 23:35 | |
| *** mriedem1 has joined #openstack-infra | 23:35 | |
| fungi | by rerunning with a different output directory, turning off extra-atcs and adding a branch regex | 23:35 |
| *** SumitNaiksatam has quit IRC | 23:35 | |
| fungi | complicated, but i tried to document it as thoroughly as possible | 23:36 |
| *** achanda has quit IRC | 23:36 | |
| tonyb | fungi: Thanks! I'm not sure how I missed that. | 23:36 |
| *** tsg_ has quit IRC | 23:36 | |
| fungi | which is how i handled it with the predecessor script as well, but it could stand some better encoding | 23:36 |
| fungi | for next cycle i'd like to integrate that and also zanata querying, if possible by then | 23:36 |
| *** dizquierdo has quit IRC | 23:37 | |
| *** mriedem has quit IRC | 23:37 | |
| *** asettle has joined #openstack-infra | 23:37 | |
| fungi | because the stable branch electorate is so "special" i tested that again as recently as today | 23:37 |
| tonyb | fungi: so querying zanata for APC status rather then relying on extra-atcs? | 23:37 |
| fungi | yep | 23:38 |
| *** denisra has quit IRC | 23:38 | |
| fungi | as for stable branch, they are the second largest electorate (right after infra) | 23:38 |
| tonyb | fungi: yeah it's very special ;P I knew you had a script from the first special election | 23:38 |
| fungi | i was surprised initially, but then realized i shouldn't be | 23:38 |
| fungi | lots of people do backports | 23:38 |
| fungi | neutron has the third largest, then docs, then nova, then qa | 23:38 |
| tonyb | fungi: Yeah | 23:38 |
| fungi | tc electorate is well over 3k now | 23:39 |
| *** smarcet has joined #openstack-infra | 23:40 | |
| mgagne | fungi: I think the issue is really related to expiration of token. I see 4 images uploaded today. 1) 00:05:02 (success) 2) 00:32:03 (failed) 3) 00:32:54 (success) 4) 00:33:58 (failed). Failure occurred as followed 2) 01:11:55 4) 01:12:49. I suspect token used for 1) was also used for 2-3-4. | 23:40 |
| fungi | other things which we could improve with the script but which i'd want to put before the tc are: scraping co-authored-by and using git author (or committer?) instead of gerrit change owner | 23:40 |
| tonyb | fungi: wow | 23:40 |
| fungi | mgagne: makes sense. i wonder if any of this changes with the pending patches to switch to using more shade in nodepool | 23:41 |
| mgagne | what I don't fully understand is *when* is the token created. Obviously not before an operation. If we bump to 24h, we guaranty use that it won't happen again if image is uploaded at 23:45 | 23:41 |
| mgagne | what* | 23:41 |
| *** chlong has quit IRC | 23:41 | |
| mgagne | what guaranty us* | 23:41 |
| tonyb | fungi: mikal had a script to do the co-authored-by thing and output an extra-atcs hunk, which we failed to run | 23:41 |
| *** asettle has quit IRC | 23:42 | |
| tonyb | fungi: I'd think both of those changes would be good, but agree need TC buy-in | 23:42 |
| fungi | tonyb: yep, it's not hard, and the gerrit api makes the commit message available so the script could be tweaked to do that pretty easily | 23:42 |
| *** doug-fish has joined #openstack-infra | 23:43 | |
| fungi | on the other hand we've seen particularly shady stuff with one-line changes that have several co-authored-by lines, so i worry that makes this even easier to game | 23:43 |
| *** adrian_otto has joined #openstack-infra | 23:43 | |
| thiagop | pabelanger: I think the problem is with the image | 23:43 |
| fungi | mostly it seems to be around people trying to get free summit passes though, not extra votes in elections (yet anyway) | 23:43 |
| tonyb | fungi: I assume that the hardest part is mapping the email addresses when people change | 23:43 |
| * tonyb is surprised that people do that | 23:44 | |
| thiagop | pabelanger: It's kinda getting stuck at some point... I launched an instance with the same image by myself and it's stuck at the same point | 23:44 |
| fungi | tonyb: yes, spotting and merging duplicate accounts is complicated. but for summit passes because we do several batches being able to track what people's addresses change to so that we don't send them additional passes is hard | 23:44 |
| *** annegentle has joined #openstack-infra | 23:45 | |
| pabelanger | thiagop: could be, in that cause nodepool cannot ssh, and it likely restarts | 23:46 |
| fungi | for author and co-authored-by there is also the added complication of them not requiring a gerrit account, so it'll probably need an account lookup pass for those | 23:46 |
| pabelanger | thiagop: which cloud are you using? | 23:46 |
| thiagop | pabelanger: is there a timeout for this process? | 23:46 |
| thiagop | pabelanger: one of the clouds in our lab at UFCG, Brazil | 23:47 |
| tonyb | fungi: yeah I was thinkign about that side of things as well | 23:47 |
| pabelanger | thiagop: yes, there are some timeouts for nodepool. eg: boot-timeout and launch-timeout | 23:47 |
| pabelanger | boot it 1hr and launch is 1min | 23:48 |
| pabelanger | IIRC | 23:48 |
| *** eil397 has quit IRC | 23:49 | |
| thiagop | pabelanger: so I think the only alternative here is update the image with nodepool image-update and wait that the image turns out to be good? | 23:49 |
| *** acruz has quit IRC | 23:51 | |
| pabelanger | thiagop: well, if the image is uploaded to the cloud, you can try and manually launch it and debug | 23:51 |
| *** annegentle has quit IRC | 23:52 | |
| pabelanger | thiagop: usually issues are networking if the instance doesn't boot properly | 23:52 |
| pabelanger | thiagop: but I think I seen an IP address on your debug.log | 23:52 |
| *** gildub has joined #openstack-infra | 23:52 | |
| fungi | mgagne: so the timing would imply that we get the token immediately prior to the first upload and reuse it for the remaining ones | 23:53 |
| mgagne | fungi: that's my hypothesis | 23:53 |
| clarkb | we get a new token when we need a new one | 23:53 |
| clarkb | keystoneauth does it all for us | 23:53 |
| *** piet has joined #openstack-infra | 23:53 | |
| mgagne | fungi: I'm trying to see if something logs the actual token so I can get the creation or expiration time | 23:53 |
| *** gyee has quit IRC | 23:53 | |
| thiagop | pabelanger: I think I'm too tired to make sense of these logs today... :/ | 23:53 |
| *** salv-orl_ has joined #openstack-infra | 23:54 | |
| mgagne | clarkb: what if it expires 30s later? glance needs the token to still be valid after the upload is completed | 23:54 |
| pabelanger | thiagop: fresh eyes always help | 23:55 |
| thiagop | pabelanger: so I'll call it a day, thank you for the help | 23:56 |
| clarkb | mgagne: I am not sure, I know it renews the token if it expires soon but don't know what soon is | 23:56 |
| mgagne | we managed to replicate a similar situation by uploading an image that would take more than 1h to complete. result: upload fails with 500 error, image stays in saving state | 23:56 |
| clarkb | mgagne: this sounds like a flaw in openstack | 23:56 |
| clarkb | if the token is valid on the initial request it should continue to be valid | 23:56 |
| clarkb | through the entirety of htat request | 23:56 |
| mgagne | clarkb: it has already been taken care of in mitaka by using trusts. we don't run mitaka | 23:56 |
| mgagne | clarkb: not if the service uses the user token to contact another service after the upload | 23:56 |
| clarkb | right that sounds like a glance bug | 23:57 |
| *** salv-orlando has quit IRC | 23:57 | |
| clarkb | I mena we can work around it if we need to, but this is something that the apis should handle imo | 23:57 |
| mgagne | clarkb: but there is little we can do on our side if caller still uses a token expiring in 30m when upload takes 40m to complete | 23:58 |
| *** ybathia has quit IRC | 23:58 | |
| clarkb | mgagne: correct but glance shouldn't fail in that case because the token was valid when the request started | 23:58 |
| clarkb | the idea that the token has to valid after is silly beucase we can't change anything at that point | 23:58 |
| mgagne | I don't think we disagree here | 23:58 |
| mgagne | there is little I can do to fix that except telling consumers to renew their token if they are about to perform a long request/task | 23:59 |
| *** roxanagh_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
