Wednesday, 2017-05-03

*** thorst_afk has joined #openstack-infra00:00
fungiwith 'From .*' initial line, headers for From:, Date:, Subject:00:00
fungishow instead gives you commit id for the initial line, with a Date: and Author: header that align the data fields with one another, and indenting the commit message00:01
fungimost of the differences are in the way the commit message and headers are represented rather than the diff itself00:02
*** rlandy is now known as rlandy|afk00:02
fungii don't think git am actually cares about that version string after the signature separator00:03
*** dingyichen has joined #openstack-infra00:04
*** harlowja has quit IRC00:04
fungimtreinish: i'm getting lots of those panics from the subscribers actually. seeing if i can track down why00:05
jeblairclarkb: git-am on that worked! thanks00:07
ianwroot@ubuntu-xenial-osic-cloud1-s3500-8688880:~# ls00:07
ianwswapfile00:07
ianwis that supposed to be there?00:07
*** lamt has joined #openstack-infra00:08
clarkbianw: if d-g ran there yes00:08
clarkbianw: we make a swapfile if no swapfile device and no ephemeral drive to repartition00:08
ianwbut in /root?00:08
ianwmaybe it was always there and i just never noticed00:09
jeblairclarkb: can you point me at rst docs for '|' ?00:09
fungijeblair: http://docutils.sourceforge.net/docs/user/rst/quickref.html#substitution-references-and-definitions00:10
fungii happened to have it up in a tab from a few minutes ago00:11
clarkbjeblair: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#line-blocks00:11
jeblairfungi: different use of '|' i think00:11
fungioh, the _other_ use for |00:11
fungiyep, also useful ;)00:12
fungimtreinish: aha! i think i must be misunderstanding how mqtt-bench is supposed to work... if the clients number on the subscriber process exceeds the clients number on the publisher process00:12
*** zz_dimtruck is now known as dimtruck00:13
fungimtreinish: so you can't use this to test fewer subscribers than publishers. i guess our graph will be triangular?00:13
*** dave-mccowan has quit IRC00:13
fungimtreinish: that's what seems to have been causing the engrish panics00:14
fungino, wait, i'm getting it on mismatches the other way too00:16
*** lamt has quit IRC00:16
*** lamt has joined #openstack-infra00:16
*** jamesmca_ has joined #openstack-infra00:17
*** mat128 has joined #openstack-infra00:18
*** sbezverk_ has joined #openstack-infra00:19
fungiyeah, this is turning into an exercise in figuring out what the tool can handle rather than benchmarking anything00:21
*** ym200 has joined #openstack-infra00:21
*** sbezverk has quit IRC00:22
*** tonytan4ever has joined #openstack-infra00:22
*** jamesmca_ has quit IRC00:22
*** sbezverk has joined #openstack-infra00:23
fungigotta be running into some load/timing issue. if i try to set pubs and subs both to -clients=100 i get that panic on the subscriber too00:24
fungieven 50 pubs and 100 subs00:25
*** sbezverk_ has quit IRC00:26
*** ym200 has quit IRC00:26
*** felipemonteiro has joined #openstack-infra00:26
ianwhmm, journalctl -f has quite a lag between output and when it decides to show it to you00:28
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for UnicodeDecodeError with journal log handler bug 1580728  https://review.openstack.org/46193900:29
openstackbug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged] https://launchpad.net/bugs/158072800:29
*** neiljerram has quit IRC00:32
*** tphummel has quit IRC00:35
*** ijw has quit IRC00:35
*** ym200 has joined #openstack-infra00:36
*** sbezverk_ has joined #openstack-infra00:36
*** ijw has joined #openstack-infra00:36
*** ijw has quit IRC00:37
*** ijw has joined #openstack-infra00:37
*** sbezverk has quit IRC00:37
*** thorst_afk has quit IRC00:37
*** dhinesh has quit IRC00:39
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for UnicodeDecodeError with journal log handler bug 1580728  https://review.openstack.org/46193900:39
openstackbug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged] https://launchpad.net/bugs/158072800:39
mtreinishfungi: hmm, I wonder I wonder if we're hitting a ulimit00:40
*** ym200 has quit IRC00:40
*** mriedem has quit IRC00:40
mtreinishI know when I did my local testing before we set up the firehose back in Aug. I was hitting open fd errors with our for loop testing00:40
mtreinishI can't remember how many clients that was though00:40
mtreinishfungi: you can also turn on guaranteed delivery with qos=1 or qos=2 (on both the pub and sub)00:41
*** markvoelker has quit IRC00:41
mtreinishqos=1 is guaranteed delivery, and qos=2 is guaranteed delivery once00:41
*** ijw has quit IRC00:41
mtreinishobviously there is much more overhead with that too00:42
*** markvoelker has joined #openstack-infra00:42
mtreinishfungi: oh it might also be bandwidth: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=4952&rra_id=5&view_type=&graph_start=1493768988&graph_end=1493771861&graph_height=120&graph_width=500&title_font_size=1000:42
mtreinish400 mbps seems like a lot for the small vm we've got deployed00:43
*** thorst_afk has joined #openstack-infra00:44
fungioh, you know what, i forgot the default size on this is 1024 bytes per message00:45
*** funzo has joined #openstack-infra00:46
*** thorst_afk has quit IRC00:46
*** markvoelker has quit IRC00:46
*** gildub has quit IRC00:47
*** dave-mccowan has joined #openstack-infra00:47
funginope, dropping the -size to 100 and setting -intervaltime=10 on the pub and with 10 pubs i still get a panic at 100 subs00:47
*** gildub has joined #openstack-infra00:48
fungifeels a lot like we need to get just the right balance of timing and volume parameters to get any useful measurements00:50
*** funzo has quit IRC00:50
*** cuongnv has joined #openstack-infra00:51
mtreinishfungi: heh, script it with nested for loops for both params?00:51
mtreinishfind the ones taht don't panic on us00:51
*** mat128 has quit IRC00:51
fungiwell, there's a separate (related?) problem which is some combinations i'm trying cause the publisher to just wait indefinitely00:52
mtreinishlol00:52
fungier, the subscriber i mean00:52
mtreinishI really should just rewrite this in python so we can understand it better00:53
mtreinishlets see if I feel motivated enough after another beer00:53
fungii'm already trying that solution too, it's not really helping either (but at least it makes it more bearable!)00:54
mtreinishfungi: the beer, or writing your own benchmark tool? :)00:55
fungithe first part of the solution hasn't caused me to want to attempt the second00:56
*** yamamoto has joined #openstack-infra00:57
openstackgerritEmilien Macchi proposed openstack/os-client-config master: DO NOT MERGE - Revert "Remove the keystoneclient auth fallback"  https://review.openstack.org/46194500:57
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: DNM - deploy tripleo with all debug  https://review.openstack.org/45703700:57
*** Apoorva_ has joined #openstack-infra01:02
ianwpabelanger: if around ... if I say "LC_ALL" does that ring any recent bells about fiddling with it on images?01:03
*** Goneri has joined #openstack-infra01:03
funginot recent, but we've had to deal with setting default locales in the past01:04
*** yamamoto has quit IRC01:05
ianwyeah ... i'm wondering if something has change.  I *think* LC_ALL=C.UTF-8 now, which centos doesn't have01:05
*** vtapia has quit IRC01:05
openstackgerritdongwenjuan proposed openstack-infra/puppet-jenkins master: use 'openjdk-8-jre-headless' for ubuntu-16.04  https://review.openstack.org/46170801:05
*** Apoorva has quit IRC01:05
*** Apoorva_ has quit IRC01:07
*** tiswanso has joined #openstack-infra01:07
fungimtreinish: confidence instilling this is https://github.com/takanorig/mqtt-bench/issues/901:08
*** EricGonczer_ has joined #openstack-infra01:09
mtreinishfungi: heh, that would be likely01:10
mtreinishfungi: I was assuming the tls code from the go stdlib there was correct01:10
mtreinishbut I didn't actually check01:10
*** vtapia has joined #openstack-infra01:11
mtreinishfungi: anyway I started writing something similar in python (where I at least understand all the bits involved)01:11
mtreinishwe'll see if I can finish it in a reasonable amount of time01:11
fungithanks, my patience for trying to figure out the sweet spot in these parameters between what causes the sub process to panic or hang indefinitely has worn thin01:11
fungioh, even better, sometimes it hangs for a while longer than it should, and then throws that panic01:12
*** pvaneck has quit IRC01:12
fungianyway, i promised to review some other talks in progress for people so need to shift gears for a bit01:12
*** xinliang has quit IRC01:13
jeblairclarkb: in looking into your patch, i found some other things about line_block that we weren't handling; i think http://git.inaugust.com/cgit/presentty/commit/?id=308a06134d7749638c7ba3afcc4031f31ba09930 may be a more complete solution01:15
*** xinliang has joined #openstack-infra01:16
jeblairclarkb: also note that line blocks should have an empty line after them; otherwise the sphinx parser throws a warning.  so if you use them just to force vertical space, i think they should look like: http://paste.openstack.org/show/608638/01:18
*** puiterwijk has quit IRC01:19
*** puiterwijk has joined #openstack-infra01:20
*** puiterwijk has quit IRC01:21
clarkbjeblair: ah ok and get flowed text is doing ehat my change dud ya?01:22
*** puiterwijk has joined #openstack-infra01:22
*** ramishra has joined #openstack-infra01:25
*** yanyanhu has joined #openstack-infra01:25
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Add github reporter status_url config option  https://review.openstack.org/44979401:27
*** yolkfull__ has joined #openstack-infra01:29
jeblairclarkb: yep, it has a check for [] that returns ''01:29
*** felipemonteiro has quit IRC01:31
*** liujiong has joined #openstack-infra01:32
clarkbcool I'll give it a try in the moning01:33
jeblairclarkb: btw, if you're curious, 'python presentty/rst.py' will show the sphinx structure for an rst file, so you can see what's going on behind the scenes.01:35
jeblair(that's how i saw how the line_block events work)01:35
*** mriedem has joined #openstack-infra01:36
clarkboh neat01:37
clarkbI was relying in tracebacks and was abojt to pdb but figured it out01:37
clarkbthat looks handy01:37
*** gouthamr has quit IRC01:37
clarkbalso looks lioe slides are composed by piles and you just construxt them using a visitor pattern?01:41
*** nicolasbock has quit IRC01:41
clarkband a pile is a set of urwid rows01:45
*** felipemonteiro has joined #openstack-infra01:45
*** thorst_afk has joined #openstack-infra01:47
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Set LC_ALL in disk-image-create  https://review.openstack.org/46195001:49
ianwfungi / pabelanger: ^ rather unsatisfying but seems to work01:49
ianwcentos doesn't have a great way to strip locales, we regenerate the locale-archive.  i'm never sure that it's not already mapped, and if that causes confusion01:50
*** zhurong has joined #openstack-infra01:50
jeblairclarkb: yep, that's about it01:51
*** aleph1 has quit IRC01:51
*** thorst_afk has quit IRC01:52
*** nicolasbock has joined #openstack-infra01:55
*** yamahata has joined #openstack-infra01:58
*** jamielennox is now known as jamielennox|away02:03
*** mat128 has joined #openstack-infra02:04
*** zhurong has quit IRC02:07
mtreinishfungi: I was thinking about it while working on my benchmark script I bet the client counts have to be the same for the mqtt-bench02:13
mtreinisherr or pubs >= subs02:13
mtreinishbecause it's probably doing a topic per worker to avoid duplicate message counts02:13
mtreinish(which is something I'm not caring about)02:13
*** nicolasbock has quit IRC02:17
*** jamielennox|away is now known as jamielennox02:17
*** rlandy|afk is now known as rlandy02:20
mtreinishfungi: I think I've got the script written02:22
mtreinishfungi: let me just setup the repo and I'll push it up02:22
*** tonytan4ever has quit IRC02:22
*** ihrachys has quit IRC02:29
mtreinishfungi: https://github.com/mtreinish/pymqttbench02:30
mtreinishthere seems to be a bug where it hangs waiting on timing info with large client counts where it gets stuck waiting on some of the threads02:31
mtreinishnot to finish, but it's results from the queue I'm not sure what's causing it02:31
mtreinishbut for small counts it seems to work fine02:31
mtreinishbut it's definitely not perfect02:32
mtreinishbut hopefully better than the go one :)02:32
*** nicolasbock has joined #openstack-infra02:33
*** ijw has joined #openstack-infra02:33
mtreinishfungi: actually just found the bug, just sloppy on my part :)02:33
*** Goneri has quit IRC02:36
*** dave-mccowan has quit IRC02:36
*** ijw has quit IRC02:38
mtreinishfungi: we can add whatever stats and output formats you think will be useful pretty easily on this02:38
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Remove the patrole heat job  https://review.openstack.org/46196202:38
*** yamahata has quit IRC02:40
mtreinishfungi: if you're curious about the results so far, this is what it does: http://paste.openstack.org/show/608657/02:46
*** funzo has joined #openstack-infra02:46
openstackgerritRabi Mishra proposed openstack-infra/project-config master: Change identity-v3-only heat nv job  https://review.openstack.org/46196802:47
*** armax has quit IRC02:48
*** thorst_afk has joined #openstack-infra02:48
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Remove the patrole heat job  https://review.openstack.org/46196202:49
*** psachin has joined #openstack-infra02:51
*** funzo has quit IRC02:51
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Kuryr] Use gate hook in rally jobs  https://review.openstack.org/46197202:55
*** hongbin has quit IRC02:59
*** gcb has quit IRC02:59
cloudnull:q03:01
*** gildub has quit IRC03:02
*** armax has joined #openstack-infra03:05
openstackgerritRabi Mishra proposed openstack-infra/project-config master: Change identity-v3-only heat nv job  https://review.openstack.org/46196803:06
*** thorst_afk has quit IRC03:08
*** nicolasbock has quit IRC03:10
*** gcb has joined #openstack-infra03:13
*** EricGonczer_ has quit IRC03:18
*** armax has quit IRC03:19
*** winggundamth has joined #openstack-infra03:22
*** felipemonteiro has quit IRC03:24
*** mriedem has quit IRC03:25
*** tphummel has joined #openstack-infra03:33
*** armax has joined #openstack-infra03:35
*** zhurong has joined #openstack-infra03:37
*** ijw has joined #openstack-infra03:38
*** vikrant has joined #openstack-infra03:39
*** zhurong has quit IRC03:40
*** knangia has quit IRC03:41
*** tiswanso has quit IRC03:42
*** ijw has quit IRC03:42
*** masber has quit IRC03:46
*** links has joined #openstack-infra03:49
*** v3i0 has joined #openstack-infra03:54
*** v3i0 has left #openstack-infra03:54
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit  https://review.openstack.org/46198103:57
*** markvoelker has joined #openstack-infra04:00
*** aeng has quit IRC04:03
*** ym200 has joined #openstack-infra04:04
*** thorst_afk has joined #openstack-infra04:04
*** ym200 has quit IRC04:08
*** thorst_afk has quit IRC04:09
openstackgerritMerged openstack/diskimage-builder master: Set LC_ALL in disk-image-create  https://review.openstack.org/46195004:16
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit  https://review.openstack.org/46198104:16
*** salv-orlando has joined #openstack-infra04:18
*** aeng has joined #openstack-infra04:20
*** salv-orlando has quit IRC04:23
*** gyee has quit IRC04:23
*** chlong has joined #openstack-infra04:23
*** armax has quit IRC04:27
*** armax has joined #openstack-infra04:28
*** nmathew has joined #openstack-infra04:30
*** jamesmca_ has joined #openstack-infra04:31
*** krtaylor has joined #openstack-infra04:32
*** jamesmca_ has quit IRC04:35
*** ykarel has joined #openstack-infra04:36
*** dingyichen has quit IRC04:36
*** zhurong has joined #openstack-infra04:37
*** armax has quit IRC04:38
*** dingyichen has joined #openstack-infra04:38
*** yamahata has joined #openstack-infra04:41
*** gildub has joined #openstack-infra04:41
*** namnh has joined #openstack-infra04:47
*** jascott1 has quit IRC04:48
*** salv-orlando has joined #openstack-infra04:49
*** yamahata has quit IRC04:55
*** huanxie has quit IRC05:02
*** huanxie has joined #openstack-infra05:03
*** huanxie has quit IRC05:04
*** huanxie has joined #openstack-infra05:05
*** mat128 has quit IRC05:05
*** pgadiya has joined #openstack-infra05:05
*** aeng has quit IRC05:08
*** gcb has quit IRC05:10
*** Julien-zte has joined #openstack-infra05:14
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921505:18
*** dhajare has joined #openstack-infra05:19
*** erikmwil_ has joined #openstack-infra05:20
*** aeng has joined #openstack-infra05:21
*** ccamacho has left #openstack-infra05:22
*** erikmwilson has quit IRC05:23
*** sree has joined #openstack-infra05:25
*** sree has quit IRC05:26
*** sree has joined #openstack-infra05:27
*** gcb has joined #openstack-infra05:27
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: Add refactor of tree-like vs graph  https://review.openstack.org/46135305:27
*** cuongnv has quit IRC05:30
*** namnh_ has joined #openstack-infra05:32
*** kjackal_ has joined #openstack-infra05:32
*** Hal has joined #openstack-infra05:34
*** namnh has quit IRC05:35
*** bhavik1 has joined #openstack-infra05:37
*** cuongnv has joined #openstack-infra05:38
*** gouthamr has joined #openstack-infra05:44
*** winggundamth has quit IRC05:45
*** salv-orl_ has joined #openstack-infra05:46
*** ltomasbo|away is now known as ltomasbo05:47
*** gouthamr has quit IRC05:48
*** salv-orlando has quit IRC05:49
*** salv-orl_ has quit IRC05:50
*** masber has joined #openstack-infra05:58
openstackgerritMerged openstack-infra/project-config master: Disable EPEL after running Bindep for OSA jobs  https://review.openstack.org/46185106:03
*** thorst_afk has joined #openstack-infra06:06
*** Hal has quit IRC06:06
*** prateek has joined #openstack-infra06:06
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration  https://review.openstack.org/46186506:11
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job  https://review.openstack.org/46186406:11
*** thorst_afk has quit IRC06:11
*** andreas-f has quit IRC06:12
*** andreas_s has joined #openstack-infra06:13
*** markus_z has joined #openstack-infra06:19
*** jaosorior_away is now known as jaosorior06:19
*** yolanda has quit IRC06:20
*** larainema has quit IRC06:20
*** gongysh has joined #openstack-infra06:21
*** tphummel has quit IRC06:21
*** rwsu has quit IRC06:23
*** pgadiya has quit IRC06:25
*** pcaruana has joined #openstack-infra06:27
*** andreas-f has joined #openstack-infra06:34
*** rwsu has joined #openstack-infra06:36
*** tommylikehu has joined #openstack-infra06:44
*** tie has joined #openstack-infra06:45
*** florianf has joined #openstack-infra06:47
*** funzo has joined #openstack-infra06:48
*** jascott1 has joined #openstack-infra06:50
*** funzo has quit IRC06:52
*** jascott1 has quit IRC06:54
*** eranrom has joined #openstack-infra06:56
*** shardy has joined #openstack-infra07:00
*** martinkopec has joined #openstack-infra07:01
*** kjackal_ has quit IRC07:04
*** ccamacho has joined #openstack-infra07:04
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Don't get ports info from unavailable neutron service  https://review.openstack.org/46190307:04
*** aarefiev has joined #openstack-infra07:06
*** thorst_afk has joined #openstack-infra07:06
*** ociuhandu has quit IRC07:08
*** dizquierdo has joined #openstack-infra07:10
*** namnh has joined #openstack-infra07:10
*** thorst_afk has quit IRC07:11
*** bauwser is now known as bauzas07:12
*** namnh_ has quit IRC07:12
*** Hal has joined #openstack-infra07:16
*** dizquierdo has quit IRC07:17
*** dizquierdo has joined #openstack-infra07:17
openstackgerritOksana Voshchana proposed openstack-infra/devstack-gate master: WIP: Test SSL wsgi  https://review.openstack.org/41498607:18
*** aviau has quit IRC07:19
*** aviau has joined #openstack-infra07:19
*** Hal has quit IRC07:23
*** zhurong has quit IRC07:24
*** Hal has joined #openstack-infra07:24
*** ijw has joined #openstack-infra07:27
*** hashar has joined #openstack-infra07:28
*** psachin has quit IRC07:29
*** ssbarnea has joined #openstack-infra07:32
*** ijw has quit IRC07:32
*** jpena|off is now known as jpena07:33
*** ssbarnea has quit IRC07:34
*** ssbarnea has joined #openstack-infra07:35
*** Qiming has quit IRC07:36
*** gaurangt has joined #openstack-infra07:38
*** salv-orlando has joined #openstack-infra07:38
*** dtantsur|afk is now known as dtantsur07:40
*** Qiming has joined #openstack-infra07:42
*** zhurong has joined #openstack-infra07:43
*** adriant has quit IRC07:43
*** salv-orlando has quit IRC07:46
*** gouthamr has joined #openstack-infra07:51
*** oanson has quit IRC07:52
*** oanson has joined #openstack-infra07:53
*** gongysh has quit IRC07:55
*** dizquierdo has quit IRC07:56
*** gouthamr has quit IRC07:56
*** ralonsoh has joined #openstack-infra07:56
*** masber has quit IRC07:56
*** masber has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** rossella_s has joined #openstack-infra08:01
*** neiljerram has joined #openstack-infra08:04
*** ijw has joined #openstack-infra08:05
*** salv-orlando has joined #openstack-infra08:05
*** thorst_afk has joined #openstack-infra08:07
*** ijw has quit IRC08:09
*** alexchadin has joined #openstack-infra08:09
*** lucas-afk is now known as lucasagomes08:26
*** e0ne has joined #openstack-infra08:26
*** thorst_afk has quit IRC08:26
*** electrofelix has joined #openstack-infra08:28
openstackgerritYuval Brik proposed openstack-infra/project-config master: karbor: mark python-karborclient to clone from git  https://review.openstack.org/46203108:30
*** gildub has quit IRC08:32
*** markvoelker has quit IRC08:32
*** jamesmca_ has joined #openstack-infra08:32
*** ggnel_t has quit IRC08:34
*** dingyichen has quit IRC08:35
*** sshnaidm|pto is now known as sshnaidm08:36
*** jamesmca_ has quit IRC08:37
*** mikal has quit IRC08:39
*** derekh has joined #openstack-infra08:39
*** mikal has joined #openstack-infra08:41
*** matrohon has joined #openstack-infra08:44
*** zhurong has quit IRC08:47
*** isaacb has joined #openstack-infra08:48
*** jascott1 has joined #openstack-infra08:51
openstackgerritThomas Herve proposed openstack-infra/project-config master: Make Heat AMQP job voting  https://review.openstack.org/46204108:54
*** jascott1 has quit IRC08:56
*** salv-orlando has quit IRC08:56
*** namnh_ has joined #openstack-infra08:56
*** namnh has quit IRC08:59
*** namnh has joined #openstack-infra08:59
andymccrmorning infra - i'm running into releasenote gate failures for OSA. The issue was reno's with duplicate hashes - but I've since fixed that, although the releasenote gate is still failing. I'm not sure how i can fix that now. any help would be super appreciated :)09:01
*** namnh_ has quit IRC09:01
*** zhurong has joined #openstack-infra09:01
openstackgerritdongwenjuan proposed openstack-infra/puppet-jenkins master: use 'openjdk-8-jre-headless' for ubuntu-16.04  https://review.openstack.org/46170809:02
AJaegerandymccr: ask dhellmann for reno, probably best on #openstack-release09:05
*** neiljerram has quit IRC09:06
*** aviau has quit IRC09:06
*** aviau has joined #openstack-infra09:07
*** namnh_ has joined #openstack-infra09:07
*** yamamoto has joined #openstack-infra09:07
*** ggnel_t has joined #openstack-infra09:07
*** hieulq_ has joined #openstack-infra09:08
andymccrAJaeger: will do - thanks for the pointer!09:09
*** ggnel_t has quit IRC09:09
andymccrprobably wait a bit until its a more sensible time for the americas ;)09:09
*** namnh has quit IRC09:09
*** yamamoto has quit IRC09:11
*** pbourke has quit IRC09:11
*** hieulq_ has quit IRC09:12
*** hieulq_ has joined #openstack-infra09:12
*** pbourke has joined #openstack-infra09:13
*** namnh has joined #openstack-infra09:14
openstackgerritColleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial  https://review.openstack.org/46204809:15
*** namnh_ has quit IRC09:16
odyssey4meandymccr AJaeger I guess it'd be possible to temporarily disable the job, merge the fix, then enable it again?09:18
odyssey4memaybe use a filter for it on the repo in question09:18
andymccrodyssey4me: the fix is in09:19
andymccrlike the duplicates are gone09:20
andymccrand no releases went out with the duplicates in (the tox job runs fine if you just clone it)09:20
*** hieulq_ has quit IRC09:20
*** tosky has joined #openstack-infra09:21
*** yee379 has joined #openstack-infra09:23
*** yee37918 has quit IRC09:23
*** cuongnv_ has joined #openstack-infra09:23
odyssey4meargh, I suspect that this has broken the OSA role jobs https://github.com/openstack-infra/project-config/commit/44ca9dd5f38db60b16512c5dba187edf8e2edb0909:23
odyssey4meandymccr ^09:23
odyssey4mesee http://logs.openstack.org/38/461938/1/gate/gate-openstack-ansible-repo_build-ansible-func-ubuntu-xenial/d77a1e1/console.html09:23
*** cuongnv has quit IRC09:24
*** thorst_afk has joined #openstack-infra09:24
*** vipul- has joined #openstack-infra09:25
*** vipuls has quit IRC09:25
*** Jeffrey4l has quit IRC09:25
*** pots has quit IRC09:25
*** Jeffrey4l has joined #openstack-infra09:27
odyssey4meit's working for centos, but not for ubuntu it seems09:27
andymccrso the -e /usr/bin/yum isnt working like we expect then09:27
odyssey4meyup09:27
andymccroh well just delete all ubuntu jobs and move to full time centos7 problem solved09:28
odyssey4methe reason that's been put in is for https://github.com/openstack/openstack-ansible-tests/blob/master/bindep.txt#L52-L53 - I reckon that we just ditch those packages from bindep09:28
odyssey4meI suspect that we don't need them for SNI support09:28
*** thorst_afk has quit IRC09:28
odyssey4mecertainly for stuff we use to get ansible started - our role will enable epel and install the package later anyway09:29
andymccrit makes more sense to fix epel install up properly i think - then surely those would work?09:29
odyssey4mesure, although I don't really see why that would be broken09:29
*** adreznec has quit IRC09:30
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use routes for URL handling in webapp  https://review.openstack.org/46133109:30
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use tenant_name to look up project key  https://review.openstack.org/46133209:30
andymccrcould it be that the task exists 1 because the command fails?09:31
*** adreznec has joined #openstack-infra09:31
andymccrwhereas on centos it'll exit 009:31
odyssey4mehmm, perhaps - maybe the older method is a better method09:32
*** ijw has joined #openstack-infra09:32
odyssey4mebut it should have enabled/disabled09:32
andymccri'll push a patch in 2 secs09:32
*** markvoelker has joined #openstack-infra09:33
*** cuongnv_ is now known as cuongnv09:34
openstackgerritAndy McCrae proposed openstack-infra/project-config master: Fix conditional for EPEL Repo enable/disable for OSA  https://review.openstack.org/46205409:35
andymccrodyssey4me: ^ take a look09:35
odyssey4melooks fine to me09:36
*** ijw has quit IRC09:37
*** markvoelker has quit IRC09:38
*** Julien-zte has quit IRC09:39
*** dizquierdo has joined #openstack-infra09:39
AJaegerandymccr: are all your jobs broken? Then I'll +3...09:39
*** hieulq_ has joined #openstack-infra09:40
*** lamt has quit IRC09:40
*** zhurong has quit IRC09:41
andymccrAJaeger: yeah they are09:42
*** namnh_ has joined #openstack-infra09:46
*** alexchadin has quit IRC09:48
*** namnh has quit IRC09:49
AJaegerandymccr: +309:50
andymccrthanks a bunch AJaeger09:50
*** sdague has joined #openstack-infra09:55
*** tesseract has joined #openstack-infra09:56
*** ym200 has joined #openstack-infra09:58
*** ym200 has quit IRC09:58
openstackgerritAlfredo Moralejo proposed openstack-infra/tripleo-ci master: Add subpackages from openstack-tripleo-common from delorean-current  https://review.openstack.org/46206709:59
*** ym200 has joined #openstack-infra10:00
openstackgerritMerged openstack-infra/project-config master: Fix conditional for EPEL Repo enable/disable for OSA  https://review.openstack.org/46205410:00
*** hieulq_ has quit IRC10:03
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use routes for URL handling in webapp  https://review.openstack.org/46133110:04
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use tenant_name to look up project key  https://review.openstack.org/46133210:04
*** gcb has quit IRC10:04
*** salv-orlando has joined #openstack-infra10:06
*** liujiong has quit IRC10:11
*** zhurong has joined #openstack-infra10:18
*** oanson has quit IRC10:19
*** oanson has joined #openstack-infra10:21
*** ym200 has quit IRC10:21
*** ym200 has joined #openstack-infra10:22
*** cuongnv has quit IRC10:23
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic  https://review.openstack.org/46207210:23
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic  https://review.openstack.org/46207210:26
*** namnh has joined #openstack-infra10:27
*** nicolasbock has joined #openstack-infra10:28
*** namnh_ has quit IRC10:30
*** salv-orlando has quit IRC10:30
*** zhurong has quit IRC10:30
*** jkilpatr has quit IRC10:31
*** ym200 has quit IRC10:33
*** namnh_ has joined #openstack-infra10:33
*** markvoelker has joined #openstack-infra10:34
*** ym200 has joined #openstack-infra10:34
*** neiljerram has joined #openstack-infra10:36
*** namnh has quit IRC10:36
*** hieulq_ has joined #openstack-infra10:39
*** markvoelker has quit IRC10:40
*** namnh has joined #openstack-infra10:40
*** yanyanhu has quit IRC10:40
*** namnh_ has quit IRC10:42
*** namnh_ has joined #openstack-infra10:43
*** ym200 has quit IRC10:44
*** ym200 has joined #openstack-infra10:44
*** namnh has quit IRC10:46
*** funzo has joined #openstack-infra10:46
*** jkilpatr has joined #openstack-infra10:49
*** ym200 has quit IRC10:49
*** funzo has quit IRC10:51
*** bhavik1 has quit IRC10:52
*** jascott1 has joined #openstack-infra10:53
*** hieulq_ has quit IRC10:54
*** jascott1 has quit IRC10:57
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic  https://review.openstack.org/46207210:58
*** ijw has joined #openstack-infra10:59
openstackgerritTom Barron proposed openstack-infra/project-config master: Experimental py35 dsvm for ceph devstack plugin  https://review.openstack.org/45489011:00
*** yolkfull__ has quit IRC11:00
*** alexchadin has joined #openstack-infra11:00
*** sree has quit IRC11:02
*** sree has joined #openstack-infra11:02
*** ijw has quit IRC11:04
*** namnh_ has quit IRC11:05
*** sree has quit IRC11:07
*** sree has joined #openstack-infra11:08
*** darvon has quit IRC11:10
*** ijw has joined #openstack-infra11:10
*** darvon has joined #openstack-infra11:11
*** sree has quit IRC11:13
*** rhallisey has joined #openstack-infra11:14
*** alexchadin has quit IRC11:14
*** alexchadin has joined #openstack-infra11:14
*** ijw has quit IRC11:15
openstackgerritMerged openstack-infra/shade master: Don't get ports info from unavailable neutron service  https://review.openstack.org/46190311:20
*** dimtruck is now known as zz_dimtruck11:20
*** ijw has joined #openstack-infra11:21
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic  https://review.openstack.org/46207211:23
*** thorst_afk has joined #openstack-infra11:25
*** ijw has quit IRC11:25
*** lucasagomes is now known as lucas-hungry11:28
*** alexchadin has quit IRC11:29
*** thorst_afk has quit IRC11:31
*** ijw has joined #openstack-infra11:31
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in ports commands  https://review.openstack.org/46135711:32
cmurphywhat does POST_FAILURE mean again? is that something the patch could have broken or something wrong with the publisher?11:32
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in ports commands  https://review.openstack.org/46135711:34
*** alexchadin has joined #openstack-infra11:34
*** tosky has quit IRC11:34
*** ijw has quit IRC11:36
*** markvoelker has joined #openstack-infra11:36
*** nmathew has quit IRC11:39
openstackgerritAndrea Frittoli proposed openstack-infra/project-config master: Remove the tempest layer4 job  https://review.openstack.org/46186711:40
*** markvoelker has quit IRC11:40
*** ldnunes has joined #openstack-infra11:40
sdaguecmurphy: my understanding is it's nearly always the system breaking down, not the patch11:41
*** pradk has joined #openstack-infra11:43
*** pradk has quit IRC11:43
cmurphyhmm11:44
*** rfolco has joined #openstack-infra11:44
*** tosky has joined #openstack-infra11:45
*** gordc has joined #openstack-infra11:46
*** hashar is now known as hasharLunch11:48
*** roflik has joined #openstack-infra11:51
*** pcaruana has quit IRC11:53
*** thorst_afk has joined #openstack-infra11:55
*** srobert_ has joined #openstack-infra11:56
*** srobert has quit IRC11:56
AJaegercmurphy: documented in the infra-manual at https://docs.openstack.org/infra/manual/developers.html#automated-testing - basically POST_FAILURE means that we could not collect any log files. It's often a CI problem11:57
*** pcaruana has joined #openstack-infra11:57
AJaegercmurphy: but it could be also your patch breaking the system very badly, e.g. with losing network11:58
roflikis this the right place for questions about jenkins-job-builder?11:58
*** wolverineav has joined #openstack-infra11:58
*** links has quit IRC12:00
*** isaacb has quit IRC12:01
*** zhurong has joined #openstack-infra12:01
*** ijw has joined #openstack-infra12:02
*** isaacb has joined #openstack-infra12:03
cmurphyAJaeger: thanks, I was looking for that12:05
*** Qiming has quit IRC12:06
openstackgerritMonty Taylor proposed openstack/os-client-config master: WIP Braindump thoughts and a tenantive plan for v2  https://review.openstack.org/46209212:06
*** watersoul_ has quit IRC12:06
*** watersoul has joined #openstack-infra12:07
*** huanxie has quit IRC12:07
*** rlandy has joined #openstack-infra12:08
*** ijw has quit IRC12:08
openstackgerritMerged openstack-infra/shade master: Move REST error_messages to error_message argument  https://review.openstack.org/46129012:09
openstackgerritMerged openstack-infra/shade master: Replace neutronclient with REST API calls in quotas commands  https://review.openstack.org/46136212:09
*** dprince has joined #openstack-infra12:09
openstackgerritOpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements  https://review.openstack.org/46137512:11
openstackgerritSagi Shnaidman proposed openstack-infra/project-config master: Exclude undercloud containers from gates  https://review.openstack.org/46211212:17
*** hasharLunch is now known as hashar12:19
*** Qiming has joined #openstack-infra12:20
*** rhallisey has quit IRC12:22
*** rhallisey has joined #openstack-infra12:22
*** isaacb has quit IRC12:23
*** ijw has joined #openstack-infra12:23
EmilienMAJaeger: hey, good afternoon. Any chance to get https://review.openstack.org/#/c/462112 ? our gate is now blocked on this. https://review.openstack.org/#/c/446940/ merged without my +1 and the job was not ready12:24
*** askb has quit IRC12:24
*** srobert_ has quit IRC12:24
*** srobert has joined #openstack-infra12:25
*** ijw has quit IRC12:28
openstackgerritMerged openstack-infra/project-config master: Remove the tempest layer4 job  https://review.openstack.org/46186712:29
*** mat128 has joined #openstack-infra12:31
*** ijw has joined #openstack-infra12:33
*** nicolasbock has quit IRC12:34
*** alexchadin has quit IRC12:34
*** markvoelker has joined #openstack-infra12:37
*** sree has joined #openstack-infra12:37
*** ggnel_t has joined #openstack-infra12:38
*** jpena is now known as jpena|lunch12:40
*** baoli has joined #openstack-infra12:41
*** links has joined #openstack-infra12:41
*** markvoelker has quit IRC12:41
EmilienMAJaeger, fungi: thank you12:41
*** lucas-hungry is now known as lucasagomes12:42
openstackgerritMerged openstack-infra/project-config master: Exclude undercloud containers from gates  https://review.openstack.org/46211212:42
*** ijw has quit IRC12:42
*** dprince has quit IRC12:43
*** eharney has joined #openstack-infra12:44
*** nicolasbock has joined #openstack-infra12:47
openstackgerritMartin AndrĂ© proposed openstack-infra/project-config master: Add undercloud container to t-h-t check  https://review.openstack.org/46214112:48
*** felipemonteiro has joined #openstack-infra12:49
pabelangermorning12:49
*** mriedem has joined #openstack-infra12:51
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config master: Enable EPEL for openstack-ansible CentOS bindeps packages  https://review.openstack.org/45991312:54
*** gcb has joined #openstack-infra12:56
*** nmathew has joined #openstack-infra12:56
*** kgiusti has joined #openstack-infra12:57
*** vikrant has quit IRC12:58
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create openstack-zuul-jobs / openstack-zuul-roles projects  https://review.openstack.org/46189712:58
AJaegerfungi, the change above needs infra PTL +1 ^12:58
*** mdrabe has joined #openstack-infra12:59
AJaegerpabelanger: I think you have an unrelated change in it ^12:59
pabelangerAJaeger: Ah,yes12:59
pabelangerthanks12:59
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create openstack-zuul-jobs / openstack-zuul-roles projects  https://review.openstack.org/46189713:00
pabelangerforgot to git stash my current working directory13:00
*** masber has quit IRC13:01
pabelangeryonglihe: cmurphy: do you mind reviewing: https://review.openstack.org/#/q/topic:puppet-lint too? some fixes for puppet lint failures13:01
*** claudiub has joined #openstack-infra13:02
*** mpaolino has joined #openstack-infra13:03
pabelangermordred: any chance on vhd-util patch today or any objections if I work on it?13:03
*** nicolasbock has quit IRC13:05
*** gouthamr has joined #openstack-infra13:05
*** alexchadin has joined #openstack-infra13:06
*** nmathew has quit IRC13:06
*** felipemonteiro_ has joined #openstack-infra13:08
*** jamesdenton has joined #openstack-infra13:08
*** lamt has joined #openstack-infra13:08
*** felipemonteiro has quit IRC13:11
openstackgerritJordan Pittier proposed openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient"  https://review.openstack.org/46214413:11
*** eharney has quit IRC13:11
openstackgerritJordan Pittier proposed openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient"  https://review.openstack.org/46214413:11
*** tosky has quit IRC13:11
*** felipemonteiro__ has joined #openstack-infra13:13
*** felipemonteiro_ has quit IRC13:13
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add reprepro for docker  https://review.openstack.org/45330413:13
*** sree has quit IRC13:14
*** sree has joined #openstack-infra13:15
*** askb has joined #openstack-infra13:15
openstackgerritJordan Pittier proposed openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient"  https://review.openstack.org/46214413:16
*** jaypipes has joined #openstack-infra13:18
fungipabelanger: one thing which i'm still unclear on (and i didn't pick up from the meeting discussion unfortunately) is the reason for splitting untrusted shared jobs and untrusted shared roles into different repos13:19
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration  https://review.openstack.org/46186513:19
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job  https://review.openstack.org/46186413:19
fungipabelanger: i mean, rather than one shared repo to contain the untrusted jobs and untrusted roles together13:19
*** sree has quit IRC13:19
*** tiswanso has joined #openstack-infra13:20
fungiis it a design constraint or just an aesthetic/cosmetic thing?13:20
*** mat128 has quit IRC13:21
*** trown|outtypewww is now known as trown13:21
*** jamesmca_ has joined #openstack-infra13:21
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add reprepro for docker  https://review.openstack.org/45330413:21
*** isaacb has joined #openstack-infra13:21
*** nicolasbock has joined #openstack-infra13:22
*** sshnaidm is now known as sshnaidm|afk13:22
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration  https://review.openstack.org/46186513:23
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job  https://review.openstack.org/46186413:23
pabelangerfungi: I think design, if we get to a point where we want to publish openstack-zuul-roles for people (maybe 3rd party CI as a example) it would make sense to break them away from jobs (playbooks).  However, it could make job a little harder to write, since changes could me over 2 repos now13:24
pabelangerfungi: I initially was going to add roles into openstack-zuul-jobs first, then work to break out later. But opted to also create openstack-zuul-roles at the same time13:25
*** salv-orlando has joined #openstack-infra13:25
pabelangerHowever, I don't have a strong preference TBH.  Single repo maybe be fine for our use case too.13:26
*** eharney has joined #openstack-infra13:26
fungiokay, it just seems similar to (using our jjb configs as an analogy) putting the macros.yaml file in a different repo from the job templates and jobs13:26
*** tongli has joined #openstack-infra13:26
pabelangerYa, thats a good way to look at it13:26
fungithough considering that in-repo jobs for other projects may reuse those roles, maybe not dragging in the shared job definitions repo makes sense, i don't really know13:27
fungianyway, not an objection, just trying to understand the logic there13:28
*** zhurong has quit IRC13:28
pabelangerYa, I think there is a little bit of experimenting here too.  We may find it too much work with 2 repos and just decide to move back to a single13:28
*** chlong has quit IRC13:28
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration  https://review.openstack.org/46186513:28
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job  https://review.openstack.org/46186413:28
pabelangerfungi: do you mind +3 on https://review.openstack.org/#/q/topic:puppet-lint change, help get some of our puppet jobs back in working order13:29
*** liusheng has quit IRC13:32
*** liusheng has joined #openstack-infra13:33
*** ltomasbo is now known as ltomasbo|away13:33
*** arxcruz has quit IRC13:33
*** jamesmca_ has quit IRC13:34
*** arxcruz has joined #openstack-infra13:34
*** EricGonczer_ has joined #openstack-infra13:34
*** dtantsur is now known as dtantsur|brb13:35
openstackgerritMerged openstack-infra/system-config master: Add reprepro for docker  https://review.openstack.org/45330413:35
*** sbezverk_ is now known as sbezverk13:35
*** ltomasbo|away is now known as ltomasbo13:36
*** lamt has quit IRC13:37
*** gcb has quit IRC13:40
*** gcb has joined #openstack-infra13:40
*** jaosorior is now known as jaosorior_away13:41
*** jpena|lunch is now known as jpena13:41
*** xyang1 has joined #openstack-infra13:41
*** tosky has joined #openstack-infra13:42
*** gcb has quit IRC13:43
openstackgerritMerged openstack-infra/puppet-infracloud master: Fix puppet-lint job  https://review.openstack.org/46178513:44
*** gcb has joined #openstack-infra13:44
*** ggnel_t has quit IRC13:44
openstackgerritMerged openstack-infra/puppet-infracloud master: Add bindep.txt file  https://review.openstack.org/46157113:44
*** psachin has joined #openstack-infra13:45
openstackgerritMerged openstack-infra/puppet-germqtt master: Fix puppet-lint job  https://review.openstack.org/46178713:45
openstackgerritMerged openstack-infra/puppet-germqtt master: Add bindep.txt file  https://review.openstack.org/46156113:45
openstackgerritMerged openstack-infra/puppet-apps_site master: Fix puppet-lint job  https://review.openstack.org/46178813:46
openstackgerritMerged openstack-infra/puppet-apps_site master: Add bindep.txt file  https://review.openstack.org/46154113:46
*** mriedem has quit IRC13:48
*** dprince has joined #openstack-infra13:49
openstackgerritRosario Di Somma proposed openstack-infra/shade master: Use REST API for volume attach and volume backup calls  https://review.openstack.org/46178113:49
*** Julien-zte has joined #openstack-infra13:50
*** mriedem has joined #openstack-infra13:51
*** armax has joined #openstack-infra13:51
pabelangerfungi: mind +3 on https://review.openstack.org/#/c/461517 removes cache-bindep element and move pip install of bindep into infra-package-needs. Cosmetic change13:52
*** Goneri has joined #openstack-infra13:52
pabelangerhttps://review.openstack.org/#/c/461525/ too if you have the time13:53
*** felipemonteiro_ has joined #openstack-infra13:53
*** Swami has joined #openstack-infra13:54
*** jascott1 has joined #openstack-infra13:56
*** felipemonteiro__ has quit IRC13:57
*** markvoelker has joined #openstack-infra13:57
*** jamesmca_ has joined #openstack-infra13:57
*** links has quit IRC13:58
*** Goneri has quit IRC13:58
*** jascott1 has quit IRC14:00
openstackgerritMerged openstack-infra/puppet-ansible master: Fix beaker on xenial  https://review.openstack.org/46181114:00
*** zhurong has joined #openstack-infra14:02
*** srobert has quit IRC14:04
openstackgerritMerged openstack-infra/project-config master: Create openstack-zuul-jobs / openstack-zuul-roles projects  https://review.openstack.org/46189714:04
*** xarses has quit IRC14:08
*** pradk has joined #openstack-infra14:12
fungipabelanger: sorry, i must be going blind. approved!14:12
*** zz_dimtruck is now known as dimtruck14:15
fungipabelanger: i think i see a potential major simplification for 46152514:16
*** isaacb has quit IRC14:17
fungipip can already install from a git repo url, no need for git clone and cleanup14:17
*** funzo has joined #openstack-infra14:18
*** rbrndt has joined #openstack-infra14:18
pabelangerfungi: sure, will look shortly14:18
*** openstackgerrit has quit IRC14:18
*** isaacb has joined #openstack-infra14:18
*** rossella_s has quit IRC14:20
*** rossella_s has joined #openstack-infra14:21
*** funzo has quit IRC14:23
*** openstackgerrit has joined #openstack-infra14:23
openstackgerritMerged openstack-infra/project-config master: Remove cache-bindep element  https://review.openstack.org/46151714:23
*** mdrabe_ has joined #openstack-infra14:23
*** isaacb has quit IRC14:24
*** mdrabe has quit IRC14:24
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Clone zuul from git.openstack.org for zuul-env  https://review.openstack.org/46152514:25
*** dave-mccowan has joined #openstack-infra14:25
*** amotoki has joined #openstack-infra14:26
*** ihrachys has joined #openstack-infra14:26
*** jamesmca_ has quit IRC14:29
*** jamesmca_ has joined #openstack-infra14:30
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci master: Use mixed release for container upgrade  https://review.openstack.org/46006114:30
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci master: Use mixed release for container upgrade  https://review.openstack.org/46006114:31
*** srobert has joined #openstack-infra14:32
*** askb has quit IRC14:36
*** marst has quit IRC14:38
*** prateek has quit IRC14:39
*** mat128 has joined #openstack-infra14:39
dmsimardclarkb, fungi, jeblair: do you happen to know if it's possible to set up nodepool to use boot from volume ?14:40
*** mat128 has quit IRC14:40
*** lamt has joined #openstack-infra14:40
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual master: New subsection on voting third-party CI permission  https://review.openstack.org/46217614:41
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Fixes needed for reprepro docker mirror  https://review.openstack.org/46217814:42
*** jordanP has joined #openstack-infra14:44
fungidmsimard: maybe mordred's played with it? as long as shade supports it, probably not hard to implement but at the moment i'm pretty sure it expects boot from image instead14:44
dmsimardfungi: ack, thanks14:45
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual master: New subsection on voting third-party CI permission  https://review.openstack.org/46217614:47
AJaegerfungi, fixed your wording ^14:47
*** EricGonczer_ has quit IRC14:48
*** marst has joined #openstack-infra14:50
*** zhurong has quit IRC14:52
fungiAJaeger: thanks! i was probably not paying close attention to what i wrote there ;)14:52
*** salv-orl_ has joined #openstack-infra14:53
*** dtantsur|brb is now known as dtantsur14:56
*** salv-orlando has quit IRC14:56
*** xarses has joined #openstack-infra14:58
*** xarses_ has joined #openstack-infra14:58
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Fixed group event serialization issue  https://review.openstack.org/46218414:58
*** erlon has joined #openstack-infra14:58
*** rkukura has joined #openstack-infra14:59
*** annegentle has joined #openstack-infra14:59
*** andreww has joined #openstack-infra15:00
*** Goneri has joined #openstack-infra15:01
*** gcb has quit IRC15:02
*** xarses has quit IRC15:02
*** xarses has joined #openstack-infra15:03
*** gcb has joined #openstack-infra15:03
*** andreww has quit IRC15:05
*** dougwig has joined #openstack-infra15:05
*** andreww has joined #openstack-infra15:06
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Increase file permissions around generate keys  https://review.openstack.org/46121815:06
*** andreww has quit IRC15:07
clarkbdmsimard: you can attach metadata to images if that allows you to boot from volume. But otherwise I think it would require code changes15:07
*** andreww has joined #openstack-infra15:07
*** xarses has quit IRC15:07
clarkbapparently image metadata can be used to instruct generic volume attachments, but not sure about boot from volume15:07
*** Goneri has quit IRC15:10
*** gcb has quit IRC15:10
*** Goneri has joined #openstack-infra15:11
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job  https://review.openstack.org/46186415:12
openstackgerritMerged openstack-infra/system-config master: Fixes needed for reprepro docker mirror  https://review.openstack.org/46217815:12
*** tjones has joined #openstack-infra15:12
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration  https://review.openstack.org/46186515:13
*** jascott1 has joined #openstack-infra15:13
*** hongbin has joined #openstack-infra15:13
openstackgerritMerged openstack-infra/project-config master: Clone zuul from git.openstack.org for zuul-env  https://review.openstack.org/46152515:14
openstackgerritMerged openstack-infra/openstackid-resources master: Fixed group event serialization issue  https://review.openstack.org/46218415:15
*** chlong has joined #openstack-infra15:15
clarkbdevstack-gate reviewers https://review.openstack.org/#/c/461883/ review on that would be much appreciated as I'm hoping to have something in place to debug all these cinder fails before we all disappear. Hopefully gather enough info to fix either during summit or after15:16
*** rcarrillocruz has quit IRC15:16
*** gcb has joined #openstack-infra15:17
*** salv-orl_ has quit IRC15:18
*** rcarrillocruz has joined #openstack-infra15:18
*** eharney has quit IRC15:18
*** salv-orlando has joined #openstack-infra15:18
amrithhi infra folks, is keystone v2 auth disabled in the CI? something appears to have changed in the last day that has caused all trove CI jobs to fail with keystone auth v2 failures.15:22
sdagueclarkb: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Setting%20instance%20vm_state%20to%20ERROR%5C%22%20AND%20message%3A%5C%22_unplug_vifs%5C%22%20AND%20message%3A%5C%22UnicodeDecodeError%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A115:22
sdaguegrok is definitely not parsing things right15:22
amrithsdague ^^ you've made some changes to keystone settings in the CI in the past couple of weeks, do you have any guidance on things that may have changed in the past 24-36 hours in this regard?15:23
*** alexchadin has quit IRC15:24
sdagueamrith: yes, keystone v2 admin api is no longer running15:24
amrithoh?15:24
sdagueauth still is15:24
amrithwhen did this change go in, was it in the past 36 hours?15:24
sdagueI thought it was longer ago than that15:24
sdaguebut in the last week15:25
amrithyes15:25
amrithI saw your changes15:25
amriththey were 13 days back15:25
amriththe failure I'm seeing is in the past day15:25
amrithlike this15:25
amrithhttp://logs.openstack.org/90/461590/3/check/gate-trove-functional-dsvm-mysql-ubuntu-xenial/1795e58/console.html#_2017-05-02_11_10_19_12377515:25
sdaguein the last 36 hours is systemd15:25
amrithwell, systemd shouldn't have caused this15:25
amrithit is a definitive error saying that the server is temporarily unable to do stuff for me, so take an asprin and call in the morning15:25
*** martinkopec has quit IRC15:26
sdagueamrith: oh... you know what15:26
amriththe service itself is running15:26
amrithv3 is responding just fine15:26
sdagueI think one thing might have been missed15:26
amrithno, did I win the lottery?15:26
amrithagain?15:26
amrithcan I retire?15:26
amrithI'm guessing I didn't. :(15:27
*** aarefiev is now known as aarefiev_afk15:27
amrithwhat was it sdague15:28
amrithand was it something in the past 36 or so hours?15:28
clarkbsdague: I see the issue in multiline rule. patch in a sec15:28
amrithbecause this all worked on may 115:28
sdagueamrith: where are you getting identity_admin from ?15:28
openstackgerritRosario Di Somma proposed openstack-infra/shade master: Use REST API for volume attach and volume backup calls  https://review.openstack.org/46178115:29
amrithlmc15:29
sdagueso, we should have effectively scrubbed all of that15:30
amrithtrove_auth_url is set in trove's config files as http://<host>/identity_admin/v2.0/tokens15:30
amrithwe set it in the devstack plugin thusly: https://github.com/openstack/trove/blob/master/devstack/plugin.sh#L21915:31
sdaguehttp://logs.openstack.org/90/461590/3/check/gate-trove-functional-dsvm-mysql-ubuntu-xenial/1795e58/console.html#_2017-05-02_11_10_00_11218215:31
*** trown is now known as trown|brb15:31
amrithyes, that's right15:31
sdaguewhere is that test.conf built?15:31
amrithgood q, let me check15:32
amrithfor the tests you are looking at, the devstack plugin link I sent you is the thing that gets used15:32
pabelangerclarkb: jeblair: fungi: https://review.openstack.org/#/c/461905 should be ready now. Adds openstack-zuul-jobs / openstack-zuul-roles into zuulv3-dev.o.o.15:33
pabelangerI could rebase to see about removing -1 from zuul15:33
*** lamt has quit IRC15:33
sdagueamrith: ok, KEYSTONE_AUTH_URI should be with /identity not /identity_admin15:33
amrithsdague, that is set: https://github.com/openstack/trove/blob/master/devstack/plugin.sh#L19715:33
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add openstack-zuul-jobs / openstack-zuul-roles to zuul.yaml  https://review.openstack.org/46190515:33
*** eharney has joined #openstack-infra15:34
amrithsdague, did the admin service stop running in the past day or so?15:34
sdaguehttps://github.com/openstack-dev/devstack/blob/8b8441f3becbae2e704932569bff384dcc5c6713/lib/keystone#L116-L12215:34
sdagueamrith: yes15:34
amrithOK, thanks. then I'll find out where we get that KEYSTONE_AUTH_URI and change that and try again15:34
*** trown|brb is now known as trown15:35
sdagueamrith: https://review.openstack.org/#/c/458226/15:35
openstackgerritClark Boylan proposed openstack-infra/system-config master: Handle journald timestamps in multiline logstash  https://review.openstack.org/46219115:35
clarkbsdague: ^ that should fix it15:35
amrithsdague, the timing is about right15:35
amriththank you so much, will go try with that.15:35
sdagueamrith: so that should have landed all the required elements at once15:35
sdagueamrith: can you try a recheck on that patch?15:36
amrithI've pushed up recent patches that have failed with this15:36
sdaguebecause the real mystery is where it's getting the old KEYSTONE_AUTH_URI value15:36
clarkbpabelanger: and once that merges you'll have to restart zuulv3 right?15:36
amrithso maybe trove is doing something wrong (that would be a first so I shudder to suggest the idea)15:36
sdagueamrith: well, the real trick is figuring out where that reference to idenity_admin is coming from15:37
sdaguebecause it really should have been fully purged15:37
amrithi think it is in project-config in the job definition15:37
amrithlet me look15:37
pabelangerclarkb: yes, I had to manual copy file onto zuulv3.o.o, then reload zuul-scheduler15:37
pabelangerI see new private keys get created too!15:37
amrithnope, not there15:37
clarkbpabelanger: gotcha so its already in place then?15:38
jeblairclarkb, pabelanger: no restart should be required for 461905.  a reconfiguration or restart is (was?) required for the change that added them to zuul.yaml.15:38
pabelangerclarkb: Yup, if not zuul would -1 again I think15:38
jeblairclarkb, pabelanger: no restart should be required for 461905.  a reconfiguration or restart is (was?) required for the change that added them to main.yaml.  <-- correction15:38
clarkbI was going to say that is zuul.yaml :) thanks for clarifying15:38
jeblairpabelanger: right, if the necessary reconfiguration has happened, zuulv3 should +1 that change, otherwise -1 it.15:39
pabelangeragree, main.yaml was the reload15:39
jeblairand it will be effective as soon as it lands15:39
pabelangerjeblair: if the reason zuul didn't +1 because we are lacking a noop-job?15:39
jeblairpabelanger: no, zuul should report on that change, it's a change to a repo it's configured to act on15:40
*** Goneri has quit IRC15:40
*** lamt has joined #openstack-infra15:40
*** jamesmca_ has quit IRC15:41
pabelangerHmm, I don't think that happened15:41
clarkbwhere is zuul configured to act on project-config?15:41
jeblairpabelanger: oh, that's not true, we are not set up to report on project-config15:41
clarkbright that15:41
pabelangerOh, right15:41
*** ykarel has quit IRC15:41
*** Goneri has joined #openstack-infra15:41
jeblairneat.  :)15:41
jeblairit reports config errors, but not anything else15:42
*** Rockyg has joined #openstack-infra15:42
clarkbpabelanger: I'm behind on private keys, did we do things related to yesterday's stuff?15:42
*** jamesmca_ has joined #openstack-infra15:42
clarkbre pabelanger | I see new private keys get created too!15:42
jeblairi think that's zuul secrets private keys15:43
pabelangerclarkb: Oh, that was zuulv3 creating private keys for secrets, if our jobs will use them15:43
clarkbah15:43
*** felipemonteiro_ has quit IRC15:43
clarkbthanks15:43
*** ltomasbo is now known as ltomasbo|away15:43
amrithsdague, I'll see what we get invoked in the plugin (https://review.openstack.org/#/c/462165/2/devstack/plugin.sh)15:43
amrithsdague, thanks15:47
*** martinkopec has joined #openstack-infra15:47
sdagueyeh, it's definitely weird, we need to figure out where that is coming from15:47
sdaguebecause I can't see anything that would set it as such15:48
clarkbinfra-root can I also get review on https://review.openstack.org/#/c/462191/1 to fix logstash parsing issue sdague pointed out?15:49
pabelangerlooking15:49
pabelanger+215:50
*** felipemonteiro has joined #openstack-infra15:50
*** isaacb has joined #openstack-infra15:51
openstackgerritMerged openstack-infra/project-config master: Add openstack-zuul-jobs / openstack-zuul-roles to zuul.yaml  https://review.openstack.org/46190515:51
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Initial commit  https://review.openstack.org/46219515:51
AJaegerpabelanger: you should add some team members to https://review.openstack.org/#/admin/groups/1759,members for your change to merge ;)15:52
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Initial commit  https://review.openstack.org/46219515:53
*** mpaolino has quit IRC15:53
pabelangerAJaeger: Ha, ya guess we should15:53
*** felipemonteiro_ has joined #openstack-infra15:53
pabelangercool, zuul noop noop that change ^ :)15:54
*** Apoorva has joined #openstack-infra15:56
*** felipemonteiro has quit IRC15:57
*** tongli has quit IRC15:59
openstackgerritMerged openstack-infra/system-config master: Handle journald timestamps in multiline logstash  https://review.openstack.org/46219115:59
pabelangermordred: we now have docker mirror for packaging: http://mirror.dfw.rax.openstack.org/deb-docker/16:01
*** felipemonteiro_ has quit IRC16:02
*** knangia has joined #openstack-infra16:02
*** felipemonteiro_ has joined #openstack-infra16:02
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add tox-linters job  https://review.openstack.org/46220216:05
*** ccamacho has quit IRC16:07
*** rossella_s has quit IRC16:08
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Do not merge: Test ARA 0.13.0rc1  https://review.openstack.org/43996616:10
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Do not merge: Test ARA 0.13.0rc1  https://review.openstack.org/43996616:10
*** martinkopec has quit IRC16:12
*** rlandy is now known as rlandy|brb16:16
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Source ci mirrors file if it exists only  https://review.openstack.org/46002216:18
*** felipemonteiro__ has joined #openstack-infra16:18
*** matrohon has quit IRC16:20
*** jordanP has quit IRC16:21
*** felipemonteiro_ has quit IRC16:21
*** tphummel has joined #openstack-infra16:21
*** lucasagomes is now known as lucas-afk16:21
*** martinkopec has joined #openstack-infra16:24
*** pcaruana has quit IRC16:24
clarkbjeblair: pabelanger fungi thanks for th ereviews16:24
*** andreas_s has quit IRC16:25
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add tox-linters job  https://review.openstack.org/46220216:25
*** shardy has quit IRC16:28
*** martinkopec has quit IRC16:28
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/infra-specs master: Revise security spec to discuss SSH keys  https://review.openstack.org/46220716:29
*** erikmwil_ has quit IRC16:31
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Move playbook out of zuul  https://review.openstack.org/46221016:31
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Import playbooks from feature/zuulv3 development  https://review.openstack.org/46220216:32
*** vhosakot has joined #openstack-infra16:33
*** mhickey has joined #openstack-infra16:34
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml  https://review.openstack.org/46221216:35
*** isaacb_ has joined #openstack-infra16:35
*** Goneri has quit IRC16:36
mtreinishfungi: so a fun thing I was hitting with https://github.com/mtreinish/pymqttbench is that switching it to use threading instead of multiprocessing DOSes mosquitto (it also has much worse performance)16:37
mtreinishbut when I tested that the mosquitto memory usage just grows until it gets OOMKilled16:37
*** rlandy|brb is now known as rlandy16:37
mtreinishI haven't figured out why though16:37
*** trown is now known as trown|lunch16:38
*** tesseract has quit IRC16:38
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Import playbooks from feature/zuulv3 development  https://review.openstack.org/46220216:38
fungimtreinish: publisher socket not closing properly maybe?16:38
*** felipemonteiro__ has quit IRC16:38
fungior are you reusing a socket there?16:38
*** felipemonteiro__ has joined #openstack-infra16:39
*** isaacb has quit IRC16:39
*** isaacb_ has quit IRC16:39
mtreinishfungi: it should close the socket after the publish16:40
mtreinishit's baked into the paho library when use use the publish module16:40
*** dtantsur is now known as dtantsur|afk16:40
fungiahh16:41
mtreinishfungi: https://github.com/eclipse/paho.mqtt.python/blob/master/src/paho/mqtt/publish.py#L65-L7016:41
fungistrange!16:41
*** armax has quit IRC16:43
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Allow openstack-infra/zuul access our jobs  https://review.openstack.org/46221416:44
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml  https://review.openstack.org/46221216:44
mtreinishfungi: oh actually it doesn't the disconnect method doesn't explicitly call close() on the socket. It just sends the disconnect mqtt command16:44
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for rescue instance sec groups test fail bug 1688038  https://review.openstack.org/46221516:44
openstackbug 1688038 in OpenStack Compute (nova) "test_rescued_vm_add_remove_security_group fails with "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process"" [Undecided,New] https://launchpad.net/bugs/168803816:44
*** dkehn has quit IRC16:44
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Initial commit  https://review.openstack.org/46219516:45
*** gyee has joined #openstack-infra16:45
mtreinishwhich means the publisher client is relying on either the broker to close the socket or python to do it automatically16:45
fungimaybe that's lagging and causing a pileup?16:45
* fungi needs to disappear for a while. pre-travel errands16:45
fungiback in an hour or two16:46
*** harlowja has joined #openstack-infra16:48
*** markus_z has quit IRC16:48
*** dprince has quit IRC16:48
*** jordanP has joined #openstack-infra16:49
*** hashar has quit IRC16:51
*** armax has joined #openstack-infra16:51
*** EricGonczer_ has joined #openstack-infra16:52
openstackgerritKen Dreyer proposed openstack/python-jenkins master: doc: add get_job_config() example  https://review.openstack.org/46221916:56
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for rescue instance sec groups test fail bug 1688038  https://review.openstack.org/46221516:56
openstackbug 1688038 in OpenStack Compute (nova) "test_rescued_vm_add_remove_security_group fails with "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process"" [Undecided,New] https://launchpad.net/bugs/168803816:56
*** e0ne has quit IRC16:57
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml  https://review.openstack.org/46221216:58
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml  https://review.openstack.org/46221216:59
*** ralonsoh has quit IRC17:00
*** dizquierdo has quit IRC17:01
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml  https://review.openstack.org/46221217:01
*** otherwiseguy has joined #openstack-infra17:01
morganseeing a spike in 502s on gerrit and very slow respoonses when commenting.17:01
morgannot sure if its something on my end though17:02
*** clenimar has quit IRC17:02
clarkbmelody says GCing has picked up17:02
*** derekh has quit IRC17:02
*** EricGonczer_ has quit IRC17:04
*** dprince has joined #openstack-infra17:04
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Readded empty group events  https://review.openstack.org/46222617:04
*** psachin has quit IRC17:05
*** jpena is now known as jpena|off17:06
*** EricGonczer_ has joined #openstack-infra17:07
openstackgerritMerged openstack-infra/openstackid-resources master: Readded empty group events  https://review.openstack.org/46222617:07
*** EricGonczer_ has quit IRC17:10
*** Swami has quit IRC17:10
openstackgerritSean Dague proposed openstack-infra/devstack-gate master: Make syslog.txt be more constrained  https://review.openstack.org/46223117:18
sdagueclarkb: ^^^ solves the 12M syslog.txt that crashes browsers17:18
*** rkukura has quit IRC17:19
clarkbsdague: will review thanks17:20
*** tosky has quit IRC17:21
*** tie has quit IRC17:22
*** trown|lunch is now known as trown17:22
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file  https://review.openstack.org/46223217:22
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml  https://review.openstack.org/46221217:24
*** annegentle has quit IRC17:24
*** zul has quit IRC17:26
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add simple tox.ini  https://review.openstack.org/46223417:27
AJaegerpabelanger: your zuul-jobs changes need a tox.ini file, I've started one ^17:27
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap  https://review.openstack.org/45385117:27
*** Rockyg has quit IRC17:27
pabelangerAJaeger: Ya, we're just working on testing zuulv3 right now. But agree17:27
AJaegerpabelanger: your zuulv3 changes need that - the linters job will not run without...17:28
pabelangerAJaeger: agree17:28
AJaegerpabelanger: feel free to abandon them if these are too early...17:28
*** zul has joined #openstack-infra17:29
pabelangerAJaeger: sure, we're just working on getting zuul jobs working again right now17:29
pabelangerbut I can review shortly17:29
*** vhosakot has quit IRC17:29
mtreinishsdague: btw, does this address your comments from a couple weeks ago about documenting the firehose interfaces: https://docs.openstack.org/infra/system-config/firehose_schema.html ?17:30
sdaguemtreinish: looks mostly like it does17:31
mtreinishsdague: ok, cool17:32
*** mat128 has joined #openstack-infra17:33
*** thorst_afk is now known as thorst17:34
*** mhickey has quit IRC17:35
*** Hal has quit IRC17:36
*** mat128 has quit IRC17:36
*** rbrndt has quit IRC17:40
clarkbsdague: btw I'm working on the real fix for logstash rules. Turns out that indexer.conf.erb file isn't used anymore...17:41
*** mat128 has joined #openstack-infra17:41
pabelangerAJaeger: if you stacked your patches on 462202, you should see jobs running17:42
pabelangerAJaeger: other then noop17:42
openstackgerritClark Boylan proposed openstack-infra/logstash-filters master: Handle the new journald based logs  https://review.openstack.org/46223817:43
clarkbsdague: ^ can you look at that? thats where we actually source the rules from. Working on cleaning up the file that confused us now17:43
clarkbpabelanger: fungi ^ you reviewed the other change(s) too sorry for having to go back and forth on this17:44
*** mtreinish has quit IRC17:46
AJaegerpabelanger: I can do it - but guess we need to approve in different order. Let's see, worth the test17:47
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file  https://review.openstack.org/46223217:48
pabelangerAJaeger: right, we don't gate on zuulv3-dev.o.o today. So, we'll be able to merge the changes17:48
openstackgerritClark Boylan proposed openstack-infra/system-config master: Remove no longer used logstash config  https://review.openstack.org/46224017:49
clarkbpabelanger: fungi ^ thats cleanup to make this less confusing in the future17:49
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add simple tox.ini  https://review.openstack.org/46223417:49
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add untrusted-projects ansible test  https://review.openstack.org/46188117:50
clarkbjesusaur: ^ FYI the logstash things may interest you17:51
fungiclarkb: had the same thing happen to me with the logserver cleanup cronjob. it got halfway migrated to a different repo and never cleaned up17:51
fungiso i ended up making emergency changes to the copy we weren't using. man that was confusing17:52
clarkbya I was really confused why the configs weren't updating17:52
*** mat128 has quit IRC17:52
*** felipemonteiro_ has joined #openstack-infra17:53
clarkbsdague: we may also need to update the date parser at the end of that file but want to get all of this in before I keep trying to track down bugs17:53
openstackgerritMerged openstack-infra/logstash-filters master: Handle the new journald based logs  https://review.openstack.org/46223817:53
*** hongbin has quit IRC17:55
*** annegentle has joined #openstack-infra17:55
*** felipemonteiro__ has quit IRC17:56
*** baoli has quit IRC17:57
*** e0ne has joined #openstack-infra17:59
*** baoli has joined #openstack-infra17:59
*** annegent_ has joined #openstack-infra17:59
pabelangerAJaeger: http://zuulv3-dev.openstack.org/logs/b2b8dbe30fcf42f6a9e862630a65bd2c/console.log18:00
*** rkukura has joined #openstack-infra18:01
*** annegentle has quit IRC18:02
*** mtreinish has joined #openstack-infra18:05
*** jamesmca_ has quit IRC18:07
openstackgerritColleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial  https://review.openstack.org/46204818:08
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add simple tox.ini  https://review.openstack.org/46223418:11
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file  https://review.openstack.org/46223218:12
*** salv-orlando has quit IRC18:12
*** krtaylor has quit IRC18:12
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-roles master: Initial commit  https://review.openstack.org/46224418:14
*** otherwiseguy has quit IRC18:14
AJaegerpabelanger: https://review.openstack.org/#/c/462234 passes now - so, time to revert dependencies?18:15
pabelangerAJaeger: not just yet. We are still debugging some failure18:16
AJaegerpabelanger: ok. Feel free to do it yourselves if I'm not around...18:16
pabelangerack, we also need to move things into 46224418:17
*** jordanP has quit IRC18:17
AJaegerlet me copy over tox.ini, bindep, and requirements to zuul-roles18:17
openstackgerritColleen Murphy proposed openstack-infra/puppet-cgit master: Fix beaker  https://review.openstack.org/46181518:18
*** mriedem is now known as mriedem_away18:19
pabelangerAJaeger: ya, we might be able to land that sooner there.18:19
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file  https://review.openstack.org/46224718:20
AJaegerargh, wrong window ;(18:20
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-roles master: WIP: Initial import of feature/zuulv3 ansible roles  https://review.openstack.org/46224918:21
pabelangerAJaeger: if you want to add bindep.txt to 462244, I don't object18:23
AJaegerpabelanger: one by one...18:24
*** david-lyle has quit IRC18:24
*** dklyle has joined #openstack-infra18:24
*** jascott1 has quit IRC18:25
*** jascott1 has joined #openstack-infra18:26
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-roles master: Add tox.ini  https://review.openstack.org/46225318:26
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-roles master: Add empty bindep.txt  https://review.openstack.org/46225418:26
*** srobert has quit IRC18:27
*** rbrndt has joined #openstack-infra18:27
*** jascott1 has quit IRC18:27
clarkbok I think logstash rules should be happy18:27
clarkbI had to manually restart the daemons too as puppet doesn't do that automatically18:27
openstackgerritMerged openstack-infra/devstack-gate master: Collect lvmdump info in logs  https://review.openstack.org/46188318:30
openstackgerritColleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial  https://review.openstack.org/46204818:30
openstackgerritClark Boylan proposed openstack-infra/logstash-filters master: Handle journald timestamps with microseconds  https://review.openstack.org/46225718:34
clarkbconfirmed ^ is necessary after getting everything working18:34
clarkbpabelanger: fungi one more review please? :)18:34
openstackgerritColleen Murphy proposed openstack-infra/puppet-cgit master: Fix beaker  https://review.openstack.org/46181518:34
openstackgerritSean Dague proposed openstack-infra/project-config master: move neutron-full-multinode jobs to experimental  https://review.openstack.org/46225818:34
sdagueclarkb / AJaeger also ^^18:35
clarkbsdague: hrm I thought those jobs had runtime about the same as single node18:36
clarkbthe extra devstack run is added time but we get it back with tempest running quicker18:36
sdaguenot the "full" job18:36
sdaguethat's running all the scenarios18:36
sdaguein serial18:37
clarkboh18:37
clarkbfun18:37
*** xarses_ has quit IRC18:37
*** xarses_ has joined #openstack-infra18:37
*** mat128 has joined #openstack-infra18:38
sdagueit's the long pole in the tent. In an ideal world the -nv jobs would post in async so that long running -nv jobs don't hold up important fixes18:38
clarkbwe could probably add that pipeline to zuul18:39
clarkbsimilar to tripleo check18:39
*** annegent_ has quit IRC18:40
*** trown is now known as trown|brb18:40
*** annegentle has joined #openstack-infra18:41
*** chem is now known as chem_gone18:41
*** jamesmca_ has joined #openstack-infra18:42
openstackgerritEric Harney proposed openstack-infra/elastic-recheck master: Add Cinder query for bug 1580728  https://review.openstack.org/46226518:44
openstackbug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged] https://launchpad.net/bugs/158072818:44
*** trown|brb is now known as trown18:46
openstackgerritMerged openstack-infra/puppet-bandersnatch master: Fix beaker on xenial  https://review.openstack.org/46180518:46
sdaguesad slow gerrit is sad and slow :(18:47
clarkboh ya that was what I was going to do before I had to go eat breakfast18:47
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers  https://review.openstack.org/46226818:48
clarkbI'm going to restart gerrit due to the memory leak18:48
clarkbnow is it $status notice or alert that I want?18:48
clarkber #status18:49
jeblairclarkb: status notice18:50
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers  https://review.openstack.org/46226818:50
*** felipemonteiro_ has quit IRC18:50
clarkbalert is the one that updates channel topics right?18:50
*** mat128 has quit IRC18:51
*** felipemonteiro_ has joined #openstack-infra18:51
clarkb#status notice Gerrit on review.openstack.org is being restarted to accomodate a memory leak in Gerrit. Service should return shortly.18:51
openstackstatusclarkb: sending notice18:51
*** annegentle has quit IRC18:51
jeblairclarkb: right18:51
jeblairclarkb: think of it like "#status red alert" :)18:51
clarkbjeblair: there is an open source remake of those games18:51
clarkbI just discovered it last week18:52
-openstackstatus- NOTICE: Gerrit on review.openstack.org is being restarted to accomodate a memory leak in Gerrit. Service should return shortly.18:52
*** slagle has quit IRC18:53
*** krtaylor has joined #openstack-infra18:54
*** annegentle has joined #openstack-infra18:54
*** Sihan has joined #openstack-infra18:54
openstackstatusclarkb: finished sending notice18:54
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers  https://review.openstack.org/46226818:54
*** jamesmca_ has quit IRC18:54
*** electrofelix has quit IRC18:55
SihanHi Guys, in zuul, how could i config some projects vote and some project not vote?18:55
openstackgerritMerged openstack-infra/project-config master: Add specs and validation jobs for service-types-authority  https://review.openstack.org/46120818:55
*** felipemonteiro_ has quit IRC18:55
SihanNot sure, could I have two check pipeline? one for voting and one for not voting18:56
jeblairSihan: yes, that's exactly how18:56
jeblairSihan: just give them different names, and add the projects you want to vote to one, and the ones you don't want to vote to the other18:56
*** jordanP has joined #openstack-infra18:56
*** baoli has quit IRC18:57
jeblairSihan: for example, in openstack we have several pre-merge pipelines: check (votes), experimental (leaves a message but does not vote), and silent (does not vote or leave a message).18:57
*** baoli has joined #openstack-infra18:57
*** hongbin has joined #openstack-infra18:59
*** florianf has quit IRC19:00
*** gordc has quit IRC19:04
*** hongbin has quit IRC19:06
*** hongbin has joined #openstack-infra19:07
*** otherwiseguy has joined #openstack-infra19:09
*** baoli has quit IRC19:10
*** baoli has joined #openstack-infra19:11
openstackgerritMerged openstack-infra/logstash-filters master: Handle journald timestamps with microseconds  https://review.openstack.org/46225719:11
jeblairfungi: have you seen this?  telnet mapscii.me19:13
*** Sihan has quit IRC19:13
mtreinishjeblair: that is awesome19:14
fungiSorry, you reached a full server, please try again.19:14
*** Hal has joined #openstack-infra19:14
fungii guess they're getting /.ed19:14
fungigot through... either it's very slow or depends on codepoints/colors my xterm is not rendering19:15
mtreinishjeblair: heh, it's a node.js app: https://github.com/rastapasta/mapscii I find something funny about using js for a telnet service :)19:16
fungiyup. does leave you wondering19:16
jeblairmtreinish: yeah... apparently it's cause fetching and manipulating osm vector map tiles is easy in nodejs19:17
mtreinishhmm, I could see that19:18
fungioho, it's using braille codepoints which either aren't rendering visibly for me or i'm completely lacking a font covering them19:18
SotKthat is a wonderful creation19:22
mtreinishpabelanger, clarkb, jeblair: if you get a sec can you take a look at: https://review.openstack.org/449294 and https://review.openstack.org/460730 to add logstash worker events to the firehose19:25
*** hashar has joined #openstack-infra19:26
fungimtreinish: at this point that's basically identical to what you've gotten working on the subunit-worker systems, right?19:27
*** baoli has quit IRC19:27
fungiso in theory, no new bugs ;)19:27
mtreinishfungi: yeah, it should be19:27
*** baoli has joined #openstack-infra19:27
mtreinishit's not a straight copy and paste though, so I might have screwed something up in the process (that's probably pretty likely)19:27
mtreinishbut I think I fixed all the bugs we found doing the subunit-worker19:27
clarkbmtreinish: I will take a look in a bit, currently fixing things for journald formatted logs19:28
clarkbI think the change I am restarting logstash for will be the last19:29
clarkbalso I wish syslog/journald used iso8601 timestamps19:29
clarkbso much better19:29
mtreinishclarkb: I don't know how to do that in journald, but rsyslog at least lets you set the template format and you could use iso8601 timestamps in that19:31
mtreinishclarkb: but yeah, it would make life a lot easier19:31
clarkbmtreinish: I read the manpage and couldnt determine a way to do it19:31
fungijournalctl has --output=short-iso19:32
clarkbooh19:33
clarkbmaybe thats what we want19:33
mtreinishclarkb: http://www.rsyslog.com/doc/v8-stable/configuration/templates.html#standard-template-for-writing-to-files19:33
clarkbsdague: ^ thoughts on making that switch? we should be able to handle both at this point19:33
mtreinishthere is a dateFormat there19:33
clarkbsdague: I personally like iso8601 as its extremly clear19:34
mtreinishclarkb: also I think RSYSLOG_TraditionalFileFormat is iso8601 timestamps19:34
mtreinishbut it's been a long time since I've had to configure rsyslog19:34
*** mat128 has joined #openstack-infra19:34
mgagnepabelanger: we will be performing a maintenance on internap-mtl01 in 30 minutes where Neutron will be updated. Almost all Neutron and Nova API calls will stop working during the maintenance window.19:34
clarkbmgagne: that should be fine if you are ok with us making requests that will fail19:35
fungipopping out to grab late lunch/early dinner before i start packing for boston, but can review logging fixes when i get back in an hour or so19:35
mtreinishclarkb: but yeah doing it directly with journalctl would be better (per fungi's suggestion)19:35
mgagneclarkb: no problem with failures =)19:35
mtreinishfungi: are we out of time for doing more firehose benchmarking (now that we've goot a script that works)?19:36
clarkbmgagne: our garbage collector should swing around and clean up any resources that failed to delete after the fact19:36
clarkbmtreinish: are you really wanting a billion messages a day from logstash processing?19:40
mtreinishclarkb: it is a firehose19:40
mtreinishif it ends up being too much we can always turn it off19:40
*** salv-orlando has joined #openstack-infra19:41
clarkbmtreinish: reviewed the puppet change19:46
clarkband the processor change19:48
mtreinishclarkb: do you want me to respin the puppet-log_processor patch to pass the cert file creation up a level to system-config?19:48
clarkbmtreinish: the problem is right now its being down by logstash::indexer where we are removing support for mqtt19:49
*** mdrabe_ is now known as mdrabe19:49
clarkbmtreinish: I'm fine if we want to push it down into log_processor instead or just set it in the system-config change logstash_worker.pp19:49
clarkbmtreinish: its really a feature of the config is system-config so may be best to set it there?19:49
*** mriedem_away is now known as mriedem19:49
mtreinishclarkb: ok, I can do that19:50
mtreinishI was conflating puppet-log_processor and logstash::indexer, but I can do the file creation in system-config and push a follow up change removing that option from logstash::indexer19:50
*** markvoelker has quit IRC19:51
clarkbcool19:52
*** mat128 has quit IRC19:55
*** gyee has quit IRC19:55
openstackgerritMatthew Treinish proposed openstack-infra/system-config master: Enable mqtt notifications from logstash gearman worker  https://review.openstack.org/46073019:57
mtreinishclarkb: ^^^ there's the system-config side19:58
*** markvoelker has joined #openstack-infra19:58
openstackgerritMatthew Treinish proposed openstack-infra/puppet-logstash master: Remove the mqtt options from logstash::indexer  https://review.openstack.org/46227920:00
mtreinishclarkb: ^^^ and the puppet-logstash20:00
*** jascott1 has joined #openstack-infra20:01
*** ijw has joined #openstack-infra20:10
openstackgerritMerged openstack-infra/tripleo-ci master: Add subpackages from openstack-tripleo-common from delorean-current  https://review.openstack.org/46206720:10
*** otherwiseguy has quit IRC20:17
*** dprince has quit IRC20:19
*** jordanP has quit IRC20:20
*** e0ne has quit IRC20:25
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add untrusted-projects ansible test  https://review.openstack.org/46188120:25
*** mat128 has joined #openstack-infra20:27
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Support heterogeneous OVB environments  https://review.openstack.org/45865120:30
*** tiswanso has quit IRC20:32
*** mat128 has quit IRC20:32
*** ramishra has quit IRC20:34
*** trown is now known as trown|outtypewww20:34
*** tiswanso has joined #openstack-infra20:36
*** krtaylor has quit IRC20:39
*** tiswanso has quit IRC20:40
*** dave-mccowan has quit IRC20:41
*** dhajare has quit IRC20:45
*** Goneri has joined #openstack-infra20:45
fungimtreinish: i can still run benchmarks and collect data if you want to build graphs20:48
*** dhinesh has joined #openstack-infra20:49
*** dimtruck is now known as zz_dimtruck20:49
mtreinishfungi: ok cool20:49
mtreinishfungi: yeah I think graphs would be good for the slides20:49
fungii'm actually not doing much else this week other than summit prep, packing, flying and trying to pitch in when stuff is horribly broken20:50
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Clean up stable jobs due to driverfixes filters  https://review.openstack.org/46228720:50
smcginnisclarkb: Hopefully that's right. ^^20:50
clarkbsmcginnis: thanks also caught one with the lvmdump http://logs.openstack.org/76/450276/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5f59209/logs/screen-c-vol.txt.gz?level=ERROR digging through that now20:52
mtreinishfungi: ok sure, I'm in a similar mode20:56
mtreinishexcept s/when stuff is horribly broken/I can't look at slides anymore/g20:56
*** salv-orl_ has joined #openstack-infra20:56
clarkbsmcginnis: in that example the cinder service thinks that the volume was deactivated properly. Then lvextend fails because lvextend says it wasn't20:57
fungimtreinish: heh, nice ;)20:57
smcginnisclarkb: I saw one of those. Looked like the retries on deactivate were fine, but something else failed with the extend.20:58
fungimtreinish: anyway, i'll give your pymqttbench thing a shot here in a bit. still have my test vm handy20:58
mtreinishfungi: ok cool20:58
clarkbsmcginnis: Snapshot origin volumes can be resized only while inactive so lvextend thinks its active not inactive20:58
clarkbsmcginnis: definitely something weird going on, especially since its lvextend and cinder both catching this situation20:58
mtreinishfungi: in about 20min I'm gonna step away and run some errands, but I should be back online after that20:59
fungimtreinish: take your time. it's python, so i'm sure i can figure it out if i need to tweak something20:59
smcginnisclarkb: Looks like the state did change to deactivated. But I wonder if there was some internal state that still wasn't complete.20:59
*** tiswanso has joined #openstack-infra20:59
*** salv-orlando has quit IRC20:59
clarkbsmcginnis: the lvmdump pv status shows it is read write visible the entire time21:00
clarkbsmcginnis: I'm guessing that at the very least an inactive pv shouldn't be writeable21:00
*** mdrabe has quit IRC21:00
smcginnisclarkb: Let me check what I get when I do that.21:00
*** Apoorva_ has joined #openstack-infra21:00
mtreinishfungi: heh, that's a big part of why I wrote a new one. The go was making it really annoying to debug and iterate on21:01
*** askb has joined #openstack-infra21:01
smcginnisclarkb: Oh wait.... pv status? The lv gets deactivated. The pv should stay rw.21:02
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add untrusted-projects ansible test  https://review.openstack.org/46188121:02
clarkbsmcginnis: sorry its the lv21:03
clarkbsmcginnis: its the section with the same name as the volume uuid21:03
smcginnisok21:03
*** Apoorva has quit IRC21:04
*** tiswanso has quit IRC21:04
*** zz_dimtruck is now known as dimtruck21:04
clarkbsmcginnis: if you grab the lvmdump file untarball it then grep you'll find it21:04
*** eharney has quit IRC21:05
*** Goneri has quit IRC21:06
clarkbsmcginnis: replied to project-config change, one small regex thing otherwise lgtm21:06
*** rfolco has quit IRC21:09
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Clean up stable jobs due to driverfixes filters  https://review.openstack.org/46228721:09
clarkbsmcginnis: ^ one last thing, have to balance parens21:11
clarkbsmcginnis: also double check em on that21:11
smcginnisclarkb: Dang it, I thought I checked that.21:11
*** ldnunes has quit IRC21:12
smcginnisclarkb: Wait, isn't it right?21:12
clarkbsmcginnis: let me look at it again21:12
*** tphummel has quit IRC21:12
*** hongbin has quit IRC21:12
clarkboh yup silly me21:12
clarkbthis is why I have people double check me :)21:12
smcginniswhew21:12
*** hongbin has joined #openstack-infra21:12
smcginnis;)21:13
*** thorst has quit IRC21:13
fungi2x +2. it's a thing (which i often ignore because we need more reviewers)21:13
*** dougwig has quit IRC21:15
clarkbfungi: can you review https://review.openstack.org/462287 will fix some stable branch job stuff21:15
smcginnisclarkb: Local test after deactivating an lv: status = ["READ", "WRITE", "VISIBLE"]21:16
smcginnisclarkb: Is that what you were seeing? Need to pull down that tgz yet.21:16
clarkbsmcginnis: yes thats what I see21:16
clarkbseems odd it would have status write if not active21:17
smcginnisyeah...21:17
*** chlong has quit IRC21:17
*** jsavak has joined #openstack-infra21:18
*** krtaylor has joined #openstack-infra21:19
*** gyee has joined #openstack-infra21:25
*** Sukhdev has joined #openstack-infra21:25
*** mat128 has joined #openstack-infra21:26
*** otherwiseguy has joined #openstack-infra21:28
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers  https://review.openstack.org/46226821:28
*** mat128 has quit IRC21:31
*** jascott1 has quit IRC21:32
openstackgerritColleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial  https://review.openstack.org/46204821:32
clarkbalso lvchange doesn't seem to trigger archival21:33
smcginnisReturn code 3 is EINVALID_CMD_LINE. Looking through the code now to see what conditions would get to that.21:33
*** esberglu has quit IRC21:34
clarkbsmcginnis: stderr says "Stderr: u"  Snapshot origin volumes can be resized only while inactive" so thats why I was going after that21:34
smcginnisclarkb: Oh right.21:35
*** rlandy is now known as rlandy|afk21:35
smcginnisclarkb: So somehow it doesn't think it has actually been deactivated, even though we return from checking that it's deactivated.21:35
openstackgerritMerged openstack-infra/project-config master: Clean up stable jobs due to driverfixes filters  https://review.openstack.org/46228721:35
clarkbya21:36
*** thorst has joined #openstack-infra21:38
fungiis that following an lvchange -a n?21:39
*** tphummel has joined #openstack-infra21:39
smcginnisfungi: Yup21:39
smcginnisfungi: We had a batch of issues with it timing out waiting for the volume to actually deactivate.21:40
fungithat i've experienced first-hand21:40
smcginnisfungi: Now it appears to comlpete, but the extend call fails saying it is not.21:40
clarkbfrom what I can tell cinder does the lvchange -a n, then checks status until it flips to inactive, then extends21:40
fungithat one is definitely new to me though21:40
clarkbin some cases the status check never shows it as inactive and that fails21:40
clarkbin others it says its good and then the extend fails on it21:40
clarkbbut guessing its same udnerlying issue in both situations21:41
smcginnisclarkb: See seem to be getting around timing out waiting now. But still wish I knew why it suddenly seems to take so much longer.21:41
fungimaybe we time out waiting for it to go inactive but move on anyway?21:41
clarkband this lvmdump isn't as hlpeful as I had hoped21:41
smcginnisfungi: That was my though, but at least in this case we do not exhaust our retries.21:41
fungibizarre21:42
smcginnisfungi: And we would get an error stating if that was the case.21:42
smcginnisclarkb: It's at least data.21:42
fungiwouldn't expect there's some other process in parallel switching it back to active behind its back21:42
smcginnisfungi: I sure hope not.21:42
fungialso, possible there's a kernel bug i guess21:43
fungi(where it could misreport volume state)21:43
*** thorst has quit IRC21:43
smcginnisclarkb: Didn't you say there was a kernel bump?21:43
fungiall of these possibilities are terrible21:43
clarkbsmcginnis: there was but it appears to have happened a day or two before we noticed the problem21:44
clarkbthats based on nodepool logs not ubuntus21:45
*** jamesmca_ has joined #openstack-infra21:46
cmurphythis puppet job I'm trying to fix https://review.openstack.org/#/c/462048/ consistently fails with POST_FAILURE, but I've run it locally and confirmed it doesn't take out the test node, so I'm really confused21:46
aengfedora openid is down?21:47
clarkbcmurphy: one trick is to recheck it then connect via telnet to the console log port and watch at what step it dies21:48
cmurphyclarkb: oh good idea21:49
*** jsavak has quit IRC21:49
smcginnisAbout how long after a project-config change merges until that gets picked up by zuul?21:51
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Report job shadow errors to users  https://review.openstack.org/46230021:51
clarkbsmcginnis: 15 minutes to 45 minutes depending where inthe ansible puppet run things are21:52
smcginnisclarkb: thanks21:52
clarkbsmcginnis: you'll see the zuul status page update its last reconfigured timestamp at the bottom of the page21:52
smcginnisclarkb: Oh right, forgot about that.21:52
aengsorry, wrong channel..lol21:53
*** jascott1 has joined #openstack-infra21:54
*** Julien-zte has quit IRC21:54
cmurphyclarkb: hmm it ends successfully after gzipping logs21:59
cmurphyand yet21:59
clarkbcmurphy: and then post failure?21:59
cmurphyyep21:59
clarkbso its likely related to copying the logs21:59
clarkbcmurphy: do you have the ip address you telnet'd to ? I can check zuul execturos22:00
cmurphyclarkb: 2001:4800:1ae1:18:f816:3eff:fef9:433 1988522:00
cmurphyclarkb: http://paste.openstack.org/show/608761/22:00
clarkbcmurphy: http://paste.openstack.org/show/608763/ its permissions22:04
clarkbI think tripleo ran into this too22:04
clarkbcmurphy: you need to make sure the jenkins user can read the logs source?22:04
*** jkilpatr has quit IRC22:05
*** ijw has quit IRC22:06
*** ijw has joined #openstack-infra22:06
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921522:06
cmurphyooh because the module is messing with the jenkins user22:07
cmurphythanks clarkb22:07
*** gouthamr has quit IRC22:07
*** wolverineav has quit IRC22:07
*** jamesdenton has quit IRC22:08
*** wolverineav has joined #openstack-infra22:08
*** baoli has quit IRC22:09
*** jamesmca_ has quit IRC22:09
*** Apoorva_ has quit IRC22:10
*** Apoorva has joined #openstack-infra22:10
*** ijw has quit IRC22:11
*** ijw has joined #openstack-infra22:11
*** marst has quit IRC22:12
*** jamesmca_ has joined #openstack-infra22:14
*** wolverineav has quit IRC22:15
clarkbjeblair: patch for | in presentty works great, thanks22:18
jeblairclarkb: yay!22:18
clarkbjeblair: ther eisn't a good way to pop out of a presentation then rsume at the slide you left off is there?22:21
*** knangia has quit IRC22:21
clarkbnot a big deal can skip ahead22:21
jeblairclarkb: no; i start up the presenters console and use it to jump22:22
clarkbjeblair: I think I can actually just switch terminals22:22
jeblairclarkb: oh, like for a demo? yeah that's what i usually do22:22
*** warpc_ has joined #openstack-infra22:23
clarkbyup22:23
*** cmurphy_ has joined #openstack-infra22:23
clarkband this looks like it will work great22:23
*** jamesmca_ has quit IRC22:23
*** Sukhdev has quit IRC22:23
*** jascott1_ has joined #openstack-infra22:23
*** ediardo_ has joined #openstack-infra22:23
*** mgkwill_ has joined #openstack-infra22:24
*** mnencia has quit IRC22:24
*** EricGonczer_ has joined #openstack-infra22:25
*** abadger2000 has joined #openstack-infra22:25
*** jgrassle1 has joined #openstack-infra22:25
*** mikal_ has joined #openstack-infra22:26
*** hashar_ has joined #openstack-infra22:26
*** mnencia has joined #openstack-infra22:26
*** SotK_ has joined #openstack-infra22:26
*** cmurphy has quit IRC22:26
*** cmurphy_ is now known as cmurphy22:26
*** apetrich_ has joined #openstack-infra22:27
*** alex_xu_ has joined #openstack-infra22:27
*** verdurin_ has joined #openstack-infra22:27
*** nibz has joined #openstack-infra22:27
*** mariojv_ has joined #openstack-infra22:27
*** rodrigod` has joined #openstack-infra22:27
*** slagle has joined #openstack-infra22:27
*** tdasilva- has joined #openstack-infra22:28
*** bexelbie- has joined #openstack-infra22:28
*** erlon has quit IRC22:28
*** alex_xu has quit IRC22:28
*** mgkwill has quit IRC22:28
*** ediardo has quit IRC22:28
*** bexelbie has quit IRC22:28
*** rodrigods has quit IRC22:28
*** jgrassler has quit IRC22:28
*** jascott1 has quit IRC22:28
*** mikal has quit IRC22:28
*** apetrich has quit IRC22:28
*** verdurin has quit IRC22:28
*** mariojv has quit IRC22:28
*** jesusaur has quit IRC22:28
*** SotK has quit IRC22:28
*** puiterwijk|cld has quit IRC22:28
*** warpc has quit IRC22:28
*** nibalizer has quit IRC22:28
*** hashar has quit IRC22:28
*** harlowja has quit IRC22:28
*** tdasilva has quit IRC22:28
*** xgerman has quit IRC22:28
*** abadger1999 has quit IRC22:28
*** d0ugal has quit IRC22:28
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Remove stale queries for invalid or fixed bugs  https://review.openstack.org/46230822:28
*** mgkwill_ is now known as mgkwill22:28
*** bexelbie- is now known as bexelbie22:29
*** warpc_ is now known as warpc22:29
*** jesusaur has joined #openstack-infra22:29
*** ediardo_ is now known as ediardo22:29
clarkbalso friendly reminder to double check you aren't vulnerable to amt mess before conferencing22:30
* clarkb is about to reboot as soon as packge updates are done t ofigure that out22:30
*** gouthamr has joined #openstack-infra22:30
*** nibz is now known as nibalizer22:31
*** salv-orl_ has quit IRC22:32
fungicmurphy: clarkb: if this permissions issue persists in zuul v3 i think we should probably take steps to blanket normalize permissions at the end of jobs or something22:36
*** jkilpatr has joined #openstack-infra22:39
* mtreinish has returned22:42
*** baoli has joined #openstack-infra22:43
*** baoli has quit IRC22:43
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: CI test - never merge  https://review.openstack.org/46100122:44
*** baoli has joined #openstack-infra22:44
*** xyang1 has quit IRC22:47
*** rbrndt has quit IRC22:51
openstackgerritColleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial  https://review.openstack.org/46204822:51
*** annegentle has quit IRC22:51
*** rbrndt has joined #openstack-infra22:51
*** rbrndt has quit IRC22:51
cmurphyfungi: I think in this case it was just the puppet-jenkins module changing the jenkins user's authorized_keys22:52
cmurphyit wouldn't generally be a problem for anything else22:52
*** d0ugal has joined #openstack-infra22:53
fungicmurphy: i would be inclined to agree _if_ it were the first time we've seen this22:53
fungibut there have been other jobs which made files unreadable by the user we're copying them as, resulting in "mysterious" POST_FAILURE job states22:53
fungii think it counts as a sharp edge we should file down (if it's still around come zuul v3)22:54
cmurphyfungi: it wasn't an unreadable file, the file's contents were literally set to the string 'sshkey'22:54
fungioh, hah. so it broke ssh access entirely?22:55
cmurphyyep22:55
fungithat's an ice cream of a different flavor entirely22:55
cmurphyi would hope no other project has a good reason to do something like that22:55
fungichanging my opinion, we might want some other means of signaling "zuul couldn't connect to the worker to retrieve the logs" distinct from other general failures at that stage22:56
openstackgerritMerged openstack-infra/elastic-recheck master: Remove stale queries for invalid or fixed bugs  https://review.openstack.org/46230822:58
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Kuryr] Make fullstack jobs voting  https://review.openstack.org/46191123:00
*** pradk has quit IRC23:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger  https://review.openstack.org/46117623:01
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor  https://review.openstack.org/46117723:01
*** sdague has quit IRC23:02
*** lamt has quit IRC23:02
*** lamt has joined #openstack-infra23:02
*** xarses has joined #openstack-infra23:03
*** xarses_ has quit IRC23:03
*** andreww has quit IRC23:03
*** csomerville has joined #openstack-infra23:04
*** hashar_ has quit IRC23:05
*** lamt has quit IRC23:05
*** lamt has joined #openstack-infra23:06
*** lamt has quit IRC23:07
*** hongbin has quit IRC23:08
*** cody-somerville has quit IRC23:09
*** larainema has joined #openstack-infra23:10
*** ym200 has joined #openstack-infra23:11
*** annegentle has joined #openstack-infra23:11
*** EricGonc_ has joined #openstack-infra23:11
*** EricGonczer_ has quit IRC23:11
*** annegentle has quit IRC23:12
*** EricGonc_ has quit IRC23:13
openstackgerritMerged openstack-infra/devstack-gate master: Only hardlink files from cache_dir if it exists  https://review.openstack.org/45976323:14
*** EricGonczer_ has joined #openstack-infra23:14
*** ramishra has joined #openstack-infra23:15
*** rkukura has quit IRC23:16
*** jamesdenton has joined #openstack-infra23:16
*** funzo has joined #openstack-infra23:21
*** dklyle has quit IRC23:23
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Remove timestamp in devstack plugin  https://review.openstack.org/46231923:23
*** ym200 has quit IRC23:23
*** rfolco has joined #openstack-infra23:24
*** rfolco has quit IRC23:24
*** jamesdenton has quit IRC23:25
*** gildub has joined #openstack-infra23:25
*** funzo has quit IRC23:26
*** jamesdenton has joined #openstack-infra23:26
*** benj_ has quit IRC23:28
*** harlowja has joined #openstack-infra23:31
*** benj_ has joined #openstack-infra23:31
*** lamt has joined #openstack-infra23:34
*** Apoorva_ has joined #openstack-infra23:37
*** Apoorva__ has joined #openstack-infra23:40
*** Apoorva_ has quit IRC23:40
*** Apoorva has quit IRC23:41
*** gcb has quit IRC23:42
*** ramishra has quit IRC23:46
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Turn of time stamps in dib log  https://review.openstack.org/46232123:52
*** ijw has quit IRC23:52
*** ijw has joined #openstack-infra23:52
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Report job shadow errors to users  https://review.openstack.org/46230023:52
*** lamt has quit IRC23:54
*** ijw has quit IRC23:55
*** lamt has joined #openstack-infra23:55
*** ijw has joined #openstack-infra23:57
*** ijw has joined #openstack-infra23:58
*** thorst has joined #openstack-infra23:58
*** thorst has quit IRC23:58
jeblairianw: do you have a link to a problem logfile?23:58
*** masber has joined #openstack-infra23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!