Thursday, 2016-10-20

*** bobh has quit IRC00:09
*** spzala has joined #openstack-release00:13
*** spzala has quit IRC00:18
*** armax has joined #openstack-release00:25
*** spzala has joined #openstack-release00:26
*** spzala has quit IRC00:32
*** david-lyle_ has joined #openstack-release00:51
*** david-lyle has quit IRC00:54
*** spzala has joined #openstack-release00:59
*** armax has quit IRC01:06
*** armax has joined #openstack-release01:06
*** armax has quit IRC01:08
*** armax has joined #openstack-release01:10
openstackgerritMerged openstack/releases: make OSC team name consistent  https://review.openstack.org/38673301:10
*** armax has quit IRC01:14
*** hongbin has joined #openstack-release01:25
*** spzala has quit IRC01:26
*** sdake has quit IRC01:27
openstackgerritMerged openstack/releases: Release python-magnumclient 2.3.1 for Newton  https://review.openstack.org/38874901:27
*** sdake has joined #openstack-release01:29
*** mriedem_away is now known as mriedem01:33
*** sdake has quit IRC01:40
openstackgerritMerged openstack/releases: Release ironic-lib 2.3.0  https://review.openstack.org/38888301:57
*** armax has joined #openstack-release02:00
*** armax has quit IRC02:15
*** mriedem has quit IRC02:17
*** spzala has joined #openstack-release02:27
*** spzala has quit IRC02:43
*** sdake has joined #openstack-release03:14
*** david-lyle_ has quit IRC03:23
*** david-lyle has joined #openstack-release03:23
tonybttx, dims, dhellmann: I looks like the release-jobs for 388749 and 388883 both failed.  I *think* the correct action would be to manually run release_from_yaml locally but I'm not certain.  I don't like to ask others to clean up my mess but ... help!03:25
tonyblogs http://logs.openstack.org/98/98f29cf8d9131c90ec922fcab73a1f143abeded6/release-post/tag-releases/30a945b/console.html and http://logs.openstack.org/8f/8f9f8de5a6177eb07ba067e78fade6c9ccde5df1/release-post/tag-releases/1823078/console.html03:26
*** dave-mccowan has quit IRC03:27
*** hongbin has quit IRC04:03
*** sdake has quit IRC04:07
*** sdake has joined #openstack-release04:13
*** csomerville has quit IRC04:19
*** csomerville has joined #openstack-release04:19
*** cody-somerville has joined #openstack-release04:23
*** cody-somerville has quit IRC04:23
*** cody-somerville has joined #openstack-release04:23
openstackgerritSteve Martinelli proposed openstack/releases: release keystoneauth 2.15.0  https://review.openstack.org/38899104:24
*** csomerville has quit IRC04:25
*** janonymous has left #openstack-release04:26
*** sdake has quit IRC04:26
*** spzala has joined #openstack-release04:44
*** spzala has quit IRC04:48
*** jtomasek has quit IRC05:39
stevemartonyb: hmm, your last comment on ^ is it related to http://lists.openstack.org/pipermail/openstack-dev/2016-October/106130.html ?05:58
tonybstevemar: I don't think so  I checked the tricircle jobs they're failing in a different was as they're failign in the tag [hase where as mine aren't gettign there :(06:01
* stevemar shrugs at tonyb06:08
*** pcaruana has joined #openstack-release06:18
*** claudiub has joined #openstack-release07:16
*** spzala has joined #openstack-release07:29
*** spzala has quit IRC07:34
ttxtonyb: let me see07:50
tonybttx: Thanks07:50
ttxtonyb: Ideally we would re-trigger the job somehow otherwise we'll tag but not publish07:53
ttxbut then we could also do the bunch of tags and the publish will catchup at the next refresh07:54
*** sdake has joined #openstack-release07:54
ttxtonyb: I'll run release_from_yaml07:55
tonybttx: Thanks.07:55
*** sdake has quit IRC07:55
*** sdake has joined #openstack-release07:56
tonybttx: The failure looked like something was/is strange on the signing instance to me07:59
ttxtonyb: any idea why the jobs initially failed ? The logs are not exactly clear07:59
tonybttx: it looks to me like rm'ing jenkins-sudo failed and then the job aborted08:00
tonybttx: I haven't looked into any changes that may have caused that08:00
ttxtonyb: release queue is pretty ugly08:01
ttx(see zuul status)08:01
tonybttx: Yeah.  It *looks* like the sudo config on signing01 is wonky08:03
ttxwatching the release jobs i just triggered to check they are executing the important steps (tarball / upload / announce) properly08:04
tonybttx: Actuallt that's wrong.  the wheel builinf is on a differen slave :(08:04
*** jpich has joined #openstack-release08:04
ttxwe'll still need to refresh releases.o.o at some point08:04
tonybttx: okay but we can do that with a "noop" chnage that rebuilds and publishes the docs right?08:06
ttxyes08:06
* ttx looks for a potential candidate08:06
*** david-lyle_ has joined #openstack-release08:07
ttxoh well, that can wait until we can get more eyes on that tag job failure08:07
ttxhmm, jobs look stuck08:08
*** david-lyle has quit IRC08:09
*** jrist has quit IRC08:09
ttxon "sudo rm -f /etc/sudoers.d/jenkins-sudo" again08:09
*** jrist has joined #openstack-release08:10
ttxtonyb: ok, so I expect the tarball-signing jobs to fail as well, now. That means stuff won't get uploaded08:11
ttxwhich means we should probably not approve https://review.openstack.org/#/c/389052/ and bump the upper-constraints to a version that won't be available on pypi08:11
ttxthe check-uc-constraints job failed, so we are safe on that side08:12
tonybYeah the u-c bumps can't merge until the packages is installable so we're pretty safe on the requirements side08:13
ttxSo... summary is:08:13
ttx"sudo rm -f /etc/sudoers.d/jenkins-sudo" takes forever on the signing node08:14
ttxtriggering the job timeout after 35min08:14
tonybAh okay08:15
ttxthat results in tagging and signing jobs failing08:15
ttxWe have two projects I just tagged that won't get tarball-signed/uploaded properly:08:15
ttx ironic-lib 2.3.0 and python-magnumclient 2.3.108:16
ttxLet's wait for fungi / dhellmann before we make things any worse :)08:17
tonybokay and we have somethign else gping on in the wheel building  slaves, which also appears to be sudo releated08:17
tonybokay.08:17
tonybstevemar: pointed me at a thread on os-dev from joehuang which seems to be related (despite what I said earlier).  Do you wnat to reply to that with these details?08:18
tonybor should I?08:18
* ttx looks08:19
tonybhttp://lists.openstack.org/pipermail/openstack-dev/2016-October/106130.html in case that helps08:19
tonybit's very light on details but seems related based on your analysis08:20
ttxtonyb: yes it's same issue -- could you quickly post a "yes, problem under investigation, don't do anything" reply ?08:20
tonybyup np08:21
*** TravT has joined #openstack-release08:21
ttxyes, so python-oneviewclient and tricircle also failed their "release jobs" and are at the same stage as the two we manually tagged08:22
ttxi.e. tagged but tarball not signed nor uploaded08:22
tonyb:(08:24
ttxtonyb: btw you posted the scarab twice on the ml08:28
*** TravT has quit IRC08:28
r1chardj0n3sttx he was just so proud of it :-D08:28
ttxwrong attachment :)08:28
ttxr1chardj0n3s: I'll say, I was skeptical we could make them look like a family, but they do08:28
tonybttx: Oh did I? Oh the second email shoudln't have had an attachment08:28
tonybI'm keen to see them all side-by-side08:29
ttxthe video from Matt gives an idea08:29
r1chardj0n3sttx: I'm looking forward to seeing the draft for Horizon's Shiba Inu (aka Doge :-)08:29
tonybYeah08:30
ttxr1chardj0n3s: didn't you receive it ? Maybe Heidi did not update her PTl list08:30
tonybttx: Heidi said they only had about 50% from the artist08:31
ttxah ok08:32
r1chardj0n3sttx: yeah, I queried Heidi and she said it's still in drafting, no worries08:32
ttxStill I'm pretty sure she targeted old PTLs08:32
ttxlike Hongbin posted the Magnum logo08:32
r1chardj0n3srighto, it's a good thing Rob and I are still on speaking terms :-D08:34
*** spzala has joined #openstack-release08:43
*** electrofelix has joined #openstack-release08:45
*** spzala has quit IRC08:47
*** dtantsur|sick is now known as dtantsur08:56
robcresswell\o/08:57
*** Rockyg has quit IRC09:06
*** sdake has quit IRC09:11
*** ihrachys has joined #openstack-release09:19
*** spzala has joined #openstack-release09:41
*** spzala has quit IRC09:45
*** amotoki has joined #openstack-release09:58
openstackgerritMerged openstack-infra/release-tools: ensure version numbers are always strings  https://review.openstack.org/38881310:31
openstackgerritMerged openstack-infra/release-tools: ensure all proposed versions include major.minor.patch values  https://review.openstack.org/38881410:31
toabctlI have a question about https://review.openstack.org/#/c/388749/10:39
toabctlisn't that enough to get a new sdist on pypi? or do I need todo a release also for master?10:40
toabctlI wonder how I get this into the stable/newton upper-contraints.txt10:40
toabctltonyb, ^^ any idea?10:42
toabctlor dhellmann maybe? ^^10:44
*** amotoki has quit IRC10:46
dimsgood morning ttx tonyb11:02
*** Rockyg has joined #openstack-release11:03
dimstoabctl : there were job failures for ironic-lib and magnumclient releases from yesterday. we will probably need today to check on those jobs and re-trigger with infra folks' help11:04
*** lucas-sick is now known as lucasagomes11:10
toabctldims, thanks for the info11:12
*** claudiub|2 has joined #openstack-release11:30
*** claudiub has quit IRC11:33
*** weshay is now known as weshay_pto12:22
*** dansmith has quit IRC12:23
*** trown|outtypewww is now known as trown12:25
*** Jeffrey4l has quit IRC12:35
*** gordc has joined #openstack-release12:49
*** rhallisey has joined #openstack-release12:51
*** bobh has joined #openstack-release12:54
*** bobh has quit IRC13:00
stevemartonyb: woo hoo, i was right13:11
robcresswellstevemar: First sign of the apocalypse :p13:12
* robcresswell goes back to lurking13:12
stevemar:)13:12
robcresswellstevemar: Was your being correct related to the gate shenanigans going on right now?13:13
stevemarrobcresswell: related to the release failures13:14
robcresswellAh, yes, I saw Tonys email.13:14
robcresswellSomeone mentioned in infra that a tag failed for python-oneviewclient IIRC13:14
robcresswellNot sure if thats on the list13:14
robcresswell(as in, was tagged, didnt get uploaded)13:15
*** mriedem has joined #openstack-release13:20
*** sdake has joined #openstack-release13:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/releases: Release OpenStack-Ansible Mitaka/13.3.6  https://review.openstack.org/38919113:21
openstackgerritThomas Bechtold proposed openstack/releases: Release oslo.rootwrap 5.2.0  https://review.openstack.org/38919313:24
dhellmanndims, ttx: good morning13:29
* dhellmann catches up on scrollback13:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/releases: Release OpenStack-Ansible Liberty/12.2.6  https://review.openstack.org/38920213:29
*** sdake has quit IRC13:33
*** TravT has joined #openstack-release13:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/releases: Release OpenStack-Ansible Liberty/12.2.6  https://review.openstack.org/38920213:38
ttxdhellmann: I posted a summary around 8:13UTC13:46
ttxissue on signing nodes makes tagging and signing/uploading jobs fail13:47
dhellmannttx: ack. pabelanger is looking at it13:48
ttxThere is a bunch of things that people tagged that failed to upload as a result13:48
dhellmanndo we have a list of those all in one place, yet?13:48
ttxand the two that tonyb approved failed to tag too, I pushed the tag manually and then they failed signing/upload13:48
ttxthe release-job-failures ML has a list.13:49
ttxlet me etherpad it13:49
*** inc0 has joined #openstack-release13:50
dimshi ttx dhellmann am back from an errand13:51
*** sdague has joined #openstack-release13:51
dhellmannhi, dims13:53
ttxhttps://etherpad.openstack.org/p/6mZZeAigiR13:53
*** sjmc7 has joined #openstack-release13:54
*** TravT has quit IRC13:56
*** bobh has joined #openstack-release13:57
ttxdhellmann: do you have questions ?14:00
*** spzala has joined #openstack-release14:01
*** bobh has quit IRC14:02
*** sdague has quit IRC14:03
dhellmannttx: no, I think I understand the situation now. I have a couple of meetings, and I'll check in again with infra after those.14:05
*** sdague has joined #openstack-release14:10
fungittx: tonyb: sorry, workout ran long this morning. looking through the discussion now14:18
fungiahh, pabelanger is looking into it? switching to #-infra then14:20
*** lucasagomes is now known as lucas-hungry14:28
*** hongbin has joined #openstack-release14:35
*** Jeffrey4l has joined #openstack-release14:35
*** amotoki has joined #openstack-release14:45
*** xavierr has joined #openstack-release14:45
xavierrgood morning release team14:47
dimshi xavierr14:48
xavierrI have done a tag in openstack/python-oneviewclient project that should be upload to pypi right after the tag. however it was not done. may you help me?14:49
xavierrhttp://logs.openstack.org/c5/c5dc97fadf24ac6ddf7075f3dbc6456878f8ee56/release/14:50
xavierris possible to retrigger the job?14:50
dhellmannxavierr : yes, the infra team is still working on fixing the problem but we will retrigger all of the failed releases when that is done14:51
dhellmannxavierr : we have python-oneviewclient on our list https://etherpad.openstack.org/p/6mZZeAigiR14:51
xavierrdhellmann: good to know! :)14:52
xavierrgood job14:52
xavierrand thank you all14:52
*** spzala has quit IRC14:53
*** xavierr has left #openstack-release14:55
*** bobh has joined #openstack-release14:58
*** amotoki has quit IRC15:02
*** bobh has quit IRC15:03
*** amotoki has joined #openstack-release15:05
*** sdague has quit IRC15:06
*** sdague has joined #openstack-release15:10
dasmdhellmann: do we know, what caused failures?15:19
*** spzala has joined #openstack-release15:20
dhellmanndasm : not yet. we're working on it in #openstack-infra15:21
dasmdhellmann: ack, thanks. will follow on #-infra15:21
*** amotoki has quit IRC15:24
*** amotoki has joined #openstack-release15:26
*** lucas-hungry is now known as lucasagomes15:32
*** amotoki has quit IRC15:34
*** spzala has quit IRC15:37
*** spzala has joined #openstack-release15:46
*** spzala has quit IRC15:50
*** bobh has joined #openstack-release15:58
*** david-lyle_ is now known as david-lyle16:01
*** bobh has quit IRC16:03
*** jpich has quit IRC16:04
ttxdhellmann: I'm signing off soon, I'll let you guys handle the fallout16:04
dhellmannttx: ack16:04
*** amotoki has joined #openstack-release16:07
*** armax has joined #openstack-release16:10
*** openstackgerrit has quit IRC16:18
*** openstackgerrit has joined #openstack-release16:19
*** cody-somerville has quit IRC16:21
*** cody-somerville has joined #openstack-release16:21
*** cody-somerville has quit IRC16:21
*** cody-somerville has joined #openstack-release16:21
*** mriedem has quit IRC16:30
*** mriedem has joined #openstack-release16:30
*** dtantsur is now known as dtantsur|afk16:38
*** bobh has joined #openstack-release16:59
*** ihrachys has quit IRC17:08
*** inc0 has quit IRC17:10
*** inc0 has joined #openstack-release17:11
*** trown is now known as trown|lunch17:12
*** rtheis has joined #openstack-release17:46
*** onovy has quit IRC17:52
*** onovy has joined #openstack-release17:57
*** sjmc7 has quit IRC18:04
*** sjmc7 has joined #openstack-release18:05
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily.18:09
*** electrofelix has quit IRC18:11
*** trown|lunch is now known as trown18:13
*** dave-mccowan has joined #openstack-release18:18
openstackgerritMerged openstack/releases: add version number verification for fuel projects  https://review.openstack.org/38880818:20
openstackgerritMerged openstack/releases: prepare newton final releases for cycle-trailing projects  https://review.openstack.org/38879918:21
stevemarany chance to get https://review.openstack.org/#/c/388991/ released? or too late?18:34
*** rtheis has quit IRC18:36
odyssey4medhellmann maybe I'm just being paranoid but https://review.openstack.org/388799 merged 30 mins ago and I still don't see some of the announcements on http://lists.openstack.org/pipermail/openstack-announce/2016-October/date.html18:52
dhellmannodyssey4me : we're seeing lots of failures. I'm working on it.18:52
odyssey4meI'm specifically wondering about the tag release for 'openstack-ansible' itself.18:52
odyssey4meah ok, thanks - will leave you to it :)18:52
*** mriedem has quit IRC18:53
*** tobias_ has joined #openstack-release18:53
*** bobh has quit IRC18:56
*** bobh has joined #openstack-release18:57
*** lucasagomes is now known as lucas-afk18:58
*** tobias_ has quit IRC18:59
*** mriedem has joined #openstack-release18:59
dhellmannodyssey4me : it appears that all of the tags were actually pushed correctly, and only doc rebuilds and announcements failed for ansible repos18:59
dhellmannodyssey4me : see https://etherpad.openstack.org/p/6mZZeAigiR19:00
odyssey4methanks - appreciate that you're taking the time so close to the summit19:02
dhellmannodyssey4me : the announce email for openstack-ansible should be coming through now. it was put in the moderation queue because of the size19:03
odyssey4medhellmann heh, ok - that makes sense - I expect that the diff output is rather large19:07
dhellmannodyssey4me : at this point all of your stuff should be tagged and released. Given the light staffing in -infra, I'm not planning to requeue the doc build jobs. Landing a patch in your repo ought to do that anyway, and if it's critical we can discuss re-running them19:20
odyssey4methanks dhellmann - we have a few patches updating the docs theme going in anyway so I don't think that'll be an issue19:20
odyssey4meappreciate the support - see you next week!19:21
dhellmannodyssey4me : cool. and yes, I'm looking forward to catching up with everyone next week!19:21
openstackgerritMerged openstack/releases: ensure pike ptg reference has a unique target name  https://review.openstack.org/38875819:29
openstackgerritMerged openstack/releases: fix the tarball-base for kuryr-lib  https://review.openstack.org/38875919:29
dimsdhellmann : yay for the pause. sorry i haven't been able to get free today19:29
*** ihrachys has joined #openstack-release19:30
dhellmanndims : ack, np19:30
dhellmannthe infra team did most of the work, really19:30
dimsthanks fungi! (and the team!)19:31
dhellmanndims : I'm going to sign off and get ready for travel tomorrow. I'll be offline tomorrow.19:36
dimsack dhellmann see you in BCN. safe travels. i'll leave home on sunday afternoon19:36
dhellmanndims : ack, I'm leaving tomorrow so I can get in a little relaxation and site-seeing before meetings start tuesday19:37
dhellmanner, monday19:37
*** claudiub|2 has quit IRC20:16
dims:)20:16
*** ihrachys has quit IRC20:24
*** ihrachys has joined #openstack-release20:24
*** dave-mccowan has quit IRC20:28
*** Rockyg has quit IRC20:38
*** askb has joined #openstack-release20:47
openstackgerritJulien Danjou proposed openstack/releases: -Release ceilometerclient 2.7.0  https://review.openstack.org/38934520:50
openstackgerritJulien Danjou proposed openstack/releases: Release aodh 3.0.1 for Newton  https://review.openstack.org/38934620:52
*** dave-mccowan has joined #openstack-release20:54
openstackgerritRob Cresswell proposed openstack/releases: Tag Django OpenStack Auth 3.0.0  https://review.openstack.org/38935020:55
openstackgerritRob Cresswell proposed openstack/releases: Tag Horizon 10.0.1  https://review.openstack.org/38935220:59
*** inc0 has quit IRC21:04
*** dave-mcc_ has joined #openstack-release21:05
*** dave-mccowan has quit IRC21:08
*** trown is now known as trown|outtypewww21:11
*** cody-somerville has quit IRC21:25
*** csomerville has joined #openstack-release21:25
*** gordc has quit IRC21:28
*** cody-somerville has joined #openstack-release21:30
*** cody-somerville has quit IRC21:30
*** cody-somerville has joined #openstack-release21:30
*** csomerville has quit IRC21:32
*** dave-mcc_ has quit IRC21:33
*** Jeffrey4l has quit IRC21:34
*** inc0 has joined #openstack-release21:37
*** claudiub|2 has joined #openstack-release21:43
tonybWhat's up with the signing/wheel nodes and rebuilding?  Anything I need to watch/do?21:47
*** cody-somerville has quit IRC21:51
*** cody-somerville has joined #openstack-release21:51
tonybdims: ^^21:52
*** bobh has quit IRC22:02
*** csomerville has joined #openstack-release22:30
*** cody-somerville has quit IRC22:33
dimstonyb : see notes in scroll back https://etherpad.openstack.org/p/6mZZeAigiR23:07
dimstonyb : i believe infra folks have handled it already and Doug has declared a freeze till Nov 1st23:08
tonybdims: Thanks.  I saw dhellmann email after I asked ...23:08
dimstonyb : no worries...see you in BCN?23:09
*** ihrachys has quit IRC23:09
tonybdims: there are still job in the relese pipeline that have been there for 42 hours :(23:09
tonybdims: Yup.23:09
dimstonyb : ack on the jobs :)23:12
*** mriedem has quit IRC23:13
tonybdims: with no release and few requirements changes to review I might make forward progress ;P23:13
*** hongbin has quit IRC23:22
*** dave-mccowan has joined #openstack-release23:24
*** inc0 has quit IRC23:24
*** sdague has quit IRC23:25
*** Jeffrey4l has joined #openstack-release23:31
*** toabctl has quit IRC23:33
*** toabctl has joined #openstack-release23:44
*** rhallisey has quit IRC23:45
*** mriedem has joined #openstack-release23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!