| *** piet has quit IRC | 00:01 | |
| pabelanger | Nice! | 00:03 |
|---|---|---|
| *** piet has joined #openstack-infra | 00:04 | |
| pabelanger | clarkb: was it you who mentioned starting dib uploads once an image has finished vs waiting until all images build | 00:06 |
| *** kzaitsev_mb has quit IRC | 00:06 | |
| *** thorst has quit IRC | 00:07 | |
| *** ddieterly[away] is now known as ddieterly | 00:07 | |
| clarkb | yes | 00:08 |
| *** rfolco has quit IRC | 00:08 | |
| pabelanger | clarkb: you have a patch up for that? | 00:08 |
| clarkb | right now the way it works is it blocks until all images complete their daily buildsthen it starts uploads | 00:08 |
| *** kengo_sakai has quit IRC | 00:08 | |
| *** mylu has quit IRC | 00:08 | |
| clarkb | pabelanger: no I have been swamped under all the bugs, havne't been able to work on anything recently | 00:08 |
| clarkb | disabling epel did not fix my issue | 00:08 |
| pabelanger | clarkb: ack | 00:08 |
| clarkb | I give up, someone else should make sure rpm-distro fixes unbound :) | 00:08 |
| *** ddieterly is now known as ddieterly[away] | 00:09 | |
| greghaynes | clarkb: Is this something I can repro with just dib elements or are the puppet element bits part of it too? | 00:09 |
| clarkb | greghaynes: I have only reprepro'd with puppet elements but I think it is failing in /tmp/in_target.d/install.d/50-download-pkgs wherever that comes from | 00:10 |
| *** mylu has joined #openstack-infra | 00:10 | |
| greghaynes | clarkb: hrm, not in dib | 00:10 |
| clarkb | http://paste.openstack.org/show/494719/ is the error output | 00:11 |
| greghaynes | clarkb: ah, can you grap the yum install command too | 00:12 |
| clarkb | ya | 00:12 |
| clarkb | http://paste.openstack.org/show/494720/ | 00:13 |
| *** asselin_ has joined #openstack-infra | 00:13 | |
| pabelanger | clarkb: also, when time is free. Updated: https://review.openstack.org/#/c/294339/ to be CLI | 00:13 |
| clarkb | I thik it might be part of the cache devstack stuff | 00:13 |
| pabelanger | which is for uploads | 00:13 |
| clarkb | so its listing all the things from devstack then trying to cache them | 00:13 |
| clarkb | and devstack is saying cache librados and friends | 00:13 |
| *** baoli has quit IRC | 00:13 | |
| clarkb | and we are breaking | 00:13 |
| clarkb | let me check devstack real quick for new chagnes that do that | 00:13 |
| greghaynes | clarkb: yea, to me that looks like it might just be broken packages | 00:13 |
| clarkb | I don't see anything in devstack's history that looks suspect | 00:14 |
| *** ddieterly[away] is now known as ddieterly | 00:14 | |
| greghaynes | clarkb: this is using centos-minimal? | 00:15 |
| clarkb | greghaynes: yes | 00:15 |
| *** thorst has joined #openstack-infra | 00:15 | |
| clarkb | centos-minimal vm growroot simple-init devuser infra-package-needs puppet node-devstack nodepool-base openstack-repos rpm-distro | 00:15 |
| clarkb | that is my element list | 00:15 |
| pabelanger | I am just testing centos-minimal vm growroot simple-init devuser rpm-distro ATM | 00:16 |
| *** vhosakot has joined #openstack-infra | 00:16 | |
| pabelanger | then, our elements | 00:16 |
| clarkb | pabelanger: that won't test unbound though | 00:16 |
| pabelanger | clarkb: true | 00:17 |
| pabelanger | just want to see setfiles run | 00:17 |
| pabelanger | then on to our stuff | 00:17 |
| pabelanger | slow interwebs connects at home | 00:17 |
| *** fernnest_ has quit IRC | 00:17 | |
| *** fernnest has quit IRC | 00:18 | |
| pabelanger | 2016-04-20 00:18:13,430 INFO nodepool.image.build.centos-7: + setfiles /etc/selinux/targeted/contexts/files/file_contexts / | 00:18 |
| clarkb | greghaynes: that yum line also has a comment in it that is broken | 00:18 |
| pabelanger | running now it looks | 00:18 |
| clarkb | greghaynes: something is really off here I tink | 00:18 |
| pabelanger | okay testing our elements | 00:18 |
| greghaynes | clarkb: huh, thats wierd | 00:20 |
| clarkb | greghaynes: its cache-devstack | 00:20 |
| *** _amrith_ is now known as amrith | 00:20 | |
| clarkb | so I think there are a few bugs here, 1) we should probably say missing is ok there like we do with apt | 00:20 |
| clarkb | because it is triyng to cache things from all the branches | 00:20 |
| clarkb | but all the branches may not work on the different releases | 00:20 |
| clarkb | and something is inserting comments into that which is wrong | 00:21 |
| *** jrmyjohn has joined #openstack-infra | 00:21 | |
| *** asselin_ has quit IRC | 00:21 | |
| greghaynes | clarkb: oh, so you think the compilation we do is creating a broken list of stuff to install | 00:21 |
| *** sdake has quit IRC | 00:21 | |
| clarkb | yes | 00:22 |
| greghaynes | clarkb: if you add a script to the build which cats /tmp/packages-to-install;bash it should be helpful to debug | 00:23 |
| greghaynes | or just add a bash and poke around there | 00:23 |
| clarkb | greghaynes: well thats not what causes the issue I am having | 00:23 |
| greghaynes | er, /tmp/pkgs-to-install | 00:23 |
| clarkb | greghaynes: because we saw the issue I was having with cache-bindep too | 00:24 |
| clarkb | I am just noticing how broken everything is in trying to figure this out | 00:24 |
| clarkb | files/rpms/general:redhat-rpm-config # missing dep for gcc hardening flags, see rhbz#1217376 is causing the commnt issue | 00:24 |
| *** piet has quit IRC | 00:24 | |
| greghaynes | clarkb: ok, so I added a dib script which did a yum install with that list you gave me (after rmoving the comment) and it works, so theres something else breaking things in the gbuild | 00:24 |
| clarkb | I will work on a patch for that because I understand this | 00:24 |
| *** fawadkhaliq has joined #openstack-infra | 00:25 | |
| clarkb | yup found that particular bug, patch in a few | 00:25 |
| *** sdake has joined #openstack-infra | 00:25 | |
| *** jrmyjohn has quit IRC | 00:25 | |
| greghaynes | clarkb: hah, you know, that probably 'just works' because of the yum behavior where if it cant find a package it silently succeeds | 00:26 |
| greghaynes | clarkb: that makes me think we shouldnt allow things with spaces to go in to that file | 00:26 |
| *** alex_xu has quit IRC | 00:26 | |
| pabelanger | clarkb: starting build now | 00:27 |
| *** alex_xu has joined #openstack-infra | 00:27 | |
| clarkb | greghaynes: ya thats why, but I don't undertsand the msising deps issue at all | 00:28 |
| *** pvaneck_ has quit IRC | 00:28 | |
| greghaynes | clarkb: I wonder if something is nuking the repos.d dir or otherwise borking it | 00:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Fix package and image list comment cleanup https://review.openstack.org/308093 | 00:29 |
| clarkb | greghaynes: pabelanger ^ that should fix the bug I do undertand | 00:29 |
| *** jaypipes has quit IRC | 00:30 | |
| *** madhu_ak has quit IRC | 00:31 | |
| pabelanger | clarkb: Ah, good to know | 00:31 |
| clarkb | now for the other thing it seems related to ceph which according to the error output is in epel | 00:32 |
| clarkb | which rules out the rdo repo causing problems | 00:33 |
| greghaynes | clarkb: Yea, I didnt have the rdo repo enabled | 00:33 |
| clarkb | and looking at the versions it is mad about those are el6 versions | 00:33 |
| greghaynes | clarkb: I just did a build with ./bin/disk-image-create centos-minimal vm growroot simple-init devuser + that yum install line and its fine | 00:33 |
| openstackgerrit | Merged openstack/requirements: Add xmltodict to global-requirements https://review.openstack.org/286316 | 00:33 |
| clarkb | I am running with DIB_RELEASE=7 set so why would it try to get el6 things? | 00:34 |
| greghaynes | so its something specific to !dib elements | 00:34 |
| greghaynes | hahaha | 00:34 |
| greghaynes | I have no idea but that is extremely suspicious | 00:34 |
| greghaynes | I wonder if your puppet or something is adding a epel6 | 00:34 |
| pabelanger | not that I've seen | 00:34 |
| pabelanger | last DIBs I did was about 2 weeks ago | 00:34 |
| pabelanger | should know more in 25mins here | 00:35 |
| ianw_ | clarkb: hmm, you probably want https://review.openstack.org/#/c/306213/ | 00:35 |
| clarkb | greghaynes: or dib is leaky and my build host has some state I don't want | 00:35 |
| greghaynes | clarkb: I am not sure what would get cached for that... | 00:35 |
| clarkb | greghaynes: I think ianw is on to something | 00:36 |
| clarkb | I am guessing that my use of the epel element leaked out somewhere? | 00:36 |
| *** pfallenop has joined #openstack-infra | 00:37 | |
| greghaynes | clarkb: epel is used by default now though | 00:37 |
| greghaynes | I thought | 00:37 |
| ianw_ | greghaynes: still in the works -> https://review.openstack.org/#/c/306220 | 00:38 |
| pabelanger | I didn't know that | 00:38 |
| greghaynes | ah | 00:38 |
| ianw_ | that change depends on the fixup though | 00:38 |
| *** mylu has quit IRC | 00:38 | |
| openstackgerrit | Nathan Reller proposed openstack/requirements: Upgrade PyKMIP to 0.5.0 https://review.openstack.org/308095 | 00:38 |
| *** sdake has quit IRC | 00:38 | |
| clarkb | greghaynes: I had manually added it in when things started failing | 00:38 |
| clarkb | I have since removed it but its still filing | 00:38 |
| clarkb | so guessing state leak somewhere | 00:38 |
| ianw_ | should we just push these through? what more review are they really going to get | 00:38 |
| pabelanger | ianw_: mind want to confirm with RDO folks, they might have comments | 00:38 |
| greghaynes | ok, then I am really concerned about our cache coherency | 00:38 |
| *** sdake has joined #openstack-infra | 00:38 | |
| pabelanger | ianw_: Last I talked with RDO, they see EPEL enabled by default as a bad thing | 00:39 |
| greghaynes | because clearly it is completely broken | 00:39 |
| ianw_ | pabelanger: hmm ,well our "base" defines packages that come from epel. so disabling epel just creates a problem somewhere else :) | 00:39 |
| clarkb | greghaynes: you might want to add epel in and find out locally | 00:39 |
| clarkb | greghaynes: because I honestly have no idea where to start looking | 00:39 |
| *** yamamot__ has joined #openstack-infra | 00:39 | |
| pabelanger | ianw_: agreed | 00:39 |
| greghaynes | clarkb: yea, so what is the ordering - you add epel, then remove it? | 00:39 |
| clarkb | greghaynes: I added epel and had these fails | 00:39 |
| ianw_ | pabelanger: but, clearly nobody is building centos-minimal without all the openstack-infra bits, because nobody noticed | 00:39 |
| clarkb | greghaynes: I removed epel continued to have thise fails | 00:40 |
| pabelanger | ianw_: I did :) But hacked around it | 00:40 |
| clarkb | greghaynes: I added epel because a couple weeks ago centos-minimal did not work if it was missing | 00:40 |
| pabelanger | ianw_: don't think I ever filled a bug | 00:40 |
| clarkb | greghaynes: I think the issue is when you then try and download packages in el6 which we do when trying to cache packages for bindep or devstack | 00:40 |
| clarkb | greghaynes: I think your reproduce steps are add epel, then add that yum install line to end of install.d in a dummy element | 00:40 |
| greghaynes | clarkb: ok, verified with epel I get that error, running without epel... | 00:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: run shell builders defined in JJB https://review.openstack.org/308099 | 00:44 |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 00:44 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Fix dependencies for openstack-repos https://review.openstack.org/308100 | 00:45 |
| ianw_ | ^ a little yak-shaving there | 00:45 |
| *** doug-fish has quit IRC | 00:46 | |
| *** maestro2 has quit IRC | 00:46 | |
| greghaynes | clarkb: hrm, I dont get the error after removing epel | 00:47 |
| *** alex_xu has quit IRC | 00:47 | |
| *** doug-fish has joined #openstack-infra | 00:49 | |
| *** alex_xu has joined #openstack-infra | 00:49 | |
| openstackgerrit | Nate Marsella proposed openstack-infra/elastic-recheck: Make the index format string configurable in the conf file. This allows query to work against indices not named the typical "logstash-%Y.%m.%d". https://review.openstack.org/306360 | 00:52 |
| jlvillal | tonyb: Regarding https://review.openstack.org/#/c/305528/ Just wondering if you had any concerns/comments about it. | 00:53 |
| clarkb | greghaynes: huh maybe rpm-distro depends on epel? | 00:53 |
| greghaynes | clarkb: dunno... | 00:53 |
| greghaynes | clarkb: can you check in your image if you have epel enabled? | 00:53 |
| greghaynes | clarkb: just drop a bash in there somewhere | 00:54 |
| jlvillal | tonyb: Luckily at the moment we appear to be at multiples of 4 for the number of meetings. So it isn't broken :) | 00:54 |
| greghaynes | clarkb: or even easier, run with break=after-error | 00:54 |
| *** mtanino_ has joined #openstack-infra | 00:54 | |
| clarkb | greghaynes: I closed laptop, couldnt keep generating heat | 00:55 |
| greghaynes | clarkb: haha | 00:55 |
| *** mtanino has quit IRC | 00:55 | |
| *** fawadkhaliq has quit IRC | 00:56 | |
| *** fawadkhaliq has joined #openstack-infra | 00:57 | |
| tonyb | jlvillal: nothing major I was going down a rat-hole which I've abandoned, so I'll probably +2 it today | 00:58 |
| *** tphummel has quit IRC | 00:59 | |
| *** mtanino_ has quit IRC | 00:59 | |
| *** ybathia has joined #openstack-infra | 00:59 | |
| *** kengo_sakai has joined #openstack-infra | 01:00 | |
| pabelanger | clarkb: will post the results in the morning. | 01:01 |
| pabelanger | I'm out for the night | 01:01 |
| *** dahoo has quit IRC | 01:02 | |
| *** zz_dimtruck is now known as dimtruck | 01:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:02 | |
| *** zhurong has joined #openstack-infra | 01:04 | |
| *** rossella_s has quit IRC | 01:04 | |
| *** rossella_s has joined #openstack-infra | 01:04 | |
| *** bpokorny has quit IRC | 01:07 | |
| *** ybathia has quit IRC | 01:08 | |
| *** amrith is now known as _amrith_ | 01:08 | |
| *** flepied has joined #openstack-infra | 01:08 | |
| *** ayoung has joined #openstack-infra | 01:09 | |
| *** Sukhdev has quit IRC | 01:11 | |
| *** _amrith_ is now known as amrith | 01:16 | |
| clarkb | rpm-distro doens't depend on epel | 01:18 |
| *** ddieterly has quit IRC | 01:18 | |
| clarkb | pabelanger: it does look like centos-minimal depends on rpm-distro already | 01:19 |
| clarkb | pabelanger: so this is even more confusing | 01:19 |
| clarkb | greghaynes: I definitely removed epel from my list and don't see anything else depending on epel | 01:21 |
| clarkb | greghaynes: so I do not know why my stuff continues to break without epel, but at least you can reproduce with epel | 01:21 |
| *** Sukhdev has joined #openstack-infra | 01:23 | |
| *** yamahata has quit IRC | 01:23 | |
| ianw_ | i just built a centos-minimal with everything git updated | 01:24 |
| *** kzaitsev_mb has quit IRC | 01:25 | |
| clarkb | ianw_: greghaynes I think that once https://review.openstack.org/#/c/306213/2 can get a +1 from jenkins it should be approved if that fixes epel for us | 01:25 |
| *** rhallisey has quit IRC | 01:26 | |
| clarkb | I do not expect rpm-distro to fix selinux for us because we already seem to have that element in the dep tree | 01:26 |
| clarkb | grepping for rpm-base in the centos-7 image build log returns nothing so maybe not | 01:27 |
| clarkb | I think it should be centos-minimal -> yum-minimal -> rpm-distro | 01:28 |
| clarkb | derp not rpm-base, rpm-distro does show up | 01:28 |
| *** zhurong has quit IRC | 01:29 | |
| clarkb | ya we run the selinux restore element | 01:29 |
| *** zhurong has joined #openstack-infra | 01:29 | |
| clarkb | the issue is the finalise for unbound happens after the selinux finalize | 01:31 |
| clarkb | greghaynes: ianw_ we run the unbound finalise under 99, because we don't want to break dns until everything else is done. The selinux finalise runs at 11 | 01:33 |
| clarkb | greghaynes: ianw_ should we maybe move the unbound to 90 and the selinux to 99? | 01:33 |
| clarkb | ideally the selinux fixing would happen dead last | 01:33 |
| clarkb | or as an alternative we could just rerun setfiles on the path that ew are changing for unbound | 01:34 |
| clarkb | maybe we should just start with that, then any change to dib will just work (tm) | 01:34 |
| sdake | how many vms are there available in the gate? | 01:34 |
| clarkb | sdake: right now just over 500, why? | 01:35 |
| sdake | just curious - 335 jobs in gerrit atm | 01:35 |
| sdake | some of our jobs have been runnig 5 hours or longner | 01:35 |
| *** mylu has joined #openstack-infra | 01:35 | |
| clarkb | sdake: http://status.openstack.org/zuul/ has a graph at the bottom of the page | 01:35 |
| clarkb | sdake: the problem is that we lost most of our capacity over the last couple days | 01:35 |
| clarkb | we were down to 2 rax regions and internap | 01:35 |
| sdake | cool i'll look at that, i am on zuuls page right atm | 01:35 |
| clarkb | so now we are playing catchup | 01:35 |
| sdake | what about bluebox? | 01:37 |
| prometheanfire | anyone have time to look at https://review.openstack.org/307493 ? | 01:37 |
| prometheanfire | bindep change | 01:37 |
| *** EricGonczer_ has joined #openstack-infra | 01:38 | |
| ianw_ | clarkb: yeah the selinux should probably be later | 01:38 |
| ianw_ | clarkb: in fact, it should probably be in here -> https://review.openstack.org/#/c/307588/ :) | 01:39 |
| *** apoorvad has joined #openstack-infra | 01:39 | |
| sdake | clarkb osic has a slew of machines, ought to reach out to them to see if they would mek them avaialble for gating | 01:40 |
| sdake | they tested kolla with 64 nodes, 1300 tb ram, 2600 cores | 01:40 |
| anteaya | apparently there is a docs toolchain infra session on the wednesday: Documentation: Docs Toolchain/Infra Session 16:30-17:10 in 87 | 01:40 |
| clarkb | sdake: we already run on osic | 01:41 |
| sdake | were able to launch 2000 vms in about 5 seconds | 01:41 |
| clarkb | sdake: and they fought me over the allocation | 01:41 |
| clarkb | sdake: they balked at 600 | 01:41 |
| clarkb | I got 100 | 01:41 |
| anteaya | but there is no session in that timeframe on the infra schedule | 01:41 |
| clarkb | I am appreciative though, their Vms are quick, I just don't expect them to give us any more quota | 01:41 |
| clarkb | sdake: aiui IPs are the big problem there | 01:41 |
| jhesketh | jeblair: ping | 01:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use setfiles to restore unbound selinux policy https://review.openstack.org/308105 | 01:42 |
| clarkb | ianw_: pabelanger ^ maybe we start with that, then when dib moves selinux setfiles to later run level we can remove that from the unbound finalise and move unbound back to 90 | 01:43 |
| sdake | clarkb i'll ask around and see if they can squeeze out a 64 node cluster for upstream gating | 01:43 |
| sdake | clarkb they are keen to run kolla in production and it serves double duty for them - testing kolla and upstream gating of openstack ;) | 01:43 |
| sdake | clarkb no promises, but i can ask around if you like | 01:43 |
| clarkb | ianw_: is virt-sysprep a magical image manipulation thing for selinux and other stuff? | 01:43 |
| clarkb | sdake: I mean I already asked | 01:44 |
| clarkb | sdake: and when they saw the number they immediately said "woah now we need to loop jonathan in" | 01:44 |
| sdake | clarkb doesn't hurt to ask from a different person for a different agenda ;) | 01:44 |
| clarkb | which is a really bad use of jbryce's time imo | 01:44 |
| clarkb | so I negotiated down to our minimum ask | 01:44 |
| anteaya | clarkb: agreed | 01:44 |
| ianw_ | clarkb: virt-sysprep is libguestfs' "cleanup image for cloning" tool. it's what we keep recreating (badly) in various parts of various elements | 01:44 |
| anteaya | mtreinish: I'm trying to decide between qa's negative testing and cruft-busters work sessions and some neutron sessions | 01:46 |
| sdake | clarkb basically osic see's openstack's #1 problem as deployment and opertion of an openstack ckloud | 01:46 |
| clarkb | sdake: I am always happy for more quota though, so it can't hurt to ask. Just trying to explain why I am not hopeful | 01:46 |
| anteaya | mtreinish: but the qa etherpads don't have any details so it is hard for me to know if I would be useful if I attend | 01:46 |
| sdake | and they think kolla is silver bullet for that problem | 01:46 |
| *** kengo_sakai has quit IRC | 01:47 | |
| *** MarkAtwood has quit IRC | 01:48 | |
| openstackgerrit | Merged openstack/requirements: Update requirements ordering https://review.openstack.org/305552 | 01:48 |
| *** ddieterly has joined #openstack-infra | 01:50 | |
| *** ddieterly is now known as ddieterly[away] | 01:51 | |
| *** kengo_sakai has joined #openstack-infra | 01:51 | |
| *** apoorvad has quit IRC | 01:52 | |
| anteaya | Wednesday is definitely my busy day next week | 01:54 |
| *** keedya has quit IRC | 01:55 | |
| *** mylu has quit IRC | 01:55 | |
| *** mylu has joined #openstack-infra | 01:58 | |
| openstackgerrit | Merged openstack/requirements: Bump upper-constraints on novaclient to 4.0.0 https://review.openstack.org/307437 | 01:59 |
| anteaya | hey linuxcon north america is going to be in toronto this year: http://events.linuxfoundation.org/events/linuxcon-north-america | 02:01 |
| *** apoorvad has joined #openstack-infra | 02:02 | |
| anteaya | cfp is open for another week: http://events.linuxfoundation.org/events/linuxcon-north-america/program/cfp | 02:02 |
| anteaya | stevemar: ^^ | 02:02 |
| *** EricGonczer_ has quit IRC | 02:02 | |
| stevemar | anteaya: oh damn! | 02:05 |
| stevemar | anteaya: nice | 02:05 |
| anteaya | stevemar: yeah was thinking you would like that | 02:06 |
| anteaya | your back yard | 02:06 |
| stevemar | anteaya: yeah, 25-30 minute walk | 02:07 |
| stevemar | nice | 02:07 |
| morgan | stevemar: toronto? | 02:07 |
| anteaya | toronto | 02:08 |
| anteaya | morgan: want to come to toronto? | 02:08 |
| stevemar | morgan: yep! | 02:08 |
| anteaya | cfp is open for another week | 02:09 |
| * morgan should figure out waht to talk abot | 02:09 | |
| anteaya | anything | 02:09 |
| stevemar | i am so happy about it! | 02:09 |
| * morgan should also learn to type. | 02:09 | |
| anteaya | toronto is nice in august | 02:09 |
| anteaya | oh symphony of fire! | 02:09 |
| * anteaya checks the dates | 02:09 | |
| *** yamamot__ has quit IRC | 02:09 | |
| *** amrith is now known as _amrith_ | 02:10 | |
| clarkb | mordred: please see comment on https://review.openstack.org/#/c/307755/4 I think that is failing tests due to that thing I point out | 02:11 |
| *** apoorvad has quit IRC | 02:12 | |
| anteaya | I guess they aren't doing them anymore :( https://en.wikipedia.org/wiki/Symphony_of_Fire | 02:13 |
| mtreinish | anteaya: well the negative testing session is about tempest policy on negative tests (tests that do something expected to fail) and whether we should allow them, remove them, or something else | 02:14 |
| *** mriedem has quit IRC | 02:14 | |
| mtreinish | anteaya: and the cruft-busters session is about some of the technical debt that's built up in tempest recently and how to clean it up | 02:15 |
| mtreinish | probably mostly an identify issues and sign up for things session | 02:15 |
| mtreinish | anteaya: I'm not sure either would be of particular interest to you (I could be wrong though) | 02:15 |
| anteaya | fair enough | 02:18 |
| anteaya | thanks for the description | 02:18 |
| *** lakshmiS has quit IRC | 02:19 | |
| anteaya | the negative testing bit probably would be | 02:19 |
| anteaya | in as much as it is a testing policy change | 02:19 |
| anteaya | and would need to be socialized | 02:19 |
| anteaya | the crufty bit probably not so much | 02:19 |
| *** r-mibu has quit IRC | 02:19 | |
| anteaya | mtreinish: oh and linuxcon na is in toronto this year! come to canada | 02:20 |
| *** sdake_ has joined #openstack-infra | 02:20 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 02:21 |
| *** kzaitsev_mb has joined #openstack-infra | 02:22 | |
| mtreinish | anteaya: yeah I know but it conflicts with: http://www.openstackeast.com/ which I was planning to go to | 02:23 |
| *** sdake has quit IRC | 02:23 | |
| anteaya | it does so | 02:23 |
| anteaya | is this the first year for openstackeast? | 02:23 |
| anteaya | I hadn't heard of it before | 02:24 |
| anteaya | and yes, I can see the attraction | 02:24 |
| mtreinish | it is, which is why I thought I should go | 02:24 |
| anteaya | good call | 02:24 |
| anteaya | they would benefit from your presence | 02:24 |
| anteaya | s/would/will | 02:24 |
| *** ddieterly[away] is now known as ddieterly | 02:25 | |
| *** _amrith_ is now known as amrith | 02:26 | |
| *** Sukhdev has quit IRC | 02:26 | |
| *** ddieterly is now known as ddieterly[away] | 02:26 | |
| *** gyee has quit IRC | 02:27 | |
| *** kzaitsev_mb has quit IRC | 02:27 | |
| *** nwkarsten has joined #openstack-infra | 02:28 | |
| clarkb | Nakato: I left a comment on https://review.openstack.org/#/c/293194/8 it is possible I don't grok what you are attmepting to do with argparse so if what I describe is already supportd fill me in and I will update my review | 02:28 |
| anteaya | mtreinish: it is very enterprisey | 02:28 |
| *** r-mibu has joined #openstack-infra | 02:29 | |
| *** sputnik13 has quit IRC | 02:30 | |
| *** keedya has joined #openstack-infra | 02:30 | |
| *** thorst has quit IRC | 02:31 | |
| clarkb | aha! I am wrong append with 1 means pass the flag multple times | 02:32 |
| *** nwkarsten has quit IRC | 02:32 | |
| clarkb | I wonder how * or + would work with that | 02:33 |
| * clarkb updates review | 02:33 | |
| Nakato | clarkb: I'd assume + and 'append' would produce something we could grok, but I used nargs=1 to conform to the other arguments. *adds that to review* | 02:35 |
| *** gildub has quit IRC | 02:35 | |
| clarkb | Nakato: ya what is there is fine, just writing a response to myself and updating my review | 02:36 |
| Nakato | clarkb: Thanks for taking a look | 02:36 |
| clarkb | and done | 02:36 |
| *** sdake_ has quit IRC | 02:36 | |
| *** EricGonczer_ has joined #openstack-infra | 02:37 | |
| clarkb | what is there seems fine to me. Tests even! | 02:37 |
| *** fedexo has joined #openstack-infra | 02:37 | |
| *** sridhar_ram has quit IRC | 02:38 | |
| *** amotoki has joined #openstack-infra | 02:44 | |
| *** sdake has joined #openstack-infra | 02:48 | |
| *** EricGonczer_ has joined #openstack-infra | 02:48 | |
| *** fawadkhaliq has quit IRC | 02:53 | |
| *** EricGonczer_ has quit IRC | 02:54 | |
| *** vhosakot has quit IRC | 02:55 | |
| *** yuanying has quit IRC | 02:56 | |
| *** yamamot__ has joined #openstack-infra | 02:56 | |
| *** vhosakot has joined #openstack-infra | 03:01 | |
| *** weshay has quit IRC | 03:06 | |
| *** pradk has quit IRC | 03:07 | |
| mwhahaha | _ody: Hunner http://logs.openstack.org/77/306677/9/gate/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/061bc66/console.html#_2016-04-20_01_23_14_212 | 03:08 |
| mwhahaha | er sorry wrong channel | 03:08 |
| *** links has joined #openstack-infra | 03:21 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:23 | |
| *** hichihara has joined #openstack-infra | 03:23 | |
| *** asettle has quit IRC | 03:27 | |
| *** ddieterly[away] has quit IRC | 03:27 | |
| *** kzaitsev_mb has quit IRC | 03:28 | |
| *** pradk has joined #openstack-infra | 03:29 | |
| *** thorst has joined #openstack-infra | 03:32 | |
| *** rkukura has joined #openstack-infra | 03:36 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Fix dependencies for openstack-repos https://review.openstack.org/308100 | 03:37 |
| clarkb | ianw_: order matters there? | 03:38 |
| ianw_ | clarkb: no, alphabetical for style ... forgot n comes before p :) | 03:38 |
| *** mylu has quit IRC | 03:38 | |
| clarkb | ianw_: is that enforced by the linter? | 03:39 |
| *** anteaya has quit IRC | 03:39 | |
| ianw_ | clarkb: yep | 03:39 |
| clarkb | gotcha | 03:39 |
| * clarkb reapplies vote | 03:39 | |
| *** thorst has quit IRC | 03:40 | |
| *** bpokorny has joined #openstack-infra | 03:40 | |
| *** ayoung has quit IRC | 03:40 | |
| *** gongysh has joined #openstack-infra | 03:40 | |
| *** arif-ali has quit IRC | 03:43 | |
| *** asettle has joined #openstack-infra | 03:43 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move selinux restore to end of finalise https://review.openstack.org/308128 | 03:45 |
| *** amrith is now known as _amrith_ | 03:45 | |
| ianw_ | clarkb: ^ so i think that's the right way to go, see comment on how i think it got that way | 03:45 |
| clarkb | ianw_: +1 | 03:46 |
| clarkb | I am happy to abandon my project-config change entirely and reaplce with with a deletionof the restorecon if we can get ^ in | 03:47 |
| *** arif-ali has joined #openstack-infra | 03:48 | |
| *** rlandy has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| greghaynes | ianw_: nitpickey -1 | 03:49 |
| ianw_ | greghaynes: aren't we kind of pointing selinux gun at foot for external elements then, since they'll miss the relabel | 03:50 |
| ianw_ | greghaynes: still investigating the virt-sysprep method, but i'm thinking an even later phase would work well for that and fstrim | 03:51 |
| ianw_ | operate on the final raw image phase | 03:51 |
| greghaynes | ianw_: an external element shouldnt be putting things past 90 without having a good reason to (and groking the implications) | 03:51 |
| greghaynes | the same argument could be said for making a 99-zsomething script, you just have to know what youre getting in to with how we do ordering | 03:52 |
| ianw_ | greghaynes: fair enough, but i'm not sure if that's documented anywhere | 03:52 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move selinux restore to end of finalise https://review.openstack.org/308128 | 03:52 |
| greghaynes | ianw_: yea, it might not be, this is something that occured to me a while ago as an 'oh crap we need to not do this', I cant remember if I wrote it down... | 03:53 |
| ianw_ | greghaynes: you want to write something, or shall i? | 03:53 |
| greghaynes | ianw_: I'mworking on it now | 03:54 |
| *** mylu has joined #openstack-infra | 03:59 | |
| *** mylu has quit IRC | 04:00 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Document upstream executable numbering convention https://review.openstack.org/308131 | 04:00 |
| *** doug-fish has quit IRC | 04:01 | |
| *** claudiub has joined #openstack-infra | 04:01 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add releasenotes https://review.openstack.org/292193 | 04:01 |
| ianw_ | greghaynes: ^ with all the chatter about being a project, would be nice to start keeping this too | 04:01 |
| greghaynes | The copyright thing is wierd, clarkb - do you know what is a reasonable thing to put there? (https://review.openstack.org/#/c/292193/7/releasenotes/source/conf.py line 23) | 04:02 |
| *** kushal has joined #openstack-infra | 04:03 | |
| clarkb | greghaynes: I would put "Diskimage-builder developers" | 04:03 |
| clarkb | or s/developers/authors/ | 04:03 |
| *** rkukura has quit IRC | 04:03 | |
| clarkb | while I have a patch or two in dib and I am sure fungi does too I am not sure that the foundation chould get exclusive (C) | 04:03 |
| greghaynes | Yea, and its always scary to change those later, at least to me | 04:04 |
| greghaynes | ianw_: ^ does 'diskimage-builder authors' sound fine? | 04:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add releasenotes https://review.openstack.org/292193 | 04:05 |
| ianw_ | ^ power to the people | 04:05 |
| * ianw_ goes to check if trump is president yet | 04:05 | |
| greghaynes | heh, that works too | 04:05 |
| ianw_ | ooh, one step closer | 04:06 |
| *** armax has quit IRC | 04:06 | |
| ianw_ | greghaynes: doc/source/conf.py still has "OpenStack" | 04:06 |
| greghaynes | ianw_: Yea, theres also another one somewhere else in docs I think. Generally everyone is afraid to change copyright lines once they are committed I think | 04:07 |
| *** piet has joined #openstack-infra | 04:07 | |
| *** gongysh has quit IRC | 04:08 | |
| *** sputnik13 has joined #openstack-infra | 04:09 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Document upstream executable numbering convention https://review.openstack.org/308131 | 04:09 |
| *** asselin_ has joined #openstack-infra | 04:10 | |
| *** ekarlso has quit IRC | 04:11 | |
| *** andreaf has quit IRC | 04:11 | |
| ianw_ | we got a lot in the pipeline, just need the CI to work :( | 04:11 |
| *** mylu has joined #openstack-infra | 04:14 | |
| *** bpokorny has quit IRC | 04:16 | |
| *** tphummel has joined #openstack-infra | 04:17 | |
| *** piet has quit IRC | 04:19 | |
| *** piet has joined #openstack-infra | 04:20 | |
| *** andreaf has joined #openstack-infra | 04:21 | |
| *** ekarlso has joined #openstack-infra | 04:24 | |
| *** maishsk has quit IRC | 04:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:25 | |
| *** fawadkhaliq has joined #openstack-infra | 04:28 | |
| *** fedexo has quit IRC | 04:29 | |
| *** kzaitsev_mb has quit IRC | 04:30 | |
| *** apoorvad has joined #openstack-infra | 04:31 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Document running Nodepool for testing https://review.openstack.org/179840 | 04:31 |
| clarkb | pleia2: ^ I haven't walked through testing that step by step, but I reall want to get this doc change in and I think what I have there is a small viable doc update | 04:32 |
| clarkb | I don't actually have local machines big enough to run devstack anymore | 04:34 |
| clarkb | :/ | 04:34 |
| *** piet has quit IRC | 04:34 | |
| *** markvoelker has quit IRC | 04:34 | |
| *** markvoelker has joined #openstack-infra | 04:35 | |
| *** sdake has quit IRC | 04:36 | |
| *** sputnik13 has quit IRC | 04:36 | |
| *** apoorvad has quit IRC | 04:36 | |
| *** thorst has joined #openstack-infra | 04:37 | |
| *** markvoelker has quit IRC | 04:40 | |
| *** imcsk8 has quit IRC | 04:42 | |
| *** psachin has joined #openstack-infra | 04:43 | |
| *** imcsk8 has joined #openstack-infra | 04:44 | |
| *** maestro1 has joined #openstack-infra | 04:45 | |
| *** thorst has quit IRC | 04:46 | |
| *** kengo_sa_ has joined #openstack-infra | 04:48 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 04:49 |
| *** kengo_sa_ has quit IRC | 04:52 | |
| *** abregman has joined #openstack-infra | 04:52 | |
| *** _amrith_ is now known as amrith | 04:53 | |
| *** piet has joined #openstack-infra | 04:54 | |
| *** mylu has quit IRC | 04:57 | |
| *** mrmartin has joined #openstack-infra | 04:59 | |
| *** mrmartin has quit IRC | 04:59 | |
| *** yamamot__ has quit IRC | 04:59 | |
| *** Sukhdev has joined #openstack-infra | 05:00 | |
| *** gildub has joined #openstack-infra | 05:01 | |
| *** doug-fish has joined #openstack-infra | 05:02 | |
| *** flwang1 has quit IRC | 05:02 | |
| *** rossella_s has quit IRC | 05:03 | |
| *** rossella_s has joined #openstack-infra | 05:04 | |
| *** yamamot__ has joined #openstack-infra | 05:05 | |
| *** doug-fish has quit IRC | 05:07 | |
| pleia2 | clarkb: I'll do a once through in the AM | 05:09 |
| *** piet has quit IRC | 05:10 | |
| clarkb | pleia2: that would be awesome, thank you | 05:10 |
| *** yamahata has joined #openstack-infra | 05:10 | |
| clarkb | pleia2: the devstack plugin is very well tested now so I mostly expect that to just work | 05:10 |
| openstackgerrit | Merged openstack/requirements: Allow maintaining extras from project https://review.openstack.org/303271 | 05:12 |
| *** avarner has joined #openstack-infra | 05:14 | |
| *** avarner has quit IRC | 05:18 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add an alternative path of Neutron tempest plugin https://review.openstack.org/308135 | 05:22 |
| *** yonglihe has quit IRC | 05:22 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add an alternative path of Neutron tempest plugin https://review.openstack.org/308135 | 05:23 |
| *** sridhar_ram has joined #openstack-infra | 05:25 | |
| *** bpokorny has joined #openstack-infra | 05:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:26 | |
| *** ramishra has quit IRC | 05:28 | |
| *** eliqiao_ is now known as eliqiao | 05:29 | |
| *** ramishra has joined #openstack-infra | 05:30 | |
| *** kzaitsev_mb has quit IRC | 05:31 | |
| *** amrith is now known as _amrith_ | 05:32 | |
| *** asettle has quit IRC | 05:32 | |
| *** asettle has joined #openstack-infra | 05:33 | |
| *** dimtruck is now known as zz_dimtruck | 05:36 | |
| *** sdake_ has joined #openstack-infra | 05:37 | |
| *** denisra has quit IRC | 05:42 | |
| *** ihrachys has joined #openstack-infra | 05:42 | |
| *** zz_dimtruck is now known as dimtruck | 05:43 | |
| stevemar | mordred: looks like we need https://github.com/openstack/os-client-config/commit/8fa55700b90e335e54cd459ea8a60578e8d27fc7 backported to get OSC building again in stable/liberty - d'oh | 05:44 |
| stevemar | mordred: le bug: https://bugs.launchpad.net/os-client-config/+bug/1535562 | 05:44 |
| openstack | Launchpad bug 1535562 in python-openstackclient "Unit test failure when buildnig debian package for 2.0.0" [High,Triaged] | 05:44 |
| *** thorst has joined #openstack-infra | 05:44 | |
| lennyb | Hi Infra, I see that sometimes checkout nova project takes too long ( almost 2h yesterday ) #link http://paste.openstack.org/show/494728/ . | 05:50 |
| *** thorst has quit IRC | 05:50 | |
| clarkb | lennyb: we cache all of our repos | 05:53 |
| clarkb | lennyb: without more info debugging that is almost impossible | 05:53 |
| clarkb | but generally you dont want to do a full clone for every job run | 05:53 |
| *** ihrachys has quit IRC | 05:54 | |
| *** asettle has quit IRC | 05:55 | |
| *** asettle has joined #openstack-infra | 05:55 | |
| *** bpokorny has quit IRC | 05:56 | |
| *** yuanying_ has joined #openstack-infra | 05:57 | |
| *** yuanying has quit IRC | 05:58 | |
| *** harlowja_at_home has joined #openstack-infra | 06:00 | |
| lennyb | clarcb: do you have mirror sites, maybe I can clone from others? How can I increase log level of our CI during ./stack.sh ? I am running CI based on Common CI solution, running stack.sh every time, if I understand you correctly I dont have to reclone devstack each time, only unstack it with new ZUUL_COMMIT? | 06:02 |
| *** flwang1 has joined #openstack-infra | 06:02 | |
| *** eliqiao has quit IRC | 06:02 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Consider to log a package list for pip3 https://review.openstack.org/246017 | 06:02 |
| *** flwang1 has quit IRC | 06:03 | |
| *** flwang1 has joined #openstack-infra | 06:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix dependencies for openstack-repos https://review.openstack.org/308100 | 06:04 |
| *** flwang1 has quit IRC | 06:04 | |
| lennyb | clarkb: ^ ( sorry for 'clarcb' :) ) | 06:04 |
| *** flwang1 has joined #openstack-infra | 06:05 | |
| *** flwang1 has quit IRC | 06:05 | |
| *** flwang1 has joined #openstack-infra | 06:06 | |
| *** maishsk has joined #openstack-infra | 06:06 | |
| *** infra-red has joined #openstack-infra | 06:07 | |
| clarkb | lennyb: git.openstack.org is a frotnend for a pool of mirrors | 06:07 |
| *** infra-re_ has joined #openstack-infra | 06:07 | |
| *** oanson has joined #openstack-infra | 06:07 | |
| clarkb | what I mean about caching is clone once a day or so then have the jobs update to the commit under test | 06:07 |
| openstackgerrit | Merged openstack-infra/project-config: Fix package and image list comment cleanup https://review.openstack.org/308093 | 06:09 |
| *** eliqiao has joined #openstack-infra | 06:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Magnum: remove some projects and configs https://review.openstack.org/305475 | 06:11 |
| *** infra-red has quit IRC | 06:11 | |
| *** aeng_ is now known as aeng | 06:11 | |
| *** thiagop has quit IRC | 06:12 | |
| lennyb | clarkb: we are running #link https://specs.openstack.org/openstack-infra/infra-specs/specs/openstackci.html as is. Thanks for the tip, I will modify our CI to use this approach. | 06:13 |
| *** ericksonsantos has quit IRC | 06:13 | |
| *** maestro2 has joined #openstack-infra | 06:14 | |
| *** maestro1 has quit IRC | 06:14 | |
| *** flwang1 has quit IRC | 06:15 | |
| *** flwang1 has joined #openstack-infra | 06:16 | |
| openstackgerrit | Dong Ma proposed openstack-infra/subunit2sql: Add db api tests for subunit2sql (2/3) https://review.openstack.org/308147 | 06:19 |
| *** rcernin has joined #openstack-infra | 06:21 | |
| *** tphummel has quit IRC | 06:22 | |
| *** nelsnelson has quit IRC | 06:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:24 | |
| *** dimtruck is now known as zz_dimtruck | 06:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/306798 | 06:25 |
| *** flwang1 has quit IRC | 06:25 | |
| *** flwang1 has joined #openstack-infra | 06:26 | |
| *** nwkarsten has joined #openstack-infra | 06:26 | |
| *** nadya has joined #openstack-infra | 06:26 | |
| *** nmagnezi has joined #openstack-infra | 06:27 | |
| *** jlanoux has joined #openstack-infra | 06:29 | |
| *** nwkarsten has quit IRC | 06:31 | |
| lennyb | clarkb: you are running 'new' VM each job, so how do you manage clone git only once a day? | 06:31 |
| *** apoorvad has joined #openstack-infra | 06:34 | |
| *** flwang1 has quit IRC | 06:36 | |
| *** flwang1 has joined #openstack-infra | 06:36 | |
| ianw_ | lennyb: we build snapshots can clone them | 06:37 |
| ianw_ | lennyb: one overview i recently put together is https://www.technovelty.org/openstack/image-building-in-openstack-ci.html which might be helpful | 06:37 |
| *** e0ne has joined #openstack-infra | 06:38 | |
| *** apoorvad has quit IRC | 06:39 | |
| *** mikelk has joined #openstack-infra | 06:40 | |
| *** infra-re_ has quit IRC | 06:40 | |
| *** Guest30844 has quit IRC | 06:41 | |
| lennyb | ianw_:10x | 06:42 |
| *** e0ne has quit IRC | 06:43 | |
| *** e0ne has joined #openstack-infra | 06:44 | |
| *** infra-red has joined #openstack-infra | 06:44 | |
| *** e0ne_ has joined #openstack-infra | 06:45 | |
| *** sridhar_ram has quit IRC | 06:45 | |
| *** flwang1 has quit IRC | 06:46 | |
| *** flwang1 has joined #openstack-infra | 06:46 | |
| *** yuanying_ has quit IRC | 06:47 | |
| *** tesseract has joined #openstack-infra | 06:47 | |
| *** yuanying has joined #openstack-infra | 06:48 | |
| *** maishsk has quit IRC | 06:48 | |
| *** e0ne has quit IRC | 06:48 | |
| *** thorst has joined #openstack-infra | 06:48 | |
| *** tesseract is now known as Guest67082 | 06:48 | |
| *** maishsk has joined #openstack-infra | 06:48 | |
| *** e0ne_ has quit IRC | 06:50 | |
| *** kzaitsev_mb has quit IRC | 06:52 | |
| *** thorst has quit IRC | 06:55 | |
| *** asettle has quit IRC | 06:55 | |
| *** asettle has joined #openstack-infra | 06:56 | |
| *** flwang1 has quit IRC | 06:56 | |
| *** flwang1 has joined #openstack-infra | 06:57 | |
| *** jaosorior has joined #openstack-infra | 06:57 | |
| *** harlowja_at_home has quit IRC | 06:59 | |
| *** infra-red has quit IRC | 06:59 | |
| *** permalac has joined #openstack-infra | 07:00 | |
| *** jrmyjohn has joined #openstack-infra | 07:01 | |
| openstackgerrit | Michele Paolino proposed openstack-infra/project-config: New project devstack-plugin-tar-installer https://review.openstack.org/308156 | 07:01 |
| *** matrohon has joined #openstack-infra | 07:02 | |
| *** asettle has quit IRC | 07:03 | |
| *** mpaolino has joined #openstack-infra | 07:04 | |
| *** Sukhdev has quit IRC | 07:04 | |
| *** jrmyjohn has quit IRC | 07:05 | |
| mpaolino | Hello folks, finally I managed to have a new project proposal (thanks also to your help) : https://review.openstack.org/#/c/308156/ | 07:06 |
| *** flwang1 has quit IRC | 07:06 | |
| mpaolino | what's the process to proceed now? | 07:06 |
| openstackgerrit | Tony Breeds proposed openstack/requirements: Cap testresources<2.0.0 https://review.openstack.org/307858 | 07:07 |
| *** flwang1 has joined #openstack-infra | 07:07 | |
| *** Rockyg has quit IRC | 07:10 | |
| *** vincentll has joined #openstack-infra | 07:10 | |
| *** vgridnev has joined #openstack-infra | 07:11 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add ironic cleaning job https://review.openstack.org/304473 | 07:13 |
| *** chmouel has joined #openstack-infra | 07:15 | |
| *** flwang1 has quit IRC | 07:17 | |
| *** flwang2 has joined #openstack-infra | 07:17 | |
| AJaeger | mpaolino: fix the failure -and wait for review ;) | 07:17 |
| mpaolino | thanks, I just saw it | 07:18 |
| AJaeger | mpaolino: and read the complete Creator's guide - you do not have ACLs defined at all. | 07:19 |
| *** nadya has quit IRC | 07:19 | |
| mpaolino | arg, I didnt push the ACLs file | 07:20 |
| *** asselin_ has quit IRC | 07:21 | |
| *** vhosakot has quit IRC | 07:21 | |
| *** Nikolay_St has joined #openstack-infra | 07:22 | |
| *** jmccrory has quit IRC | 07:22 | |
| *** jlvillal has quit IRC | 07:22 | |
| *** taron1 has quit IRC | 07:22 | |
| *** psachin_ has joined #openstack-infra | 07:22 | |
| yolanda | good morning | 07:22 |
| *** jlvillal has joined #openstack-infra | 07:22 | |
| *** melwitt has quit IRC | 07:23 | |
| *** kevinbenton has quit IRC | 07:23 | |
| *** ifarkas has joined #openstack-infra | 07:24 | |
| openstackgerrit | Michele Paolino proposed openstack-infra/project-config: New project devstack-plugin-tar-installer https://review.openstack.org/308156 | 07:24 |
| *** jmccrory has joined #openstack-infra | 07:24 | |
| *** zz_dimtruck is now known as dimtruck | 07:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Propose translations: Fix invalid bash https://review.openstack.org/308166 | 07:25 |
| AJaeger | morning, yolanda. | 07:25 |
| *** melwitt has joined #openstack-infra | 07:25 | |
| *** taron1 has joined #openstack-infra | 07:25 | |
| *** melwitt is now known as Guest17033 | 07:25 | |
| * AJaeger broke today's translation import jobs, see 308166 | 07:26 | |
| *** kevinbenton has joined #openstack-infra | 07:26 | |
| *** flwang2 has quit IRC | 07:27 | |
| *** flwang1 has joined #openstack-infra | 07:27 | |
| *** nwkarsten has joined #openstack-infra | 07:28 | |
| *** esikachev has joined #openstack-infra | 07:28 | |
| *** Yarboa has joined #openstack-infra | 07:28 | |
| *** infra-red has joined #openstack-infra | 07:29 | |
| *** infra-red has joined #openstack-infra | 07:30 | |
| *** dizquierdo has joined #openstack-infra | 07:30 | |
| openstackgerrit | Dina Belova proposed openstack/requirements: Change osprofiler version to osprofiler>=1.3.0 https://review.openstack.org/308171 | 07:30 |
| *** psachin_ has quit IRC | 07:30 | |
| *** psachin_ has joined #openstack-infra | 07:31 | |
| *** infra-red has quit IRC | 07:31 | |
| *** nwkarsten has quit IRC | 07:32 | |
| *** psachin_ has quit IRC | 07:34 | |
| *** dimtruck is now known as zz_dimtruck | 07:34 | |
| *** psachin_ has joined #openstack-infra | 07:34 | |
| *** apoorvad has joined #openstack-infra | 07:35 | |
| *** shardy has joined #openstack-infra | 07:36 | |
| *** flwang1 has quit IRC | 07:37 | |
| *** scheuran has joined #openstack-infra | 07:37 | |
| *** ihrachys has joined #openstack-infra | 07:38 | |
| *** psachin_ has quit IRC | 07:38 | |
| *** psachin_ has joined #openstack-infra | 07:38 | |
| openstackgerrit | Michele Paolino proposed openstack-infra/project-config: New project devstack-plugin-tar-installer https://review.openstack.org/308156 | 07:38 |
| *** psachin_ has quit IRC | 07:40 | |
| *** apoorvad has quit IRC | 07:40 | |
| *** jlanoux has quit IRC | 07:41 | |
| *** psachin_ has joined #openstack-infra | 07:44 | |
| *** fawadkhaliq has quit IRC | 07:45 | |
| *** fawadkhaliq has joined #openstack-infra | 07:45 | |
| *** psachin_ has quit IRC | 07:46 | |
| *** psachin_ has joined #openstack-infra | 07:46 | |
| *** psachin_ has quit IRC | 07:47 | |
| *** Hal has joined #openstack-infra | 07:47 | |
| *** Hal is now known as Guest74966 | 07:48 | |
| *** yuanying has quit IRC | 07:49 | |
| *** andymaier has joined #openstack-infra | 07:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 07:51 |
| *** yuanying has joined #openstack-infra | 07:52 | |
| *** thorst has joined #openstack-infra | 07:53 | |
| *** lucas-dinner is now known as lucasagomes | 07:55 | |
| yolanda | ttx, or other requirements core, can you take a look at https://review.openstack.org/307000 ? it already has +2 from mordred | 07:57 |
| *** hichihara has quit IRC | 07:58 | |
| *** elo has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** thorst has quit IRC | 08:01 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 08:01 |
| *** elo has joined #openstack-infra | 08:02 | |
| yolanda | btw, nice to see that we have good times on gate now, yesterday's queue was long | 08:02 |
| *** fawadkhaliq has quit IRC | 08:02 | |
| ttx | yolanda: looking | 08:03 |
| yolanda | thx | 08:03 |
| AJaeger | yolanda: it was the longest queue in post I've ever seen... | 08:03 |
| ttx | approved | 08:03 |
| yolanda | ovh loss, combined with rax, was so bad | 08:04 |
| *** zzzeek has joined #openstack-infra | 08:04 | |
| yolanda | ttx, thanks | 08:04 |
| *** maestro2 has quit IRC | 08:05 | |
| *** yuanying has quit IRC | 08:05 | |
| *** arxcruz has joined #openstack-infra | 08:05 | |
| yolanda | AJaeger, my changes for nodepool finally were merged, after a day... going to remove nodepool from emergency | 08:05 |
| *** zz_dimtruck is now known as dimtruck | 08:07 | |
| AJaeger | great, yolanda ! | 08:09 |
| *** HeOS has joined #openstack-infra | 08:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Run slave-cleanup on all proposal jobs https://review.openstack.org/306467 | 08:12 |
| *** jordanP has joined #openstack-infra | 08:14 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 08:16 |
| *** dimtruck is now known as zz_dimtruck | 08:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Propose translations: Fix invalid bash https://review.openstack.org/308166 | 08:17 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 08:17 |
| *** jistr has joined #openstack-infra | 08:18 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 08:19 |
| *** cody-somerville has quit IRC | 08:20 | |
| *** Yarboa has quit IRC | 08:21 | |
| *** claudiub has quit IRC | 08:22 | |
| *** jlanoux has joined #openstack-infra | 08:22 | |
| *** dtardivel has joined #openstack-infra | 08:23 | |
| *** dosaboy_ is now known as dosaboy | 08:24 | |
| *** maestro1 has joined #openstack-infra | 08:24 | |
| *** _amrith_ is now known as amrith | 08:25 | |
| *** vgridnev has quit IRC | 08:26 | |
| *** yonglihe has joined #openstack-infra | 08:27 | |
| *** keedya has quit IRC | 08:28 | |
| openstackgerrit | Merged openstack-infra/project-config: grafana: added grenade dvr multinode job to neutron dashboard https://review.openstack.org/307825 | 08:29 |
| *** zeih has joined #openstack-infra | 08:29 | |
| *** eric_lopez has joined #openstack-infra | 08:32 | |
| *** Yarboa has joined #openstack-infra | 08:34 | |
| *** dizquierdo has quit IRC | 08:35 | |
| *** elo has quit IRC | 08:35 | |
| *** elo has joined #openstack-infra | 08:36 | |
| *** Qiming has joined #openstack-infra | 08:37 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** shardy has quit IRC | 08:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Add script to facilitate project renaming in project-config repo" https://review.openstack.org/298410 | 08:38 |
| *** shardy has joined #openstack-infra | 08:38 | |
| *** eric_lopez has quit IRC | 08:39 | |
| *** _ody has quit IRC | 08:41 | |
| *** gildub has quit IRC | 08:41 | |
| *** psachin_ has joined #openstack-infra | 08:42 | |
| *** asettle has joined #openstack-infra | 08:42 | |
| *** jrmyjohn has joined #openstack-infra | 08:42 | |
| *** psachin_ has quit IRC | 08:43 | |
| *** psachin_ has joined #openstack-infra | 08:43 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Remove the old path of Neutron tempest plugin https://review.openstack.org/308202 | 08:45 |
| *** jrmyjohn has quit IRC | 08:47 | |
| *** fhubik has joined #openstack-infra | 08:52 | |
| *** electrofelix has joined #openstack-infra | 08:54 | |
| *** psachin has quit IRC | 08:55 | |
| *** psachin_ has quit IRC | 08:56 | |
| *** psachin has joined #openstack-infra | 08:56 | |
| *** zeih has quit IRC | 08:58 | |
| *** eric_lopez has joined #openstack-infra | 08:58 | |
| *** thorst has joined #openstack-infra | 08:58 | |
| *** elo has quit IRC | 08:58 | |
| *** david-lyle has quit IRC | 08:59 | |
| *** david-lyle has joined #openstack-infra | 09:00 | |
| *** psachin_ has joined #openstack-infra | 09:01 | |
| *** eric_lopez has quit IRC | 09:03 | |
| *** thorst has quit IRC | 09:05 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Don't add duplicate entries to metadata.txt https://review.openstack.org/296364 | 09:07 |
| *** zz_dimtruck is now known as dimtruck | 09:08 | |
| *** asettle has quit IRC | 09:11 | |
| *** vgridnev has joined #openstack-infra | 09:12 | |
| *** psachin has quit IRC | 09:13 | |
| *** sambetts|afk is now known as sambetts | 09:14 | |
| *** psachin_ has quit IRC | 09:14 | |
| *** psachin has joined #openstack-infra | 09:14 | |
| *** dingyichen has quit IRC | 09:17 | |
| *** asettle has joined #openstack-infra | 09:17 | |
| *** dimtruck is now known as zz_dimtruck | 09:17 | |
| *** dtantsur|afk is now known as dtantsur | 09:18 | |
| *** _degorenko|afk is now known as degorenko | 09:20 | |
| *** mpaolino has quit IRC | 09:23 | |
| *** mpaolino has joined #openstack-infra | 09:23 | |
| *** yamahata has quit IRC | 09:26 | |
| *** nwkarsten has joined #openstack-infra | 09:29 | |
| *** zeih has joined #openstack-infra | 09:29 | |
| *** yamamot__ has quit IRC | 09:30 | |
| openstackgerrit | Merged openstack/diskimage-builder: Document upstream executable numbering convention https://review.openstack.org/308131 | 09:30 |
| openstackgerrit | Merged openstack/requirements: Update dependency for betamax to 0.6.0 https://review.openstack.org/307000 | 09:33 |
| *** zeih has quit IRC | 09:33 | |
| *** nwkarsten has quit IRC | 09:34 | |
| *** markmcd__ has quit IRC | 09:36 | |
| *** jlanoux has quit IRC | 09:37 | |
| *** apoorvad has joined #openstack-infra | 09:37 | |
| *** markmcd has joined #openstack-infra | 09:38 | |
| *** markmcd1 has joined #openstack-infra | 09:38 | |
| *** jlanoux has joined #openstack-infra | 09:39 | |
| *** apoorvad has quit IRC | 09:42 | |
| openstackgerrit | Merged openstack/requirements: Block broken netmiko release 0.4.3 https://review.openstack.org/306206 | 09:42 |
| *** nadya has joined #openstack-infra | 09:50 | |
| *** _ody has joined #openstack-infra | 09:50 | |
| *** amrith is now known as _amrith_ | 09:52 | |
| *** asettle has quit IRC | 09:54 | |
| shardy | Hi all, anyone know why meeting minutes may not have been logged yesterday? | 09:54 |
| shardy | http://eavesdrop.openstack.org/meetings/tripleo/2016/ | 09:54 |
| shardy | we had a meeting yesterday, where the bot appeared to be working, but it's not logged our meeting | 09:54 |
| *** asettle has joined #openstack-infra | 09:54 | |
| shardy | nvm, there's a stray character in the startmeeting command :( | 09:55 |
| shardy | <doh!> | 09:55 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 09:56 |
| *** _ody has quit IRC | 09:56 | |
| *** _ody has joined #openstack-infra | 09:56 | |
| *** ivc_ has joined #openstack-infra | 09:58 | |
| *** asettle has quit IRC | 09:59 | |
| *** tosky has joined #openstack-infra | 10:02 | |
| *** thorst has joined #openstack-infra | 10:03 | |
| *** yamamoto has joined #openstack-infra | 10:05 | |
| *** sdague has joined #openstack-infra | 10:08 | |
| *** nelsnelson has joined #openstack-infra | 10:08 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 10:08 |
| *** zz_dimtruck is now known as dimtruck | 10:08 | |
| *** asettle has joined #openstack-infra | 10:10 | |
| *** thorst has quit IRC | 10:11 | |
| *** nelsnelson has quit IRC | 10:13 | |
| *** jlanoux has quit IRC | 10:15 | |
| *** dimtruck is now known as zz_dimtruck | 10:18 | |
| *** doug-fish has joined #openstack-infra | 10:26 | |
| *** doug-fish has quit IRC | 10:31 | |
| *** maestro1 has quit IRC | 10:39 | |
| *** maishsk has quit IRC | 10:42 | |
| *** jrmyjohn has joined #openstack-infra | 10:44 | |
| *** pradk has quit IRC | 10:45 | |
| openstackgerrit | Sergei Chipiga proposed openstack-infra/project-config: Publish test_reports dir https://review.openstack.org/308261 | 10:46 |
| *** ivc_ has quit IRC | 10:46 | |
| *** jrmyjohn has quit IRC | 10:49 | |
| *** ociuhandu has quit IRC | 10:51 | |
| *** jaosorior has quit IRC | 10:52 | |
| *** jlanoux has joined #openstack-infra | 10:53 | |
| openstackgerrit | Sergei Chipiga proposed openstack-infra/project-config: Publish test_reports dir https://review.openstack.org/308261 | 10:53 |
| *** jaosorior has joined #openstack-infra | 10:53 | |
| *** jordanP has quit IRC | 10:55 | |
| *** pradk has joined #openstack-infra | 10:55 | |
| *** gongysh has joined #openstack-infra | 10:57 | |
| *** nwkarsten has joined #openstack-infra | 10:58 | |
| *** cfarquhar has quit IRC | 10:59 | |
| *** eliqiao_ has joined #openstack-infra | 10:59 | |
| *** cfarquhar has joined #openstack-infra | 11:00 | |
| *** cfarquhar has quit IRC | 11:00 | |
| *** cfarquhar has joined #openstack-infra | 11:00 | |
| *** eliqiao has quit IRC | 11:01 | |
| *** zeih has joined #openstack-infra | 11:02 | |
| *** nadya has quit IRC | 11:02 | |
| *** yamamoto has quit IRC | 11:02 | |
| *** claudiub has joined #openstack-infra | 11:03 | |
| *** ddieterly has joined #openstack-infra | 11:06 | |
| *** maishsk has joined #openstack-infra | 11:07 | |
| *** mikelk has quit IRC | 11:08 | |
| *** thorst has joined #openstack-infra | 11:08 | |
| *** mikelk has joined #openstack-infra | 11:09 | |
| *** zz_dimtruck is now known as dimtruck | 11:09 | |
| *** elo has joined #openstack-infra | 11:10 | |
| *** doug-fish has joined #openstack-infra | 11:11 | |
| *** sshnaidm has quit IRC | 11:14 | |
| *** ociuhandu has joined #openstack-infra | 11:14 | |
| *** sshnaidm has joined #openstack-infra | 11:14 | |
| *** thorst has quit IRC | 11:16 | |
| *** gildub has joined #openstack-infra | 11:18 | |
| *** zeih has quit IRC | 11:18 | |
| *** dimtruck is now known as zz_dimtruck | 11:19 | |
| *** asettle has quit IRC | 11:20 | |
| pabelanger | morning | 11:20 |
| *** asettle has joined #openstack-infra | 11:21 | |
| pabelanger | clarkb: So, centos-minimal built, but think I see an issue with our finalise.d script. 99-unbound is getting called after 11-setfiles in rpm-distro | 11:21 |
| pabelanger | so if we reorder, I think it will work | 11:21 |
| pabelanger | that is what I am testig now | 11:21 |
| pabelanger | testing* | 11:21 |
| *** zeih has joined #openstack-infra | 11:22 | |
| AJaeger | morning, pabelanger | 11:22 |
| *** ldnunes has joined #openstack-infra | 11:23 | |
| *** nadya has joined #openstack-infra | 11:24 | |
| pabelanger | bluebox looks angry: http://grafana.openstack.org/dashboard/db/nodepool-blue-box | 11:24 |
| pabelanger | checking FIPs | 11:24 |
| pabelanger | mordred: might be interested ^ | 11:24 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Update documentation about infra cloud https://review.openstack.org/308277 | 11:24 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic too | 11:25 |
| *** ddieterly is now known as ddieterly[away] | 11:25 | |
| *** trown|outtypewww is now known as trown | 11:25 | |
| *** yamamoto has joined #openstack-infra | 11:26 | |
| *** _amrith_ is now known as amrith | 11:26 | |
| *** gordc has joined #openstack-infra | 11:26 | |
| pabelanger | 51 leaked IPs in bluebox, deleting now | 11:26 |
| *** lucasagomes is now known as lucas-hungry | 11:28 | |
| *** ddieterly[away] is now known as ddieterly | 11:29 | |
| *** electrofelix has quit IRC | 11:29 | |
| AJaeger | thanks, pabelanger ! | 11:29 |
| pabelanger | and 99 leaked in OSIC | 11:29 |
| pabelanger | so something not right | 11:29 |
| *** electrofelix has joined #openstack-infra | 11:31 | |
| AJaeger | that'S a lot ;( | 11:34 |
| pabelanger | I think we still need to land some patches in nodepool to fix FIPs | 11:34 |
| pabelanger | but bluebox and osic back online | 11:35 |
| *** esikachev has quit IRC | 11:35 | |
| *** amrith is now known as _amrith_ | 11:36 | |
| *** electrofelix has quit IRC | 11:37 | |
| *** electrofelix has joined #openstack-infra | 11:37 | |
| *** electrofelix has quit IRC | 11:38 | |
| *** ddieterly is now known as ddieterly[away] | 11:38 | |
| *** electrofelix has joined #openstack-infra | 11:38 | |
| AJaeger | And the post queue still not cleaned up ;( "Only" 136 jobs, still some for 49hours ;( | 11:38 |
| *** apoorvad has joined #openstack-infra | 11:39 | |
| *** rhallisey has joined #openstack-infra | 11:39 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 11:40 |
| *** gordc has quit IRC | 11:40 | |
| *** gordc has joined #openstack-infra | 11:41 | |
| yolanda | wow, that's so much | 11:42 |
| *** electrofelix has quit IRC | 11:44 | |
| *** electrofelix has joined #openstack-infra | 11:44 | |
| *** apoorvad has quit IRC | 11:44 | |
| *** thorst has joined #openstack-infra | 11:45 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Make it clear when user reaches end of events timeline https://review.openstack.org/305483 | 11:45 |
| *** rfolco has joined #openstack-infra | 11:46 | |
| Shrews | pabelanger: do we know if those leaks are with mordred's latest nodepool and shade changes? specifically, nodepool with https://review.openstack.org/307755 | 11:46 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 11:46 |
| pabelanger | Shrews: no, I don't know | 11:47 |
| pabelanger | not sure which patches we are running ATM | 11:47 |
| Shrews | k. me neither. we'll wait to see what he says when he shows up | 11:47 |
| *** baoli has joined #openstack-infra | 11:47 | |
| *** binbincong has quit IRC | 11:48 | |
| *** esikachev has joined #openstack-infra | 11:49 | |
| *** baoli has quit IRC | 11:49 | |
| *** asettle has quit IRC | 11:49 | |
| *** baoli has joined #openstack-infra | 11:49 | |
| *** baoli has quit IRC | 11:49 | |
| *** daemontool has joined #openstack-infra | 11:50 | |
| *** dprince has joined #openstack-infra | 11:57 | |
| *** nwkarste_ has joined #openstack-infra | 12:01 | |
| AJaeger | pabelanger: could you review today a change for reviewday, please? https://review.openstack.org/307510 | 12:03 |
| AJaeger | pabelanger: that one should obsolete your change https://review.openstack.org/#/c/305996/ | 12:04 |
| *** nwkarsten has quit IRC | 12:04 | |
| *** baoli has joined #openstack-infra | 12:04 | |
| pabelanger | AJaeger: LGTM | 12:04 |
| AJaeger | pabelanger: thanks! rossella_s, FYI ^ | 12:04 |
| rossella_s | AJaeger, pabelanger great thanks! | 12:05 |
| *** aysyd has joined #openstack-infra | 12:05 | |
| *** baoli_ has joined #openstack-infra | 12:06 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Update documentation about infra cloud https://review.openstack.org/308277 | 12:06 |
| AJaeger | rossella_s: let's wait an hour after it'S merged until puppet is run - and then check again... | 12:06 |
| rossella_s | AJaeger, yeah hope now it will finally work, thanks a lot again | 12:06 |
| *** jheroux has joined #openstack-infra | 12:06 | |
| openstackgerrit | Merged openstack-infra/reviewday: Use pip to install gerrit-dash-creator https://review.openstack.org/307510 | 12:07 |
| *** binbincong has joined #openstack-infra | 12:07 | |
| *** psilvad has joined #openstack-infra | 12:08 | |
| *** baoli has quit IRC | 12:09 | |
| *** zz_dimtruck is now known as dimtruck | 12:10 | |
| *** kengo_sakai has quit IRC | 12:11 | |
| *** ddieterly[away] is now known as ddieterly | 12:18 | |
| *** ddieterly has quit IRC | 12:18 | |
| *** markvoelker has joined #openstack-infra | 12:18 | |
| *** piet has joined #openstack-infra | 12:18 | |
| *** eharney has quit IRC | 12:19 | |
| *** dimtruck is now known as zz_dimtruck | 12:21 | |
| *** nadya has quit IRC | 12:26 | |
| openstackgerrit | Merged openstack-infra/project-config: restore tagging ACLs for independent projects https://review.openstack.org/308044 | 12:29 |
| *** vponomaryov1 has joined #openstack-infra | 12:36 | |
| *** asettle has joined #openstack-infra | 12:37 | |
| *** keedya has joined #openstack-infra | 12:37 | |
| *** GnomeKing has joined #openstack-infra | 12:38 | |
| *** GnomeKing is now known as BobBall | 12:40 | |
| *** flepied1 has joined #openstack-infra | 12:40 | |
| openstackgerrit | Merged openstack-infra/release-tools: [WIP] Compare local and remote tarballs https://review.openstack.org/307178 | 12:41 |
| openstackgerrit | Merged openstack-infra/release-tools: add latest-deliverable-versions command https://review.openstack.org/307229 | 12:41 |
| *** asettle has quit IRC | 12:42 | |
| *** flepied has quit IRC | 12:43 | |
| pabelanger | clarkb: okay, redording our finalise.d scripts helped unbound | 12:44 |
| *** EricGonczer_ has joined #openstack-infra | 12:46 | |
| *** links has quit IRC | 12:46 | |
| *** ddieterly has joined #openstack-infra | 12:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Lower our nodepool-base finalise.d script https://review.openstack.org/308309 | 12:47 |
| pabelanger | clarkb: ^ | 12:47 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 12:47 |
| *** EricGonc_ has joined #openstack-infra | 12:47 | |
| *** rlandy has joined #openstack-infra | 12:47 | |
| pabelanger | clarkb: confirmed unbound started properly | 12:47 |
| pabelanger | with: -rw-r--r--. root root unconfined_u:object_r:named_conf_t:s0 forwarding.conf security context | 12:47 |
| *** salv-orlando has joined #openstack-infra | 12:48 | |
| AJaeger | pabelanger: should we push this through for today's images? | 12:48 |
| pabelanger | greghaynes: ianw_: we should also consider bumping 11-selinux-fixfiles-restore later in the finalise process | 12:49 |
| pabelanger | AJaeger: possible, I'd like clarkb, maybe fungi to comment | 12:49 |
| *** kgiusti has joined #openstack-infra | 12:49 | |
| *** matt-borland has joined #openstack-infra | 12:49 | |
| pabelanger | we have about 90mins until next round of image builds | 12:50 |
| pabelanger | only OSIC is affected for centos-7 | 12:50 |
| *** EricGonczer_ has quit IRC | 12:50 | |
| *** lucas-hungry is now known as lucasagomes | 12:51 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Make it clear when user reaches end of events timeline https://review.openstack.org/305483 | 12:51 |
| *** maishsk_ has joined #openstack-infra | 12:53 | |
| *** flepied1 is now known as flepied | 12:53 | |
| *** netsin has quit IRC | 12:54 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade https://review.openstack.org/243239 | 12:54 |
| *** |-paul-| has joined #openstack-infra | 12:55 | |
| *** maishsk has quit IRC | 12:55 | |
| *** maishsk_ is now known as maishsk | 12:55 | |
| *** piet has quit IRC | 12:56 | |
| *** nelsnelson has joined #openstack-infra | 12:57 | |
| *** piet has joined #openstack-infra | 12:58 | |
| *** pradk_ has joined #openstack-infra | 12:59 | |
| *** trown is now known as trown|brb | 13:00 | |
| *** aimeeU has joined #openstack-infra | 13:02 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: tripleo.sh --delorean-build handle oslo.* package builds https://review.openstack.org/302312 | 13:03 |
| *** rossella_s has quit IRC | 13:03 | |
| *** rossella_s has joined #openstack-infra | 13:04 | |
| AJaeger | pabelanger: could you check whether reviewday cron job run fine, please? IMHO it should be finished by now and I see no output, so seems something went wrong with the changes ;( | 13:06 |
| *** Goneri has joined #openstack-infra | 13:06 | |
| *** zz_dimtruck is now known as dimtruck | 13:07 | |
| pabelanger | AJaeger: sure | 13:08 |
| *** ivc_ has joined #openstack-infra | 13:08 | |
| rossella_s | AJaeger, pabelanger I am here to help if something needs to be fixed, thanks! | 13:10 |
| pabelanger | AJaeger: okay, puppet just ran successfully | 13:10 |
| fungi | pabelanger: AJaeger: talking about 308309? | 13:10 |
| AJaeger | fungi, yes, that was what we talked earlier about - right now we talk about reviewday | 13:11 |
| fungi | i mean as far as what pabelanger was wanting input on before image updates start in an hour | 13:12 |
| AJaeger | pabelanger: puppet run already an hour ago, I expected that to deploy everything, so waiting for cron job now. | 13:12 |
| AJaeger | fungi: correct! | 13:12 |
| EmilienM | pabelanger: we have tarballs! \o/ http://tarballs.openstack.org/puppet-aodh/ | 13:12 |
| AJaeger | EmilienM: woot! | 13:12 |
| pabelanger | fungi: AJaeger: We're limping long already. So if we miss, I think we are okay | 13:13 |
| AJaeger | pabelanger, rossella_s: I know I'm impatient on reviewday, so if cron is not finished yet or even started... | 13:13 |
| pabelanger | AJaeger: rossella_s: I'm just looking for the log file now | 13:13 |
| *** weshay has joined #openstack-infra | 13:15 | |
| EmilienM | but puppet periodic jobs are still broken | 13:15 |
| *** psachin has quit IRC | 13:15 | |
| EmilienM | we're exporting ZUUL_REF to None http://logs.openstack.org/periodic/periodic-puppet-openstack-integration-4-scenario001-tempest-centos-7/3ab3f15/console.html#_2016-04-20_06_07_19_324 | 13:15 |
| fungi | pabelanger: well, it looks simple on its face, but i need to skim through everything else that runs from finalise.d between 10 and 99 to make sure those don't have dependencies we'll start missing due to the reorder. have you tested it locally? | 13:15 |
| *** sc68cal has quit IRC | 13:16 | |
| EmilienM | exporting ZUUL_REF in other pipelines works fine, but not in periodic. I might miss something | 13:17 |
| *** asselin_ has joined #openstack-infra | 13:17 | |
| *** njohnston has quit IRC | 13:17 | |
| *** hrybacki has quit IRC | 13:17 | |
| *** dansmith has quit IRC | 13:17 | |
| *** sheeprine has quit IRC | 13:17 | |
| *** sheeprine has joined #openstack-infra | 13:18 | |
| *** piet has quit IRC | 13:18 | |
| *** michaelxin has quit IRC | 13:18 | |
| AJaeger | http://logs.openstack.org/periodic/periodic-puppet-openstack-integration-4-scenario001-tempest-centos-7/3ab3f15/console.html#_2016-04-20_06_10_31_567 has "--zuul-ref --zuul-branch --zuul-url http:/" - that'S the problem | 13:18 |
| pabelanger | AJaeger: just running it manually now | 13:18 |
| *** hrybacki has joined #openstack-infra | 13:18 | |
| *** ivc_ has quit IRC | 13:19 | |
| *** trown|brb is now known as trown | 13:19 | |
| *** sc68cal has joined #openstack-infra | 13:19 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add job validation and latest link to magnum image upload https://review.openstack.org/308323 | 13:19 |
| *** njohnston has joined #openstack-infra | 13:20 | |
| *** dansmith has joined #openstack-infra | 13:20 | |
| *** dansmith is now known as Guest64767 | 13:20 | |
| pabelanger | fungi: Ya, looks like we currently run setfiles before we modify unbound, so moving them setup the security contexts properly. And ya, tested it locally and uploaded into devstack to confirm | 13:20 |
| EmilienM | AJaeger: the /p you mean? | 13:20 |
| EmilienM | --zuul-url http://zm01.openstack.org/p | 13:20 |
| *** samueldmq has joined #openstack-infra | 13:20 | |
| pabelanger | AJaeger: seems to be working properly when I run it manually | 13:20 |
| *** |-paul-| has quit IRC | 13:20 | |
| EmilienM | AJaeger: the problem is with zuul_ref | 13:21 |
| pabelanger | AJaeger: where is the URL for this? | 13:21 |
| samueldmq | dhellmann: jeblair: hi, are you okay with patch 264398 now ? looks like it's already good to go | 13:21 |
| samueldmq | thanks | 13:21 |
| pabelanger | AJaeger: found it | 13:22 |
| *** jlanoux has quit IRC | 13:22 | |
| EmilienM | should I use gerrit-git-prep for periodic jobs? | 13:23 |
| *** lezbar__ has joined #openstack-infra | 13:23 | |
| *** lezbar has quit IRC | 13:23 | |
| *** maestro1 has joined #openstack-infra | 13:23 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add releasenote jobs to OpenStack-Ansible repositories https://review.openstack.org/308325 | 13:23 |
| *** piet has joined #openstack-infra | 13:23 | |
| pabelanger | EmilienM: you need to fix your zuul-clone script to not setup --zuul-ref and --zuul-branch if the zuul ENV are missing | 13:23 |
| *** xyang1 has joined #openstack-infra | 13:23 | |
| pabelanger | EmilienM: in install_modules.sh | 13:24 |
| *** bdemers has joined #openstack-infra | 13:24 | |
| EmilienM | ok | 13:25 |
| *** hashar has joined #openstack-infra | 13:25 | |
| AJaeger | EmilienM: your builder has ZUUL_REF set but run-tests and functions is a separate shell | 13:25 |
| AJaeger | EmilienM: as pabelanger says... | 13:25 |
| AJaeger | EmilienM: I guess in puppet-openstack-integration/functions | 13:25 |
| *** mriedem has joined #openstack-infra | 13:25 | |
| EmilienM | yeah | 13:25 |
| fungi | you may still need to set --zuul-branch if you're passing in a branch parameter to run the periodic job against non-master branches, but i haven't looked at what you have set up just commenting generally about zuul-cloner in periodic jobs | 13:25 |
| pabelanger | AJaeger: EmilienM: Yup. I'd default then to None | 13:26 |
| AJaeger | pabelanger: anything rossella_s or me can help with the reviewday? Sorry, was too slow to answer... | 13:26 |
| *** asselin_ has quit IRC | 13:26 | |
| *** wznoinsk has joined #openstack-infra | 13:28 | |
| wznoinsk | hi all | 13:28 |
| *** sc68cal has quit IRC | 13:28 | |
| *** michaelxin has joined #openstack-infra | 13:28 | |
| *** thiagop has joined #openstack-infra | 13:28 | |
| *** _ari_ has joined #openstack-infra | 13:29 | |
| wznoinsk | would anyone have problems with jenkins (when using zuul/gearman or not) when using OFFLINE_NODE_WHEN_COMPLETE parameter that a slave is being picked up by 2nd build when the 1st build is just finishing ? | 13:29 |
| wznoinsk | s/would/does | 13:29 |
| fungi | pabelanger: i'm mildly worried some of the network/dns config file blanking may happen too early now in 308309 but i couldn't spot anything in other elements we use that might be putting content there after it so probably unfounded. i went ahead and approved | 13:30 |
| pabelanger | AJaeger: rossella_s: It ran successfully, so just check the results are fine? | 13:30 |
| AJaeger | woot! | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on nova refresh failure https://review.openstack.org/308024 | 13:30 |
| * AJaeger is impatient | 13:30 | |
| AJaeger | rossella_s: party time! | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 13:31 |
| AJaeger | "Gerrit Dashboard links " is at the top of http://status.openstack.org//reviews/ now! | 13:31 |
| *** jordanP has joined #openstack-infra | 13:31 | |
| *** sc68cal has joined #openstack-infra | 13:31 | |
| pabelanger | fungi: Understood. Maybe we should move the setfiles script later in the rpm-distro element? | 13:31 |
| rossella_s | AJaeger, yaeh!! party time!!! | 13:32 |
| AJaeger | pabelanger: how long did the job run? | 13:32 |
| pabelanger | AJaeger: Check again in 30mins to make sure it updates again | 13:32 |
| pabelanger | AJaeger: not sure, may 5-10mins | 13:32 |
| fungi | wznoinsk: OFFLINE_NODE_WHEN_COMPLETE=1 is supposed to be handled by the jenkins-gearman plugin to make sure the worker is offline before it can pick up new jobs, so unless there's a bug i'm unaware of then no. the only times we see it happen are when restarting jenkins, since it will attempt to bring any offline slaves it knows about back online as it starts | 13:32 |
| AJaeger | pabelanger: ah, that's ok... | 13:32 |
| *** amitgandhinz has joined #openstack-infra | 13:32 | |
| rossella_s | pabelanger, the results are perfect, thanks a lot!!! thanks everybody for your help fungi clarkb jeblair AJaeger ! | 13:32 |
| AJaeger | rossella_s: let's monitor that it runs fine via cron now... | 13:33 |
| pabelanger | AJaeger: rossella_s: okay, just confirmed cron is running it right now | 13:33 |
| *** piet has quit IRC | 13:33 | |
| AJaeger | pabelanger: next time remind me that I'm impatient and should wait an hour, please ;) | 13:33 |
| pabelanger | AJaeger: np | 13:33 |
| fungi | pabelanger: we'll see how this goes, but yes i have a feeling some of the file blanking needs to go back to 99 just for future-proofing. also i spotted an incorrect comment in one of those scripts which could be cleaned up | 13:34 |
| rossella_s | AJaeger, pabelanger I have to go now, I will be back in a couple of hours, hope it runs fine otherwise I can fix it later | 13:34 |
| *** ivc_ has joined #openstack-infra | 13:34 | |
| AJaeger | thanks, rossella_s | 13:34 |
| pabelanger | fungi: okay | 13:34 |
| *** vgridnev has quit IRC | 13:34 | |
| *** psilvad has quit IRC | 13:34 | |
| EmilienM | pabelanger: ZUUL_BRANCH is also empty in periodic jobs. Where should I set it? | 13:34 |
| *** psilvad_ has joined #openstack-infra | 13:34 | |
| *** psilvad_ has quit IRC | 13:34 | |
| *** rossella_s has quit IRC | 13:34 | |
| *** psilvad_ has joined #openstack-infra | 13:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Lower our nodepool-base finalise.d script https://review.openstack.org/308309 | 13:35 |
| pabelanger | EmilienM: I'd have to see how other periodic jobs work. I've never set one up | 13:35 |
| pabelanger | EmilienM: do you need something other then master? | 13:35 |
| EmilienM | I don't understand why ZUUL_REF & ZUUL_BRANCH are empty in periodic jobs | 13:35 |
| EmilienM | well, for now, I'm happy if we deploy master | 13:35 |
| AJaeger | EmilienM: there'S no reference to check out, so it should take HEAD | 13:35 |
| AJaeger | It'S not associated with a specific change that is specified by reference and branch, EmilienM | 13:36 |
| EmilienM | so is it fine if I set both --zuul-ref and --zuul-branch to None? | 13:36 |
| AJaeger | ref to None and branch to master | 13:36 |
| EmilienM | oki | 13:36 |
| wznoinsk | fungi thanks, I'll verify it's actually a problem or some misconfig | 13:37 |
| clarkb | pabelanger: ya it was an ordering thing. ianw_ has a change up to dib to push the selinux script back to 99ish then we can move ours to 90ish and be good | 13:37 |
| *** nmagnezi has quit IRC | 13:37 | |
| *** obondarev_ has quit IRC | 13:37 | |
| *** esker has quit IRC | 13:37 | |
| *** vgridnev has joined #openstack-infra | 13:38 | |
| *** edmondsw has joined #openstack-infra | 13:38 | |
| fungi | clarkb: should i emergency revert 308309 then and wait for ianw_'s patch in dib to land? | 13:38 |
| *** tonytan4ever has joined #openstack-infra | 13:39 | |
| fungi | we still have a few minutes before image builds kick off for the day | 13:39 |
| *** obondarev has joined #openstack-infra | 13:39 | |
| pabelanger | clarkb: Ah, missed that. Let me find the review | 13:39 |
| fungi | or just take this as an opportunity to see how it plays out and then reorder them again after the patch lands in dib and gets a release? | 13:39 |
| clarkb | ya I dont think we want 308309 | 13:40 |
| fungi | reverting. i'll directly enqueue into the gate | 13:40 |
| *** abregman has quit IRC | 13:40 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Revert "Lower our nodepool-base finalise.d script" https://review.openstack.org/308331 | 13:40 |
| *** samueldmq has quit IRC | 13:41 | |
| *** abregman has joined #openstack-infra | 13:41 | |
| clarkb | https://review.openstack.org/#/c/308105/ was my interim option | 13:41 |
| fungi | actually i'll just emergency merge that. it's unwinding the last commit on the branch so doesn't actually need retesting | 13:41 |
| *** denisra has joined #openstack-infra | 13:41 | |
| *** samueldmq has joined #openstack-infra | 13:41 | |
| EmilienM | pabelanger, AJaeger: https://review.openstack.org/308333 | 13:41 |
| *** apoorvad has joined #openstack-infra | 13:41 | |
| clarkb | but ya it needs a small update as auggested | 13:42 |
| AJaeger | EmilienM: go for it ;) | 13:42 |
| wznoinsk | fungi would that be expected when I manually stop a build i njenkins build queue and hit rebuild or something new comes in it may happen the same slave is being used (assuming gearman plugin didn't make it to put this slave offline yet) ? | 13:43 |
| pabelanger | EmilienM: LGTM | 13:43 |
| *** samueldmq has quit IRC | 13:43 | |
| EmilienM | pabelanger, AJaeger: thanks for your help guys | 13:43 |
| fungi | grr, getting gerrit caches flushed so it picks up my membership is taking longer than running jobs in the gate would have | 13:43 |
| *** psilvad_ has quit IRC | 13:44 | |
| *** psilvad_ has joined #openstack-infra | 13:44 | |
| *** ivc_ has quit IRC | 13:44 | |
| *** dimtruck is now known as zz_dimtruck | 13:44 | |
| mtreinish | oh, cool it looks like github understands gpg signed commits now | 13:45 |
| mordred | fungi: you want me to emergency merge? | 13:45 |
| mtreinish | most of my commits now say "verified" next to them... | 13:45 |
| fungi | mordred: just did. gertty is syncing now | 13:45 |
| mordred | kk | 13:45 |
| mordred | mtreinish: I just started signing my commits last week too | 13:46 |
| fungi | oh, though gertty has decided it's in an offline sync suddenly | 13:46 |
| *** pradk has quit IRC | 13:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Lower our nodepool-base finalise.d script" https://review.openstack.org/308331 | 13:46 |
| *** apoorvad has quit IRC | 13:46 | |
| fungi | okay, done | 13:46 |
| fungi | should be in place well before the nodepool builder needs it | 13:47 |
| *** njohnston is now known as njohnston__ | 13:47 | |
| *** njohnston__ is now known as njohnston_afk | 13:47 | |
| mtreinish | mordred: cool, I started doing a while ago but there was never any easily visible artifact of that before | 13:47 |
| fungi | wznoinsk: i don't think i've ever seen manually stopping a build cause that, but it's also probably not a situation we test so maybe? | 13:49 |
| *** gongysh has quit IRC | 13:50 | |
| *** arxcruz has quit IRC | 13:50 | |
| *** ayoung has joined #openstack-infra | 13:50 | |
| *** psilvad_ has quit IRC | 13:51 | |
| *** psilvad_ has joined #openstack-infra | 13:51 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones https://review.openstack.org/302770 | 13:51 |
| mordred | mtreinish: that is neat | 13:51 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: WIP: Add job validation and latest link to magnum image upload https://review.openstack.org/308323 | 13:51 |
| wznoinsk | fungi: ok, I'll gather more info next time I see it | 13:51 |
| *** psilvad_ has quit IRC | 13:51 | |
| *** psilvad_ has joined #openstack-infra | 13:51 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 13:52 |
| *** links has joined #openstack-infra | 13:52 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: WIP: Add job validation and latest link to magnum image upload https://review.openstack.org/308323 | 13:53 |
| *** kengo_sakai has joined #openstack-infra | 13:54 | |
| *** _amrith_ is now known as amrith | 13:55 | |
| openstackgerrit | Merged openstack-dev/pbr: Fix soabi tests with pypy https://review.openstack.org/302045 | 13:55 |
| *** avarner has joined #openstack-infra | 13:56 | |
| *** kengo_sakai has quit IRC | 13:56 | |
| *** kengo_sakai has joined #openstack-infra | 13:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add releasenote jobs to OpenStack-Ansible repositories https://review.openstack.org/308325 | 13:58 |
| *** arxcruz has joined #openstack-infra | 13:58 | |
| *** xarses has quit IRC | 14:00 | |
| *** tlian has joined #openstack-infra | 14:01 | |
| AJaeger | pabelanger: the cron job did not update the web page so far | 14:02 |
| pabelanger | AJaeger: it is running again now | 14:03 |
| *** MarkAtwood has joined #openstack-infra | 14:03 | |
| *** cody-somerville has joined #openstack-infra | 14:04 | |
| *** denisra has quit IRC | 14:04 | |
| clarkb | pabelanger: the other issue I found was epel element uses el6 on el7 which also has a fix proposed to dib. And then the cache devstack script fix I wrote which you reviewed | 14:04 |
| clarkb | ~4 bugs in one go. :/ | 14:04 |
| AJaeger | pabelanger: ok, let me check in a few minutes again whether the website has changed. Thanks | 14:05 |
| *** gildub has quit IRC | 14:05 | |
| pabelanger | clarkb: ack | 14:05 |
| *** michaelxin has quit IRC | 14:05 | |
| *** denisra has joined #openstack-infra | 14:06 | |
| mordred | clarkb: btw - Shrews came up with a better idea for the volume invalidation thing that I think you're going to like better | 14:06 |
| *** baoli_ has quit IRC | 14:08 | |
| clarkb | mordred is that why things got wip'd? | 14:08 |
| *** vgridnev has quit IRC | 14:08 | |
| *** vgridnev has joined #openstack-infra | 14:09 | |
| mordred | yah | 14:09 |
| *** jaosorior has quit IRC | 14:09 | |
| clarkb | mordred I am going to have to ask for a test on https://review.openstack.org/#/c/307755/ too I think. Set up the fake so that you return an unattached fip, call cleanup floaters, check list | 14:09 |
| *** jaosorior has joined #openstack-infra | 14:09 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Have delete_server use the timed server list cache https://review.openstack.org/307887 | 14:10 |
| *** michaelxin has joined #openstack-infra | 14:10 | |
| Shrews | mordred: clarkb: ^^^ | 14:11 |
| *** cloudtrainme has joined #openstack-infra | 14:12 | |
| *** zz_dimtruck is now known as dimtruck | 14:12 | |
| *** gildub has joined #openstack-infra | 14:12 | |
| mordred | clarkb: hrm. there are no tests of cleanupLeakedInstances that I can copy from | 14:14 |
| *** kengo_sakai has quit IRC | 14:14 | |
| mordred | clarkb: ah - nevermind - htere is one of periodicCleanup | 14:15 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:15 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Retry floating ip deletion before deleting server https://review.openstack.org/307439 | 14:16 |
| *** haypo has joined #openstack-infra | 14:16 | |
| *** jaosorior has quit IRC | 14:16 | |
| haypo | hi. i would like to search | 14:16 |
| *** 14WAATBIX has joined #openstack-infra | 14:16 | |
| haypo | hi. i would like to search when gate-neutron-lbaasv1-dsvm-api failed for the first time on neutron_lbaas.tests.tempest.v1.api.admin.test_quotas.QuotasTest.test_lbaas_quotas using http://logstash.openstack.org/ | 14:16 |
| *** jaypipes has joined #openstack-infra | 14:16 | |
| AJaeger | pabelanger: is that cronjob finishing with success? I do not see the webpage getting updated ;( | 14:17 |
| haypo | here is a quick guide to use this tool? i would like to filter on the gate "gate-neutron-lbaasv1-dsvm-api" and search the pattern "neutron_lbaas.tests.tempest.v1.api.admin.test_quotas.QuotasTest.test_lbaas_quotas" | 14:17 |
| *** nwkarste_ has quit IRC | 14:17 | |
| *** nwkarsten has joined #openstack-infra | 14:18 | |
| *** gildub has quit IRC | 14:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Delete floating IP on nova refresh failure https://review.openstack.org/308024 | 14:19 |
| clarkb | haypo: you can AND together search terms. so build_name:"foo" AND message:"bar" AND build_status:"FAILURE" | 14:20 |
| Shrews | clarkb: basically, we are bad reviewers. one of mordred's changes uncovered some really bad code that caused the test failures | 14:20 |
| Shrews | 307887 fixes that | 14:20 |
| *** ivc_ has joined #openstack-infra | 14:21 | |
| clarkb | haypo we have some simple docs at http://docs.openstack.org/infra/system-config/logstash.html#query-logstash | 14:21 |
| clarkb | haypo I notice what @fields is in use there which is out of date we have a flat schema now, but the principles should hold | 14:21 |
| *** xarses has joined #openstack-infra | 14:21 | |
| haypo | clarkb: i always get the message "There were no results because no indices were found that match your selected time span". i tried your syntax, and then just message:"neutron_lbaas.tests.tempest.v1.api.admin.test_quotas" using "Last 1h" timeframe | 14:22 |
| haypo | clarkb: yeah, i was reading http://docs.openstack.org/infra/system-config/logstash.html i tried @message"error" syntax which doesn't work -:p | 14:22 |
| pabelanger | AJaeger: it finished, but I have no way to see if it was successful. I don't see any logginc | 14:22 |
| pabelanger | logging* | 14:22 |
| clarkb | you will probably have to query further in the past the cluster has been unhappy amd recoverying from a failed node for a few days | 14:22 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/306798 | 14:23 |
| haypo | clarkb: oh | 14:23 |
| pabelanger | AJaeger: I think we need to pipe to a log file | 14:23 |
| *** vgridnev has quit IRC | 14:24 | |
| AJaeger | pabelanger: ;( | 14:24 |
| *** denisra_ has joined #openstack-infra | 14:24 | |
| mordred | Shrews: I still have to think that the code in question was some sort of horrible rebase accident | 14:24 |
| openstackgerrit | Merged openstack/requirements: Change osprofiler version to osprofiler>=1.3.0 https://review.openstack.org/308171 | 14:24 |
| pabelanger | AJaeger: I see that milestone.dash and data_table.html were updated, but that appears to be it | 14:24 |
| pabelanger | Will help more once I get back from my errand | 14:25 |
| * pabelanger & | 14:25 | |
| AJaeger | pabelanger: thanks! | 14:25 |
| *** denisra has quit IRC | 14:26 | |
| Shrews | mordred: you gotta love broken code that works until you fix it :) | 14:27 |
| *** vgridnev has joined #openstack-infra | 14:28 | |
| clarkb | I was trying to figure out what was broken then firefox mobile crashed, again | 14:28 |
| clarkb | clearly gerrit should not be used on anything with less than 8gbof ram | 14:28 |
| haypo | clarkb: something is wrong. build_name:"gate-neutron-lbaasv1-dsvm-api" AND message:"test_quotas" gives me 0 hit, but I saw https://bugs.launchpad.net/neutron/+bug/1572592 on at least 5 recent changesets | 14:28 |
| openstack | Launchpad bug 1572592 in neutron "gate-neutron-lbaasv1-dsvm-api gate broken on Neutron LBaaS: fail to find a quota" [Undecided,New] | 14:28 |
| mordred | clarkb: one of these days I expect you to rage code an android version of gertty | 14:28 |
| clarkb | haypo: the cluster is unhappy and not indexing events quickly right now | 14:28 |
| clarkb | mordred: I have actually looked at running gertty over ssh | 14:29 |
| *** esker has joined #openstack-infra | 14:29 | |
| clarkb | problem is the keyboard takes up so much real estate | 14:29 |
| *** esker has quit IRC | 14:29 | |
| clarkb | haypo which is why I suggested changing the timeframe to look at the past | 14:30 |
| *** inc0 has joined #openstack-infra | 14:30 | |
| *** esker has joined #openstack-infra | 14:30 | |
| clarkb | I havent had time to figure out why the cluster recovery is so slow this time around | 14:30 |
| *** mylu has joined #openstack-infra | 14:31 | |
| Shrews | clarkb: if you need me to give you a tl;dr of the brokenness, i can attempt to do so | 14:31 |
| haypo | clarkb: it looks like the bug only occurs since a few hours | 14:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-reviewday: Create logs for update_review https://review.openstack.org/308367 | 14:31 |
| AJaeger | pabelanger: is this what you meant? ^ | 14:31 |
| *** zhurong has quit IRC | 14:32 | |
| *** Sukhdev has joined #openstack-infra | 14:32 | |
| *** dprince has quit IRC | 14:33 | |
| jeblair | jhesketh: pong | 14:33 |
| *** nwkarsten has quit IRC | 14:33 | |
| *** nwkarsten has joined #openstack-infra | 14:34 | |
| jhesketh | jeblair: just wanted to touch base on some zuul stuff if you have time? | 14:34 |
| jeblair | clarkb: gertty does handle mouse clicks... | 14:34 |
| *** tonytan4ever has quit IRC | 14:34 | |
| jeblair | jhesketh: yep; i briefly glanced at your patches, and i think they look good, i only didn't review them because i need a bit more time on the one where lots of stuff gets renamed | 14:34 |
| yolanda | infra-root, seems we have a problem with beaker-centos tests | 14:35 |
| yolanda | 2016-04-20 14:14:12.594 | centos-70-x64 14:14:12$ rpm --replacepkgs -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm | 14:35 |
| yolanda | 2016-04-20 14:14:12.689 | Retrieving http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm | 14:35 |
| yolanda | 2016-04-20 14:14:12.689 | Preparing... ######################################## | 14:35 |
| yolanda | 2016-04-20 14:14:12.690 | file /etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs from install of puppetlabs-release-7-12.noarch conflicts with file from package puppetlabs-release-7-10.noarch | 14:35 |
| jhesketh | jeblair: cool, thanks | 14:35 |
| *** tonytan4ever has joined #openstack-infra | 14:35 | |
| fungi | yolanda: interesting. i wonder why it's trying to install both of those packages | 14:35 |
| jhesketh | jeblair: with the v2 patches, I noticed you merged some and had a look at others. Do you think it'd be a good idea to add a few notes to that etherpad on which ones you think should be reviewed for merging? | 14:35 |
| yolanda | i will debug that | 14:36 |
| jhesketh | jeblair: so other core can +w where appropriate | 14:36 |
| jeblair | jhesketh: ah, i want to get through all of them, but then the jenkins thing happened | 14:36 |
| jhesketh | all good | 14:36 |
| AJaeger | yolanda: -i can install in parallel, -U updates | 14:36 |
| jeblair | jhesketh: (and now i'm focusing heavily on zuulv2.5, which is what i'm calling the effort to quickly replace jenkins with a bunch of code cobbled together from v3 and other places) | 14:36 |
| wznoinsk | fungi: http://paste.openstack.org/?reply_to=494828 | 14:36 |
| AJaeger | yolanda: not sure about --replacepkgs | 14:36 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Conditionally request a test env from geard https://review.openstack.org/306506 | 14:37 |
| *** yamahata has joined #openstack-infra | 14:37 | |
| *** Qiming has quit IRC | 14:37 | |
| jhesketh | jeblair: once more information is released about jenkins, won't we be able to turn console log access back on? | 14:37 |
| jhesketh | jeblair: couldn't we alternatively put a proxy in front of jenkins that allowed streaming console logs and no other pages? | 14:37 |
| wznoinsk | fungi: http://paste.openstack.org/show/494829/ even | 14:38 |
| jhesketh | jeblair: or lastly we could throw turbo-hipster at it | 14:38 |
| fungi | wznoinsk: did this start recently? have you confirmed that the 3rdparty-nfv-ci-intel-tests job run shows OFFLINE_NODE_WHEN_COMPLETE with a value of 1 in its job parameters? | 14:38 |
| jeblair | jhesketh: turbo-hipster would require us to finish the work to stop using scp or ftp at all; and regardless of where we are with logs, docs still isn't started. | 14:39 |
| *** electrofelix has quit IRC | 14:39 | |
| *** pradk_ is now known as pradk | 14:39 | |
| fungi | wznoinsk: like by going directly to the parameters page in the jenkins webui for that first completed job run? | 14:39 |
| *** jaosorior has joined #openstack-infra | 14:40 | |
| jhesketh | jeblair: yeah it's not a great solution, just thowing it out there if you're considering a "quick replace jenkins" | 14:40 |
| *** denisra_ has quit IRC | 14:40 | |
| jeblair | jhesketh: definitely considered it. as you know, this isn't my first choice. :) | 14:40 |
| jhesketh | jeblair: but I actually think a proxy would get access to logs and help protect us | 14:40 |
| wznoinsk | fungi yes, it is set in both jobs parameters, it's our sandbox and it was happening like this from time to time since we've deployed it 3weeks ago | 14:40 |
| *** kengo_sakai has joined #openstack-infra | 14:41 | |
| *** denisra has joined #openstack-infra | 14:41 | |
| fungi | wznoinsk: what version of jenkins and of the gearman-jenkins plugin are you running there? | 14:41 |
| wznoinsk | fungi open the 494828 | 14:41 |
| wznoinsk | 494829* | 14:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add FIP cleaning script https://review.openstack.org/308376 | 14:42 |
| mordred | clarkb, nibalizer: ^^ fwiw, this is what I use to clean fips from osic and bluebox currently | 14:42 |
| *** electrofelix has joined #openstack-infra | 14:42 | |
| *** sdake_ has quit IRC | 14:42 | |
| mordred | it could obviously be made much more robust, but it's been doing the job so far | 14:43 |
| *** apoorvad has joined #openstack-infra | 14:43 | |
| fungi | wznoinsk: oh, i see it at the bottom. missed it initially | 14:43 |
| *** ajmiller has quit IRC | 14:44 | |
| *** roxanagh_ has joined #openstack-infra | 14:44 | |
| *** claudiub has quit IRC | 14:44 | |
| *** ddieterly is now known as ddieterly[away] | 14:45 | |
| *** sdake_ has joined #openstack-infra | 14:45 | |
| *** tonytan_brb has joined #openstack-infra | 14:46 | |
| *** ddieterly[away] is now known as ddieterly | 14:46 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Make it easier to find 'add task' button https://review.openstack.org/294221 | 14:46 |
| *** apoorvad has quit IRC | 14:47 | |
| *** tosky has quit IRC | 14:47 | |
| *** Hal has joined #openstack-infra | 14:47 | |
| *** Guest74966 has quit IRC | 14:48 | |
| *** Hal is now known as Guest15985 | 14:48 | |
| *** watanabe_isao has joined #openstack-infra | 14:48 | |
| nibalizer | mordred: python you say | 14:48 |
| clarkb | arg ya our last index is from the 16th | 14:48 |
| clarkb | elasticsearch what is wrong with you | 14:48 |
| *** arxcruz has quit IRC | 14:48 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 14:48 |
| nibalizer | also that simply isn't the algorithm that clark and i do | 14:49 |
| fungi | clarkb: jeblair: do either of you recall what the gearman-plugin build we're running was testing out? it's versioned 0.1.3.9.9f3adc9 but there are only 8 commits on master since the 0.1.3 tag | 14:49 |
| nibalizer | we list all untattached ips, then list them again in a few minutes, anything that isn't attached at both times we elete | 14:49 |
| mordred | nibalizer: yes. you do not need to do that | 14:49 |
| clarkb | it is trying to recover an index from the 15th | 14:49 |
| mordred | nibalizer: at least, not for those two clouds | 14:49 |
| clarkb | and going nowhere | 14:49 |
| mordred | they use the neutron port-id floating ip creation routine | 14:49 |
| nibalizer | its not possible that a fip has been allocated but not attached | 14:49 |
| mordred | nope | 14:50 |
| mordred | not with the new code | 14:50 |
| *** kengo_sakai has quit IRC | 14:50 | |
| *** tonytan4ever has quit IRC | 14:50 | |
| clarkb | I "closed" the 15th's index which seems to have it doing new recovery steps | 14:50 |
| *** armax has joined #openstack-infra | 14:50 | |
| tlian | To run Keystone's test via tox with constraints, what is the correct way to run? Below is not running OK. | 14:50 |
| tlian | + tox --recreate -e py27-constraints,pep8 | 14:50 |
| tlian | ERROR: unknown environment 'py27-constraints' | 14:50 |
| *** apoorvad has joined #openstack-infra | 14:51 | |
| *** denisra has quit IRC | 14:51 | |
| jeblair | fungi: i thought i tagged master with 014? | 14:51 |
| nibalizer | mordred: aight | 14:51 |
| *** denisra has joined #openstack-infra | 14:51 | |
| *** haypo has left #openstack-infra | 14:51 | |
| fungi | jeblair: master is tagged 0.2.0 but regardless that's still a commit shy of the count on the version we're using | 14:51 |
| *** haypo has joined #openstack-infra | 14:51 | |
| mordred | nibalizer: you DO have to do your trick for the tripleo cloud, as it's using the nova fip api | 14:51 |
| nibalizer | ok | 14:51 |
| *** roxanagh_ has quit IRC | 14:51 | |
| AJaeger | tlian: tox -e py27 will use constraints, won't it? | 14:52 |
| clarkb | I have since reopened the index and now that shard is unassigned hopeflly the next time it attempts to assign it it will do things | 14:52 |
| haypo | clarkb: anyway, thanks for your help on logstash. FYI it looks like i found the root cause of the bug, and i proposed a fix: https://bugs.launchpad.net/neutron/+bug/1572592 | 14:52 |
| openstack | Launchpad bug 1572592 in neutron "gate-neutron-lbaasv1-dsvm-api gate broken on Neutron LBaaS: fail to find a quota" [Critical,In progress] - Assigned to Victor Stinner (victor-stinner) | 14:52 |
| *** anteaya has joined #openstack-infra | 14:52 | |
| jeblair | fungi: i have no idea then. | 14:52 |
| AJaeger | tlian: yes, it should. no need to do anything special... | 14:52 |
| tlian | Ajaeger: No, it's not. That's why I gave a try with -constraints | 14:52 |
| tlian | Ajaeger: let me try again. | 14:52 |
| dstanek | tlian: you mean upper constraints? | 14:53 |
| fungi | jeblair: the timestamp for /var/lib/jenkins/plugins/gearman-plugin on jenkins01 is about 30 minutes before that last change merged to the tip of master, so maybe that's all we were trying and the commit count is misleading for some reason | 14:53 |
| tlian | right | 14:53 |
| *** nwkarsten has quit IRC | 14:53 | |
| dstanek | there is no py27-constaints target in keystone | 14:53 |
| dstanek | tlian: that was recently added to master | 14:53 |
| dstanek | tlian: http://git.openstack.org/cgit/openstack/keystone/tree/tox.ini#n8 | 14:53 |
| AJaeger | dstanek, tlian: Current master head should use constraints | 14:53 |
| *** tosky has joined #openstack-infra | 14:54 | |
| *** Sukhdev has quit IRC | 14:54 | |
| fungi | wznoinsk: so your installed versions look basically the same as ours (we're on jenkins 1.642.3 instead of 1.642.4 and are running what i think is a pre-release equivalent of gearman-plugin 0.2.0) | 14:54 |
| AJaeger | tlian: -constraints jobs were a short time experiment, no project should have them. | 14:55 |
| *** armax_ has joined #openstack-infra | 14:55 | |
| *** nwkarsten has joined #openstack-infra | 14:55 | |
| *** apoorvad has quit IRC | 14:55 | |
| *** oanson has quit IRC | 14:55 | |
| clarkb | haypo: good to hear, sorry the cluster is in a bad state and didn't have what you are looking for | 14:55 |
| *** armax has quit IRC | 14:56 | |
| *** armax_ is now known as armax | 14:56 | |
| tlian | My real problem is that tests are throwing import error on testresources, so I thought tox was not installing https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt | 14:56 |
| fungi | wznoinsk: have you checked /var/log/jenkins/jenkins.log for any hints? the gearman plugin logs details in there | 14:56 |
| *** denisra_ has joined #openstack-infra | 14:56 | |
| fungi | along with the rest of jenkins | 14:56 |
| AJaeger | tlian: my .tox/py27/log/py27-1.log file for keystone shows the useage of constraints | 14:56 |
| *** ajmiller has joined #openstack-infra | 14:56 | |
| *** gongysh has joined #openstack-infra | 14:56 | |
| *** denisra has quit IRC | 14:56 | |
| fungi | tlian: are you trying to test a different branch than master? i think stable/kilo may have import issues at the moment | 14:57 |
| tlian | fungi: I am trying on master | 14:57 |
| tlian | .tox/py27/python is not importing 'testresources'. The package should have been installed by tox, isn't it? | 14:58 |
| *** vgridnev has quit IRC | 14:58 | |
| *** denisra_ has quit IRC | 14:58 | |
| *** denisra_ has joined #openstack-infra | 14:58 | |
| *** haypo has left #openstack-infra | 14:59 | |
| *** Swami has joined #openstack-infra | 14:59 | |
| tlian | I think testresources should get installed per | 14:59 |
| tlian | install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} | 14:59 |
| *** amitgandhinz has quit IRC | 14:59 | |
| *** dprince has joined #openstack-infra | 14:59 | |
| clarkb | tlian: you also need a requirements file | 14:59 |
| *** Guest64767 is now known as dansmith | 15:00 | |
| *** bpokorny has joined #openstack-infra | 15:00 | |
| tlian | I just use tox.ini from the master branch - nothing is modified. | 15:01 |
| AJaeger | tlian: The constraints are used, testresources is an additional problem. | 15:01 |
| tlian | deps = -r{toxinidir}/test-requirements.txt | 15:01 |
| *** denisra_ has quit IRC | 15:01 | |
| *** denisra_ has joined #openstack-infra | 15:01 | |
| anteaya | AJaeger: in this session are you needing more infra people than yourself? https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=toolchain | 15:02 |
| *** maishsk has quit IRC | 15:02 | |
| AJaeger | anteaya: that'S more docs toolchain than infra - and if infra questions come up, I can handle them. | 15:03 |
| anteaya | great, just confirming as it has infra in the title but wasn't on the infra schedule | 15:03 |
| AJaeger | anteaya: thanks for asking. No need for people to represent infra - still everybody that wants to learn about the scripts and tools we use to publish are fine... | 15:03 |
| fungi | yeah, the main reason i noted that as a potential conflict is so that i didn't book it at the same time as an infra session AJaeger was leading | 15:03 |
| anteaya | wonderful thank you | 15:03 |
| anteaya | fungi: ah how kind | 15:04 |
| AJaeger | anteaya: the tools are invoked by infra scripts. | 15:04 |
| AJaeger | fungi: thanks, appreciated! | 15:04 |
| anteaya | AJaeger: that makes sense | 15:04 |
| fungi | part of the variety of inputs i was taking into consideration when trying out various shuffling in our session schedule | 15:04 |
| * AJaeger will be back later... | 15:04 | |
| anteaya | I went through my schedule last night and so far minimal conflicts | 15:04 |
| anteaya | yay | 15:04 |
| fungi | sounds like i've mostly succeeded in that goal | 15:05 |
| *** asselin_ has joined #openstack-infra | 15:05 | |
| fungi | as noted in my announcement, puppet/ansible conflicts were the main problem (and that was really just around work sessions, not fishbowls) | 15:05 |
| anteaya | fungi: yay for mostly succeeded | 15:05 |
| crinkle | nibalizer: did you see yolanda's beaker centos issue in sb ^ do you think it's related to the repo derp they had last night? | 15:05 |
| yolanda | crinkle, i have a holded node now, and trying to debug | 15:06 |
| *** amit213 has quit IRC | 15:06 | |
| *** matrohon has quit IRC | 15:06 | |
| nibalizer | i did not see anything | 15:06 |
| nibalizer | yolanda: link? | 15:06 |
| yolanda | http://logs.openstack.org/77/308277/2/check/gate-openstackci-beaker-centos-7/0736982/console.html | 15:06 |
| *** tonytan_brb is now known as tonytan4ever | 15:06 | |
| crinkle | yolanda: they were missing a package last night and it was a result of something they changed in their release pipeline, so i would most likely blame puppet | 15:07 |
| fungi | in this case the error is specific to two puppet-release packages with different versions coexisting and conflicting over at least one file on the filesystem | 15:07 |
| fungi | but maybe that's a transitional issue | 15:08 |
| nibalizer | im also curious why we null route updates.puppetlabs.com | 15:08 |
| *** baoli has joined #openstack-infra | 15:08 | |
| yolanda | i have been hitting this during all day | 15:08 |
| *** amit213 has joined #openstack-infra | 15:08 | |
| fungi | i wonder if we're leaving that package preinstalled on the image, and the newer one is being installed by the job and conflicting? | 15:08 |
| *** dimtruck is now known as zz_dimtruck | 15:09 | |
| dstanek | anteaya: i got that personal ical generator done | 15:09 |
| fungi | in which case image updates will possibly cause the bug to be hidden again | 15:09 |
| *** jrmyjohn has joined #openstack-infra | 15:09 | |
| fungi | for whatever reason yum/rpm seems to be trying to add teh new version rather than replacing/upgrading the old one | 15:10 |
| anteaya | dstanek: I saw last night, dhellmann linked it when MarkAtwood asked about it | 15:10 |
| clarkb | nibalizer: we remove the puppet repo from our images when we are done with it to avoid hitting puppet on every update on every test | 15:10 |
| tlian | At run time, how should testresources get installed in the virtualevn? The package is NOT listed in test-requirment.txt. | 15:10 |
| *** cloudtrainme has quit IRC | 15:10 | |
| anteaya | dstanek: nicely done, does anne know about your script yet? | 15:10 |
| *** denisra has joined #openstack-infra | 15:10 | |
| dstanek | anteaya: perfect. if you try it and have issues let me know | 15:10 |
| clarkb | tlian: probably as a dependency of testr/subunit | 15:10 |
| *** mylu has quit IRC | 15:10 | |
| dstanek | anteaya: nope, but i'll email her now | 15:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: run-tarball/puppet: fix MODULE_NAME & support stable builds https://review.openstack.org/308394 | 15:10 |
| *** mylu has joined #openstack-infra | 15:11 | |
| anteaya | dstanek: I will do so, I put my schedule in a tomboy note as that is my preference, I dislike signing into anything I don't have to | 15:11 |
| nibalizer | clarkb: what does that have to do will null routing updates.puppetlabs.com? | 15:11 |
| anteaya | dstanek: but should that change I will let you know | 15:11 |
| *** denisra_ has quit IRC | 15:11 | |
| dstanek | tlian: are you having issues getting testresources installed? | 15:11 |
| clarkb | nibalizer: I do not know where or how we are null routing updates.puppetlabs.com | 15:11 |
| nibalizer | yolanda: they think the fixed it about 8am utc so if you are still hitting it then its not related | 15:11 |
| anteaya | dstanek: in any case, thanks for doing that | 15:11 |
| openstackgerrit | Sunjoo Park proposed openstack/python-jenkins: Fix get_running_builds for url with prefix https://review.openstack.org/308396 | 15:11 |
| clarkb | nibalizer: I am just pointing out that we do have code that disables the puppet repo in places | 15:11 |
| nibalizer | clarkb: http://logs.openstack.org/77/308277/2/check/gate-openstackci-beaker-centos-7/0736982/console.html#_2016-04-20_14_14_12_586 | 15:11 |
| yolanda | fungi, so the error complains about the key actually rpm --replacepkgs -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm | 15:11 |
| *** rajinir has joined #openstack-infra | 15:12 | |
| nibalizer | that's not where it fails | 15:12 |
| yolanda | bad paste | 15:12 |
| yolanda | file /etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs from install of puppetlabs-release-7-12.noarch conflicts with file ... | 15:12 |
| fungi | yolanda: it complains about the _file_ containing the key | 15:12 |
| tlian | dstanek: yes - testresources is not installed in virtual env. | 15:12 |
| clarkb | nibalizer: oh that is beaker not us I don't think | 15:12 |
| fungi | yolanda: it's a duplicate file conflict | 15:12 |
| fungi | yolanda: both those packages claim to install that same file | 15:12 |
| yolanda | yep, i meant that , conflict in installing that key | 15:13 |
| *** jaypipes has quit IRC | 15:13 | |
| *** hashar has quit IRC | 15:13 | |
| clarkb | ok I think the spice is flowing again re elasticsearch. It is very far behind but with weekend and summit coming up hopefully recovers | 15:14 |
| fungi | and the question is why it's seeing that as a conflict, since both of those seem like they should be different versions of the same package rather than different packages | 15:14 |
| *** mylu has quit IRC | 15:14 | |
| clarkb | of course this assumes it is able to place that shard without stalling again :/ | 15:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 15:14 |
| dstanek | tlian: i was having issues with that yesterday and had to update my tox and pip to the latest | 15:14 |
| *** mylu has joined #openstack-infra | 15:15 | |
| dstanek | tlian: i haven't had time to see why that actually worked, but it appears that the dep resolution isn't listening to our test-requirements.txt | 15:15 |
| tlian | dstanek: yeah I am not having them the latest. Let me trying upgrading them first. Thank | 15:15 |
| *** cloudtrainme has joined #openstack-infra | 15:15 | |
| *** maishsk has joined #openstack-infra | 15:16 | |
| dstanek | tlian: 'oslo.db[fixtures,mysql,postgresql]>=4.1.0 # Apache-2.0' appears to throw it off | 15:16 |
| yolanda | fungi looks as a problem with packagign? | 15:16 |
| fungi | yolanda: crinkle: nibalizer: could this be related to the centos issue we've got where epel 6 was being enabled as well as epel 7 | 15:16 |
| fungi | ? | 15:16 |
| * fungi is unsure the current state of that bug | 15:17 | |
| nibalizer | im unaware of that bug | 15:17 |
| tlian | dstanek: yes, exactly. oslo_db/sqlalchemy/test_base.py ... | 15:17 |
| *** Nikolay_St has quit IRC | 15:17 | |
| clarkb | fungi: no I don't think that bug ever affected our production images | 15:18 |
| nibalizer | infra-root per conversations in #openstack-meeting i'm going to cleanup runaway storyboard workers and restart the workers service | 15:18 |
| fungi | just checking, since i see some puppetlabs bug reports from last year indicating el6 and el7 can be installed side-by-side | 15:18 |
| clarkb | fungi: unless it was independently bugged somewhere else. I ran into it on my local builds because I added epel in | 15:18 |
| fungi | thanks nibalizer | 15:18 |
| *** e0ne has joined #openstack-infra | 15:18 | |
| nibalizer | SotK: Zara we are down to 5 wokres | 15:19 |
| SotK | \o/ thanks! | 15:19 |
| *** jordanP has quit IRC | 15:21 | |
| *** mylu_ has joined #openstack-infra | 15:22 | |
| *** mylu has quit IRC | 15:23 | |
| *** zz_dimtruck is now known as dimtruck | 15:24 | |
| yolanda | infra-root, so the solution i can think about, is to upgrade to 7-12 in our install_puppet, and rebuild | 15:24 |
| fungi | yolanda: crinkle: nibalizer: i checked a not-yet-used centos-7 worker and the puppetlabs-release-7-10.noarch is showing preinstalled on it along with puppet-3.6.2-3.el7.noarch | 15:24 |
| yolanda | install_puppet preinstalls 7-10 yes | 15:25 |
| fungi | yolanda: unless i'm completely mistaken, the solution is to make sure we uninstall puppet packages properly in our images | 15:25 |
| clarkb | isn't the real fix to get puppet to have packages that can upgrade? | 15:25 |
| clarkb | otherwise this is an issue on the git mirrors | 15:26 |
| *** salv-orlando has quit IRC | 15:26 | |
| fungi | clarkb: that too. or perhaps this is beaker not calling rpm correctly | 15:26 |
| fungi | but regardless, the puppeting we do to configure our worker images should _not_ be allowed to leak packages into the images | 15:26 |
| yolanda | rpm can be called with a flag to ignore file conflicts, but not doing that | 15:26 |
| morgan | clarkb: i like your idea of having packages that can be upgraded | 15:27 |
| morgan | yolanda: ++, oh god --no-deps --force, NEVER | 15:27 |
| fungi | yolanda: i wonder if -i with --replace-pkgs really does teh same thing as -U | 15:27 |
| *** esker has quit IRC | 15:28 | |
| mwhahaha | by the way the beaker thing has broken the puppet-openstack beaker stuff as well on centos7 | 15:28 |
| *** hashar has joined #openstack-infra | 15:28 | |
| *** esker has joined #openstack-infra | 15:28 | |
| yolanda | fungi , it needs a --replacefiles to ignore that kind of conflicts | 15:28 |
| yolanda | but we don't control it | 15:28 |
| fungi | yolanda: right, i'm asking why it's doing -i instead of -U | 15:28 |
| openstackgerrit | Dean Troyer proposed openstack/requirements: bump python-openstackclient in u-c to 2.4.0 https://review.openstack.org/308411 | 15:28 |
| *** burgerk has joined #openstack-infra | 15:29 | |
| fungi | i'm pretty sure -i doesn't imply -U when there's an existing package of the same name | 15:29 |
| yolanda | fugni, so in prepre node, we have an install_puppet there, shall we have a uninstall_puppet when finishing? | 15:29 |
| fungi | so on its face, it looks like a bug in beaker rather than in the puppetlabs packaging | 15:29 |
| *** jaosorior has quit IRC | 15:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Rework floating ip use test to be neutron based https://review.openstack.org/308412 | 15:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add public helper method for cleaning floating ips https://review.openstack.org/308413 | 15:29 |
| fungi | yolanda: that's a good question. i feel like we shouldn't leave puppet packaging preinstalled on any of our workers, but haven't looked at our other images yet to see if this is affecting more than just centos-7 | 15:30 |
| yolanda | fungi, i wonder if we can break some job if we remove puppet | 15:30 |
| yolanda | so, if they rely on puppet being present | 15:30 |
| fungi | right, i think jobs which use puppet should install puppet | 15:30 |
| *** baoli has quit IRC | 15:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 15:31 |
| fungi | indeed, our ubuntu-trusty images have facter, hiera, puppet, puppet-common and puppetlabs-release preinstalled | 15:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 15:31 |
| yolanda | yes, it installs puppet but never uninstalls it | 15:32 |
| mordred | clarkb, Shrews: ^^ those three above | 15:32 |
| yolanda | it will happen on all base images | 15:32 |
| fungi | so this is consistent, though i still think we should clean it up. others may disagree | 15:32 |
| crinkle | i think this is probably something that should be reported to puppetlabs whether or not we decide to uninstall things | 15:32 |
| mwhahaha | so last night they broke the release rpms I wonder if they changed something that has affected the beaker jobs | 15:32 |
| mordred | clarkb, Shrews: I tink that exposing the logic of doing that to nodepool is a layer violation - nodepool should not need to know about neutron fips just in that one spot | 15:32 |
| crinkle | mwhahaha: that's what i was thinking, that they broke something else | 15:32 |
| *** links has quit IRC | 15:32 | |
| mwhahaha | sounds like it's doing something different in the new packaging | 15:32 |
| yolanda | sounds good to me, but if we verify that we any of our jobs rely on puppet being present | 15:33 |
| mwhahaha | as the install helper/beaker code hasn't changed in the last 24 hours | 15:33 |
| yolanda | i bet that at least downstream, not having puppet should have broken us | 15:33 |
| * mwhahaha pokes Hunner | 15:33 | |
| crinkle | i don't think the puppet-openstack jobs rely on puppet being present (if they do that's a bug) | 15:33 |
| fungi | crinkle: yolanda: nibalizer: clarkb: i just tested, it's not a problem with the package itself. `rpm --replacepkgs -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm` fails for me while `rpm -Uvh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm` works fine | 15:33 |
| crinkle | interesting | 15:33 |
| fungi | i think beaker needs to be fixed to use rpm correctly | 15:33 |
| yolanda | either with -U, or at least , the --replacefiles flag | 15:34 |
| mordred | yah. -Uvh is what I'd use personally | 15:34 |
| fungi | my guess is that they thought --replacepkgs would do the same thing as -U | 15:34 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add message pointing to postci.log https://review.openstack.org/308419 | 15:34 |
| crinkle | i can look into that today and submit either a bug or a pr | 15:35 |
| *** ajmiller_ has joined #openstack-infra | 15:35 | |
| *** ajmiller_ has quit IRC | 15:35 | |
| *** esker has quit IRC | 15:36 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Don't remove packages from the delorean-deps repository https://review.openstack.org/308421 | 15:36 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove metric from previous runs https://review.openstack.org/308422 | 15:36 |
| *** ajmiller has quit IRC | 15:37 | |
| *** Sukhdev has joined #openstack-infra | 15:37 | |
| fungi | crinkle: just as a point of reference, -U works fine whether there is an existing version of the package installed or not, so it doesn't look like you have to choose between -U and -i (-U effectively implies -i) | 15:37 |
| crinkle | that makes sense | 15:37 |
| fungi | i just tested it by ininstalling the package and then installing it with rpm -U instead of rpm -i | 15:37 |
| *** scheuran has quit IRC | 15:37 | |
| fungi | to be sure (though it was my recollection already) | 15:37 |
| *** 14WAATBIX has quit IRC | 15:38 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add message pointing to postci.txt https://review.openstack.org/308419 | 15:38 |
| *** nikhil_k has joined #openstack-infra | 15:38 | |
| *** ajmiller has joined #openstack-infra | 15:38 | |
| dhellmann | did we lose the ability to look at draft html output as part of the jenkins lockdown, or did some files from https://review.openstack.org/#/c/305924/ just time out and get removed? (we're seeing 404 errors) | 15:39 |
| *** mriedem is now known as mriedem_haircut | 15:39 | |
| anteaya | dhellmann: I confirm http://docs-draft.openstack.org/24/305924/1/check/gate-releases-docs/7a08922//doc/build/html/ 404's for me | 15:40 |
| *** nikhil has quit IRC | 15:40 | |
| *** esikachev has quit IRC | 15:40 | |
| *** gongysh has quit IRC | 15:40 | |
| anteaya | dhellmann: that doesn't have anything to do with the jenkins opacity as far as I know | 15:40 |
| anteaya | dhellmann: I would suspect a log upload issue | 15:40 |
| dhellmann | anteaya : ok, thanks. I'll run a recheck and see if that regenerates the files | 15:41 |
| yolanda | fungi, so what do you think it can be better? for short term, i can see the beaker fix, or the rpm bump on preinstalled images | 15:41 |
| openstackgerrit | Daniel Berrange proposed openstack-infra/project-config: Add release-notes job for os-vif https://review.openstack.org/308423 | 15:41 |
| yolanda | and to be looked more carefully, uninstall puppet on base images | 15:41 |
| clarkb | recovering shards on the node that crashed doesn't really seem to go anywhere. It is indexing new shards but relocated shards or recovered shards don't seem to do anything | 15:41 |
| anteaya | okay, I'm looking at the job definition on that for how the logs are meant to be uploaded | 15:41 |
| fungi | yolanda: shortest term this goes away as soon as our running image updates complete | 15:42 |
| clarkb | I am going to try the close open trick on one of those indexes | 15:42 |
| yolanda | fungi, not sure. Because in install_puppet script, it is pinned to 7-10 | 15:42 |
| yolanda | function setup_puppet_rhel7 { | 15:42 |
| yolanda | local puppet_pkg="https://yum.puppetlabs.com/el/7/products/x86_64/puppetlabs-release-+346056416397-10.noarch.rpm" | 15:42 |
| yolanda | so if we want to get rid of conflicts, we need to bump there | 15:43 |
| fungi | yolanda: longer term, someone (like crinkle) getting a fix into beaker for their bug is a good idea and also we should have a broader discussion about making sure we clean up/remove lingering build-related packages on our worker images | 15:43 |
| crinkle | i think both fixing beaker and cleaning up build packages are good things | 15:43 |
| clarkb | ok the original recovering shard that wasn't going anywhere has an upward moving byte counter | 15:43 |
| fungi | yolanda: oh, we're explicitly installing -10 in our images, not latest puppetlabs-release? | 15:43 |
| clarkb | I think it may be actually recovering this time | 15:43 |
| *** maishsk has quit IRC | 15:43 | |
| yolanda | fungi, that's it | 15:43 |
| nibalizer | crinkle: ++ to fixing beaker | 15:44 |
| anteaya | dhellmann: to confirm, this job used to offer the log for gate-releases-docs and started 404'ing recently? | 15:44 |
| *** thiagop has quit IRC | 15:44 | |
| *** mylu_ has quit IRC | 15:44 | |
| *** mylu has joined #openstack-infra | 15:44 | |
| yolanda | crinkle, nibalizer, do you know why we pin explicitly to a rpm version' | 15:44 |
| yolanda | ? | 15:44 |
| fungi | yolanda: so yeah, fixing that would be good too. i wonder whether there's a reason we hardcode the specific package rev. i guess it's because we're relying on a download url | 15:44 |
| nibalizer | im expecting all of that to be handled by an install_puppet() macro anyways | 15:45 |
| *** sridhar_ram has joined #openstack-infra | 15:45 | |
| fungi | rather than adding a package channel to yum install it | 15:45 |
| nibalizer | so probably outside our direct control, unless we want to write/use our own install_puppet macro | 15:45 |
| crinkle | yolanda: i have no idea, i can't think of a good reason | 15:45 |
| yolanda | what fungi said... and there is no -latest version to rely on | 15:46 |
| yolanda | ok, i'll fix that and we can update images | 15:46 |
| mwhahaha | http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm | 15:46 |
| mwhahaha | that's the latest | 15:46 |
| mwhahaha | not sure why you're going down into el/7/products/... | 15:46 |
| yolanda | ah, there is a good one at upper level.. i honestly don't know | 15:47 |
| fungi | mwhahaha: oh, nice! | 15:47 |
| fungi | thanks for the tip | 15:47 |
| mwhahaha | that was the file that went missing last night :D | 15:47 |
| mwhahaha | but that's the 'latest' | 15:47 |
| mwhahaha | and what is getting installed by beaker | 15:47 |
| yolanda | yep, that will solve our issues | 15:47 |
| fungi | we likely didn't know it existed | 15:47 |
| fungi | or maybe it didn't exist when we originally wrote this bootstrapper | 15:48 |
| fungi | in the long-ago, in the before-time | 15:48 |
| *** mylu has quit IRC | 15:48 | |
| *** esikachev has joined #openstack-infra | 15:48 | |
| mwhahaha | :D | 15:48 |
| *** lakshmiS has joined #openstack-infra | 15:49 | |
| *** dimtruck is now known as zz_dimtruck | 15:49 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Consume the latest puppetlabs rpm when installing puppet https://review.openstack.org/308426 | 15:50 |
| anteaya | dhellmann: I'm not seeing a problem in the job setup, your setup for releases is very similar to governance and it is generating html: http://docs-draft.openstack.org/27/268727/9/check/gate-governance-docs/0a70e8c//doc/build/html/ | 15:50 |
| yolanda | ok pushed patch | 15:50 |
| *** tphummel has joined #openstack-infra | 15:51 | |
| *** baoli has joined #openstack-infra | 15:51 | |
| *** mikelk has quit IRC | 15:52 | |
| anteaya | dhellmann: sorry that was generated April 15th, I'll recheck that patch | 15:52 |
| dhellmann | anteaya : yeah, we've definitely had the output usable before | 15:52 |
| *** mylu has joined #openstack-infra | 15:53 | |
| anteaya | dhellmann: great, thank you | 15:53 |
| *** gyee has joined #openstack-infra | 15:54 | |
| *** nwkarsten has quit IRC | 15:54 | |
| *** nt has quit IRC | 15:54 | |
| *** nwkarsten has joined #openstack-infra | 15:55 | |
| clarkb | dhellmann: anteaya see http://logs.openstack.org/24/305924/1/check/gate-releases-docs/7a08922/console.html#_2016-04-18_17_16_26_084 | 15:55 |
| *** xyang1 has quit IRC | 15:56 | |
| anteaya | so the html file disappeared? | 15:56 |
| fungi | should be build/sphinx/html ? | 15:57 |
| *** mylu has quit IRC | 15:57 | |
| fungi | 2016-04-18 17:16:02.352 | creating /home/jenkins/workspace/gate-releases-docs/build/sphinx/html | 15:57 |
| clarkb | if so that is new | 15:58 |
| *** salv-orlando has joined #openstack-infra | 15:58 | |
| *** xyang1 has joined #openstack-infra | 15:58 | |
| *** mylu has joined #openstack-infra | 15:58 | |
| *** dizquierdo has joined #openstack-infra | 15:59 | |
| *** fernnest has joined #openstack-infra | 15:59 | |
| anteaya | yes | 15:59 |
| fungi | yeah, if it's supposed to copy contents into doc/build i'm not seeing that step output in the log | 15:59 |
| clarkb | fungi: no sphinx is supposed to write to doc/build | 15:59 |
| clarkb | and has since I originally configured the docs jobs almost 4 years ago | 16:00 |
| *** maestro1 has quit IRC | 16:00 | |
| fungi | did a recent pbr change it? are other projects being affected? | 16:00 |
| *** Rockyg has joined #openstack-infra | 16:00 | |
| clarkb | or maybe new sphinx? | 16:00 |
| *** vincentll has quit IRC | 16:01 | |
| yolanda | infra-root, can you review https://review.openstack.org/308426? | 16:01 |
| fungi | clarkb: sphinx 1.4.1 released on the 12th | 16:02 |
| fungi | dhellmann: when was the last time it seemed to work? | 16:03 |
| dhellmann | fungi : I'm not sure I could say. This is the first time we noticed it not working, but I don't look at the output for every patch and I haven't changed the way things render in a while | 16:04 |
| *** zz_dimtruck is now known as dimtruck | 16:04 | |
| *** ddieterly is now known as ddieterly[away] | 16:04 | |
| *** vhosakot has joined #openstack-infra | 16:05 | |
| *** abregman is now known as abregman|afk | 16:05 | |
| *** flepied1 has joined #openstack-infra | 16:06 | |
| anteaya | this is the right timeframe: https://review.openstack.org/#/c/306537/2/jenkins/jobs/python-jobs.yaml | 16:06 |
| anteaya | but I can't see how that would have changed the name of the path | 16:07 |
| *** dtantsur is now known as dtantsur|afk | 16:08 | |
| *** abregman|afk has quit IRC | 16:10 | |
| *** flepied has quit IRC | 16:10 | |
| *** austin81 has joined #openstack-infra | 16:10 | |
| anteaya | the console-log macro: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n893 | 16:10 |
| *** nt has joined #openstack-infra | 16:10 | |
| anteaya | is looking for 'logs/$LOG_PATH' | 16:11 |
| *** ddieterly[away] is now known as ddieterly | 16:11 | |
| *** ddieterly is now known as ddieterly[away] | 16:12 | |
| *** jrmyjohn has quit IRC | 16:12 | |
| anteaya | does source need to be specified for the -docs jobs? | 16:12 |
| *** zeih has quit IRC | 16:13 | |
| *** nwkarsten has quit IRC | 16:14 | |
| *** nwkarsten has joined #openstack-infra | 16:14 | |
| anteaya | fungi: I had a link above that produced html output on the 15th | 16:14 |
| *** ams_ has joined #openstack-infra | 16:15 | |
| *** lakshmiS has quit IRC | 16:15 | |
| fungi | anteaya: in that case, probably not caused by the sphinx release on the 12th | 16:15 |
| *** jrmyjohn has joined #openstack-infra | 16:15 | |
| anteaya | I would agree with that conclusion | 16:15 |
| *** lakshmiS has joined #openstack-infra | 16:15 | |
| anteaya | patchset 9 https://review.openstack.org/#/c/268727/9 | 16:15 |
| fungi | https://review.openstack.org/279238 changed a fair amount of logic in the docs build slave script, but merged back in february so also not a likely cause | 16:16 |
| *** hdd has joined #openstack-infra | 16:17 | |
| *** nadya has joined #openstack-infra | 16:17 | |
| Yarboa | wznoinsk, ping | 16:17 |
| anteaya | fungi: 279238 doesn't look like it changed anything that was pre-existing just added some new job templates | 16:18 |
| *** bpokorny_ has joined #openstack-infra | 16:18 | |
| anteaya | I'm stumped and need to search for food | 16:19 |
| *** jrmyjohn has quit IRC | 16:19 | |
| *** watanabe_isao has quit IRC | 16:21 | |
| mpaolino | anteaya, clarkb thanks also to your help I finally managed to request the creation of a new project through project-config | 16:21 |
| *** apoorvad has joined #openstack-infra | 16:21 | |
| mpaolino | now I am waiting for a review ;) | 16:21 |
| mpaolino | https://review.openstack.org/#/c/308156/3 | 16:22 |
| *** apoorvad has quit IRC | 16:22 | |
| *** bpokorny has quit IRC | 16:22 | |
| *** jistr has quit IRC | 16:22 | |
| *** apoorvad has joined #openstack-infra | 16:22 | |
| wznoinsk | Yarboa pong | 16:22 |
| wznoinsk | fungi: http://paste.openstack.org/show/494850/ | 16:22 |
| *** rlandy has quit IRC | 16:23 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 16:24 |
| pabelanger | and back, catching up on back scroll | 16:26 |
| pabelanger | AJaeger: Yup! Will review now | 16:26 |
| *** dizquierdo has quit IRC | 16:26 | |
| *** bnemec has quit IRC | 16:26 | |
| *** erlon_ has joined #openstack-infra | 16:29 | |
| *** Nikolay_St has joined #openstack-infra | 16:30 | |
| openstackgerrit | Chris Denneen proposed openstack-infra/jenkins-job-builder: Related-Bug: 2000560 https://review.openstack.org/308457 | 16:32 |
| pabelanger | yolanda: left comment about https | 16:33 |
| *** ddieterly[away] is now known as ddieterly | 16:33 | |
| *** psilvad has joined #openstack-infra | 16:33 | |
| *** mpaolino has quit IRC | 16:35 | |
| *** rlandy has joined #openstack-infra | 16:35 | |
| *** bnemec has joined #openstack-infra | 16:35 | |
| *** cschwede has quit IRC | 16:36 | |
| *** psilvad_ has quit IRC | 16:36 | |
| *** nemish has joined #openstack-infra | 16:38 | |
| nemish | Was wondering if someone could help me with Story 2000560 | 16:38 |
| nemish | I'm not good with python at all but think I'm most of the way there in creating this PR | 16:38 |
| nibalizer | nemish: are you the chris on that story? | 16:40 |
| *** mylu has quit IRC | 16:40 | |
| nibalizer | we use the gerrit workflow here, http://docs.openstack.org/infra/manual/developers.html | 16:40 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Consume the latest puppetlabs rpm when installing puppet https://review.openstack.org/308426 | 16:40 |
| *** Yarboa has quit IRC | 16:40 | |
| nibalizer | requires some setup, doesn't use github | 16:40 |
| *** ihrachys has quit IRC | 16:40 | |
| *** lucasagomes is now known as lucas-dinner | 16:40 | |
| nemish | nibalizer: yes i am Chris... I submitted the current state as Change 308457 in Gerrit | 16:41 |
| *** amrith is now known as _amrith_ | 16:41 | |
| nibalizer | cool | 16:41 |
| pleia2 | nemish: you'll want to update the story with the gerrit link, since that's what jjb contributors will look for :) | 16:42 |
| nibalizer | the tests are currently running | 16:42 |
| pleia2 | not github links | 16:42 |
| nibalizer | yes that | 16:42 |
| nemish | I just know it will fail :P since I can't figure out the error message... also the python tests fail since inside the yaml the {} needs to be escaped by having {{}} but in xml it's single | 16:42 |
| *** cschwede has joined #openstack-infra | 16:43 | |
| nemish | pleia2: done | 16:43 |
| anteaya | nemish: you might enjoy looking at https://wiki.openstack.org/wiki/GitCommitMessages | 16:43 |
| nemish | anteaya: don't like my commit message? I added the Related-Bug tag | 16:44 |
| anteaya | well I'm not a jjb reviewer | 16:44 |
| *** derekh has quit IRC | 16:44 | |
| pleia2 | nemish: a bug link is additional information, your commit message does need to say what it's doing and why anyway | 16:44 |
| anteaya | but generally I wouldn't consider a link to another resource a commit message, no | 16:44 |
| pleia2 | people need to be able to look at a commit log and glean what changed without clicking on links in a bug tracker | 16:45 |
| pleia2 | say, when they're stuck in a datacenter at the command line and jjb is broken ;) | 16:45 |
| * pleia2 back to testing nodepool docs | 16:46 | |
| *** Swami has quit IRC | 16:46 | |
| anteaya | dhellmann: I'm dry on ideas on where to look next | 16:46 |
| anteaya | pleia2: go you | 16:46 |
| *** mriedem_haircut is now known as mriedem | 16:46 | |
| *** Guest67082 has quit IRC | 16:46 | |
| anteaya | dhellmann: but there does appear to be a naming/path mismatch going on | 16:47 |
| anteaya | mriedem: how do you look? | 16:47 |
| anteaya | mriedem: will I recognize you? | 16:47 |
| *** esikachev has quit IRC | 16:47 | |
| *** _amrith_ is now known as amrith | 16:47 | |
| mriedem | anteaya: marvelous | 16:47 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Don't try to cancel the timer thread if it isn't alive https://review.openstack.org/308471 | 16:47 |
| anteaya | mriedem: oh so spiffy for summit | 16:47 |
| openstackgerrit | Ian Cordasco proposed openstack-dev/hacking: Add reno and release notes tracking https://review.openstack.org/308472 | 16:47 |
| *** pfallenop has quit IRC | 16:48 | |
| openstackgerrit | Chris Denneen proposed openstack-infra/jenkins-job-builder: Added BuildName to Builders in order to provide ability to update build name during build time https://review.openstack.org/308457 | 16:48 |
| pabelanger | EmilienM: lets some comments on your puppet tarballs patch | 16:48 |
| *** zul has quit IRC | 16:48 | |
| nemish | ok I updated the commit message | 16:49 |
| fungi | wznoinsk: does jenkins lose connectivity with the worker as a part of that job? e.g., does it take down the network interfaces, default routes, anything? | 16:50 |
| fungi | the reconnects are interesting | 16:50 |
| EmilienM | pabelanger: ok looking | 16:50 |
| *** watanabe_isao has joined #openstack-infra | 16:50 | |
| openstackgerrit | Merged openstack/requirements: Cap testresources<2.0.0 https://review.openstack.org/307858 | 16:52 |
| *** mylu has joined #openstack-infra | 16:52 | |
| *** ddieterly is now known as ddieterly[away] | 16:52 | |
| crinkle | yolanda: fungi https://github.com/puppetlabs/beaker/pull/1104 | 16:52 |
| *** dahoo has joined #openstack-infra | 16:52 | |
| clarkb | https://discuss.elastic.co/t/es-marking-and-sending-shard-failed-due-to-failed-recovery-in-enabling-replication/41742/7 this seems to be what is affecting our recovery | 16:53 |
| clarkb | we have plenty of disk so it isn't a disk issue | 16:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 16:53 |
| EmilienM | pabelanger: ^ | 16:53 |
| fungi | wznoinsk: anyway, it looks like either shutDownWorker is coming too late or fails silently (hard to tell since your jenkins and job logs don't seem to be synchronized, and there's no subsecond resolution) | 16:53 |
| tlian | tox tests per Keystone stable/mitaka is failing with VersionConflict onf Babel | 16:53 |
| tlian | pkg_resources.ContextualVersionConflict: (Babel 2.3.3 (/home/jenkins/jobs/stable-keystone-merge/workspace/.tox/py27/lib/python2.7/site-packages), Requirement.parse('Babel!=2.3.0,!=2.3.1,!=2.3.2,!=2.3.3,>=1.3'), set(['oslo.log'])) | 16:53 |
| tlian | Tox is pulling the Newton version of oslo.log from pypi since pypi doesn't segregate by OpenStack release and when that gets used in the virtual env for Mitaka testing badness ensues. | 16:53 |
| tlian | Souldn't stable/mitaka have install_command as the following so it pulls in specific version of a package ? | 16:53 |
| *** roxanagh_ has joined #openstack-infra | 16:53 | |
| fungi | wznoinsk: zaro might have insight into it once he's around | 16:53 |
| fungi | crinkle: thanks! | 16:53 |
| clarkb | we have java heap space issues on jenkins07 :/ | 16:54 |
| clarkb | I am tempted to just attempt restarting the service there | 16:54 |
| clarkb | any objects? | 16:54 |
| clarkb | er objections | 16:54 |
| *** chuck__ has joined #openstack-infra | 16:54 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Upload the overcloud images to cache https://review.openstack.org/306499 | 16:54 |
| dstanek | tlian: this issue was discussed on the dev list and i think there are patches being worked on to fix the stable branches | 16:54 |
| *** maishsk has joined #openstack-infra | 16:55 | |
| fungi | clarkb: i say got for it | 16:55 |
| tlian | dstanek: by any change do you have the ticket number? :) | 16:55 |
| *** Sukhdev has quit IRC | 16:55 | |
| EmilienM | pabelanger: err, I missed something | 16:56 |
| dstanek | tlian: no bug that i know of, but you can start here https://review.openstack.org/#/q/I8636e7c86c6c5c608429fab88e181108ae615db9 | 16:56 |
| pabelanger | EmilienM: ack | 16:56 |
| *** madhu_ak has joined #openstack-infra | 16:56 | |
| tlian | dstanke: tahnks | 16:57 |
| *** andymaier has quit IRC | 16:57 | |
| *** ddieterly[away] is now known as ddieterly | 16:58 | |
| dstanek | tlian: yw | 16:58 |
| nemish | question... maybe it's my browser but the Toggle CI button doesn't seem to do anything for me | 16:58 |
| *** amotoki has quit IRC | 16:59 | |
| nemish | I've never used Gerrit before so not sure what it's supposed to do | 16:59 |
| clarkb | and now we have lock failures... | 16:59 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 16:59 |
| fungi | nemish: it just hides/exposes comments left by ci systems if there are any | 16:59 |
| EmilienM | pabelanger: should be reviewable noww ^ | 16:59 |
| fungi | nemish: if the change was just uploaded, it can take some time for job results to appear | 16:59 |
| pabelanger | EmilienM: looking | 17:00 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Set author and committer explicitly https://review.openstack.org/222601 | 17:00 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Refactor Isolated Env to use in unit tests https://review.openstack.org/308476 | 17:00 |
| nemish | fungi: ok didn't know if it took me to the CI build process... thanks | 17:00 |
| clarkb | I wonder if the lock failures are the actual underlying issue | 17:00 |
| fungi | nemish: upwards of a couple hours at the moment since we're still a little backed up | 17:00 |
| nibalizer | clarkb: lock failrures in jenkins? | 17:01 |
| nemish | fungi: ok thanks | 17:01 |
| clarkb | nibalizer: no elasticsearch. I actually understand this now. 07 hasn't rejoined the cluster and 07 is the only one with a copy of 03 | 17:01 |
| clarkb | so the cluster can't lock the 03 shard of this index | 17:01 |
| clarkb | hence lock fails | 17:01 |
| clarkb | 07 is timing out joining the cluster :/ | 17:01 |
| anteaya | :( | 17:02 |
| clarkb | I have tcp connectivity between master and 07 so dunno what is going on | 17:02 |
| pabelanger | EmilienM: what about moving your post pipeline into puppet-check-jobs template? I think all your modules use it | 17:02 |
| pabelanger | then we can remove puppet-tarball-jobs | 17:02 |
| nibalizer | huh | 17:03 |
| * nibalizer pokes | 17:03 | |
| pabelanger | EmilienM: actually, looks like you are only doing tarballs for specific projects | 17:03 |
| EmilienM | pabelanger: puppet-check-jobs is used by infra puppet modules, no? | 17:04 |
| *** MarkAtwood has quit IRC | 17:04 | |
| jesusaur | clarkb: hows the load on the elected master? ive seen issues where the master took too long to respond to the join | 17:04 |
| EmilienM | looking at jenkins/jobs/projects.yaml | 17:04 |
| pabelanger | EmilienM: don't think so, we have infra-puppet-check-jobs | 17:04 |
| jesusaur | clarkb: iirc, it looked like a failed join followed by a dropped ping in the logs | 17:04 |
| clarkb | jesusaur: ~2 not bad | 17:04 |
| *** johnny has joined #openstack-infra | 17:04 | |
| EmilienM | pabelanger: the problem is some repos that run puppet-check-jobs are not modules | 17:04 |
| EmilienM | pabelanger: instack-undercloud for example | 17:05 |
| *** johnny is now known as Guest56909 | 17:05 | |
| *** electrofelix has quit IRC | 17:05 | |
| EmilienM | pabelanger: also puppet-accessbot is not released, etc | 17:05 |
| pabelanger | EmilienM: Ya, seeing that now. Any other template we could mash it into ? | 17:05 |
| *** maishsk has quit IRC | 17:05 | |
| EmilienM | pabelanger: not really | 17:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add PortList to Floating IP Clouds https://review.openstack.org/308480 | 17:05 |
| nibalizer | clarkb: ya nibz@elasticsearch07:~$ curl http://elasticsearch03.openstack.org:9200 works so | 17:06 |
| nibalizer | huh | 17:06 |
| *** bpokorny_ has quit IRC | 17:06 | |
| pabelanger | EmilienM: puppet-module-unit-jobs I guess, I mean, not a deal break. Just helps minimize the templates used by puppet jobs | 17:06 |
| AJaeger | dhellmann: If I build releases locally - using "tox -e python setup.py build_sphinx", I get a doc/build/html directory as expected | 17:07 |
| *** tosky has quit IRC | 17:07 | |
| *** bpokorny has joined #openstack-infra | 17:07 | |
| EmilienM | pabelanger: yeah, me too I like minimizing. The issue is just our templates are re-used by some stuff that are not released | 17:07 |
| anteaya | AJaeger: thank you | 17:07 |
| clarkb | nibalizer: ya, anyways I am about to hop on bus to meet you so maybe you can poke at it more? | 17:07 |
| mordred | pabelanger: ^^ added another stat to the graphs | 17:07 |
| * AJaeger doesn't understand why it fails ;( | 17:07 | |
| pabelanger | EmilienM: Lets wait to see what AJaeger says. Otherwise +2 | 17:07 |
| anteaya | AJaeger: so somehow the job is doing something else, or did on dhellmann's patch | 17:07 |
| clarkb | nibalizer: see tail -f /var/log/elasticsearch/elasticsearch.log on es07 | 17:07 |
| nibalizer | clarkb: fo-sho | 17:08 |
| clarkb | oh sweet it just found the master | 17:08 |
| AJaeger | anteaya: yeah - let me checkout dhellmann's change... | 17:08 |
| pabelanger | mordred: nice | 17:08 |
| anteaya | AJaeger: good idea | 17:08 |
| *** maishsk has joined #openstack-infra | 17:08 | |
| dhellmann | AJaeger , anteaya : we did make a change to the "docs" target in our tox.ini recently to run sphinx directly, but that's not used in jenkins, right? | 17:08 |
| EmilienM | pabelanger: nice suggestion on https://review.openstack.org/#/c/308394/1/jenkins/scripts/run-tarball.sh | 17:08 |
| anteaya | dhellmann: oh | 17:08 |
| anteaya | dhellmann: let's see that change | 17:08 |
| AJaeger | dhellmann: correct, see above for the comment to be used | 17:09 |
| *** maestro1 has joined #openstack-infra | 17:09 | |
| AJaeger | pabelanger: which chnage of EmilienM's should I review? | 17:09 |
| pabelanger | EmilienM: ya, seen below you were doing some magic with MODULE_VERSION | 17:09 |
| pabelanger | AJaeger: 307931 was looking at a way to reduce the amount of templates for puppet jobs | 17:10 |
| AJaeger | dhellmann: If I build releases locally with your change - using "tox -e python setup.py build_sphinx", you will get no html directory, but instead a man directory | 17:10 |
| AJaeger | dhellmann: so, something produces man by default instead of html | 17:10 |
| pabelanger | AJaeger: if you had thought. Nothing major | 17:10 |
| dhellmann | hmm | 17:10 |
| *** sputnik13 has joined #openstack-infra | 17:10 | |
| dhellmann | I'll bet I broke this, let me see | 17:10 |
| fungi | mordred: 97 leaked floating ips in osic. should i clean them up or do you want to dig deeper? | 17:10 |
| AJaeger | dhellmann: change I2aa0d844855d5a18646d0bc3907620544ae71be2 removes build_sphinx from setup.cfg | 17:11 |
| *** claudiub has joined #openstack-infra | 17:11 | |
| AJaeger | dhellmann: revert and be happy ;) | 17:11 |
| pabelanger | fungi: mordred: we had 99 this morning too | 17:11 |
| dhellmann | yep | 17:11 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-openstackid: Order of the classes parameters is refactored https://review.openstack.org/295320 | 17:11 |
| AJaeger | dhellmann: we do need build_sphinx in setup.cfg | 17:11 |
| anteaya | dhellmann AJaeger and what is https://review.openstack.org/#/c/305923/ expected to do? | 17:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add ServerGet task to nodepool graphs https://review.openstack.org/308483 | 17:12 |
| mordred | pabelanger: one more | 17:12 |
| mordred | fungi: I got em | 17:12 |
| fungi | k | 17:12 |
| AJaeger | anteaya: that's the broken patch. | 17:12 |
| *** yamahata has quit IRC | 17:12 | |
| AJaeger | dhellmann's change is stacked on top of it. | 17:12 |
| AJaeger | Thanks, anteaya | 17:12 |
| anteaya | ah | 17:13 |
| EmilienM | pabelanger: is there a python thing I can use to extract it? or just some magic bash? | 17:13 |
| anteaya | sorry I missed the dependency there | 17:13 |
| pabelanger | EmilienM: I'd have to google | 17:14 |
| *** mtanino has joined #openstack-infra | 17:14 | |
| *** icey has quit IRC | 17:14 | |
| EmilienM | pabelanger: i'll google it :-P | 17:15 |
| clarkb | nibalizer: the elastic recheck status page claims the cluster is green can you confirm directly? | 17:15 |
| *** maestro1 has quit IRC | 17:15 | |
| AJaeger | pabelanger, EmilienM you could add the post job to one of the existing puppet templates - and then perhaps rename it. | 17:15 |
| nibalizer | i have yellow | 17:15 |
| *** icey has joined #openstack-infra | 17:16 | |
| AJaeger | We have python-jobs that includes the branch-tarball, so why not have puppet-check-jobs include branch-tarball? | 17:16 |
| dhellmann | AJaeger, anteaya : thanks for digging into that one! | 17:16 |
| openstackgerrit | Brent Eagles proposed openstack-infra/tripleo-ci: Fail pingtest more quickly https://review.openstack.org/308486 | 17:17 |
| EmilienM | AJaeger: because puppet-check-jobs are used by some projects that 1) we don't want tarballs 2) are not puppet modules 3) are not managed by Puppet OpenStack | 17:17 |
| clarkb | nibalizer: ok thanks, it was red when I closrd laptop so that is an i.rpovement | 17:17 |
| pabelanger | AJaeger: EmilienM I think the best choice could be puppet-module-unit-jobs. But would need to double check | 17:19 |
| *** Guest17033 is now known as melwitt | 17:19 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-refstack: Order of the classes parameters is refactored https://review.openstack.org/295351 | 17:20 |
| nibalizer | yay | 17:20 |
| wznoinsk | my jenkins server time matches the ones in jenkins.log and in the paste I've sent... Inside the build I restart unbound but that hsouldn't disconnect the slave from network... I'll try to enable more logging and see | 17:23 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-reviewday: Create logs for update_review https://review.openstack.org/308367 | 17:23 |
| wznoinsk | ^ fungi | 17:24 |
| AJaeger | EmilienM: if there's no job that you use everywhere, then leave it as two templates... | 17:24 |
| pabelanger | clarkb: If interested: http://logs.openstack.org/44/301944/7/check/gate-tempest-dsvm-full-devstack-plugin-ceph/277d418/logs/devstacklog.txt.gz#_2016-04-19_11_50_34_704 ceph-deb-hammer mirror working in the gate. Thanks for driving it | 17:24 |
| AJaeger | EmilienM: your change looks fine... | 17:24 |
| anteaya | dhellmann: welcome | 17:24 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-openstackci: Order of the classes parameters is refactored https://review.openstack.org/295309 | 17:24 |
| AJaeger | pabelanger: fixed 308367 to handle the log file - could you review again,please? Note that I'm not a puppet expert... | 17:24 |
| clarkb | pabelanger: cool, we will have to do it all over again when the release after hammer happens :/ | 17:25 |
| pabelanger | AJaeger: sure | 17:25 |
| pabelanger | clarkb: good to know | 17:25 |
| clarkb | pabelanger: also we will need to think about how xenial factors in if they dont make xenial packages | 17:26 |
| fungi | wznoinsk: oh, i see where i was confused. in that paste your job fails at 15:14:25 but nodepool doesn't delete the worker until 15:15:26 (that is actually a built-in delay) | 17:26 |
| clarkb | (not sure what the ceph plan is maybe those will show up tomorrow at release) | 17:26 |
| fungi | wznoinsk: yet the second job starts immediately after the first ends, implying that OFFLINE_NODE_WHEN_COMPLETE is not being respected | 17:26 |
| *** nadya has quit IRC | 17:26 | |
| AJaeger | thanks, pabelanger | 17:27 |
| *** trown is now known as trown|lunch | 17:27 | |
| fungi | wznoinsk: so, yeah, i wonder whether that ioexception from the worker and subsequent reconnect is undoing the offlining | 17:28 |
| pabelanger | AJaeger: sorry, 1 nit | 17:28 |
| pabelanger | AJaeger: but good work | 17:28 |
| fungi | wznoinsk: if so, that seems like a bug in the plugin | 17:28 |
| *** rlandy has quit IRC | 17:28 | |
| pabelanger | clarkb: I vote for ceph team to poke the upstream mirror to get builds, then we just mirror them | 17:28 |
| clarkb | pabelanger: ya but we want to swit h to xenial | 17:29 |
| *** sridhar_ram has quit IRC | 17:29 | |
| *** MarkAtwood has joined #openstack-infra | 17:29 | |
| clarkb | do we wait for ceph or not? | 17:29 |
| clarkb | becomes bigger problem if/when we start doing mariadb and other deps | 17:29 |
| pabelanger | clarkb: not? _shrugs_ | 17:29 |
| EmilienM | pabelanger: in fact, we can't use setup.cfg, because not all modules (yet) have this file. | 17:30 |
| wznoinsk | fungi that was my first thought, also the WARNING about IOException from the paste is before the 1st build (that's doesn't put the slave offline at the end) starts | 17:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support job timeouts https://review.openstack.org/308488 | 17:30 |
| pabelanger | clarkb: I mean, it comes down to us poking upstream mirrors or passing it to our projects. | 17:30 |
| wznoinsk | fungi I though removing the jenkins label from the slave for the time of the build would probably do a good job in not using the slave | 17:30 |
| EmilienM | I have a question for infra-root. Is it possible for you to remove wrong puppet tarballs on tarballs.o.o when our scripts are in place and we know they work? | 17:31 |
| pabelanger | clarkb: but, I if ceph is not rolling xenial packages now, they won't be ready day 1. | 17:31 |
| fungi | wznoinsk: nevermind, you're right. i keep getting dyslexic mapping between 12-hour and 24-hour times in the two logs | 17:31 |
| pabelanger | dansmith: you have thoughts for ceph on xenial ^? | 17:31 |
| nibalizer | EmilienM: someone else can say more, but I have never gone in there and manually changed things | 17:31 |
| fungi | EmilienM: i'm happy to if you give me a heads up when you need it | 17:31 |
| clarkb | pabelanger: ya we wont switch until after aummit anyways so not a huge deal right now | 17:32 |
| *** maestro1 has joined #openstack-infra | 17:32 | |
| fungi | nibalizer: there's not really anything magic about tarballs.o.o. it's just serving up a bunch of files out of /srv/static/tarballs | 17:32 |
| EmilienM | fungi: thanks a lot, I'll when things will be clean-up | 17:32 |
| dansmith | pabelanger: yeah, not surprised if they don't have xenial packages yet, but we should make sure those guys know we're going to want that | 17:32 |
| *** yarkot has joined #openstack-infra | 17:32 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/jeepyb: Update mechanism of creating groups in database and managing access rights with jeepyb https://review.openstack.org/284843 | 17:33 |
| pabelanger | clarkb: so, how migrating to xenial on master look like? infra blindly moves all projects or is there a migration period? | 17:33 |
| *** yamahata has joined #openstack-infra | 17:33 | |
| pabelanger | how does* | 17:33 |
| fungi | dansmith: pabelanger: clarkb: any chance the wily packages install cleanly on xenial as a short-term solution? | 17:33 |
| dansmith | fungi: I have no idea | 17:33 |
| clarkb | pabelanger: last time we tested a cross section of the jobs then switched | 17:33 |
| clarkb | pabelanger: we wont do a split horizon across projects | 17:34 |
| fungi | pabelanger: it'll be only for changes to branches newer than stable/mitaka | 17:34 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Stop overriding the signal module in terminate() https://review.openstack.org/308492 | 17:34 |
| nibalizer | fungi: ok | 17:34 |
| clarkb | fungi thinking about the mess with the gearman function to select based on branch. one alternative is to have a trusty job and a xenial job | 17:35 |
| pabelanger | fungi: clarkb: Right, I guess my question is do we hold up migrating to ubuntu-xential for master is a project fails to build on it? I'm guess no | 17:35 |
| clarkb | then use the job branch regex to force them to the correct branches | 17:35 |
| clarkb | this explodes the job count | 17:35 |
| fungi | clarkb: yeah, that was one i considered as well | 17:35 |
| fungi | though maybe job count is less of an issue for zuul 2.5+? | 17:35 |
| EmilienM | AJaeger: I've worked on https://review.openstack.org/#/c/308394/ also, in case you wanted to check | 17:35 |
| clarkb | fungi: ya possibly | 17:35 |
| jeblair | fungi, clarkb: job count is still an issue | 17:36 |
| fungi | thanks, noted ;) | 17:36 |
| *** tosky has joined #openstack-infra | 17:36 | |
| clarkb | pabelanger: last time we tried to fix the bugs as brst as possible | 17:36 |
| jeblair | fungi, clarkb: we won't have the jjb part to worry about as much, but overwhelming gearman is still a concern | 17:36 |
| wznoinsk | out of curiosity... did you guys get around using grok at some stage? | 17:36 |
| openstackgerrit | Madhusudhan Kandadai proposed openstack-infra/project-config: Correct function name to run the right script https://review.openstack.org/308023 | 17:36 |
| clarkb | pabelanger: and in a couple cases gave up/moved on | 17:36 |
| fungi | jeblair: ahh, right, twice the gearman registrations | 17:36 |
| fungi | wznoinsk: nobody has worked on proposing an imlementation for it yet afaik | 17:36 |
| nibalizer | clarkb: did elasticsearch-head stop working? | 17:37 |
| pabelanger | clarkb: understood | 17:37 |
| nibalizer | it doesn't connect any more | 17:37 |
| fungi | nibalizer: are you using an ssh tunnel? | 17:37 |
| nibalizer | yes | 17:37 |
| nibalizer | bigdesk works | 17:37 |
| fungi | ahh | 17:37 |
| *** rlandy has joined #openstack-infra | 17:38 | |
| clarkb | nibalizer: are tou using master? yhat may be 2.x only | 17:38 |
| *** mriedem has quit IRC | 17:38 | |
| clarkb | I was using it earlier with whatever I had checked out | 17:38 |
| nibalizer | ahhh | 17:38 |
| *** mriedem has joined #openstack-infra | 17:38 | |
| jeblair | fungi, clarkb: having said that, we are going to increase the gearman function registration no matter what. for the set of affected jobs, we will increase by 1.5x if we use the parameter function. we will increase by 2x if we use duplicate jobs with branch matching. | 17:39 |
| clarkb | jeblair: fungi I almost thing we need a meta node type in zuul. 'default' | 17:40 |
| jeblair | (er, not so much 'increase by' but 'multiply the existing count by') | 17:40 |
| clarkb | and that is set per branch | 17:40 |
| pabelanger | yolanda: clarkb: nibalizer: when you have spare cycles: https://review.openstack.org/#/c/284874/ for our puppetboard.o.o migration to ubuntu-trusty | 17:40 |
| clarkb | then only overlay the extra selection on top of that | 17:40 |
| jeblair | clarkb: i think you could write the parameter function that way? | 17:41 |
| clarkb | maybe, I guess encode the default into the function | 17:41 |
| clarkb | I will have to poke at it | 17:41 |
| *** sdake__ has joined #openstack-infra | 17:41 | |
| *** ybathia has joined #openstack-infra | 17:42 | |
| jeblair | let's pseudo-code what we want to do, then figure out the best way to do it | 17:42 |
| *** sdake__ has quit IRC | 17:43 | |
| *** sdake_ has quit IRC | 17:43 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add validate_tarballs.sh https://review.openstack.org/307230 | 17:45 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make compare_tarball_diff.sh quieter https://review.openstack.org/307390 | 17:45 |
| *** sdake_ has joined #openstack-infra | 17:45 | |
| nibalizer | clarkb: initialzing_shards is dropping | 17:45 |
| nibalizer | so i think it will self heal | 17:46 |
| clarkb | awesome | 17:46 |
| openstackgerrit | Abel Lopez proposed openstack/diskimage-builder: Change to latest CentOS-6 image https://review.openstack.org/308504 | 17:46 |
| *** sambetts is now known as sambetts|afk | 17:46 | |
| *** kengo_sakai has joined #openstack-infra | 17:47 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add message pointing to postci.txt https://review.openstack.org/308419 | 17:48 |
| *** nwkarsten has quit IRC | 17:49 | |
| *** nwkarste_ has joined #openstack-infra | 17:49 | |
| *** avarner has quit IRC | 17:51 | |
| *** ifarkas has quit IRC | 17:51 | |
| *** kengo_sakai has quit IRC | 17:51 | |
| *** jaypipes has joined #openstack-infra | 17:52 | |
| *** ivc_ has quit IRC | 17:52 | |
| *** sdake__ has joined #openstack-infra | 17:53 | |
| *** jed56 has quit IRC | 17:53 | |
| *** daemontool has quit IRC | 17:53 | |
| anteaya | dhellmann: just to confirm, governance docs build html still: http://docs-draft.openstack.org/27/268727/9/check/gate-governance-docs/493023f//doc/build/html/ | 17:53 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-reviewday: Create logs for update_review https://review.openstack.org/308367 | 17:54 |
| AJaeger | pabelanger: fixed, thanks ^ | 17:54 |
| dhellmann | anteaya : yeah, the issue with the releases repo was with a patch where I removed some of the sphinx instructions, and then the patch I showed you depended on the bad patch | 17:55 |
| *** sdake_ has quit IRC | 17:57 | |
| *** ddieterly is now known as ddieterly[away] | 17:57 | |
| *** rama_y has joined #openstack-infra | 17:57 | |
| *** esker has joined #openstack-infra | 17:59 | |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Bumps os-win version to 0.4.2 https://review.openstack.org/308506 | 17:59 |
| *** pvaneck has joined #openstack-infra | 17:59 | |
| AJaeger | EmilienM: check the definition of '{name}-branch-tarball' in jenkins/jobs/python-jobs.yaml - and do it the same way. | 18:00 |
| openstackgerrit | Merged openstack/diskimage-builder: Don't stop dib-lint on first flake8 failure https://review.openstack.org/306510 | 18:00 |
| EmilienM | AJaeger: ok, will do | 18:00 |
| AJaeger | EmilienM: I want the same handling for both python and puppet tarballs in that script | 18:01 |
| *** ybathia has quit IRC | 18:01 | |
| *** rmoats has joined #openstack-infra | 18:02 | |
| *** dloi has left #openstack-infra | 18:03 | |
| *** tphummel has quit IRC | 18:03 | |
| *** rmoats has quit IRC | 18:04 | |
| *** jswarren_ has quit IRC | 18:04 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add message pointing to postci.txt https://review.openstack.org/308419 | 18:05 |
| wznoinsk | what's the best way to inject arbitrary variables (not supported by devstack-gate) into localrc in devstack_gate? write localrc in pre_test_hook and set KEEP_LOCALRC is one option I can see but not ideal possibly? | 18:06 |
| *** psilvad_ has joined #openstack-infra | 18:06 | |
| *** psilvad has quit IRC | 18:06 | |
| AJaeger | armax: so, https://review.openstack.org/#/c/268726/ neutron-dynamic-routing will be part of the neutron tent, correct? Could you +1 the governance change https://review.openstack.org/#/c/268727/ as well, please? | 18:06 |
| EmilienM | AJaeger: so you want me to add a job-group? | 18:07 |
| EmilienM | oh just saw your comment | 18:08 |
| AJaeger | EmilienM: not necessarily, I want the semantics of the shell script to stay the same. | 18:08 |
| armax | AJaeger: I was going to, but I imagined that 268726 won’t need 268727 to land at the same time, right? | 18:08 |
| EmilienM | AJaeger: sure thing | 18:08 |
| AJaeger | armax: no, it does not. But there was so much going back and forth on this change that I wanted to check with you ;) | 18:09 |
| armax | AJaeger: fair enough | 18:09 |
| AJaeger | armax: Ok, taking your word on it and +2 it now... | 18:09 |
| krotscheck | jeblair, mordred: EIther of you around to check a file for me? I need to know whether what I see in http://mirror.dfw.rax.openstack.org/npm/-/ is the same as /afs/.openstack.org/mirror/npm/-/index.json on update-mirror.openstack.org | 18:13 |
| *** nwkarste_ has quit IRC | 18:13 | |
| krotscheck | Particulary the sequence and latestSeq properties. | 18:13 |
| *** nwkarsten has joined #openstack-infra | 18:13 | |
| *** avarner has joined #openstack-infra | 18:14 | |
| *** sdake__ has quit IRC | 18:14 | |
| anteaya | dhellmann: yup | 18:15 |
| *** watanabe_isao has quit IRC | 18:15 | |
| *** MarkAtwood has quit IRC | 18:15 | |
| *** amitgandhinz has joined #openstack-infra | 18:15 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: run-tarball/puppet: fix MODULE_NAME https://review.openstack.org/308394 | 18:16 |
| *** tphummel has joined #openstack-infra | 18:17 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 18:17 |
| EmilienM | AJaeger: how does it look now? ^ | 18:17 |
| *** avarner_ has joined #openstack-infra | 18:18 | |
| *** mylu has quit IRC | 18:18 | |
| EmilienM | AJaeger: the real fix, will be in the last patch https://review.openstack.org/#/c/307931/ | 18:18 |
| *** hashar has quit IRC | 18:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Add releasenote jobs to OpenStack-Ansible repositories https://review.openstack.org/308325 | 18:19 |
| AJaeger | please rebase 307931 | 18:19 |
| AJaeger | EmilienM: ^ | 18:20 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 18:20 |
| *** ybathia has joined #openstack-infra | 18:20 | |
| AJaeger | ah, thanks | 18:20 |
| EmilienM | thanks to you | 18:20 |
| *** avarner has quit IRC | 18:21 | |
| AJaeger | EmilienM: I'm happy now ;) | 18:22 |
| *** Sukhdev has joined #openstack-infra | 18:23 | |
| *** hashar has joined #openstack-infra | 18:23 | |
| EmilienM | AJaeger: me too, we have some operators who want tarballs | 18:23 |
| *** Sukhdev has quit IRC | 18:24 | |
| pleia2 | clarkb: ran through the nodepool docs patch (179840) earlier and left comments | 18:25 |
| *** ayoung has quit IRC | 18:25 | |
| *** mylu has joined #openstack-infra | 18:25 | |
| *** yolanda has quit IRC | 18:26 | |
| *** Sukhdev has joined #openstack-infra | 18:28 | |
| clarkb | pleia2: thanks | 18:28 |
| AJaeger | fungi, pabelanger, nibalizer: Looking at the puppet install patch, I wonder about the logic. | 18:29 |
| *** yolanda has joined #openstack-infra | 18:29 | |
| AJaeger | We do in line 144: "yum install -y redhat-lsb-core git puppet" | 18:29 |
| AJaeger | and then two lines below "rpm -ihv $puppet_pkg" | 18:29 |
| AJaeger | Why the puppet arleady earlier? | 18:29 |
| *** mylu has quit IRC | 18:30 | |
| *** sputnik13 has quit IRC | 18:30 | |
| *** yamamoto has quit IRC | 18:31 | |
| krotscheck | jeblair, mordred: n/m - the datestamp gives it away | 18:32 |
| *** mylu has joined #openstack-infra | 18:32 | |
| *** degorenko is now known as _degorenko|afk | 18:32 | |
| *** nwkarste_ has joined #openstack-infra | 18:33 | |
| *** nwkarsten has quit IRC | 18:34 | |
| pabelanger | AJaeger: I'd have to test, but possible picking up some dependencies that puppetlabs is / was missing. | 18:34 |
| *** mylu has quit IRC | 18:35 | |
| pabelanger | AJaeger: we should likely review the functionality and maybe remove it in a follow up patch | 18:35 |
| mtreinish | jeblair, clarkb: did you want to try rolling out the serialized subunit worker on 1 of the 2 workers today? | 18:35 |
| *** daemontool has joined #openstack-infra | 18:35 | |
| AJaeger | pabelanger: I double checked the log file, the failure is elsewhere- still I'm not sure what's going on there. And yolanda's patch is failing in the gate, so it does not see to help ;( | 18:36 |
| *** mylu has joined #openstack-infra | 18:36 | |
| pabelanger | AJaeger: Ya, we'll need new dibs | 18:36 |
| *** rama_y has left #openstack-infra | 18:37 | |
| AJaeger | argh;( | 18:37 |
| *** sridhar_ram has joined #openstack-infra | 18:39 | |
| *** woodster_ has joined #openstack-infra | 18:39 | |
| pabelanger | clarkb: greghaynes: do you mind looking at https://review.openstack.org/#/c/308128/ again? setfiles renames for dib | 18:39 |
| *** HeOS has quit IRC | 18:40 | |
| *** maestro1 has quit IRC | 18:40 | |
| *** hashar has quit IRC | 18:41 | |
| *** mylu has quit IRC | 18:41 | |
| *** hashar has joined #openstack-infra | 18:42 | |
| *** oanson has joined #openstack-infra | 18:42 | |
| *** hashar has quit IRC | 18:43 | |
| *** ociuhandu has quit IRC | 18:43 | |
| *** ociuhandu has joined #openstack-infra | 18:43 | |
| *** austin81 has left #openstack-infra | 18:44 | |
| *** hashar has joined #openstack-infra | 18:44 | |
| *** mylu has joined #openstack-infra | 18:45 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:45 | |
| *** trown|lunch is now known as trown | 18:46 | |
| *** doug-fish has quit IRC | 18:47 | |
| *** pfallenop has joined #openstack-infra | 18:47 | |
| *** kengo_sakai has joined #openstack-infra | 18:48 | |
| *** sputnik13 has joined #openstack-infra | 18:50 | |
| *** kengo_sakai has quit IRC | 18:52 | |
| *** ddieterly[away] is now known as ddieterly | 18:54 | |
| *** nwkarste_ has quit IRC | 18:54 | |
| *** stewie925_ has joined #openstack-infra | 18:55 | |
| *** nwkarsten has joined #openstack-infra | 18:55 | |
| *** e0ne has quit IRC | 18:55 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 18:56 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:56 | |
| *** yolanda has quit IRC | 18:58 | |
| *** ociuhandu has quit IRC | 18:59 | |
| *** dimtruck is now known as zz_dimtruck | 18:59 | |
| *** tqtran has joined #openstack-infra | 19:00 | |
| *** salv-orlando has quit IRC | 19:03 | |
| *** zz_dimtruck is now known as dimtruck | 19:03 | |
| *** pcrews_ has quit IRC | 19:04 | |
| *** pcrews_ has joined #openstack-infra | 19:05 | |
| *** pcrews_ has quit IRC | 19:05 | |
| *** rcernin has quit IRC | 19:05 | |
| *** aimeeU has quit IRC | 19:06 | |
| anteaya | I am offline until Friday | 19:07 |
| *** anteaya has quit IRC | 19:07 | |
| *** doug-fis_ has joined #openstack-infra | 19:07 | |
| mordred | AJaeger: have fun! | 19:07 |
| mordred | Shrews: you hate me and progress and bunny rabbits | 19:08 |
| sridhar_ram | infra team: looking for guidance.. tacker dsvm functional gate tests spawns lots of VMs and it is hitting the max 10 VM instance quota limit. Any easy way to bump the limit up in the gate tests ? | 19:08 |
| *** jrmyjohn has joined #openstack-infra | 19:08 | |
| mordred | Shrews: (just kidding) | 19:09 |
| mordred | Shrews: for https://review.openstack.org/#/c/307887 - I +2'd because you changed it. maybe you +2 too because I wrote some of it - and then we get clarkb or jeblair to +A? | 19:09 |
| Shrews | mordred: not true. i love bunny rabbits | 19:09 |
| *** mylu has quit IRC | 19:10 | |
| *** sridhar_ram_ has joined #openstack-infra | 19:11 | |
| *** avarner_ has quit IRC | 19:11 | |
| *** rocky_g has joined #openstack-infra | 19:12 | |
| mordred | sridhar_ram: so - you should have access to the admin credentials of the devstack you're running against | 19:12 |
| *** avarner_ has joined #openstack-infra | 19:12 | |
| *** jrmyjohn has quit IRC | 19:13 | |
| mordred | sridhar_ram: they'll be in the ~/.config/openstack/clouds.yaml file | 19:13 |
| mordred | which will allow you to do whatever you want to | 19:13 |
| *** nwkarsten has quit IRC | 19:13 | |
| mordred | sridhar_ram: however, be aware that the machines we use for devstack installation in the first place only have 8G of RAM | 19:13 |
| mordred | sridhar_ram: so there will be a real limit as to how many things you can boot just due to physical resource limitations | 19:13 |
| *** nwkarsten has joined #openstack-infra | 19:14 | |
| *** ihrachys has joined #openstack-infra | 19:14 | |
| fungi | and also there's no nested virt acceleration so you can get some nasty contention if you're doing much with the booted instances | 19:14 |
| sridhar_ram | mordred: alright, so I can just update my post_test_hook.sh to bump up the limit... fair enough. good to know the 8GB limit.. | 19:15 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/nodepool: Add a fake-secure.conf for use in testing https://review.openstack.org/308536 | 19:15 |
| mordred | sridhar_ram: yup! | 19:15 |
| mtreinish | fungi: so I gave the mod_proxy for rss thing a try, but I'm pretty sure I screwed up the regex: https://review.openstack.org/307905 | 19:15 |
| sridhar_ram | mordred: .. we are mostly using cirros now.. will be switching to starting using openwrt sometime soon.. | 19:15 |
| fungi | mtreinish: fwiw i usually end up having to try them out on a real webserver to be sure | 19:16 |
| sridhar_ram | mordred: btw, do you know a way to run func test serially ? i.e. disable concurrency | 19:16 |
| mordred | sridhar_ram: yah - it's a flag to testr | 19:16 |
| mordred | one sec | 19:16 |
| *** matrohon has joined #openstack-infra | 19:17 | |
| mtreinish | fungi: yeah I do the same | 19:17 |
| fungi | mtreinish: i usually install apache on my workstation and set up a dummy debug cgi or something to call from the config if i'm really concerned | 19:17 |
| mordred | sridhar_ram: testr run --concurrency=1 | 19:17 |
| sridhar_ram | mordred: cool.. will keep in my kitty.. | 19:17 |
| mtreinish | fungi: oh, ok I didn't do that | 19:17 |
| mtreinish | fungi: that requires effort :) | 19:17 |
| sridhar_ram | mordred: ideally didn't want to use it.. but it will help to unblock us at times | 19:18 |
| mtreinish | fungi: I can play around with that in a bit though, it shouldn't be too hard | 19:18 |
| *** ociuhandu has joined #openstack-infra | 19:18 | |
| sridhar_ram | mordred: thanks so much | 19:18 |
| fungi | mtreinish: or for lighter duty testing i just try similar patterns on a low-importance webserver i already run for personal sites | 19:18 |
| *** ihrachys_ has joined #openstack-infra | 19:18 | |
| mtreinish | ugh and I just noticed that the url from the js uses ~2F which urllib.unquote won't handle. I'm pretty sure it only likes %2F | 19:18 |
| mtreinish | I might have to mod_rewrite it... :( | 19:19 |
| fungi | oh, ew | 19:19 |
| sridhar_ram | mordred: I also see keystone .testr.conf referring to TEST_RUN_CONCURRENCY .. perhaps it feeds into testr concurrency flag | 19:19 |
| *** abregman has joined #openstack-infra | 19:19 | |
| *** mylu has joined #openstack-infra | 19:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Add PortList to Floating IP Clouds https://review.openstack.org/308480 | 19:20 |
| *** ihrachys has quit IRC | 19:21 | |
| fungi | mtreinish: i'd be concerned about relying on url escapes to prevent a / from being interpreted. may be better off to go with some other replacement character instead? | 19:21 |
| fungi | mtreinish: particularly if the request gets double-unescaped at some point when you only meant to single-unescape or something you could wind up with a different number of path components than you expected | 19:22 |
| mtreinish | fungi: the api server handles a real / I added the unescaping support to the api server to make the proxy work with just a regex | 19:22 |
| fungi | oh, got it | 19:23 |
| *** yolanda has joined #openstack-infra | 19:23 | |
| mtreinish | fungi: https://review.openstack.org/307915 | 19:23 |
| *** matrohon has quit IRC | 19:23 | |
| mtreinish | which is the depends on patch in that mod_proxy review | 19:23 |
| openstackgerrit | Ye Na Rhee proposed openstack-infra/grafyaml: Add support for time range https://review.openstack.org/308539 | 19:24 |
| *** pradk has quit IRC | 19:24 | |
| fungi | mtreinish: and yeah, (.+\/.+) might not match foo%2Fbar either, i'm not sure | 19:24 |
| fungi | also that's sure to be some sort of obscure emoticon | 19:25 |
| *** matrohon has joined #openstack-infra | 19:25 | |
| mtreinish | fungi: well I'm trying to match 'key/foo%2Fbar' there | 19:25 |
| fungi | aha | 19:25 |
| mtreinish | because the api request needs the key and value | 19:25 |
| mtreinish | and the value is often something like openstack/tempest | 19:25 |
| fungi | in which case, other way around, i also don't know for certain apache won't turn %2F into / before comparing | 19:25 |
| *** roxanagh_ has quit IRC | 19:26 | |
| *** salv-orlando has joined #openstack-infra | 19:26 | |
| fungi | mtreinish: though that explanation also doesn't jive with the pattern you gave as an example | 19:26 |
| *** sridhar_ram is now known as sridhar_ram_afk | 19:27 | |
| mtreinish | fungi: I just copied the url from my browser when navigating to the basic page I want to proxy: http://status.openstack.org/openstack-health/#/g/project/openstack~2Ftempest | 19:27 |
| fungi | since you're matching on project\/$(.+\/.+) | 19:27 |
| mtreinish | fungi: oh, yep that's wrong | 19:27 |
| fungi | and implying that you want the first .+ there to also match project | 19:27 |
| fungi | hence my confusion over what you were trying to make it do | 19:28 |
| mtreinish | yeah, I didn't realize I left project in there, my bad | 19:28 |
| mtreinish | that first .+ is supposed to match project :) | 19:28 |
| *** avarner_ has quit IRC | 19:28 | |
| *** Na3iL has joined #openstack-infra | 19:29 | |
| *** nwkarsten has quit IRC | 19:29 | |
| *** avarner has joined #openstack-infra | 19:29 | |
| fungi | okay, now i'm starting to understand the goal a little better | 19:29 |
| fungi | so my only remaining concern is over whether apache will do any transparent %-unescaping on its own | 19:31 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add openstack-health RSS proxy to api server https://review.openstack.org/307905 | 19:31 |
| mtreinish | fungi: ^^^ ok, fixed that first issue | 19:31 |
| fungi | which isn't something i've played with in the past to suss out | 19:31 |
| *** yamamoto has joined #openstack-infra | 19:31 | |
| mtreinish | fungi: yeah, I dunno, right now it doesn't look like it's even using %- anything on the frontend. the angular stuff seems to escape with ~-something | 19:32 |
| *** rcernin has joined #openstack-infra | 19:32 | |
| timothyb89 | mtreinish: will that proxy work with '#' characters? | 19:32 |
| timothyb89 | mtreinish: afaik browsers never send anything after the '#' to the server at all | 19:32 |
| *** e0ne has joined #openstack-infra | 19:33 | |
| mtreinish | timothyb89: then how does the frontend work? | 19:33 |
| timothyb89 | mtreinish: the frontend is just index.html, the rest is all figured out in js | 19:33 |
| mtreinish | timothyb89: I haven't tested it at all, I just copied a url added rss to it and tried to do a regex substition on it | 19:33 |
| timothyb89 | mtreinish: that said, we could probably turn on html5 links mode and remove the '#' entirely | 19:34 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Enable lbaasv2 jobs into check queue https://review.openstack.org/303017 | 19:35 |
| *** pradk has joined #openstack-infra | 19:35 | |
| *** matrohon has quit IRC | 19:36 | |
| *** nwkarsten has joined #openstack-infra | 19:36 | |
| mtreinish | fungi: unrelated, looking at o-h its saying the subunit2sql is > 1 day old. Is there anything in the worker logs? | 19:36 |
| fungi | mtreinish: i'll check. also i can get you and timothyb89 an access log entry for what one of those urls currently gets interpreted as on the apache side | 19:37 |
| mtreinish | fungi: oh cool, that would probably help figure out what we should be doing for this | 19:37 |
| fungi | mtreinish: is it still running on logstash.o.o normally, or did it move? | 19:38 |
| fungi | oh, subunit-worker01 | 19:38 |
| mtreinish | fungi: and subunit-worker02 now too | 19:38 |
| fungi | yeah, brain mush | 19:38 |
| *** matrohon has joined #openstack-infra | 19:39 | |
| *** yamamoto has quit IRC | 19:39 | |
| fungi | `ps auxww|grep -i gearman` on 01 returns no hits | 19:39 |
| *** apoorvad has quit IRC | 19:39 | |
| fungi | same on 02 | 19:39 |
| *** apoorvad has joined #openstack-infra | 19:40 | |
| fungi | seeing if they logged anything fun on the way down | 19:40 |
| *** ybathia has quit IRC | 19:40 | |
| mtreinish | fungi: I think the proc is: jenkins-subunit-worker | 19:40 |
| fungi | the initscript says DAEMON=/usr/local/bin/subunit-gearman-worker.py | 19:41 |
| mtreinish | oh but the python exec is subunit-gearman-worker.py | 19:41 |
| mtreinish | yep, so I guess it's not running :( | 19:41 |
| fungi | anyway, last log entry on 01 is this: | 19:41 |
| fungi | 2016-04-18 19:38:52,747 Get job; number of threads waiting for jobs: 1 | 19:41 |
| fungi | so at some point this weekend it silently died | 19:41 |
| fungi | on 02 it's... | 19:42 |
| fungi | 2016-04-18 19:38:38,398 Setting state to: GRAB_WAIT | 19:42 |
| fungi | so they both died around the same times | 19:42 |
| *** jrmyjohn has joined #openstack-infra | 19:42 | |
| fungi | with nothing logged to indicate they did so | 19:43 |
| mtreinish | fungi: hmm, yeah that's not a lot to go on. Anything in syslog or dmesg, maybe oomkiller (although if its waiting for a job I would expect an empty queue so no big memory consumption | 19:43 |
| fungi | yeah, i've moved on to checking those possibilities | 19:43 |
| fungi | nothing at all recent in dmesg on either machine. they weren't rebooted and there was no oom-killer invoked | 19:44 |
| mtreinish | fungi: hmm, ok. I guess it's just cosmic rays or something | 19:44 |
| fungi | puppet was updating them at Apr 18 19:38 | 19:45 |
| mtreinish | fungi: so on the 15th we did land a single threaded refactor: https://review.openstack.org/#/c/303583/ | 19:45 |
| fungi | (/Stage[main]/Subunit2sql/Package[subunit2sql]/ensure) ensure changed '1.4.0' to '1.5.0' | 19:46 |
| mtreinish | but I wouldn't have expected it to take 3 days to apply (also it shouldn't do anything to the running process) | 19:46 |
| *** Sukhdev has quit IRC | 19:46 | |
| fungi | (/Stage[main]/Openstack_project::Subunit_worker/Subunit2sql::Worker[A]/Service[jenkins-subunit-worker-A]) Triggered 'refresh' from 1 events | 19:46 |
| fungi | those happened at 19:38:52 on 01 | 19:46 |
| mtreinish | ah, there was that. I did push a subunit2sql release | 19:46 |
| fungi | "refresh" in puppetland usually means service restart | 19:46 |
| mtreinish | fungi: hmm, I thought things weren't configured to restart automagically | 19:47 |
| mtreinish | it doesn't have ensure => running set or anything | 19:47 |
| *** jrmyjohn has quit IRC | 19:48 | |
| *** dtardivel has quit IRC | 19:48 | |
| fungi | mtreinish: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/manifests/worker.pp#n91 | 19:49 |
| *** brad_behle has joined #openstack-infra | 19:49 | |
| fungi | the subscribe => Package['subunit2sql'] there is what triggered that | 19:49 |
| mtreinish | ah ok | 19:50 |
| fungi | so since we upgraded to a new version of subunit2sql, puppet (at least tried) to restart the service | 19:50 |
| mtreinish | fungi: yeah and that likely had the single threaded refactor on it too | 19:50 |
| mtreinish | s/likely/did | 19:50 |
| fungi | shall i try to start it and see if it fails somehow? | 19:51 |
| mtreinish | fungi: does anything get logged when you go to restart it? | 19:51 |
| mtreinish | :) | 19:51 |
| fungi | no stdout but the process is not running. digging deeper, will let you know | 19:52 |
| mtreinish | hmm, ok | 19:52 |
| *** mrmartin has joined #openstack-infra | 19:52 | |
| fungi | huh, the log claims it started, but i see no pidfile and no process | 19:54 |
| mtreinish | fungi: I wonder if we screwed up the refactor and it just exists instead of actually running | 19:55 |
| fungi | that would explain all the witnessed symptoms, so it a good bet | 19:55 |
| fungi | here's the entirety of the log http://paste.openstack.org/show/494883/ | 19:56 |
| mtreinish | fungi: yeah looking at the patch I dropped a while loop at the end of main() | 19:56 |
| mtreinish | it starts the gearman worker thread but then doesn't have anything else in main() so it's exiting I guess | 19:57 |
| fungi | no waitpid, et cetera? | 19:57 |
| mtreinish | fungi: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py#n238 | 19:57 |
| fungi | oh, thread, i'm thinking fork. duh | 19:57 |
| jlvillal | If https://jenkins06.openstack.org/job/gate-tempest-dsvm-neutron-dvr/3429/ doesn't respond. Could that mean that jenkins server is down? | 19:57 |
| *** avarner_ has joined #openstack-infra | 19:58 | |
| mtreinish | jlvillal: that was turned off for sec concerns | 19:58 |
| mtreinish | jlvillal: there was a ml post from mordred about it | 19:58 |
| fungi | jlvillal: http://lists.openstack.org/pipermail/openstack-dev/2016-April/092482.html | 19:58 |
| *** jrmyjohn has joined #openstack-infra | 19:58 | |
| mtreinish | fungi: I'll push a patch to fix that, one sec | 19:58 |
| jlvillal | Oh, we can't see the logfile | 19:58 |
| *** avarner has quit IRC | 19:58 | |
| jlvillal | fungi: mtreinish thanks | 19:58 |
| fungi | jlvillal: it'll still get uploaded to logs.openstack.org once the job completes, and linked in a change comment | 19:59 |
| mtreinish | fungi: should I just add a while True: pass to the end of main()? | 19:59 |
| fungi | jlvillal: jeblair has a couple of changes we're reviewing to get early linking to the job log uploads working from the zuul status page as an interim fix | 19:59 |
| jlvillal | fungi: Thanks. Any plans in place to change the links on status.openstack.org to maybe go to an info page? | 19:59 |
| *** dimtruck is now known as zz_dimtruck | 19:59 | |
| fungi | yep | 20:00 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 20:00 |
| *** ddieterly is now known as ddieterly[away] | 20:01 | |
| *** zz_dimtruck is now known as dimtruck | 20:01 | |
| *** HeOS has joined #openstack-infra | 20:02 | |
| zigo | fungi: Hello! | 20:02 |
| zigo | fungi: Would you have time to help me with fixing the ACLs of https://review.openstack.org/#/c/307742/ ? | 20:02 |
| zigo | I don't really understand what the problem is. | 20:02 |
| *** shardy has quit IRC | 20:03 | |
| zigo | I've set the ACL in gerrit/projects.yaml | 20:03 |
| zigo | +- project: openstack/deb-spice-html5 | 20:03 |
| zigo | + acl-config: /home/gerrit2/acls/openstack/deb-packaging.config | 20:03 |
| zigo | + [...] | 20:03 |
| zigo | Is this wrong? | 20:03 |
| fungi | zigo: http://logs.openstack.org/42/307742/4/check/gate-project-config-projects-yaml/32bee46/console.html#_2016-04-20_08_32_43_607 "Error: No master branch exists" | 20:03 |
| jeblair | jlvillal: if you reload the status page, you should stop seeing links | 20:04 |
| jeblair | mtreinish, fungi: back from lunch | 20:04 |
| jlvillal | Thanks | 20:04 |
| zigo | fungi: Yeah, there's no master branch, so what? | 20:04 |
| zigo | fungi: The master branch is reserved for upstream, so it doesn't conflict... | 20:04 |
| *** ddieterly[away] is now known as ddieterly | 20:04 | |
| fungi | zigo: gerrit needs master to be the default HEAD when importing or things break | 20:04 |
| fungi | mtreinish: i'm a terrible person to ask for advice on multi-threaded application design | 20:05 |
| fungi | haven't really done much of it to speak of | 20:05 |
| zigo | fungi: Well, is this fixeable, or is it a hard requirement due to the design of Gerrit? | 20:05 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Add loop to the end of main() https://review.openstack.org/308556 | 20:05 |
| zigo | I could add a dumb master branch ... | 20:05 |
| mtreinish | jeblair: fungi: ^^^ | 20:05 |
| jeblair | mtreinish: i have an alternate suggestion | 20:06 |
| mtreinish | I;ll work on the patch to stop using threads there (there isn't any reason now I don't think) | 20:06 |
| fungi | zigo: yeah, that's what we suggested previously. just add a master branch with maybe only a README that says to look at the other branches for content | 20:06 |
| * AJaeger is glad that we implemented the test for master after the previous project that had no master branch screwed up quite a few things... | 20:06 | |
| mtreinish | jeblair: ok, I wasn't set on that approach, it just seemed fastest to get things going again | 20:07 |
| zigo | fungi: Well, I'll just push the content of upstream then (as this is what we would do to fetch the latest tag). | 20:07 |
| fungi | zigo: i guess that's fine, but might be confusing since it's actually content/history from a different repo | 20:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-subunit2sql: Make main thread non-daemon https://review.openstack.org/308558 | 20:08 |
| zigo | fungi: not really, we may import from the master branch into the packaging branch if we want to package trunk. | 20:08 |
| jeblair | mtreinish: ^ that doesn't involve a busy-wait. i agree, just dropping threads entirely is better long term. | 20:09 |
| zigo | fungi: It's now fixed in the Debian repo where it will import from. So, if I get this well, it was unrelated to ACLs, right? | 20:10 |
| mtreinish | jeblair: ok, that is a better fast fix. +2 | 20:10 |
| fungi | zigo: yes, i didn't see any errors related to the proposed acl | 20:10 |
| zigo | fungi: Thanks. | 20:10 |
| jeblair | mtreinish: should we go ahead and merge it? | 20:11 |
| mtreinish | jeblair: yeah I think so, there's no reason not to | 20:11 |
| *** oanson has quit IRC | 20:11 | |
| *** shardy has joined #openstack-infra | 20:12 | |
| fungi | zigo: as an example of a project that doesn't really use its master branch, http://git.openstack.org/cgit/openstack-infra/publications/tree/?h=master just contains some documentation about the branching workflow for that repo | 20:12 |
| fungi | but, up to you what you keep in the master branch really | 20:13 |
| zigo | fungi: As I wrote earlier, I can't do that, I *must* reserve the master branch for getting commits from the upstream projects. | 20:13 |
| *** nwkarste_ has joined #openstack-infra | 20:13 | |
| fungi | zigo: questioning the "must" with regards to teh published copy of the git repository. couldn't you pull into a local clone from the relevant branch when packaging? | 20:14 |
| zigo | fungi: The workflow would be: git checkout master && git pull upstream && git tag <something> && git checkout debian/trunk && git merge -X theirs <something> | 20:14 |
| *** comstud has quit IRC | 20:14 | |
| *** nwkarsten has quit IRC | 20:15 | |
| zigo | fungi: Yeah, though the master branch shall stay empty then. | 20:15 |
| zigo | fungi: If you think it's better to keep it empty, we can do that. | 20:15 |
| fungi | zigo: why not git checkout upstream/master instead of git checkout master && git pull upstream? | 20:15 |
| zigo | Hum... | 20:16 |
| fungi | you're working on a headless tree at that point, but that way it doesn't conflate branches from two different repositories | 20:16 |
| zigo | it's just that I prefer that, at any time, not conflict with upstream branch names. | 20:16 |
| *** vgridnev has joined #openstack-infra | 20:17 | |
| zigo | Maybe there's workaround using non-upstream branch names, but then it becomes too complicated for my small brains ... :P | 20:17 |
| fungi | right, that's another option. something like git checkout -B upstream-master upstream/master | 20:18 |
| fungi | creates a local branch called upstream-master with the contents of the master branch from the upstream remote | 20:19 |
| *** apoorvad has quit IRC | 20:20 | |
| *** apoorvad has joined #openstack-infra | 20:20 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Stop using a seperate retreiver thread https://review.openstack.org/308567 | 20:21 |
| mtreinish | jeblair: ^^^ I think that'll do it (it was easier than I thought) | 20:21 |
| mtreinish | I'm trying to start it locally right now, but starbucks wifi is slow, building a venv is going at 14kB/s | 20:22 |
| jeblair | mtreinish: lgtm; maybe i can test it | 20:24 |
| *** infra-red has joined #openstack-infra | 20:24 | |
| *** infra-red has quit IRC | 20:24 | |
| mtreinish | jeblair: sure go for it, I'm still downloading babel... | 20:24 |
| *** roxanagh_ has joined #openstack-infra | 20:25 | |
| mtreinish | actually, I'm gonna head home. I think that might be faster | 20:25 |
| *** tonytan4ever has quit IRC | 20:25 | |
| jeblair | i have the venv, writing a config file | 20:25 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config: Replace the packstack-dev channel by rdo https://review.openstack.org/308572 | 20:27 |
| jeblair | mtreinish: it starts locally without a traceback | 20:29 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Move packstack notifications from packstack-dev to rdo https://review.openstack.org/308576 | 20:29 |
| *** doug-fish has joined #openstack-infra | 20:30 | |
| *** ybathia has joined #openstack-infra | 20:31 | |
| *** doug-fi__ has joined #openstack-infra | 20:31 | |
| *** doug-f___ has joined #openstack-infra | 20:32 | |
| *** mylu has quit IRC | 20:33 | |
| *** Sukhdev has joined #openstack-infra | 20:33 | |
| *** doug-fis_ has quit IRC | 20:33 | |
| nibalizer | es is still yellow, but only initializing one shard | 20:34 |
| *** hashar has quit IRC | 20:34 | |
| *** doug-fish has quit IRC | 20:34 | |
| *** doug-fish has joined #openstack-infra | 20:35 | |
| *** doug-fi__ has quit IRC | 20:35 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Replace the packstack-dev channel by rdo https://review.openstack.org/308578 | 20:35 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 20:36 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Replace the packstack-dev channel by rdo https://review.openstack.org/308576 | 20:37 |
| *** doug-f___ has quit IRC | 20:37 | |
| *** lucas-dinner has quit IRC | 20:40 | |
| *** vponomaryov1 has left #openstack-infra | 20:41 | |
| *** tonytan_brb has joined #openstack-infra | 20:41 | |
| *** asettle has joined #openstack-infra | 20:42 | |
| *** tonytan_brb has quit IRC | 20:42 | |
| mtreinish | jeblair: ok cool, that's as far as I was probably gonna test it | 20:43 |
| *** tonytan_brb has joined #openstack-infra | 20:44 | |
| *** tonytan_brb is now known as tonytan4ever_ | 20:44 | |
| *** asettle has quit IRC | 20:45 | |
| *** rajinir has quit IRC | 20:45 | |
| *** lucasagomes has joined #openstack-infra | 20:47 | |
| *** _ari_ has quit IRC | 20:48 | |
| *** doug-fish has quit IRC | 20:50 | |
| *** matrohon has quit IRC | 20:50 | |
| clarkb | mordred: I am looking at 307887 and I have no idea what was wrong with the old code | 20:51 |
| clarkb | would be nice if we could leave comments on the review if there are major issues | 20:51 |
| *** Swami has joined #openstack-infra | 20:51 | |
| *** doug-fis_ has joined #openstack-infra | 20:52 | |
| mtreinish | jeblair: do we want to just merge the thread dropping patch? and restart the workers after both inprogress patches land? | 20:53 |
| jeblair | mtreinish: yeah, lets | 20:53 |
| *** trown is now known as trown|outtypewww | 20:53 | |
| *** nwkarste_ has quit IRC | 20:54 | |
| prometheanfire | anyone able to review this bindep change to add gentoo support to bindep? https://review.openstack.org/307493 | 20:54 |
| *** ihrachys has joined #openstack-infra | 20:54 | |
| *** ihrachys_ has quit IRC | 20:55 | |
| *** mylu has joined #openstack-infra | 20:55 | |
| *** shardy has quit IRC | 20:55 | |
| *** tonytan4ever_ is now known as tonytan4ever | 20:56 | |
| openstackgerrit | Merged openstack/diskimage-builder: Allow skipping the md docs check https://review.openstack.org/306514 | 20:56 |
| clarkb | mordred: is the problem that get_server can raise? | 20:56 |
| *** daemontool has quit IRC | 20:56 | |
| *** esker has quit IRC | 20:56 | |
| *** mylu has quit IRC | 20:56 | |
| *** dprince has quit IRC | 20:57 | |
| mordred | clarkb: no- it's the thing you were taling about yesterday | 20:57 |
| mordred | clarkb: about invalidating the cache based on the cache | 20:58 |
| mordred | clarkb: tis is "check to see before we delete the server if the volume cache should be invalidated, then, when we actually do delete the server, invalidate the vlume cache" | 20:58 |
| mordred | clarkb: we also added in "don't bother doing the check on the volumes if caching is not enabled" | 20:58 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Slowly turn rax-ord back on in nodepool https://review.openstack.org/308585 | 20:58 |
| fungi | rax noc updated the network issues ticket moments ago suggesting we try again since they just had some maintenance fixing issues with the private circuits between ord and dfw, so... ^ | 20:59 |
| *** mylu has joined #openstack-infra | 20:59 | |
| krotscheck | infra-core: Can someone check on the npm mirror vos-release for me? | 20:59 |
| *** ybathia has quit IRC | 20:59 | |
| clarkb | mordred: oh because get_volume(server) after the server is dead is ya | 20:59 |
| clarkb | ok | 20:59 |
| clarkb | fungi: yay | 20:59 |
| *** MarkAtwood has joined #openstack-infra | 21:00 | |
| *** baoli has quit IRC | 21:01 | |
| clarkb | mordred: oh hey did the test on that one change catch something? I see the exception name changed :) | 21:01 |
| *** mylu has quit IRC | 21:02 | |
| *** bpokorny_ has joined #openstack-infra | 21:02 | |
| mordred | clarkb: yah. it did. ... ssssssshhhhhhh | 21:03 |
| *** nwkarsten has joined #openstack-infra | 21:04 | |
| * morgan hands mordred the "we wont tell that the new tests caught bugs" card. | 21:04 | |
| clarkb | mordred: I am approving things | 21:04 |
| rocky_g | mordred, do you need me to do any prep for the cross project session on discovery? | 21:05 |
| *** bpokorny has quit IRC | 21:07 | |
| *** mrmartin has quit IRC | 21:07 | |
| clarkb | mordred: 308413 has -1 on it from shrews, i have approved everything before that change | 21:08 |
| mordred | clarkb: yah - I need to write a test for that one | 21:08 |
| mordred | you know - for reasons | 21:08 |
| *** jheroux has quit IRC | 21:09 | |
| clarkb | sometimes they catch bugs | 21:09 |
| *** thorst has quit IRC | 21:09 | |
| clarkb | mordred: also http://logs.openstack.org/55/307755/7/check/gate-nodepool-pep8/507c31f/console.html doesn't like the sahde version and still no test ;) | 21:09 |
| *** ybathia has joined #openstack-infra | 21:10 | |
| *** psilvad_ has quit IRC | 21:10 | |
| *** mylu has joined #openstack-infra | 21:10 | |
| *** ybathia has quit IRC | 21:12 | |
| *** rfolco has quit IRC | 21:12 | |
| *** sridhar_ram_afk is now known as sridhar_ram | 21:12 | |
| *** ybathia has joined #openstack-infra | 21:12 | |
| mordred | clarkb: yah - I'm still working on the test | 21:13 |
| mordred | it's been a distracting day | 21:13 |
| *** Goneri has quit IRC | 21:13 | |
| clarkb | I can sympthize | 21:13 |
| fungi | prometheanfire: i put it to workflow -1 because that really needs equivalents of the same basic unit tests we have for dpkg and rpm support | 21:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use setfiles to restore unbound selinux policy https://review.openstack.org/308105 | 21:15 |
| *** Sukhdev has quit IRC | 21:15 | |
| fungi | prometheanfire: should be able to add that even without having gentoo workers and a functional test job available | 21:15 |
| clarkb | pabelanger: fungi ianw_ ^ that can be the intermediate step for selinux fixing on centos7, then switch to running unbound things at 89 which is before level 90 which is where dib will do it for us | 21:15 |
| EmilienM | pabelanger: in case you have some time, please review again https://review.openstack.org/#/c/308394/ and https://review.openstack.org/#/c/307931/ - thanks! | 21:16 |
| *** weshay has quit IRC | 21:17 | |
| *** inc0 has quit IRC | 21:17 | |
| *** ldnunes has quit IRC | 21:18 | |
| prometheanfire | fungi: ok | 21:18 |
| *** abregman has quit IRC | 21:19 | |
| mtreinish | is there something missing from puppet on picking up: https://review.openstack.org/#/c/305496/ on health.o.o | 21:20 |
| *** e0ne has quit IRC | 21:20 | |
| mtreinish | I checked puppetboard: http://puppetboard.openstack.org/node/health.openstack.org and didn't see it get picked up anywhere | 21:20 |
| fungi | prometheanfire: part of my insistence is because i'm unlikely to spin up a gentoo chroot to test modifications in, so having tox -e py27 give me a little confidence i'm not breaking gentoo support is useful | 21:20 |
| fungi | without having to push the change up to gerrit first and wait for functional testing on a real gentoo vm | 21:22 |
| fungi | also what we have as a "functional" test for bindep today doesn't really cover much other than making sure we don't break the one specific way we use it in a lot of gate jobs, so at some point we should consider extending our functional testing to a broader set of exercises | 21:23 |
| *** Nikolay_St has quit IRC | 21:23 | |
| clarkb | mtreinish: neither pip nor the not installed pbr command know about the openstack health install | 21:25 |
| clarkb | mtreinish: the commit that is checked out is that commit | 21:25 |
| clarkb | is this using a virtualenv? | 21:25 |
| mtreinish | clarkb: it is, it's been on my list to refactor it to stop doing that | 21:25 |
| mtreinish | but I haven't gotten around to it yet | 21:25 |
| clarkb | yes .venv | 21:25 |
| prometheanfire | fungi: no, that's fine I should have done that | 21:25 |
| clarkb | mtreinish: openstack-health==0.0.1.dev576 # git sha 8093b5b | 21:26 |
| clarkb | that isn't the commit that is checked out | 21:26 |
| clarkb | it is the previous commit | 21:26 |
| clarkb | mtreinish: Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed | 21:27 |
| mtreinish | clarkb: hmm, ok. That would explain why: http://health.openstack.org/runs/key/project/openstack/puppet-keystone/recent/rss isn't working | 21:27 |
| clarkb | failed building wheel for lxml | 21:27 |
| mtreinish | clarkb: it probably isn't, that's likely a new req being pulled in by the rss libs | 21:27 |
| mtreinish | there wasn't anything xml before | 21:27 |
| clarkb | you probably need to intsall libxml2-dev and libxslt-dev or whatever the deps are | 21:27 |
| mtreinish | clarkb: right, I'll push a puppet patch up to add those packages | 21:28 |
| *** sdake_ has joined #openstack-infra | 21:28 | |
| ianw_ | clarkb: ok, want to put the mv in with a depends on so we don't forget? | 21:28 |
| *** tonytan4ever has quit IRC | 21:28 | |
| clarkb | ianw_: sure | 21:28 |
| ianw_ | dib has a lot of stuff half-approved atm, and can now do releases again, so we should get things going for a release | 21:31 |
| *** Na3iL has quit IRC | 21:31 | |
| *** HeOS has quit IRC | 21:31 | |
| fungi | git-review, gerritbot and bindep releases were on my to do list for this week, but the week is rapidly evaporating out from under me | 21:32 |
| *** ihrachys has quit IRC | 21:32 | |
| *** ihrachys has joined #openstack-infra | 21:33 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-openstack_health: Add packages needed to build lxml https://review.openstack.org/308628 | 21:33 |
| mtreinish | clarkb, fungi: ^^^ | 21:33 |
| fungi | lxml will never die | 21:34 |
| mtreinish | fungi: it's because I used feedgen to do the rss/atom generation | 21:34 |
| fungi | sure | 21:34 |
| mtreinish | if I wanted to hand craft the xml to generate a feed I probably could have gotten away without it | 21:35 |
| fungi | i know why it's there, just expressing my inner pessimist | 21:35 |
| mtreinish | but I think I'd hate myself more if I did that, then pull in lxml :) | 21:35 |
| openstackgerrit | Merged openstack-infra/shade: Have delete_server use the timed server list cache https://review.openstack.org/307887 | 21:35 |
| openstackgerrit | Merged openstack-infra/shade: Retry floating ip deletion before deleting server https://review.openstack.org/307439 | 21:35 |
| fungi | indeed. if only feedgen could get by on the built-in xml support in stdlib, but c'est la python | 21:36 |
| pabelanger | fungi: yay, rax | 21:36 |
| pabelanger | clarkb: ack, +2 | 21:36 |
| *** rhallisey has quit IRC | 21:37 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Improved External Order Error https://review.openstack.org/308629 | 21:37 |
| *** ddieterly is now known as ddieterly[away] | 21:37 | |
| *** ddieterly[away] is now known as ddieterly | 21:38 | |
| *** Sukhdev has joined #openstack-infra | 21:38 | |
| *** amitgandhinz has quit IRC | 21:38 | |
| *** kgiusti has left #openstack-infra | 21:39 | |
| openstackgerrit | Merged openstack-infra/shade: Delete floating IP on nova refresh failure https://review.openstack.org/308024 | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Let dib fix selinux policy for us https://review.openstack.org/308630 | 21:40 |
| clarkb | ianw_: ^ | 21:40 |
| *** thorst has joined #openstack-infra | 21:41 | |
| *** e0ne has joined #openstack-infra | 21:41 | |
| *** Na3iL has joined #openstack-infra | 21:43 | |
| openstackgerrit | Merged openstack-infra/shade: Rework floating ip use test to be neutron based https://review.openstack.org/308412 | 21:43 |
| *** Sukhdev has quit IRC | 21:43 | |
| *** ihrachys has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack-infra/project-config: run-tarball/puppet: fix MODULE_NAME https://review.openstack.org/308394 | 21:44 |
| *** sdake_ has quit IRC | 21:45 | |
| *** thorst has quit IRC | 21:46 | |
| *** mylu has quit IRC | 21:46 | |
| mtreinish | fungi, jeblair: :/ both puppet-subunit2sql patches failed on centos beaker. Looks unrelated: http://logs.openstack.org/58/308558/1/check/gate-openstackci-beaker-centos-7/176b231/console.html#_2016-04-20_20_16_07_695 | 21:46 |
| *** sdake_ has joined #openstack-infra | 21:47 | |
| *** Sukhdev has joined #openstack-infra | 21:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 21:50 |
| *** e0ne has quit IRC | 21:51 | |
| openstackgerrit | Merged openstack/diskimage-builder: Change to latest CentOS-6 image https://review.openstack.org/308504 | 21:52 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 21:52 |
| *** lakshmiS has quit IRC | 21:53 | |
| *** nwkarsten has quit IRC | 21:54 | |
| *** nwkarsten has joined #openstack-infra | 21:54 | |
| *** sridhar_ram_ has quit IRC | 21:54 | |
| *** vgridnev_ has joined #openstack-infra | 21:55 | |
| *** |-paul-| has joined #openstack-infra | 21:55 | |
| *** salv-orl_ has joined #openstack-infra | 21:56 | |
| *** jrmyjohn has quit IRC | 21:57 | |
| *** roxanagh_ has quit IRC | 21:57 | |
| *** piet has joined #openstack-infra | 21:57 | |
| *** Na3iL has quit IRC | 21:58 | |
| *** nikhil_k is now known as nikhil | 21:58 | |
| *** salv-orlando has quit IRC | 21:59 | |
| *** Sukhdev has quit IRC | 21:59 | |
| *** vgridnev_ has quit IRC | 22:00 | |
| *** asettle has joined #openstack-infra | 22:01 | |
| clarkb | https://etherpad.openstack.org/p/newton-infra-distro-upgrade-plans has been updated with information on what machines are running what | 22:01 |
| *** matt-borland has quit IRC | 22:02 | |
| fungi | thanks! | 22:02 |
| *** ddieterly is now known as ddieterly[away] | 22:02 | |
| fungi | mtreinish: yeah, there's a latent bug in beaker exposed by a new puppet release | 22:03 |
| fungi | yolanda: proposed a fix for it but will need an image update to have the workaround in place | 22:03 |
| *** sridhar_ram1 has joined #openstack-infra | 22:03 | |
| fungi | er, yolanda proposed a fix | 22:04 |
| *** sridhar_ram has quit IRC | 22:04 | |
| *** burgerk has quit IRC | 22:04 | |
| clarkb | fungi: is there a chnage up to put rax-ord back into service? | 22:04 |
| *** Sukhdev has joined #openstack-infra | 22:05 | |
| *** roxanagh_ has joined #openstack-infra | 22:05 | |
| fungi | clarkb: yeah, i uploaded and was pointing it out when i commented | 22:05 |
| fungi | clarkb: 308585 | 22:06 |
| clarkb | approved | 22:06 |
| *** fitoduarte has joined #openstack-infra | 22:07 | |
| *** ybathia has quit IRC | 22:08 | |
| clarkb | mordred: we have leaked fips right now | 22:09 |
| clarkb | mordred: I am guessing we are not deployed with the latest code? | 22:09 |
| *** Na3iL has joined #openstack-infra | 22:09 | |
| mordred | clarkb: not with the fip cleanup code | 22:09 |
| mordred | clarkb: I believe there is also likely still a leak point somewhere we haven't identified yet | 22:10 |
| mordred | clarkb: I'm cleaning them now | 22:10 |
| mordred | clarkb: I had intended to do another hunt today on one of the leaks | 22:10 |
| fungi | pour diluted dish soap over it and look for bubbles | 22:10 |
| mordred | but today got away from me | 22:10 |
| mordred | fungi: exactly | 22:10 |
| *** rocky_g has quit IRC | 22:12 | |
| *** nwkarsten has quit IRC | 22:13 | |
| *** nwkarste_ has joined #openstack-infra | 22:13 | |
| *** marcusvrn_ has quit IRC | 22:13 | |
| morgan | clarkb: leaked fips are fips that are allocated but unassigned and untracked? | 22:15 |
| *** mylu has joined #openstack-infra | 22:15 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:15 | |
| *** nwkarste_ has quit IRC | 22:15 | |
| * morgan is curious because... Sounds like something interesting to chase down if it is still occurring. (Or assumed to be occurring even with new code ) | 22:16 | |
| *** nwkarsten has joined #openstack-infra | 22:16 | |
| clarkb | morgan: correct | 22:16 |
| clarkb | morgan: basically becaues we don't have atomic resource management | 22:17 |
| morgan | Right. | 22:17 |
| clarkb | morgan: if you aren't super careful you leak resources | 22:17 |
| clarkb | the fun thing is we do have that for cinder volumes | 22:17 |
| clarkb | just not for fips | 22:17 |
| morgan | Yar. And lots of other things. | 22:17 |
| *** doug-fis_ has quit IRC | 22:17 | |
| *** aysyd has quit IRC | 22:17 | |
| morgan | OK I might have something to glare at tomorrow then. ;) | 22:17 |
| *** mylu has quit IRC | 22:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support scp publisher https://review.openstack.org/308656 | 22:20 |
| *** dimtruck is now known as zz_dimtruck | 22:20 | |
| *** nwkarsten has quit IRC | 22:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Slowly turn rax-ord back on in nodepool https://review.openstack.org/308585 | 22:20 |
| morgan | Have I mentioned I dislike floating ips? | 22:20 |
| morgan | :P | 22:21 |
| jeblair | okay. i've saved the best for last. anyone know if ansible has an ftp upload module? ;) | 22:21 |
| mordred | morgan: you are in good company | 22:22 |
| morgan | mordred: hah | 22:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add public helper method for cleaning floating ips https://review.openstack.org/308413 | 22:22 |
| mordred | Shrews, clarkb: there you go. with tests | 22:22 |
| jeblair | (my backup plan is basically: rsync to localhost tmpdir then "shell: ftp ...") | 22:22 |
| morgan | jeblair: old school style ;) | 22:23 |
| *** gordc has quit IRC | 22:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add ServerGet task to nodepool graphs https://review.openstack.org/308483 | 22:23 |
| mtreinish | fungi: ok, so we're just waiting for a nodepool image refresh before I recheck things? | 22:23 |
| *** mylu has joined #openstack-infra | 22:24 | |
| mordred | jeblair: it looks like only ftp support for fetching | 22:24 |
| morgan | Out of curiosity does infra (yet) run ZK cluster? Or that waiting for other software support (aka zuul etc)? | 22:24 |
| *** edmondsw has quit IRC | 22:24 | |
| mordred | morgan: none of our things use it yet, so we have yet to spin one up | 22:25 |
| * morgan knows it is possible to just go look at configs but people are active and know off the top of their head. | 22:25 | |
| *** ddieterly[away] is now known as ddieterly | 22:25 | |
| morgan | mordred: OK. Good to know. | 22:25 |
| jeblair | morgan: the alma schola is my alma mater | 22:25 |
| mordred | morgan: it is definitely a task that will need done before we can grow the zk depends | 22:25 |
| morgan | mordred: well I was just wondering the order of things. | 22:25 |
| *** gyee has quit IRC | 22:25 | |
| jeblair | i'd suggest we write the nodepool code first, then spin up the cluster | 22:26 |
| morgan | If ZK was in use, it can grow to a depends, if not, it is "add function and then rely on it" | 22:26 |
| morgan | jeblair: that is my view. | 22:26 |
| *** rossella_s has joined #openstack-infra | 22:26 | |
| morgan | But if ZK was already there... Just easy to start leaning on it. | 22:26 |
| jeblair | morgan, mordred: i think we should implement this first: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html | 22:26 |
| *** asettle has quit IRC | 22:26 | |
| morgan | It was more of a where are we, than a best laid plans for impl question. | 22:27 |
| morgan | jeblair: ++ | 22:27 |
| jeblair | morgan, mordred: i also have this standy by for when we need it: https://review.openstack.org/293176 | 22:27 |
| jeblair | er standing by | 22:27 |
| *** asettle has joined #openstack-infra | 22:27 | |
| *** Na3iL has quit IRC | 22:27 | |
| morgan | Nice | 22:27 |
| *** Adri2000 has quit IRC | 22:27 | |
| jeblair | (for me, making sure we could run it in tests was the unknown -- i'm pretty sure we can figure out how to run it in prod :) | 22:28 |
| *** rbergeron has joined #openstack-infra | 22:28 | |
| * morgan feels like lurking in infra sessions at the summit is going to be fun this time around ;) | 22:28 | |
| morgan | Not to imply I disliked it last time. | 22:29 |
| morgan | ;) | 22:29 |
| jeblair | morgan: then this is the next step (it's an ammendment to the zuulv3 spec): https://review.openstack.org/305506 | 22:29 |
| bkero | (morgan has registered an allegiance with 'Peanut Gallery') | 22:29 |
| jeblair | i'd like to do the nodepool worker thing first just to get it estableshed, then do the zuulv3 thing, and not implement the zuul-nodepool protocol as described currently in the v3 spec. | 22:30 |
| jeblair | established | 22:30 |
| jeblair | sheesh | 22:30 |
| morgan | jeblair: hehe. | 22:30 |
| jeblair | E_STABLE_SHED | 22:30 |
| morgan | That's like an e_bike_shed but for code that doesn't change? | 22:31 |
| jeblair | i believe you've got it | 22:32 |
| morgan | Nice. | 22:32 |
| rbergeron | morgan: lurker. :) | 22:32 |
| * rbergeron lurks | 22:32 | |
| morgan | I am totally going to make one of the errors in something I develop e_stable_shed now. | 22:32 |
| *** tosky has quit IRC | 22:32 | |
| *** ayoung has joined #openstack-infra | 22:33 | |
| * jeblair watches b5 | 22:33 | |
| morgan | rbergeron: I suck at lurking. I am too loud. | 22:33 |
| *** burgerk has joined #openstack-infra | 22:34 | |
| rbergeron | :) | 22:34 |
| fungi | mtreinish: i'll check and see if that fix got approved, but yeah if so images tomorrow should be fine or if it's an emergency we can try to get fresh image updates in place sooner | 22:36 |
| mtreinish | fungi: well it's blocking the subunit worker fixes | 22:36 |
| clarkb | need to get mordred to stop killing builds too when you do it:) | 22:36 |
| mordred | clarkb: when did I kill a build? :) | 22:36 |
| mtreinish | fungi: but gearman will probably hold all the events until things start up again, so I don't think we'll lose anything | 22:37 |
| clarkb | mordred: everytime you restart nodepool when an image is building | 22:37 |
| mordred | look, it's a rbergeron ... can we have an ftp module for ansible? | 22:37 |
| mordred | clarkb: oh. that | 22:37 |
| mordred | clarkb: bah. | 22:37 |
| jeblair | mordred: the zk stuff will fix it :) | 22:37 |
| mtreinish | fungi: it just means the yellow warning on: http://status.openstack.org/openstack-health/#/ will be there a little longer | 22:38 |
| mordred | jeblair: yay for successing! | 22:38 |
| jeblair | mordred, rbergeron: ftp *upload* that is :) | 22:38 |
| *** thorst has joined #openstack-infra | 22:39 | |
| fungi | ftp _put_ (to be even more pedantic) | 22:39 |
| clarkb | maybe cloud sites has grown new protocol support? | 22:39 |
| jeblair | fungi is in charge of ftp | 22:39 |
| rbergeron | jeblair: yeah, downloading is a thing already | 22:39 |
| * mordred looks forward to replacing cloud sites with afs | 22:39 | |
| mordred | we need a fungi module! | 22:40 |
| morgan | mordred: ++ | 22:40 |
| rbergeron | jeblair, mordred: i guess the official crappy answer is "patches welcome" but ... that's a terrible answer that i hate giving :) | 22:40 |
| mordred | heh | 22:40 |
| pabelanger | clarkb: morgan I just cleaned up bluebox about 25mins ago. But something odd going on, we are always building more nodes then using. I assume you already know that | 22:40 |
| jeblair | mordred: i have a spec for that too :) https://review.openstack.org/269928 | 22:40 |
| *** ybathia has joined #openstack-infra | 22:40 | |
| rbergeron | you could hvae one, but i think it's a matter of finding a nice human with the cycles to do so -- and convincing them | 22:40 |
| mordred | to be fair, I suppose I do already own a bunch of modules in ansible | 22:40 |
| mordred | jeblair: if you write an ftp module, I will merge it | 22:41 |
| fungi | jeblair: ftp is one of those things i will probably never forget. like xmodem | 22:41 |
| mtreinish | mordred: https://pypi.python.org/pypi/Fungiform was the closest on pypi :) | 22:41 |
| clarkb | pabelanger: not sure I understand ehat you mean about building more than we are using | 22:41 |
| mordred | fungi: if you write an ftp module, I will merge it | 22:41 |
| fungi | heh | 22:41 |
| clarkb | can we have tftp too? | 22:41 |
| jeblair | rbergeron: no worries :) i think that synchronize + shell will work for now; if we find that we're going to keep using ftp for a while, it might be worth writing a module. but honestly, we'd like to just stop using it, so i kind of hope that happens first :) | 22:41 |
| mordred | clarkb: if you write a tftp module, I will merge it | 22:42 |
| clarkb | woot | 22:42 |
| fungi | clarkb: i have much more terrible memories of tftp, so hopefully not? | 22:42 |
| rbergeron | also, i suspect that maybe with cookies and hugs and we could probably wheedle someone into wanting to write it while everyone is seeing each other next week | 22:42 |
| mordred | s/cookies/booze/ | 22:42 |
| mordred | s/hugs/booze/ | 22:42 |
| rbergeron | and by that i mean "not someone currently here" | 22:42 |
| mordred | s/and/booze/g | 22:42 |
| rbergeron | or with booze and booze | 22:42 |
| fungi | yeah, austin has cue and booze. dunno if they also have cookies and hugs | 22:43 |
| rbergeron | well, if i have booze, then i give hugs, so | 22:43 |
| * jeblair looks forlornly at mordred and eats raw cookie dough | 22:43 | |
| *** rbrndt has joined #openstack-infra | 22:43 | |
| rbergeron | cookies, tho | 22:43 |
| mordred | "also, i suspect that maybe with booze booze booze booze booze we could probably wheedle someone into wanting to write it while everyone is seeing each other next week" | 22:43 |
| *** xyang1 has quit IRC | 22:43 | |
| *** ddieterly has quit IRC | 22:43 | |
| pabelanger | clarkb: so, if you look at http://grafana.openstack.org/dashboard/db/nodepool-blue-box. I've noticed, we always now have more nodes in the build state, then ready / used. If you look on nodepool, grep building | grep bluebox, you'll see we have nodes +28mins. They seem to be just idle, not attaching to an IP | 22:43 |
| fungi | booze, booze, booze, eggs and spam | 22:44 |
| clarkb | pabelanger: ya its the fip problem | 22:44 |
| pabelanger | clarkb: even though we now how FIPs free | 22:44 |
| mordred | fungi: ++ | 22:44 |
| * fungi starts humming | 22:44 | |
| clarkb | pabelanger: its bad all around | 22:44 |
| *** mriedem has quit IRC | 22:44 | |
| pabelanger | clarkb: okay | 22:44 |
| mordred | hopefully will be fixed fixed tomorrow | 22:44 |
| pabelanger | clarkb: is it wroth dropping our default build timeout? | 22:45 |
| jeblair | mordred: boozed boozed? | 22:45 |
| mordred | jeblair: ++ | 22:45 |
| clarkb | pabelanger: probably not those should start to complete as the fips attach I think | 22:45 |
| clarkb | pabelanger: mordred will know more | 22:45 |
| pabelanger | okay, np | 22:45 |
| pabelanger | will see what the fix is tomorrow | 22:45 |
| fungi | mtreinish: so the short answer is that https://review.openstack.org/308426 is not self-testing so it is failing on the bug it is needed to fix. i'm going to bypass gating on it | 22:47 |
| fungi | that way it's at least in place for tomorrow's image updates | 22:47 |
| openstackgerrit | Merged openstack-infra/system-config: Consume the latest puppetlabs rpm when installing puppet https://review.openstack.org/308426 | 22:49 |
| fungi | there ^ it is | 22:49 |
| fungi | mordred: are you done being in project bootstrappers for now? or should i leave you in it for a bit longer? | 22:49 |
| *** yamahata has quit IRC | 22:49 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add js-openstack-registry-hooks (openstack-registry-hooks) to openstack-infra https://review.openstack.org/308662 | 22:51 |
| *** |-paul-| has quit IRC | 22:52 | |
| krotscheck | jeblair: ^^ I believe that should assist with the clean exit. | 22:53 |
| * jeblair disguises his voice as mordred and says "fungi: oh, yes, i'm totally like a space cadet and you should definitely remove me!" | 22:53 | |
| jeblair | oops, that wasn't the right voice, but oh well | 22:54 |
| fungi | sounded more like stimpy | 22:54 |
| fungi | er, i mean billy west | 22:54 |
| jeblair | was billy west really stimpy? | 22:55 |
| jeblair | wow | 22:55 |
| fungi | yep | 22:55 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added clean exit hooks to npm registry script https://review.openstack.org/308669 | 22:56 |
| fungi | if i know only two things, it's ftp and cartoons | 22:56 |
| krotscheck | And there's the other part | 22:56 |
| *** claudiub has quit IRC | 22:56 | |
| *** krotscheck is now known as krotscheck_dcm | 22:57 | |
| jeblair | krotscheck: neat! that is not the form i expected that solution to take. :) | 22:57 |
| rbergeron | billy west is everyone. | 22:57 |
| fungi | krotscheck_dcm: are you the data center manager now? | 22:57 |
| fungi | or is that don't care mode? | 22:57 |
| fungi | oh... day care mode | 22:58 |
| jeblair | fungi: i'm familiar with him from, basically half the futurama cast | 22:58 |
| * fungi is slow | 22:58 | |
| * clarkb waves in morgans genral area | 22:58 | |
| *** carolbarrett has joined #openstack-infra | 22:58 | |
| *** carolbarrett has quit IRC | 22:59 | |
| fungi | jeblair: yeah, everyone on futurama sounds like fry, except maybe katey sagal | 22:59 |
| clarkb | morbo | 23:00 |
| fungi | little-known fact, i was the voice of the hypnotoad | 23:00 |
| fungi | (prove me wrong!) | 23:00 |
| rbergeron | except for bender, who sounds like jake from adventure time | 23:00 |
| fungi | so true | 23:00 |
| *** thorst has quit IRC | 23:01 | |
| *** Qiming has joined #openstack-infra | 23:01 | |
| rbergeron | and those are the same person but man. if i am not actually watching and theyre not saying obvious character things, i cant actually tell sometimes | 23:02 |
| *** kjorgensen has joined #openstack-infra | 23:02 | |
| *** ams_ has quit IRC | 23:03 | |
| *** piet has quit IRC | 23:03 | |
| *** cloudtrainme has quit IRC | 23:03 | |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Bump oslo.log and oslo.il8n for april 18th releases https://review.openstack.org/308680 | 23:04 |
| *** jrmyjohn has joined #openstack-infra | 23:06 | |
| *** burgerk has quit IRC | 23:06 | |
| mtreinish | fungi: ok cool, thanks | 23:07 |
| *** rbrndt has quit IRC | 23:07 | |
| fungi | i'm transferring to evening mode, wherein i stop openstacking for a bit and see if i can make progress toward the "fungi" laptop model ascribed to me at the midcycle a reality before next week | 23:08 |
| fungi | this will probably need more beer | 23:09 |
| openstackgerrit | Merged openstack/requirements: bump python-openstackclient in u-c to 2.4.0 https://review.openstack.org/308411 | 23:10 |
| mtreinish | fungi: "fungi" laptop model? | 23:10 |
| *** jrmyjohn has quit IRC | 23:10 | |
| *** Adri2000 has joined #openstack-infra | 23:10 | |
| openstackgerrit | melanie witt proposed openstack-infra/project-config: Add a new devstack plugin based src job for ceph https://review.openstack.org/308686 | 23:11 |
| openstackgerrit | melanie witt proposed openstack-infra/project-config: Replace ceph-src jobs with ceph-plugin-src jobs https://review.openstack.org/308687 | 23:11 |
| fungi | mtreinish: there was an informal poll of laptop models at the infra mid-cycle and i was out of the room while it was performed, so since nobody knew what my asus netbook was they just put "fungi" down for mine | 23:11 |
| clarkb | and no macbooks | 23:11 |
| fungi | but i've got a half-complete netbook running an open hardware sbc (radxa rock pro), an lvds panel i picked up from adafruit and some other less important bits | 23:12 |
| *** apoorvad has quit IRC | 23:12 | |
| fungi | so i'm hopeful i can knock it out in the next couple evenings | 23:12 |
| *** sdague has quit IRC | 23:12 | |
| jeblair | i'm pretty sure if we remembered 'asus netbook' we still would have listed it as 'fungi' anyway | 23:12 |
| mtreinish | fungi: haha, ok. Fwiw, I have a friend in nicaragua for the peace corp and he uses a similar asus netbook. (if not the same, there are a few models) Mostly because it was unlikely to get stolen | 23:13 |
| mtreinish | clarkb: that's a good thing | 23:13 |
| mtreinish | fungi: oh, cool that sounds liek a fun project | 23:13 |
| fungi | mtreinish: yeah, i bought that particular asus (i have several) for something like us$200 ~10 years ago. definitely not a theft target | 23:13 |
| fungi | it's fine as a mostly thin client ssh/mosh terminal and for very lightweight web browsing, but it struggles a bit for i/o bandwidth to do luks on its flash storage | 23:14 |
| ianw_ | what's up with the weird jenkins comment on : https://review.openstack.org/#/c/292193/ | 23:14 |
| jeblair | ianw_: what's weird? | 23:15 |
| *** ddieterly has joined #openstack-infra | 23:15 | |
| mtreinish | fungi: well he's had like 2 cameras and 3 cell phones stolen in his ~2yrs there, and those weren't particularly nice either. So I think it's just luck that someone hasn't targetted the netbook | 23:15 |
| ianw_ | jeblair: well it's messed up the formatting by putting log messages in there about the requirements result | 23:16 |
| mtreinish | fungi: heh, and he runs windows on it, which is just so terribad | 23:16 |
| fungi | ianw_: it's intended to make it more obvious why the requirements job failed, i think. you're looking for http://logs.openstack.org/93/292193/8/check/gate-diskimage-builder-requirements/cabb107/console.html#_2016-04-20_22_29_49_929 | 23:16 |
| jeblair | ianw_: oh, you mean there's something wrong in the web ui? | 23:17 |
| ianw_ | fungi: yeah, but shouldn't it just be a failure? | 23:17 |
| fungi | mtreinish: oh, it's newer then. the one i had with me at the mid-cycle only supported linux )shipped with "lindows" before they renamed it) | 23:17 |
| ianw_ | i mean, with CI comments compressed, you don't see it at all? | 23:17 |
| jeblair | ianw_: that means there is a bug in the javascript that munges that | 23:17 |
| ianw_ | ok, if we dont' know about it i'll take a closer look | 23:18 |
| fungi | mtreinish: microsoft got all snippy about asus shipping a product that couldn't run windows and actually made a stripped down windows for netbooks which the next model was capable of running, just barely | 23:18 |
| jeblair | ianw_: probably in http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/gerrit/hideci.js | 23:18 |
| fungi | mtreinish: though to asus's credit, you could order those later models with linux preinstalled and not get dinged for the "windows tax" (which cost almost as much as teh hardware) | 23:19 |
| *** mylu has quit IRC | 23:20 | |
| fungi | mtreinish: and then microsoft essentially licensed the windows for netbooks or whatever they were calling it for free for a while so asus wouldn't sell something cheaper without it | 23:20 |
| *** mylu has joined #openstack-infra | 23:20 | |
| fungi | was actually pretty funny to watch that situation play out | 23:21 |
| mtreinish | fungi: haha, why I'm not surprised. That sounds like pretty typical behavior for microsoft back then | 23:22 |
| *** Sukhdev has quit IRC | 23:22 | |
| fungi | maybe they only lowered the bundling license for it to something nominal like $25-50 i don't completely remember it's been so long | 23:22 |
| fungi | anyway, to the lab! | 23:22 |
| fungi | futurama conversation has convinced me to pull out the box set and watch it in the background while i hack | 23:23 |
| *** Na3iL has joined #openstack-infra | 23:23 | |
| *** ddieterly is now known as ddieterly[away] | 23:23 | |
| fungi | mordred: apologies in advance, but i'm going to assume you've forgotten you're still in project bootstrappers and am removing you for your own safety | 23:24 |
| jeblair | fungi: thanks! | 23:25 |
| *** dingyichen has joined #openstack-infra | 23:25 | |
| *** bpokorny_ has quit IRC | 23:25 | |
| * jeblair disguised his voice again | 23:25 | |
| krotscheck_dcm | jeblair: yeah, me neither. | 23:25 |
| * krotscheck_dcm has an infant battling him for the keyboard. Singing off. | 23:25 | |
| *** salv-orl_ has quit IRC | 23:26 | |
| *** bpokorny has joined #openstack-infra | 23:26 | |
| fungi | jeblair: also, zuul still looks memory-leaky http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 23:26 |
| fungi | jeblair: we're running with connections patched out? or no? | 23:26 |
| fungi | okay, really going afk for a while. night all! | 23:29 |
| melwitt | does anyone know why testr_results.html.gz in logs changed from showing only failures as the default view to showing pass and fail all together? | 23:29 |
| *** jaypipes has quit IRC | 23:29 | |
| *** mat128 has quit IRC | 23:29 | |
| crinkle | krotscheck_dcm: you need this https://twitter.com/abock/status/718103213619208192 | 23:30 |
| *** mat128 has joined #openstack-infra | 23:30 | |
| *** apoorvad has joined #openstack-infra | 23:30 | |
| *** roxanagh_ has quit IRC | 23:30 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Replace the packstack-dev channel by rdo https://review.openstack.org/308576 | 23:37 |
| *** mylu has quit IRC | 23:38 | |
| *** mylu has joined #openstack-infra | 23:38 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Changed ACL group in js-afs-blob-store to a more generic group https://review.openstack.org/308695 | 23:38 |
| clarkb | melwitt: which job(s)? | 23:39 |
| *** sridhar_ram1 has quit IRC | 23:41 | |
| *** Sukhdev has joined #openstack-infra | 23:41 | |
| *** lakshmiS has joined #openstack-infra | 23:41 | |
| melwitt | clarkb: gate-nova-python34-db for example. it's been awhile since it changed http://logs.openstack.org/33/266233/6/gate/gate-nova-python34-db/843ab2d/testr_results.html.gz | 23:42 |
| melwitt | clarkb: it used to default to Failed but even when I click Failed it doesn't change the view | 23:43 |
| clarkb | melwitt: looks like that is generated by os-testr now | 23:44 |
| clarkb | I am guessing os-testr updated that? | 23:45 |
| clarkb | I would look at its commit log | 23:45 |
| melwitt | clarkb: okay, thanks for the hint. I'll check it out | 23:45 |
| mtreinish | melwitt: probably this: http://git.openstack.org/cgit/openstack/os-testr/commit/?id=2b5c0175fc0cf9eddf570b743bf3c89ad7e0fd3a | 23:45 |
| *** sridhar_ram has joined #openstack-infra | 23:46 | |
| mtreinish | which is the only change that has seen since I added the apache header back when I first created the repo | 23:46 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/project-config: Support multi backends for Zaqar tempest job https://review.openstack.org/308698 | 23:46 |
| *** fitoduarte has quit IRC | 23:46 | |
| *** asettle has quit IRC | 23:47 | |
| melwitt | mtreinish: cool, thanks | 23:47 |
| *** asettle has joined #openstack-infra | 23:47 | |
| *** Sukhdev has quit IRC | 23:48 | |
| mtreinish | melwitt: I think the old behavior you saw was just a bug and things weren't be rendered as intended | 23:50 |
| *** kengo_sakai has joined #openstack-infra | 23:50 | |
| mtreinish | it just happened to be more convienent especially for larger test suites (like nova unit tests) | 23:50 |
| *** Guest56909 has left #openstack-infra | 23:50 | |
| melwitt | mtreinish: hm. it used to show failed first and you could expand it to show all. unless I dreamed it | 23:51 |
| *** jamielennox is now known as jamielennox|away | 23:51 | |
| melwitt | there are links now to Summary, Failed, and All and clicking Failed seems to do nothing | 23:51 |
| melwitt | I guess it thinks Summary == Failed | 23:52 |
| melwitt | when I click on All it changes to a different view and then click on Failed and it shows the view it started with | 23:52 |
| mtreinish | melwitt: right because it wasn't parsing the subunit correctly. So it put everything in 1 giant group (instead of splitting by class like it was supposed to) | 23:53 |
| mtreinish | and by default it only shows failed tests in a group | 23:53 |
| melwitt | oh | 23:53 |
| *** kengo_sakai has quit IRC | 23:54 | |
| *** xarses has quit IRC | 23:54 | |
| mtreinish | melwitt: I reverted that fix on the subunit out from that run you linked and reran it: http://blog.kortar.org/wp-content/uploads/2016/04/results.html | 23:56 |
| mtreinish | melwitt: I think it'd be totally fair to move all the failures to the top instead of just doing a straight alphabetical sort on the classes | 23:57 |
| melwitt | mtreinish: yeah, that's it! thanks. I can rest now | 23:57 |
| mtreinish | it defintitely makes things more convenient | 23:57 |
| melwitt | yeah, that would help a lot | 23:58 |
| mtreinish | heh, but good luck dealing with the implementation. It's not exactly an easy to read script :) | 23:58 |
| melwitt | :) | 23:58 |
| *** sdake__ has joined #openstack-infra | 23:58 | |
| melwitt | the only time I ever click on the results is when they fail so I thought the failure view was intentional | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!