2015-03-19T00:00:11 StevenK: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/proposal_slave.pp 2015-03-19T00:01:26 clarkb let me know when u think i can recheck https://review.openstack.org/#/c/164836/ and it might work 2015-03-19T00:01:31 *** alexpilotti has quit IRC 2015-03-19T00:01:34 I can confirm with AJaeger that script-wise that's the only place it's needed 2015-03-19T00:01:49 harlowja_: I have no idea, I was hoping mordred would have some great insight for us 2015-03-19T00:01:50 we'll need to know where to put this puppet change anyway 2015-03-19T00:01:51 k 2015-03-19T00:02:03 mordred great insight required! 2015-03-19T00:02:28 clarkb: why would i have insight? 2015-03-19T00:02:31 i don't think any py26 supporting project can update its requirements with those '~=' style until that gets addressed 2015-03-19T00:02:34 *** claudiub has joined #openstack-infra 2015-03-19T00:02:51 *** alexpilotti has joined #openstack-infra 2015-03-19T00:02:58 mordred: because you wrote all the recent changes to that pin 2015-03-19T00:03:06 clarkb: oh - that was the pin thing 2015-03-19T00:03:09 harlowja_: maybe we shouldn't use ~= then 2015-03-19T00:03:21 harlowja_: because we can't garuntee everyone using this sfotware has pip 6.0 2015-03-19T00:03:24 clarkb: is that pinning to older or newer? 2015-03-19T00:03:29 mordred: older 2015-03-19T00:03:33 clarkb u won't get disagreement from me :-P 2015-03-19T00:03:40 clarkb: yes, we can. we do not make any guarantees about non-most-recent pip 2015-03-19T00:03:41 *** Ryan_Lane has quit IRC 2015-03-19T00:03:50 clarkb: in fact, we explicitly don't work on it, i believe 2015-03-19T00:03:56 https://github.com/openstack/requirements/commit/25ef10 put a bunch in though, lol 2015-03-19T00:03:59 *** alexpilotti has quit IRC 2015-03-19T00:04:01 mordred: we do we know this because centos6 2015-03-19T00:04:01 *** yamamoto has joined #openstack-infra 2015-03-19T00:04:09 I need to run off to an evening event, thanks jeblair, StevenK and mordred 2015-03-19T00:04:14 clarkb: no - if you're installing from pip, you need to install latest pip 2015-03-19T00:04:18 *** jp_at_hp has joined #openstack-infra 2015-03-19T00:04:20 we do not support the pip in centos6 2015-03-19T00:04:22 mordred: that is not how we test 2015-03-19T00:04:27 clarkb: then we're testing wrong 2015-03-19T00:04:33 mordred: thats fine we can fix that 2015-03-19T00:04:36 clarkb: woot 2015-03-19T00:04:40 mordred: just pointing out that your assertion is false as of right now 2015-03-19T00:04:48 *** sdake_ has joined #openstack-infra 2015-03-19T00:04:58 *** alexpilotti has joined #openstack-infra 2015-03-19T00:05:00 clarkb: cool. well - I refuse to support old pip except for transitionally 2015-03-19T00:05:09 because it makes absolutely no sense 2015-03-19T00:05:19 the reason I suggest not using ~= is because >= and < should also work fine 2015-03-19T00:05:30 so there is little benefit in supporting fancy new symbol that isn't available everwhere 2015-03-19T00:05:33 clarkb: right - but openstack is adopting ~= across the board pretty aggressively 2015-03-19T00:05:40 mordred: yes but there is no reason to 2015-03-19T00:05:44 mordred: why would we do that? 2015-03-19T00:05:45 clarkb: there is no reason not to 2015-03-19T00:05:56 clarkb: supporting out of date pip has no point 2015-03-19T00:05:57 mordred: there is 2015-03-19T00:06:01 there is literally no benefit 2015-03-19T00:06:03 mordred: code change churn 2015-03-19T00:06:11 there is no benefit to change to ~= so lets just leave it be 2015-03-19T00:06:11 tehn don't pip install from master 2015-03-19T00:06:15 *** alexpilotti has quit IRC 2015-03-19T00:06:17 clarkb: not our call here 2015-03-19T00:06:20 mordred: this has nothing to do with pip 2015-03-19T00:06:23 * harlowja_ has to go fix anvil to convert ~= -> rpm versions, lol, (code churn there, lol) 2015-03-19T00:06:26 sure it does 2015-03-19T00:06:33 mordred: and everything to do with we don't need hundreds of requirements updates to support ~= unnecessarily 2015-03-19T00:06:41 mordred: it just happens that old pip also fails at this 2015-03-19T00:06:45 I'm not going to police that 2015-03-19T00:06:52 if the project wants to adopt it, great 2015-03-19T00:07:00 sure, I haven't -1'd anything 2015-03-19T00:07:02 it fits in with my understanding of our pip support 2015-03-19T00:07:07 just pointing out that there are valid reasons to not do this 2015-03-19T00:07:09 which is "anything other than latest pip is insane" 2015-03-19T00:07:11 sure 2015-03-19T00:07:28 clarkb, mordred: 155072 includes a formal adoption of it 2015-03-19T00:07:30 I'm just saying that if we're pinning to an out of date venv - sorry, that's my bad 2015-03-19T00:08:04 and we should stop doing it 2015-03-19T00:08:09 *** sdake has quit IRC 2015-03-19T00:08:11 because we test with latest pip in devstack 2015-03-19T00:08:13 *** mtanino has quit IRC 2015-03-19T00:08:30 and testing with older pip elsewhere because I suck is a bad policy :) 2015-03-19T00:08:39 mordred please suck less, thx 2015-03-19T00:08:52 harlowja_: listen man - there is only so much suck I can remove on a given day 2015-03-19T00:08:57 whoa, pip has ~= now? what does that do? 2015-03-19T00:09:03 * jroll not sure if he wants to know 2015-03-19T00:09:05 also Istill don't know why trusty works 2015-03-19T00:09:17 mordred: ^ which was mostly I think what I wanted clarification on before we changed anything 2015-03-19T00:09:24 jroll https://www.python.org/dev/peps/pep-0440/#compatible-release 2015-03-19T00:09:32 it appears the only place we install virtualenv is in the base.pp manifest 2015-03-19T00:09:40 and yet trusty nodes don't seem to get downgraded 2015-03-19T00:09:55 *** jamielennox is now known as jamielennox|away 2015-03-19T00:10:00 *** liusheng has quit IRC 2015-03-19T00:10:00 huh 2015-03-19T00:10:21 clarkb: hrm 2015-03-19T00:10:22 lol, jroll same response, idk either :-P 2015-03-19T00:10:50 *** liusheng has joined #openstack-infra 2015-03-19T00:10:55 00:06:11 clarkb | there is no benefit to change to ~= so lets just leave it be <- I like this argument 2015-03-19T00:11:07 *** SumitNaiksatam has quit IRC 2015-03-19T00:11:18 *** jeffgdl has joined #openstack-infra 2015-03-19T00:11:19 but then again I would rather == would work 2015-03-19T00:12:05 *** garyh has quit IRC 2015-03-19T00:12:07 *** ddieterly has joined #openstack-infra 2015-03-19T00:12:13 i start to feel that someone was taking a little to many drugs when making up ~= as a version identifier 2015-03-19T00:12:13 lol 2015-03-19T00:12:30 *** mrmartin has quit IRC 2015-03-19T00:12:37 I've only ever seen that in npm things 2015-03-19T00:12:42 so.... you may be right 2015-03-19T00:12:55 maybe they share(d) drugs then 2015-03-19T00:12:57 ha 2015-03-19T00:13:17 *** otter768 has joined #openstack-infra 2015-03-19T00:13:18 *** dims__ is now known as dims 2015-03-19T00:13:35 i 2015-03-19T00:13:50 *** sdake has joined #openstack-infra 2015-03-19T00:13:55 *i'd rather have the versions be simple, have a nice solid dependency resolver, and then maybe start adding these edge-cases 2015-03-19T00:14:23 fungi: is anyone actively looking at https://review.openstack.org/#/c/165620/ (Temporarily stop adding devstack-f20 nodes in rax)? 2015-03-19T00:14:34 harlowja_: yeah 2015-03-19T00:14:48 fungi: clarkb: some oslo.messaging -docs, -tarball are stuck in the -post queue for a while. do i need to re-jiggle them some how? 2015-03-19T00:15:11 *** achanda has quit IRC 2015-03-19T00:15:43 dims: we need more nodes, I am currently cleaning up leaked nodes in hpcloud to hopefully free up capacity 2015-03-19T00:16:22 Spencer Krum proposed openstack-infra/puppet-zuul: Modernize template syntax https://review.openstack.org/157136 2015-03-19T00:17:07 clarkb: thanks (i saw the "zuul jobs" and "gerrit events" charts go south and thought i may have to do something) 2015-03-19T00:17:41 *** otter768 has quit IRC 2015-03-19T00:17:47 dims: thanks for keeping an eye on that 2015-03-19T00:17:49 *** sdake_ has quit IRC 2015-03-19T00:18:08 anteaya: :) 2015-03-19T00:18:13 dims: when the gate shifts and zuul needs to calculate the merge-check queue it can spike for a few minutes 2015-03-19T00:18:23 I've seen event go up over 8000 2015-03-19T00:18:25 asselin: Hey, im working on some fakes testing and im hitting the bug you patch https://review.openstack.org/#/c/162313 fixes, id like to base a test off it but I need to rebase your change off master if thats ok? 2015-03-19T00:18:26 gotcha 2015-03-19T00:18:29 Nan Liu proposed openstack-infra/puppet-elasticsearch: Use archive to fetch/checksum instead of wget. https://review.openstack.org/164908 2015-03-19T00:18:29 but they don't last long 2015-03-19T00:18:40 dims: but you are looking in the right spots 2015-03-19T00:19:03 greghaynes, sure go ahead 2015-03-19T00:20:11 thanks anteaya 2015-03-19T00:21:11 :) 2015-03-19T00:21:46 *** rfolco has quit IRC 2015-03-19T00:21:57 Jeremy Stanley proposed openstack-infra/project-config: Add docs jobs for openstack-infra/openstackid repo https://review.openstack.org/165661 2015-03-19T00:22:57 ianw: i have no idea. it was agreed as a good thing to try, but got no reviews once uploaded 2015-03-19T00:23:04 *** jamielennox|away is now known as jamielennox 2015-03-19T00:23:58 fungi: i mean the failure to boot; just trying to grab some logs on this now 2015-03-19T00:24:21 fungi: maybe just apply by hand and see if it makes any difference? 2015-03-19T00:24:46 ianw: we get no logs. we never get an instance returned by nova when we request it to boot 2015-03-19T00:25:25 clarkb: i was tempted to do that, but then worried i might accidentally leave puppet disabled on nodepool so thought better of it 2015-03-19T00:25:58 clarkb: so if i was going to apply it by hand, better to just self-approve that change i guess (i proposed a wip revert along with it for everyone's convenience) 2015-03-19T00:26:06 which i guess i can do 2015-03-19T00:26:52 *** tkelsey has joined #openstack-infra 2015-03-19T00:30:19 *** gyee has quit IRC 2015-03-19T00:31:04 *** tkelsey has quit IRC 2015-03-19T00:32:01 *** __TheDodd__ has quit IRC 2015-03-19T00:32:40 *** MarkAtwood has quit IRC 2015-03-19T00:32:45 fungi: alright, well if i can help i will, obviously console logs, if it gets that far, would be good 2015-03-19T00:33:58 *** jeffgdl has quit IRC 2015-03-19T00:34:00 *** ZZelle_ has quit IRC 2015-03-19T00:34:27 *** tonytan4ever has quit IRC 2015-03-19T00:34:56 *** Sukhdev has quit IRC 2015-03-19T00:37:02 *** markvoelker has quit IRC 2015-03-19T00:37:18 ianw: i connected to the console of one at random. i'm getting "boot: No DEFAULT or UI configuration directive found!" looping over and over there 2015-03-19T00:37:36 but i'm about to knock off for the night. getting lateish here 2015-03-19T00:38:18 *** e0ne has quit IRC 2015-03-19T00:38:52 *** tonytan4ever has joined #openstack-infra 2015-03-19T00:39:47 ianw: i'm guessing it's something not quite right with the bootloader installed to the image 2015-03-19T00:41:32 hi, i have a nodepool question… is there a way to setup nodes with multiple floating ip’s each coming from a different pool? 2015-03-19T00:41:58 i can make nodes with multiple networks, and a single floating ip from the pool specified in the providers ‘pool’ setting 2015-03-19T00:42:11 but didn’t see anything for other nics that are configured 2015-03-19T00:42:30 patrickeast: did you get a private network for nodepool? 2015-03-19T00:42:31 fungi: alright, i'll try to replicate a little 2015-03-19T00:43:09 anteaya: heh still waiting on that, in the mean time i am playing with setting up iscsi multipathing to our flash array on test nodes 2015-03-19T00:43:35 anteaya: should be soon, waiting on some new cables or something they needed to put it into the dmz 2015-03-19T00:43:40 ianw: thanks! 2015-03-19T00:43:52 *** baoli has quit IRC 2015-03-19T00:43:57 patrickeast: okay 2015-03-19T00:44:26 patrickeast: and I have no idea how to answer your original question 2015-03-19T00:45:00 *** marun has quit IRC 2015-03-19T00:45:08 *** asettle is now known as asettle-lunch 2015-03-19T00:45:47 alien nodes are cleaned up in hpcloud now 2015-03-19T00:46:32 patrickeast: there is a way to specify multiple networks I don't think you can specify multiple floating ips though 2015-03-19T00:46:51 clarkb: yea that lines up with what i’m seeing in the docs 2015-03-19T00:47:04 figured I would check before hacking away at the code 2015-03-19T00:47:05 clarkb: yay 2015-03-19T00:47:17 patrickeast: what are you trying to accomplish with multiple floating ips? 2015-03-19T00:48:05 we need to route traffic through a handful of 10Gb nic’s on the compute nodes to the test nodes so they can reach the iscsi ports on our flash array to test out multipathing 2015-03-19T00:48:14 so each node would have 3 networks, management and 2x data 2015-03-19T00:48:19 *** cdent has quit IRC 2015-03-19T00:48:38 Merged openstack-infra/system-config: Temporarily stop adding devstack-f20 nodes in rax https://review.openstack.org/165620 2015-03-19T00:48:41 all 3 of which are on different “external” networks as far as openstack is concerned 2015-03-19T00:49:06 *** mfink_ has joined #openstack-infra 2015-03-19T00:49:38 patrickeast: couldn't you do that all without floating ips? 2015-03-19T00:49:48 patrickeast: just put them on the appropriate network 2015-03-19T00:50:49 *** claudiub has quit IRC 2015-03-19T00:50:51 *** ibiris is now known as ibiris_away 2015-03-19T00:51:00 clarkb: mm that would be easier, can I put instances directly on a neutron network/subnet that is external w/ a bridge? 2015-03-19T00:52:04 fungi: yeah, just upgrading kernel & rebooting f20 on rax it's not happy with that error, so that's kind of good in that it's not specific ... 2015-03-19T00:52:46 * patrickeast goes back to reading neutron docs 2015-03-19T00:53:09 ianw: awesome--glad it's not hard to reproduce. hopefully shortly we'll know whether hpcloud does any better in that regard 2015-03-19T00:53:44 patrickeast: I am not sure how neutron does it but if neutron can set up NAT rules in iptables for internal:external mappings then it should be able to just use IPs on those networks direclty 2015-03-19T00:53:46 *** nanliu has quit IRC 2015-03-19T00:53:49 I certainly hope that neutron allows that :/ 2015-03-19T00:53:50 *** yamamoto has quit IRC 2015-03-19T00:53:53 ianw: is the depends-on in https://review.openstack.org/164901 a typo? 2015-03-19T00:54:28 clarkb: hehe yea i would think so too 2015-03-19T00:54:58 *** bhuvan has quit IRC 2015-03-19T00:55:00 Merged openstack-infra/jenkins-job-builder: Add support to configure boolean parameter in PBT https://review.openstack.org/146665 2015-03-19T00:55:03 fungi: oh, that got abandoned ... should be the previous one; just a tick 2015-03-19T00:55:19 otherwise lgtm 2015-03-19T00:55:26 *** ddieterl_ has joined #openstack-infra 2015-03-19T00:55:41 Ian Wienand proposed openstack-infra/system-config: Add nodepool config-validation environment https://review.openstack.org/164901 2015-03-19T00:56:01 fungi: ^ updated, thanks 2015-03-19T00:56:14 *** Rockyg has quit IRC 2015-03-19T00:56:29 *** ddieterly has quit IRC 2015-03-19T00:57:34 i see a devstack-f20 node building in hpcloud-b5 for 0.8 hours, so i think it's just unhappy everywheres 2015-03-19T00:57:40 everwheres 2015-03-19T00:57:54 *** isq has quit IRC 2015-03-19T00:58:30 i'll go ahead and approve the revert 2015-03-19T00:58:35 *** jp_at_hp has quit IRC 2015-03-19T00:58:40 ianw: so - you enjoy diskimage-builder, right? 2015-03-19T00:58:42 :) 2015-03-19T00:59:36 ianw: I made this a while back: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/ubuntu-minimal ... and a similar one http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/centos-minimal 2015-03-19T00:59:52 mordred: well ... sometimes. my last change took about 6 months to merge 2015-03-19T00:59:58 ianw: :) 2015-03-19T01:00:09 ianw: the ubuntu one does what Id expect - which is make a super minimal base image that's similar in size and scope to the ubuntu docker image 2015-03-19T01:00:23 ianw: the centos one is ... larger - and I think I did it wrong ... and I don't have a fedora one at all 2015-03-19T01:00:43 *** bknudson has joined #openstack-infra 2015-03-19T01:00:56 *** Sukhdev has joined #openstack-infra 2015-03-19T01:00:58 ianw: if you get bored at some point and wanted to make a better one for centos/fedora that wound up with something tiny like a docker image ... I will send a freshly dried alpaca to your house 2015-03-19T01:01:20 mordred: so one thing that stands out immediately is http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/centos-minimal/pkg-map <- redhat-lsb brings in a whole bunch of stuff, X and cups etc 2015-03-19T01:01:34 that should probably be redhat-lsb-core 2015-03-19T01:01:38 ianw: yeah - it's the only way I could figure out how to get the lsb_release command ... AH 2015-03-19T01:01:44 see - this is you knowing more things than me 2015-03-19T01:01:47 *** tonytan4ever has quit IRC 2015-03-19T01:02:30 ianw: I was also told that perhaps rinse is not actually a great way to accomplish this ... and perhaps something about just using rpm with an alternate root *me waves hands wildly* 2015-03-19T01:02:36 mordred: also `rpm -qf lsb_release` 2015-03-19T01:03:16 that is assuming you already have the packages installed, that will tell you which one is actually providing it 2015-03-19T01:04:08 i'm sure yum has something akin to apt-file for searching filenames of packages available on the package mirror but that you haven't installed, i just don't know the cli 2015-03-19T01:04:47 mordred: so don't we have the whole atomic thing? is there overlap here? 2015-03-19T01:05:02 ianw: it's not quite the same thing ... I don't actually want docker 2015-03-19T01:05:26 I want actual base images - just small without the things that my distro providers think I want in a "cloud image" 2015-03-19T01:05:39 because I have this whole puppet infrastructure where I declare the dependencies I actually want :) 2015-03-19T01:05:43 clarkb: hah nice, thanks for the suggestion… I was making that much more painful than it had to be, although now we need to setup the /etc/network/interface.d/ script for the node with its static ip (the floating ips let us use the neutron dhcp), do you know if the /etc/nodepool/node_private file has *all* of the private ip addresses for a node so we can set it up with a ready script? 2015-03-19T01:06:59 patrickeast: fyi - I'm in the middle of replacing the openstack calls in nodepool with the shade library - so if you do find that there are some places where you need enhancements to the openstack interactions - like the floating-ips thing or richer handling of multiple nics - it's probably functionality we want to add to shade too ... 2015-03-19T01:07:10 *** bhunter71 has joined #openstack-infra 2015-03-19T01:08:33 mordred: oo nice, is there a launchpad or something for it where I could document any feature requests? 2015-03-19T01:09:08 mordred: you don't want a bootable image though? 2015-03-19T01:09:19 patrickeast: https://storyboard.openstack.org/#!/project/760 2015-03-19T01:09:32 *** Longgeek has joined #openstack-infra 2015-03-19T01:09:33 ianw: no, bootable is good - but there are already other elements to handle things like grub and whatnot 2015-03-19T01:10:26 ianw: the 'vm' element handles that 2015-03-19T01:11:04 mordred: right, ok, so rinse gives you even less than downloading the upstream .qcow 2015-03-19T01:11:09 *** isq has joined #openstack-infra 2015-03-19T01:11:30 upstream .qcow cloud images 2015-03-19T01:12:45 *** garyh has joined #openstack-infra 2015-03-19T01:14:18 *** pelix has quit IRC 2015-03-19T01:15:14 *** YorikSar has quit IRC 2015-03-19T01:15:44 *** achanda has joined #openstack-infra 2015-03-19T01:17:20 *** markvoelker has joined #openstack-infra 2015-03-19T01:17:31 oh! not approving the revert yet. we've got a used devstack-f20 node in hpcloud now 2015-03-19T01:17:42 so it does seem we can still boot them there 2015-03-19T01:18:14 fungi: so it's saying "boot: No DEFAULT or UI configuration directive found!" because there's no default configuration directive. who'd have thunk ... 2015-03-19T01:18:30 fungi: this looks like rackspace fiddling with extlinux.conf again ... similar thing happened with f21 2015-03-19T01:19:14 ianw: yeah, i've also had problems with them replacing grub2 with pv-grub in the mbr of my personal instances as well 2015-03-19T01:20:09 ianw, fungi: is this file-injection feeling things? 2015-03-19T01:20:22 not sure, but it would be nice if they would just not feel obligated to mess with bootloaders 2015-03-19T01:20:57 ianw, fungi: I think Ng got glance turned on on his rax account - so hopefully we shoudl have the non-nova-agent based element soon 2015-03-19T01:20:58 mordred: no, i think they edit /boot/extlinux.conf for some reason, and then the scripts in the rpm files that update it don't work properly 2015-03-19T01:21:09 *** achanda has quit IRC 2015-03-19T01:21:17 oh - you mean in the images they make for us 2015-03-19T01:21:43 yep 2015-03-19T01:21:44 yeah - I'm very much looking forward to getting our cloud providers out of the pipeline for making images for us altogether 2015-03-19T01:22:02 * mordred promises he's working on that 2015-03-19T01:22:28 *** markvoelker has quit IRC 2015-03-19T01:22:46 fungi: i'll communicate to jroll the issue, i can put a work-around into the prepare scripts 2015-03-19T01:22:48 *** mwagner_lap has joined #openstack-infra 2015-03-19T01:23:22 *** garyh has quit IRC 2015-03-19T01:23:52 ianw: ? 2015-03-19T01:24:02 that was fast ;) 2015-03-19T01:24:12 >.> 2015-03-19T01:24:14 * jroll runs away 2015-03-19T01:24:37 jroll: the rax f20 image is broken in a similar way to the f21 where extlinux.conf gets mangled by kernel updates. i'll get exact details and email 2015-03-19T01:24:41 ianw: shoot me an email or else I'll forget 2015-03-19T01:24:43 yeah 2015-03-19T01:24:45 thanks! 2015-03-19T01:27:25 *** mwagner_lap has quit IRC 2015-03-19T01:28:13 *** bhuvan has joined #openstack-infra 2015-03-19T01:28:42 *** garyh has joined #openstack-infra 2015-03-19T01:30:30 thanks as always jroll! 2015-03-19T01:34:16 :) 2015-03-19T01:35:26 Joshua Hesketh proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 2015-03-19T01:36:01 Sukhdev Kapur proposed openstack-infra/project-config: Adding create permissions for networking-arista https://review.openstack.org/165673 2015-03-19T01:38:57 *** garyh has quit IRC 2015-03-19T01:40:44 *** mwagner_lap has joined #openstack-infra 2015-03-19T01:43:10 *** asettle-lunch is now known as asettle 2015-03-19T01:43:22 *** ivar-laz_ has joined #openstack-infra 2015-03-19T01:44:44 fungi: Hi there - I just pushed the patch based upon your suggestion. Can you please give it a review - https://review.openstack.org/#/c/165673/ 2015-03-19T01:46:30 *** spzala has quit IRC 2015-03-19T01:46:54 *** ivar-lazzaro has quit IRC 2015-03-19T01:47:44 *** ivar-laz_ has quit IRC 2015-03-19T01:48:03 *** isq has quit IRC 2015-03-19T01:49:06 *** ChuckC has joined #openstack-infra 2015-03-19T01:51:06 *** unicell has quit IRC 2015-03-19T01:52:03 hey guys, i linked my ci to the sandbox repo to test, and i checked back and both of my slaves are offline saying "offline due to gearman request"... anyone know what that means? 2015-03-19T01:53:40 rlucio: it means you've copied our configuration and are using an OFFLINE_NODE_WHEN_COMPLETE parameter in your jobs. we do this because we don't reuse workers, but delete them after a job completes and use a fresh one for each new job 2015-03-19T01:54:14 *** unicell has joined #openstack-infra 2015-03-19T01:54:39 *** unicell has quit IRC 2015-03-19T01:54:52 *** unicell has joined #openstack-infra 2015-03-19T01:55:02 fungi: ah ha, cool. yea i have static workers so i'll have to change that 2015-03-19T01:55:05 anteaya: Ping 2015-03-19T01:55:11 fungi: thanks! 2015-03-19T01:55:57 rlucio: we have some static workers too, and for those we have a "reusable_node" parameter function which unsets that parameter for us 2015-03-19T01:55:58 *** isq has joined #openstack-infra 2015-03-19T01:56:43 fungi: yea i see that defined in zuul/openstack_functions.py 2015-03-19T01:56:49 *** marun has joined #openstack-infra 2015-03-19T01:57:27 rlucio: if you look at our zuul/layout.yaml you'll see where we set it for jobs we're running on reusable nodes 2015-03-19T01:57:36 *** bhunter71 has quit IRC 2015-03-19T01:58:05 *** mwagner_lap has quit IRC 2015-03-19T01:58:09 Hi Infra cores - I need one more blessing on this patch, please - https://review.openstack.org/#/c/165673/ 2015-03-19T01:59:14 rlucio: i take that back. we just apply it based on pattern matches of the job names these days. we used to set it from layout.yaml but i guess not any longer 2015-03-19T01:59:49 fungi: right, just looking at that now, in the jobs: section of layout.yaml 2015-03-19T02:00:05 *** SumitNaiksatam has joined #openstack-infra 2015-03-19T02:00:11 fungi: simple enough to fix :) 2015-03-19T02:00:14 Merged openstack-infra/nodepool: Fix Nodepool DIB Uploads to Glance w/o region_name https://review.openstack.org/162313 2015-03-19T02:01:28 *** marun has quit IRC 2015-03-19T02:01:33 rlucio: yep. or just don't bother with the parameter function script we use. depending on your situation it's like a lot of unnecessary complication 2015-03-19T02:01:54 s/like/likely/ 2015-03-19T02:01:57 *** ajmiller has joined #openstack-infra 2015-03-19T02:02:59 *** unicell has quit IRC 2015-03-19T02:06:20 *** ddieterly has joined #openstack-infra 2015-03-19T02:06:36 *** ddieterl_ has quit IRC 2015-03-19T02:11:18 *** mwagner_lap has joined #openstack-infra 2015-03-19T02:11:35 *** Sukhdev has quit IRC 2015-03-19T02:14:08 *** harlowja_ is now known as harlowja_away 2015-03-19T02:14:09 *** otter768 has joined #openstack-infra 2015-03-19T02:16:08 *** Alexandra_ has joined #openstack-infra 2015-03-19T02:16:37 *** asettle has quit IRC 2015-03-19T02:18:24 *** markvoelker has joined #openstack-infra 2015-03-19T02:18:50 *** otter768 has quit IRC 2015-03-19T02:20:28 *** reed has quit IRC 2015-03-19T02:22:40 *** macjack1 has joined #openstack-infra 2015-03-19T02:23:02 *** markvoelker has quit IRC 2015-03-19T02:23:17 *** macjack1 has quit IRC 2015-03-19T02:24:25 *** yamahata has quit IRC 2015-03-19T02:27:14 *** achanda has joined #openstack-infra 2015-03-19T02:27:20 *** sigmavirus24_awa is now known as sigmavirus24 2015-03-19T02:28:07 *** tkelsey has joined #openstack-infra 2015-03-19T02:28:33 *** baoli has joined #openstack-infra 2015-03-19T02:28:38 *** dims has quit IRC 2015-03-19T02:29:19 clarkb: we need a good novaclient fake for this to work 2015-03-19T02:29:20 :( 2015-03-19T02:32:19 *** tkelsey has quit IRC 2015-03-19T02:33:12 *** kaisers has joined #openstack-infra 2015-03-19T02:33:25 *** kaisers1 has quit IRC 2015-03-19T02:35:23 *** zz_dimtruck is now known as dimtruck 2015-03-19T02:36:46 *** dmorita_ has quit IRC 2015-03-19T02:38:46 *** otter768 has joined #openstack-infra 2015-03-19T02:40:19 *** unicell has joined #openstack-infra 2015-03-19T02:42:52 *** macjack1 has joined #openstack-infra 2015-03-19T02:44:49 *** unicell has quit IRC 2015-03-19T02:45:13 *** unicell has joined #openstack-infra 2015-03-19T02:48:18 *** macjack1 has quit IRC 2015-03-19T02:49:59 jroll: looks like it's a fedora bug 2015-03-19T02:50:23 ianw: cool, I don't feel so bad now! 2015-03-19T02:50:27 * jroll deflects to redhat employees 2015-03-19T02:51:20 *** erlon has quit IRC 2015-03-19T02:52:23 * ianw curses anyone who thinks "i've got a need to open and close a bunch of files and mangle a lot of strings ... i think i'll write this in C" 2015-03-19T02:52:52 s/open and close a bunch of files and mangle a lot of strings/do things with computers/ 2015-03-19T02:52:57 (mostly joking) 2015-03-19T02:55:26 *** mrmartin has joined #openstack-infra 2015-03-19T02:56:21 *** harlowja_at_home has joined #openstack-infra 2015-03-19T02:58:52 *** patrickeast has quit IRC 2015-03-19T03:01:32 *** spzala has joined #openstack-infra 2015-03-19T03:04:25 *** coolsvap|afk is now known as coolsvap 2015-03-19T03:08:27 *** stevemar has joined #openstack-infra 2015-03-19T03:11:53 *** harlowja_at_home has quit IRC 2015-03-19T03:11:57 *** spzala has quit IRC 2015-03-19T03:13:07 *** otter768 has quit IRC 2015-03-19T03:14:36 hi infra. is there an easy way to see a list of a particular job? i tried here, but it's empty: https://jenkins.openstack.org/job/check-neutron-lbaas-dsvm-api/ i'm wanting to monitor it for stability. 2015-03-19T03:17:05 *** mmedvede has quit IRC 2015-03-19T03:19:15 *** Somay has joined #openstack-infra 2015-03-19T03:22:22 *** sdake_ has joined #openstack-infra 2015-03-19T03:25:24 dougwig: i think you probably want to consult http://graphite.openstack.org/ 2015-03-19T03:26:20 *** sdake has quit IRC 2015-03-19T03:28:32 ianw: ty 2015-03-19T03:29:19 *** camunoz_gone is now known as camunoz 2015-03-19T03:32:25 *** coolsvap is now known as coolsvap|afk 2015-03-19T03:33:42 *** camunoz has joined #openstack-infra 2015-03-19T03:34:24 *** changbl has joined #openstack-infra 2015-03-19T03:36:03 *** sputnik13 has joined #openstack-infra 2015-03-19T03:38:32 Ian Wienand proposed openstack-infra/system-config: Work-around grubby boot failure on Fedora 20 https://review.openstack.org/165681 2015-03-19T03:38:33 *** camunoz has quit IRC 2015-03-19T03:39:46 *** dimtruck is now known as zz_dimtruck 2015-03-19T03:39:49 *** coolsvap|afk is now known as coolsvap 2015-03-19T03:40:53 fungi: ^ when you look at scrollback 2015-03-19T03:40:55 *** yamamoto has joined #openstack-infra 2015-03-19T03:41:55 Ian Wienand proposed openstack-infra/system-config: Revert "Temporarily stop adding devstack-f20 nodes in rax" https://review.openstack.org/165621 2015-03-19T03:42:59 *** achanda has quit IRC 2015-03-19T03:43:33 *** greghaynes has quit IRC 2015-03-19T03:44:43 *** achanda has joined #openstack-infra 2015-03-19T03:46:31 *** macjack has joined #openstack-infra 2015-03-19T03:47:43 *** mrmartin has quit IRC 2015-03-19T03:47:49 *** ddieterly has quit IRC 2015-03-19T03:49:04 greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 2015-03-19T03:49:04 greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 2015-03-19T03:50:23 greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 2015-03-19T03:50:28 *** woodster_ has quit IRC 2015-03-19T03:51:48 *** camunoz has joined #openstack-infra 2015-03-19T03:54:27 greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 2015-03-19T03:54:55 *** garyh has joined #openstack-infra 2015-03-19T03:55:04 *** tqtran has quit IRC 2015-03-19T03:55:16 *** sdake has joined #openstack-infra 2015-03-19T03:56:22 *** macjack has quit IRC 2015-03-19T03:56:22 *** tqtran has joined #openstack-infra 2015-03-19T03:56:41 greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 2015-03-19T03:56:47 greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 2015-03-19T03:57:27 *** sigmavirus24 is now known as sigmavirus24_awa 2015-03-19T03:59:22 *** sdake_ has quit IRC 2015-03-19T04:00:14 *** garyh has quit IRC 2015-03-19T04:04:39 *** sdake has quit IRC 2015-03-19T04:05:11 *** achanda has quit IRC 2015-03-19T04:06:00 greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 2015-03-19T04:06:29 *** jamielennox is now known as jamielennox|away 2015-03-19T04:06:53 greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 2015-03-19T04:09:04 *** VijayTripathi has joined #openstack-infra 2015-03-19T04:10:00 *** baoli has quit IRC 2015-03-19T04:10:35 *** mmedvede has joined #openstack-infra 2015-03-19T04:11:26 greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 2015-03-19T04:15:01 *** Alexandra_ has quit IRC 2015-03-19T04:19:30 *** dims has joined #openstack-infra 2015-03-19T04:19:50 *** markvoelker has joined #openstack-infra 2015-03-19T04:22:14 *** koolhead17 has joined #openstack-infra 2015-03-19T04:24:16 *** markvoelker has quit IRC 2015-03-19T04:26:16 *** carl_baldwin has joined #openstack-infra 2015-03-19T04:27:25 *** dims has quit IRC 2015-03-19T04:31:55 *** Sukhdev has joined #openstack-infra 2015-03-19T04:33:57 *** markvan has quit IRC 2015-03-19T04:33:58 Khai Do proposed openstack-infra/system-config: Fix jquery setup on Gerrit server. https://review.openstack.org/165145 2015-03-19T04:34:04 *** mmedvede has quit IRC 2015-03-19T04:35:08 *** markvan has joined #openstack-infra 2015-03-19T04:37:22 *** achanda has joined #openstack-infra 2015-03-19T04:38:47 *** sdake has joined #openstack-infra 2015-03-19T04:39:34 *** dboik_ has quit IRC 2015-03-19T04:40:16 *** amotoki has joined #openstack-infra 2015-03-19T04:40:32 greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 2015-03-19T04:41:14 *** achanda has quit IRC 2015-03-19T04:45:12 greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 2015-03-19T04:46:54 *** dboik has joined #openstack-infra 2015-03-19T04:47:21 *** harlowja_at_home has joined #openstack-infra 2015-03-19T04:48:36 *** ddieterly has joined #openstack-infra 2015-03-19T04:51:19 greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 2015-03-19T04:51:46 greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 2015-03-19T04:53:03 *** ddieterly has quit IRC 2015-03-19T05:00:50 *** garyh has joined #openstack-infra 2015-03-19T05:03:15 *** __afazekas is now known as afazekas 2015-03-19T05:07:57 *** yamamoto has quit IRC 2015-03-19T05:08:46 *** achanda has joined #openstack-infra 2015-03-19T05:09:30 *** garyh has quit IRC 2015-03-19T05:11:05 Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 2015-03-19T05:12:36 *** sdake has quit IRC 2015-03-19T05:14:12 *** otter768 has joined #openstack-infra 2015-03-19T05:18:37 *** otter768 has quit IRC 2015-03-19T05:20:40 *** markvoelker has joined #openstack-infra 2015-03-19T05:25:33 *** jyuso1 has joined #openstack-infra 2015-03-19T05:25:37 *** markvoelker has quit IRC 2015-03-19T05:26:27 *** liusheng has quit IRC 2015-03-19T05:26:49 *** liusheng has joined #openstack-infra 2015-03-19T05:27:48 *** armax has quit IRC 2015-03-19T05:28:33 *** mmedvede has joined #openstack-infra 2015-03-19T05:31:59 *** yfried|afk is now known as yfried 2015-03-19T05:32:24 *** yamamoto has joined #openstack-infra 2015-03-19T05:34:28 *** dannywilson has quit IRC 2015-03-19T05:39:38 *** greghaynes has joined #openstack-infra 2015-03-19T05:39:53 *** carl_baldwin has quit IRC 2015-03-19T05:40:06 *** harlowja_at_home has quit IRC 2015-03-19T05:41:08 *** coolsvap is now known as coolsvap|afk 2015-03-19T05:45:08 *** ChuckC has quit IRC 2015-03-19T05:45:16 *** MarkAtwood has joined #openstack-infra 2015-03-19T05:48:42 *** coolsvap|afk is now known as coolsvap 2015-03-19T05:49:21 *** ddieterly has joined #openstack-infra 2015-03-19T05:53:55 *** ddieterly has quit IRC 2015-03-19T05:55:38 *** MarkAtwood has quit IRC 2015-03-19T05:58:33 *** ibiris_away is now known as ibiris 2015-03-19T05:59:18 *** yfried is now known as yfried|afk 2015-03-19T06:00:38 *** stevemar has quit IRC 2015-03-19T06:05:13 *** sarob has quit IRC 2015-03-19T06:09:19 *** subscope_ has joined #openstack-infra 2015-03-19T06:10:07 *** garyh has joined #openstack-infra 2015-03-19T06:10:59 *** BharatK has joined #openstack-infra 2015-03-19T06:12:54 *** yfried|afk has quit IRC 2015-03-19T06:13:04 *** dims has joined #openstack-infra 2015-03-19T06:19:48 *** dims has quit IRC 2015-03-19T06:20:10 *** garyh has quit IRC 2015-03-19T06:21:22 *** markvoelker has joined #openstack-infra 2015-03-19T06:24:26 *** bswartz has quit IRC 2015-03-19T06:26:12 *** markvoelker has quit IRC 2015-03-19T06:28:03 *** achanda has quit IRC 2015-03-19T06:29:10 *** tkelsey has joined #openstack-infra 2015-03-19T06:31:49 *** Sukhdev has quit IRC 2015-03-19T06:32:38 *** bswartz has joined #openstack-infra 2015-03-19T06:33:32 *** pcaruana has quit IRC 2015-03-19T06:33:34 *** tkelsey has quit IRC 2015-03-19T06:35:03 *** mfink_ has quit IRC 2015-03-19T06:37:31 *** mfink_ has joined #openstack-infra 2015-03-19T06:50:34 *** ddieterly has joined #openstack-infra 2015-03-19T06:50:48 *** kiseok7_ has joined #openstack-infra 2015-03-19T06:52:49 *** achanda has joined #openstack-infra 2015-03-19T06:53:17 *** jcoufal has joined #openstack-infra 2015-03-19T06:55:00 *** ddieterly has quit IRC 2015-03-19T07:00:01 *** achanda has quit IRC 2015-03-19T07:04:14 *** ujuc has joined #openstack-infra 2015-03-19T07:04:45 *** achanda has joined #openstack-infra 2015-03-19T07:05:41 *** sarob has joined #openstack-infra 2015-03-19T07:07:21 *** subscope_ has quit IRC 2015-03-19T07:08:59 *** jyuso1 has quit IRC 2015-03-19T07:09:25 *** pblaho has joined #openstack-infra 2015-03-19T07:11:15 *** sarob has quit IRC 2015-03-19T07:11:58 *** kiseok7_ has quit IRC 2015-03-19T07:12:47 Fatih Degirmenci proposed openstack-infra/jenkins-job-builder: Pass through git revision to triggered jobs https://review.openstack.org/163149 2015-03-19T07:12:50 Andreas Jaeger proposed openstack-infra/project-config: Fix failure detection in irc access check https://review.openstack.org/165508 2015-03-19T07:13:04 Andreas Jaeger proposed openstack-infra/project-config: Fix failure detection in irc access check https://review.openstack.org/165508 2015-03-19T07:13:57 *** tnovacik has joined #openstack-infra 2015-03-19T07:15:02 *** otter768 has joined #openstack-infra 2015-03-19T07:18:18 *** bhuvan has quit IRC 2015-03-19T07:19:57 *** otter768 has quit IRC 2015-03-19T07:20:46 *** garyh has joined #openstack-infra 2015-03-19T07:22:11 *** markvoelker has joined #openstack-infra 2015-03-19T07:25:48 *** yfried|afk has joined #openstack-infra 2015-03-19T07:26:29 *** markvoelker has quit IRC 2015-03-19T07:27:49 Merged openstack-infra/project-config: Move barbican client experimental job to voting gate job https://review.openstack.org/165534 2015-03-19T07:28:07 Merged openstack-infra/project-config: Adding create permissions for networking-arista https://review.openstack.org/165673 2015-03-19T07:30:15 *** yfried has joined #openstack-infra 2015-03-19T07:30:35 *** yfried|afk has quit IRC 2015-03-19T07:31:01 *** garyh has quit IRC 2015-03-19T07:33:30 *** scheuran has joined #openstack-infra 2015-03-19T07:35:02 *** VijayTripathi has quit IRC 2015-03-19T07:37:18 *** achanda has quit IRC 2015-03-19T07:38:13 *** HeOS has quit IRC 2015-03-19T07:38:21 *** ildikov has quit IRC 2015-03-19T07:42:21 *** mpaolino has joined #openstack-infra 2015-03-19T07:45:29 *** Bsony has joined #openstack-infra 2015-03-19T07:47:54 *** chlong has quit IRC 2015-03-19T07:50:46 *** mpavone has joined #openstack-infra 2015-03-19T07:51:20 *** ddieterly has joined #openstack-infra 2015-03-19T07:51:29 *** SumitNaiksatam has quit IRC 2015-03-19T07:53:15 *** belmoreira has joined #openstack-infra 2015-03-19T07:54:07 *** ominakov has joined #openstack-infra 2015-03-19T07:54:15 *** yfried is now known as yfried|afk 2015-03-19T07:54:35 *** ominakov has quit IRC 2015-03-19T07:54:44 *** ominakov_ has joined #openstack-infra 2015-03-19T07:55:53 *** ddieterly has quit IRC 2015-03-19T07:56:05 *** teran_ has joined #openstack-infra 2015-03-19T07:56:33 *** flaper87 has quit IRC 2015-03-19T07:56:33 *** flaper87 has joined #openstack-infra 2015-03-19T07:57:58 *** teran has quit IRC 2015-03-19T07:58:24 *** scheuran has quit IRC 2015-03-19T07:58:42 *** scheuran has joined #openstack-infra 2015-03-19T07:58:49 *** mrda is now known as mrda-away 2015-03-19T08:01:54 *** yfried|afk is now known as yfried 2015-03-19T08:02:04 *** mpaolino has quit IRC 2015-03-19T08:03:44 *** mpaolino has joined #openstack-infra 2015-03-19T08:04:23 *** dims has joined #openstack-infra 2015-03-19T08:07:00 *** markus_z has joined #openstack-infra 2015-03-19T08:10:14 *** dims has quit IRC 2015-03-19T08:11:44 *** hashar has joined #openstack-infra 2015-03-19T08:14:09 *** e0ne has joined #openstack-infra 2015-03-19T08:14:47 *** ildikov has joined #openstack-infra 2015-03-19T08:15:11 *** koolhead17 has quit IRC 2015-03-19T08:15:32 *** e0ne has quit IRC 2015-03-19T08:16:09 mornig all 2015-03-19T08:16:45 *** koolhead17 has joined #openstack-infra 2015-03-19T08:17:21 *** jp_at_hp has joined #openstack-infra 2015-03-19T08:17:47 *** arxcruz has joined #openstack-infra 2015-03-19T08:18:45 *** jp_at_hp has quit IRC 2015-03-19T08:19:46 *** yfried is now known as yfried|afk 2015-03-19T08:21:18 *** koolhead17 has quit IRC 2015-03-19T08:28:12 *** tkelsey has joined #openstack-infra 2015-03-19T08:29:06 *** deepakcs has joined #openstack-infra 2015-03-19T08:31:18 *** dboik has quit IRC 2015-03-19T08:31:39 *** garyh has joined #openstack-infra 2015-03-19T08:36:03 *** mpaolino has quit IRC 2015-03-19T08:37:07 *** jistr has joined #openstack-infra 2015-03-19T08:38:17 *** mpaolino has joined #openstack-infra 2015-03-19T08:39:17 *** spredzy|afk is now known as spredzy 2015-03-19T08:39:57 *** yfried|afk is now known as yfried 2015-03-19T08:40:30 *** teran_ has quit IRC 2015-03-19T08:41:39 *** garyh has quit IRC 2015-03-19T08:43:01 *** fabo has quit IRC 2015-03-19T08:49:30 *** mfink_ has quit IRC 2015-03-19T08:53:15 yolanda.robla proposed openstack-infra/puppet-cgit: Add configurable setting for MaxClients https://review.openstack.org/160885 2015-03-19T08:55:36 *** shardy_z is now known as shardy 2015-03-19T08:56:51 *** jlanoux has joined #openstack-infra 2015-03-19T08:57:58 *** flip214 has joined #openstack-infra 2015-03-19T08:58:28 *** andreykurilin_ has joined #openstack-infra 2015-03-19T08:59:23 *** nilasae has joined #openstack-infra 2015-03-19T09:00:03 clarkb: ping 2015-03-19T09:02:58 *** ujuc has quit IRC 2015-03-19T09:04:10 *** koolhead17 has joined #openstack-infra 2015-03-19T09:05:02 *** YorikSar has joined #openstack-infra 2015-03-19T09:05:20 *** fabo has joined #openstack-infra 2015-03-19T09:09:58 *** tqtran has quit IRC 2015-03-19T09:11:59 *** mfink_ has joined #openstack-infra 2015-03-19T09:13:42 *** koolhead17 has quit IRC 2015-03-19T09:14:55 *** skolekonov has joined #openstack-infra 2015-03-19T09:16:00 *** otter768 has joined #openstack-infra 2015-03-19T09:19:13 Anthony Michon proposed openstack-infra/project-config: Add cerberus-dashboard, python-cerberusclient, sticks, sticks-dashboard, python-sticksclient https://review.openstack.org/163876 2015-03-19T09:20:40 *** otter768 has quit IRC 2015-03-19T09:20:54 *** zz_johnthetubagu is now known as johnthetubaguy 2015-03-19T09:22:15 *** dims has joined #openstack-infra 2015-03-19T09:23:41 *** markvoelker has joined #openstack-infra 2015-03-19T09:23:55 *** HeOS has joined #openstack-infra 2015-03-19T09:26:50 *** dims has quit IRC 2015-03-19T09:27:16 *** Longgeek has quit IRC 2015-03-19T09:27:24 *** andreykurilin_ has quit IRC 2015-03-19T09:28:05 *** andreykurilin_ has joined #openstack-infra 2015-03-19T09:28:36 *** markvoelker has quit IRC 2015-03-19T09:28:36 *** freerunner has quit IRC 2015-03-19T09:29:17 *** Longgeek has joined #openstack-infra 2015-03-19T09:29:34 *** freerunner has joined #openstack-infra 2015-03-19T09:29:46 *** freerunner has quit IRC 2015-03-19T09:31:12 *** freerunner has joined #openstack-infra 2015-03-19T09:31:14 *** Longgeek has quit IRC 2015-03-19T09:31:29 *** freerunner has quit IRC 2015-03-19T09:33:43 *** freerunner has joined #openstack-infra 2015-03-19T09:33:51 *** mfink_ has quit IRC 2015-03-19T09:33:57 *** freerunner has quit IRC 2015-03-19T09:35:23 *** Somay has quit IRC 2015-03-19T09:35:55 *** ihrachyshka has joined #openstack-infra 2015-03-19T09:37:15 *** freerunner has joined #openstack-infra 2015-03-19T09:37:41 *** derekh has joined #openstack-infra 2015-03-19T09:37:41 *** freerunner has quit IRC 2015-03-19T09:37:55 *** Longgeek has joined #openstack-infra 2015-03-19T09:38:41 *** freerunner has joined #openstack-infra 2015-03-19T09:38:50 *** freerunner has quit IRC 2015-03-19T09:39:40 *** freerunner has joined #openstack-infra 2015-03-19T09:39:53 *** freerunner has quit IRC 2015-03-19T09:40:35 *** teran has joined #openstack-infra 2015-03-19T09:40:44 *** freerunner has joined #openstack-infra 2015-03-19T09:40:48 *** freerunner has quit IRC 2015-03-19T09:41:38 *** liusheng has quit IRC 2015-03-19T09:42:15 *** garyh has joined #openstack-infra 2015-03-19T09:42:41 *** liusheng has joined #openstack-infra 2015-03-19T09:43:23 *** teran_ has joined #openstack-infra 2015-03-19T09:43:23 *** ZZelle has quit IRC 2015-03-19T09:43:37 *** mfink_ has joined #openstack-infra 2015-03-19T09:43:49 *** ZZelle has joined #openstack-infra 2015-03-19T09:44:28 *** ihrachyshka has quit IRC 2015-03-19T09:44:45 *** freerunner has joined #openstack-infra 2015-03-19T09:45:14 *** teran has quit IRC 2015-03-19T09:45:20 *** freerunner has quit IRC 2015-03-19T09:45:24 yolanda.robla proposed openstack-infra/puppet-graphite: Add ability to configure retention policy https://review.openstack.org/161679 2015-03-19T09:45:29 *** ihrachyshka has joined #openstack-infra 2015-03-19T09:45:46 *** dims has joined #openstack-infra 2015-03-19T09:46:28 *** freerunner has joined #openstack-infra 2015-03-19T09:46:42 *** freerunner has quit IRC 2015-03-19T09:47:05 *** rlucio has quit IRC 2015-03-19T09:47:49 *** andreykurilin_ has quit IRC 2015-03-19T09:47:51 *** freerunner has joined #openstack-infra 2015-03-19T09:48:14 *** freerunner has quit IRC 2015-03-19T09:48:35 *** amotoki has quit IRC 2015-03-19T09:50:36 *** freerunner has joined #openstack-infra 2015-03-19T09:50:57 *** Trozz has quit IRC 2015-03-19T09:52:09 *** mpaolino has quit IRC 2015-03-19T09:52:39 *** ddieterly has joined #openstack-infra 2015-03-19T09:52:59 *** garyh has quit IRC 2015-03-19T09:53:22 *** Trozz has joined #openstack-infra 2015-03-19T09:53:39 *** mpavone has quit IRC 2015-03-19T09:54:06 *** dtantsur|afk is now known as dtantsur 2015-03-19T09:54:08 *** yamamoto has quit IRC 2015-03-19T09:54:16 *** Hal1 has joined #openstack-infra 2015-03-19T09:54:57 *** ujuc has joined #openstack-infra 2015-03-19T09:55:13 *** Trozz is now known as Trozz_ 2015-03-19T09:55:16 *** Trozz_ is now known as Trozz 2015-03-19T09:56:07 *** ssam2 has joined #openstack-infra 2015-03-19T09:57:19 *** ddieterly has quit IRC 2015-03-19T10:02:36 *** pelix has joined #openstack-infra 2015-03-19T10:04:08 *** ghostpl_ has joined #openstack-infra 2015-03-19T10:04:32 yolanda.robla proposed openstack-infra/puppet-jenkins: Add extension settings in jjb https://review.openstack.org/162152 2015-03-19T10:10:10 *** zzz_pczesno has joined #openstack-infra 2015-03-19T10:10:19 *** wznoinsk_ has joined #openstack-infra 2015-03-19T10:10:46 yolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created https://review.openstack.org/165029 2015-03-19T10:11:37 *** hichihara has quit IRC 2015-03-19T10:12:07 *** alex_xu has quit IRC 2015-03-19T10:12:12 *** pczesno has quit IRC 2015-03-19T10:13:07 *** Longgeek has quit IRC 2015-03-19T10:13:25 *** Longgeek has joined #openstack-infra 2015-03-19T10:13:28 *** wznoinsk has quit IRC 2015-03-19T10:14:05 *** alex_xu has joined #openstack-infra 2015-03-19T10:17:49 *** ihrachyshka has quit IRC 2015-03-19T10:18:00 *** mmedvede has quit IRC 2015-03-19T10:18:01 *** yamamoto has joined #openstack-infra 2015-03-19T10:18:38 *** claudiub has joined #openstack-infra 2015-03-19T10:19:17 *** pc_m has joined #openstack-infra 2015-03-19T10:23:09 *** e0ne has joined #openstack-infra 2015-03-19T10:24:25 *** markvoelker has joined #openstack-infra 2015-03-19T10:25:49 *** Hal1 has quit IRC 2015-03-19T10:26:10 *** Hal1 has joined #openstack-infra 2015-03-19T10:29:06 *** markvoelker has quit IRC 2015-03-19T10:29:10 *** rlandy has joined #openstack-infra 2015-03-19T10:29:35 *** ihrachyshka has joined #openstack-infra 2015-03-19T10:33:57 *** rlucio has joined #openstack-infra 2015-03-19T10:34:11 *** yfried is now known as yfried|afk 2015-03-19T10:39:48 *** mpavone has joined #openstack-infra 2015-03-19T10:40:02 *** ujuc_ has joined #openstack-infra 2015-03-19T10:40:09 *** ujuc has quit IRC 2015-03-19T10:41:08 *** coolsvap is now known as coolsvap|afk 2015-03-19T10:46:39 hey guys, are there any other options for hosting ci logs besides rolling your own server? like can you use dropbox or ftp or something? just wondering 2015-03-19T10:49:28 *** dalgaaf has joined #openstack-infra 2015-03-19T10:51:48 *** mmedvede has joined #openstack-infra 2015-03-19T10:53:34 *** garyh has joined #openstack-infra 2015-03-19T10:53:41 *** ddieterly has joined #openstack-infra 2015-03-19T10:54:21 *** yfried|afk is now known as yfried 2015-03-19T10:57:14 *** mpaolino has joined #openstack-infra 2015-03-19T10:58:01 *** ddieterly has quit IRC 2015-03-19T10:58:42 *** mmedvede has quit IRC 2015-03-19T10:59:18 *** Hal1 has quit IRC 2015-03-19T10:59:22 *** Hal2 has joined #openstack-infra 2015-03-19T11:00:00 *** amotoki has joined #openstack-infra 2015-03-19T11:00:14 *** mfink_ has quit IRC 2015-03-19T11:02:06 *** mpaolino has quit IRC 2015-03-19T11:03:01 *** mpaolino has joined #openstack-infra 2015-03-19T11:04:01 *** jaypipes has joined #openstack-infra 2015-03-19T11:04:03 *** garyh has quit IRC 2015-03-19T11:05:11 *** pcaruana has joined #openstack-infra 2015-03-19T11:07:16 *** ominakov_ has quit IRC 2015-03-19T11:07:33 *** ominakov_ has joined #openstack-infra 2015-03-19T11:09:27 *** sambetts has quit IRC 2015-03-19T11:11:01 *** mpaolino has quit IRC 2015-03-19T11:11:19 *** sambetts has joined #openstack-infra 2015-03-19T11:11:38 *** mattfari_ has joined #openstack-infra 2015-03-19T11:12:28 *** mattfari_ has quit IRC 2015-03-19T11:13:08 *** macjack has joined #openstack-infra 2015-03-19T11:16:44 *** otter768 has joined #openstack-infra 2015-03-19T11:17:56 *** Somay has joined #openstack-infra 2015-03-19T11:21:25 *** otter768 has quit IRC 2015-03-19T11:21:39 *** mwagner_lap has quit IRC 2015-03-19T11:24:45 rlucio: I haven't heard about any restrictions on where you want to host the artifacts, as long as it follows the requirements from http://ci.openstack.org/third_party.html 2015-03-19T11:24:51 *** wznoinsk_ is now known as wznoinsk 2015-03-19T11:25:12 *** yamamoto has quit IRC 2015-03-19T11:25:13 *** markvoelker has joined #openstack-infra 2015-03-19T11:28:41 *** Somay has quit IRC 2015-03-19T11:30:01 *** markvoelker has quit IRC 2015-03-19T11:30:04 *** Somay has joined #openstack-infra 2015-03-19T11:30:22 *** deepakcs has quit IRC 2015-03-19T11:30:49 *** Hal2 has quit IRC 2015-03-19T11:30:50 *** Hal1 has joined #openstack-infra 2015-03-19T11:33:22 morning GheRivero 2015-03-19T11:36:12 *** coolsvap|afk is now known as coolsvap 2015-03-19T11:36:18 Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the nofifications plugin to support 1.9 https://review.openstack.org/165785 2015-03-19T11:37:39 jhesketh: you still up and about? 2015-03-19T11:37:40 *** e0ne is now known as e0ne_ 2015-03-19T11:38:08 I can no longer run os-loganalyze unit tests locally due to the swift code not failing gracefully 2015-03-19T11:38:45 Adrian Moisey proposed openstack-infra/jenkins-job-builder: Add missing ' https://review.openstack.org/165786 2015-03-19T11:42:43 *** e0ne_ has quit IRC 2015-03-19T11:47:00 *** mmedvede has joined #openstack-infra 2015-03-19T11:47:41 *** macjack has quit IRC 2015-03-19T11:48:49 *** e0ne has joined #openstack-infra 2015-03-19T11:49:05 *** spredzy is now known as spredzy|afk 2015-03-19T11:50:00 *** yamamoto has joined #openstack-infra 2015-03-19T11:50:28 Antoine Musso proposed openstack-infra/jenkins-job-builder: Stop mkdir output on each job write https://review.openstack.org/165789 2015-03-19T11:52:08 mordred: any idea why the # of usable test nodes seems so low? 2015-03-19T11:52:28 looks like we've only got about 150 in flight 2015-03-19T11:53:00 sdague: we've been having cloud issues with HP - we may have leaked zombies again ... lemme look 2015-03-19T11:53:07 *** koolhead17 has joined #openstack-infra 2015-03-19T11:53:37 oh. my 2015-03-19T11:53:41 that's ... interesting 2015-03-19T11:53:45 novaclient.exceptions.ClientException: An unexpected error prevented the server from fulfilling your request: [Errno 12] Cannot allocate memory (Disable debug mode to suppress these details.) (HTTP 500) 2015-03-19T11:54:05 *** weshay has joined #openstack-infra 2015-03-19T11:54:18 *** markvoelker has joined #openstack-infra 2015-03-19T11:54:22 sdague: is that actually passed through from the server? ^^ ?? 2015-03-19T11:54:23 *** ddieterly has joined #openstack-infra 2015-03-19T11:55:06 apparently if debug is enabled? 2015-03-19T11:55:09 that's interesting 2015-03-19T11:55:10 *** mmedvede has quit IRC 2015-03-19T11:57:40 nevermind - that's not hpcloud 2015-03-19T11:57:49 that's hp1.tripleo 2015-03-19T11:58:10 *** kgiusti has joined #openstack-infra 2015-03-19T11:59:08 *** ddieterly has quit IRC 2015-03-19T11:59:27 *** cdent has joined #openstack-infra 2015-03-19T11:59:31 heh 2015-03-19T11:59:38 *** lucasagomes has joined #openstack-infra 2015-03-19T12:00:36 *** aysyd has joined #openstack-infra 2015-03-19T12:00:42 *** woodster_ has joined #openstack-infra 2015-03-19T12:00:54 hi, if you have some time please take a look at https://review.openstack.org/#/c/163344/ it's adding a gate check for openstack/ironic-python-agent (non-voting) to test PXE drivers using the IPA ramdisk 2015-03-19T12:01:38 *** Hal1 has quit IRC 2015-03-19T12:01:43 we want IPA to be our default deploy ramdisk, and it's already being tested on the Ironic gate. So it would be good to also check on the ironic-python-agent gate so we can capture failures quicker 2015-03-19T12:01:44 *** Hal2 has joined #openstack-infra 2015-03-19T12:04:40 *** garyh has joined #openstack-infra 2015-03-19T12:04:57 Monty Taylor proposed openstack-infra/nodepool: Don't die while doing alien list https://review.openstack.org/165792 2015-03-19T12:05:37 *** claudiub_ has joined #openstack-infra 2015-03-19T12:07:03 *** Somay has quit IRC 2015-03-19T12:07:22 *** mwagner_lap has joined #openstack-infra 2015-03-19T12:07:22 *** claudiub has quit IRC 2015-03-19T12:10:03 *** rfolco has joined #openstack-infra 2015-03-19T12:13:05 *** ibiris is now known as ibiris_away 2015-03-19T12:13:27 *** dprince has joined #openstack-infra 2015-03-19T12:14:17 *** chlong has joined #openstack-infra 2015-03-19T12:14:24 *** garyh has quit IRC 2015-03-19T12:15:59 *** adalbas has joined #openstack-infra 2015-03-19T12:19:49 ianw: good morning! I updated this patch https://review.openstack.org/160068 hope you can take a look at, just left few hours for freeze~ And the patch depend on this passed the ci tests https://review.openstack.org/150267 Thanks in advance! 2015-03-19T12:20:01 *** Longgeek has quit IRC 2015-03-19T12:20:39 *** Longgeek has joined #openstack-infra 2015-03-19T12:20:43 sdague: ok - I'm deleting a bunch of alien nodes on hpcloud 2015-03-19T12:20:52 mordred: thanks 2015-03-19T12:21:27 clarkb, fungi, jeblair: I'm a bit concerned about these aliens - I do not see the pattern clarkb was talking about yesterday where they'd been deleted and then came back 2015-03-19T12:21:41 I see no delete calls for them in teh logs at all, in fact - but they sure are not things nodepool groks 2015-03-19T12:22:47 sdague: btw - I just learned about block_device_mapping_v2 2015-03-19T12:23:04 Jaroslav Henner proposed openstack-infra/jenkins-job-builder: Add checking of multijob phase condition. https://review.openstack.org/164669 2015-03-19T12:23:10 sdague: a) dude what a terrible name ... but b) exposes the type of capability I really want in things, so awesome 2015-03-19T12:23:47 *** ajmiller has quit IRC 2015-03-19T12:24:59 *** sdake has joined #openstack-infra 2015-03-19T12:25:08 *** bknudson has quit IRC 2015-03-19T12:25:14 sdague: honestly, if there was a similar parameter for floating_ips, like "floating_ip_mapping = [{ delete_on_terminate: True, id = , pool = }] and if id was left out it would just make one for you - I think I'd have everything I need 2015-03-19T12:25:33 sdague: well, except for the ssh hostkey thing - but I'll write that in the next cycle 2015-03-19T12:25:58 *** sdake_ has joined #openstack-infra 2015-03-19T12:25:59 sdague: do I need to be writing a spec for that for liberty already? or is it still too early (never written a nova spec before) 2015-03-19T12:26:23 *** chlong has quit IRC 2015-03-19T12:26:38 spec is fine at this point, I think they are effectively already open for liberty 2015-03-19T12:29:12 *** spredzy|afk is now known as spredzy 2015-03-19T12:29:36 Sean Dague proposed openstack-infra/os-loganalyze: fix supports_sev matching https://review.openstack.org/165542 2015-03-19T12:29:36 Sean Dague proposed openstack-infra/os-loganalyze: let tests be run from test file location https://review.openstack.org/165799 2015-03-19T12:29:46 *** sdake has quit IRC 2015-03-19T12:30:27 clarkb: I respun your fix with some test cases just to convince myself that \b would work in this case 2015-03-19T12:31:35 *** BobBall_AWOL is now known as BobBall 2015-03-19T12:32:06 Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the nofifications plugin to support 1.9 https://review.openstack.org/165785 2015-03-19T12:32:07 *** Hal2 has quit IRC 2015-03-19T12:32:13 *** Hal1 has joined #openstack-infra 2015-03-19T12:32:33 *** dkranz has joined #openstack-infra 2015-03-19T12:33:27 *** e0ne is now known as e0ne_ 2015-03-19T12:34:49 *** gordc has joined #openstack-infra 2015-03-19T12:35:32 *** ghostpl_ has quit IRC 2015-03-19T12:37:26 *** gordc has joined #openstack-infra 2015-03-19T12:37:59 *** Longgeek_ has joined #openstack-infra 2015-03-19T12:39:51 Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the nofifications plugin to support 1.9 https://review.openstack.org/165785 2015-03-19T12:41:05 *** Longgeek has quit IRC 2015-03-19T12:41:30 *** mrmartin has joined #openstack-infra 2015-03-19T12:42:21 *** achanda has joined #openstack-infra 2015-03-19T12:43:07 *** chlong has joined #openstack-infra 2015-03-19T12:43:07 *** bswartz has quit IRC 2015-03-19T12:43:08 *** unicell1 has joined #openstack-infra 2015-03-19T12:43:13 *** e0ne_ has quit IRC 2015-03-19T12:44:45 *** unicell has quit IRC 2015-03-19T12:45:01 *** bknudson has joined #openstack-infra 2015-03-19T12:47:22 *** achanda has quit IRC 2015-03-19T12:47:46 *** ujuc_ has quit IRC 2015-03-19T12:53:42 *** eharney has quit IRC 2015-03-19T12:53:59 *** mfink_ has joined #openstack-infra 2015-03-19T12:55:08 *** ddieterly has joined #openstack-infra 2015-03-19T12:59:38 *** ddieterly has quit IRC 2015-03-19T13:00:02 Monty Taylor proposed openstack-infra/system-config: Remove HP TripleO region https://review.openstack.org/165814 2015-03-19T13:01:28 *** dims has quit IRC 2015-03-19T13:01:37 *** sdake_ has quit IRC 2015-03-19T13:02:04 *** dims has joined #openstack-infra 2015-03-19T13:03:18 *** dboik has joined #openstack-infra 2015-03-19T13:04:40 Ng: any luck with rax glance? 2015-03-19T13:05:56 Monty Taylor proposed openstack-infra/shade: Add 'rebuild' to shade https://review.openstack.org/161067 2015-03-19T13:06:20 *** mjturek1 has joined #openstack-infra 2015-03-19T13:08:19 *** jistr has quit IRC 2015-03-19T13:08:33 *** ldnunes has joined #openstack-infra 2015-03-19T13:09:00 *** jistr has joined #openstack-infra 2015-03-19T13:09:25 *** sdake has joined #openstack-infra 2015-03-19T13:12:39 *** sdake_ has joined #openstack-infra 2015-03-19T13:12:52 *** dustins has joined #openstack-infra 2015-03-19T13:14:45 *** bswartz has joined #openstack-infra 2015-03-19T13:15:03 *** garyh has joined #openstack-infra 2015-03-19T13:16:14 *** sdake has quit IRC 2015-03-19T13:17:02 *** mrmartin has quit IRC 2015-03-19T13:17:10 *** marcusvrn1 has joined #openstack-infra 2015-03-19T13:17:21 *** ddieterly has joined #openstack-infra 2015-03-19T13:17:42 *** otter768 has joined #openstack-infra 2015-03-19T13:17:54 *** marcusvrn has quit IRC 2015-03-19T13:18:27 mordred: any luck in freeing up servers? 2015-03-19T13:18:40 *** dboik has quit IRC 2015-03-19T13:19:01 looks like changes are waiting about an hour to get nodes in check right now, would be nice to help with that 2015-03-19T13:19:14 especially as very few folks are awake atm 2015-03-19T13:19:23 *** ddieterly has quit IRC 2015-03-19T13:19:29 afazekas, rumor has it you've done some work with multi-node jobs with nodepool? I was curious if you didn't mind pointing to some code for that 2015-03-19T13:19:54 *** ibiris_away is now known as ibiris 2015-03-19T13:21:11 *** chlong has quit IRC 2015-03-19T13:21:47 *** mrmartin has joined #openstack-infra 2015-03-19T13:22:27 *** otter768 has quit IRC 2015-03-19T13:22:55 sdague: some check jobs have been in queue for 13 hours now 2015-03-19T13:23:07 Monty Taylor proposed openstack-infra/system-config: Remove tripleo-test-cloud-hp1 from nodepool config https://review.openstack.org/156615 2015-03-19T13:23:15 and jenkins04 fails to respond 2015-03-19T13:23:37 oh, that's a different issue then 2015-03-19T13:24:00 we might have a stuck jenkins instance 2015-03-19T13:24:02 sdague: I'm not convinced the thing I was looking at was the thing 2015-03-19T13:24:35 all stuck jobs seem to be on jenkins04 2015-03-19T13:24:39 yah 2015-03-19T13:24:48 jenkins04 is hanging when I try to ssh to it 2015-03-19T13:25:18 *** sdake has joined #openstack-infra 2015-03-19T13:25:18 yeah, confirmed all jobs >12hours wait are all on jenkins04 2015-03-19T13:25:19 yeah, confirmed all jobs >12hours wait are all on jenkins04 2015-03-19T13:25:42 *** garyh has quit IRC 2015-03-19T13:25:56 *** ilyashakhat has quit IRC 2015-03-19T13:26:46 *** peristeri has joined #openstack-infra 2015-03-19T13:26:56 stuck ttx? 2015-03-19T13:27:12 jenkins crash probably 2015-03-19T13:27:27 *** e0ne has joined #openstack-infra 2015-03-19T13:27:52 * mordred poking it with a stick 2015-03-19T13:28:42 anteaya: some check jobs have been waiting for test results for 14 hours now 2015-03-19T13:28:59 *** sdake_ has quit IRC 2015-03-19T13:29:08 ttx: you posted the same thing twice 2015-03-19T13:29:09 *** ghostpl_ has joined #openstack-infra 2015-03-19T13:29:16 ttx: my lame attempt at making a funny 2015-03-19T13:29:21 mordred: nope. starting to get quite annoyed at it. AFAICS I have the latest python-glanceclient installed fine in a venv, I'm exporting all the environment stuff I can, and still just 401. If you have a working environment, can we figure out what's different about mine from yours? like redact your password from whatever environment stuff you source, and a 2015-03-19T13:29:21 pip freeze? 2015-03-19T13:29:32 ttx: and yes looking at the status page I confirm your findings 2015-03-19T13:29:42 Ng: yup - did you get my paste yesterday? 2015-03-19T13:29:46 *** garyh has joined #openstack-infra 2015-03-19T13:29:51 I think https://jenkins04.openstack.org/job/check-tempest-dsvm-ironic-pxe_ipa-nv/40/ is stuck ? 2015-03-19T13:30:06 mordred: hrm, I must have missed that 2015-03-19T13:30:08 devananda: yes. everything on jenkins04 went south 2015-03-19T13:30:11 is there a way that it can be ignored at this point, as a non-voting job 2015-03-19T13:30:16 tha tis now blocking a patch 2015-03-19T13:30:28 devananda: not really - sorry - it's going to have to get rekicked 2015-03-19T13:30:31 mordred: ah. lovely 2015-03-19T13:30:40 * mordred just hard-rebooted jenkins04 2015-03-19T13:30:45 gotcha 2015-03-19T13:30:46 there will be a bit of fallout from that 2015-03-19T13:30:56 Ng: you're probably missing export OS_IMAGE_API_VERSION=2 2015-03-19T13:31:23 mordred: I definitely have that exported 2015-03-19T13:31:35 mordred: so, how can I initiate a recheck, when zuul hasn't finished the last job run? 2015-03-19T13:31:38 ok. let me give you redacted 2015-03-19T13:31:47 ta 2015-03-19T13:32:11 *** baoli has joined #openstack-infra 2015-03-19T13:32:13 or will zuul figure that out in a minute on its own? 2015-03-19T13:32:18 -(cmsj@stonker)-(~)- glance --version 2015-03-19T13:32:18 0.16.1 2015-03-19T13:32:18 -(cmsj@stonker)-(~)- echo $OS_IMAGE_API_VERSION 2015-03-19T13:32:18 2 2015-03-19T13:32:21 devananda: well, you could do something silly like change a punctuation in the commit message ttw 2015-03-19T13:32:33 gah. right. 2015-03-19T13:32:46 Ng: OS_TENANT_NAME ? 2015-03-19T13:32:54 * devananda forgets these silly tricks sometimes 2015-03-19T13:33:01 mordred: I have that set to my account number 2015-03-19T13:33:09 great. like 6 digits or whatever? 2015-03-19T13:33:13 yeah 2015-03-19T13:34:09 Ng: http://paste.openstack.org/show/193563/ 2015-03-19T13:35:37 *** Longgeek_ has quit IRC 2015-03-19T13:36:00 Ng: and 'glance image-list' works fine for me with those on glanceclient 0.16.1 2015-03-19T13:36:46 Ng: if you do 'keystone catalog' - do you get an entry for Service: image ? 2015-03-19T13:37:58 ttx: we're definitely feature freeze week, aren't we? :) 2015-03-19T13:38:30 mordred: even FF day. The day I need that queue thing to work :) 2015-03-19T13:38:45 mordred: nup, keystone is 401ing all over town too. Comparing to my env, I'm on a different region (their UK one). I'm wondering if they're not actually all the same 2015-03-19T13:39:38 so far the only api client I've managed to get working with it, is their hacked up version of novaclient 2015-03-19T13:39:46 (rackspace-novaclient in pypi) 2015-03-19T13:40:14 *** dboik has joined #openstack-infra 2015-03-19T13:40:34 oh gross. do you have access to the non-UK regions? like, can you try doing against DFW? 2015-03-19T13:40:49 ttx: the day you need it to work is the day it never will 2015-03-19T13:42:00 *** nelsnelson has joined #openstack-infra 2015-03-19T13:43:31 mordred: same. I wonder if the reverse would work - if you set OS_REGION_NAME=LON and OS_AUTH_URL=https://lon.identity.api.rackspacecloud.com/v2.0/ 2015-03-19T13:43:38 *** sarob has joined #openstack-infra 2015-03-19T13:43:38 let me try 2015-03-19T13:44:22 *** mrmartin has quit IRC 2015-03-19T13:44:38 the web console seems to suggest taht accounts are region-locked, in that I can't do stuff in DFW via that, it tells me to create a new account 2015-03-19T13:44:47 Ng: HEADDESK 2015-03-19T13:45:22 I'm gonna go back to the support people and say I know for a fact this works against DFW, and if this somehow isn't possible against LON, can I have my account switched over/recreated 2015-03-19T13:45:40 public endpoint for image service in LON region not found 2015-03-19T13:45:58 is what I get when I try london 2015-03-19T13:46:08 *** tnovacik has quit IRC 2015-03-19T13:46:30 jroll, JayF: you both have magical rackspace powers, right? pvo isn't ever on IRC anymore, so I can't poke him ... 2015-03-19T13:46:41 *** ddieterly has joined #openstack-infra 2015-03-19T13:47:44 *** mtanino has joined #openstack-infra 2015-03-19T13:47:49 *** dkliban_afk is now known as dkliban 2015-03-19T13:48:36 mordred: what is the status on jenkins04? 2015-03-19T13:48:39 *** sarob has quit IRC 2015-03-19T13:48:45 since you rebooted 2015-03-19T13:49:32 and it looks to me like we might need to clean up more alien nodes like clarkb did last night 2015-03-19T13:49:34 zuul still shows them as stuck 2015-03-19T13:49:40 mordred: do you know how to do that? 2015-03-19T13:49:53 * mordred is cleaning up alien nodes 2015-03-19T13:50:09 thank you, that will help 2015-03-19T13:50:37 however, there may be additional cleanup tasks that jeblair and clarkb suggest 2015-03-19T13:50:47 doing what you can is appreciated 2015-03-19T13:51:02 it being feature freeze day and all 2015-03-19T13:51:04 yah 2015-03-19T13:51:08 thanks 2015-03-19T13:52:31 *** amitgandhinz has joined #openstack-infra 2015-03-19T13:53:39 *** eharney has joined #openstack-infra 2015-03-19T13:53:50 *** larsks has joined #openstack-infra 2015-03-19T13:53:52 *** mfink__ has joined #openstack-infra 2015-03-19T13:54:37 the number of zuul workers looks funny to me 2015-03-19T13:54:46 in that it looks like there aren't many 2015-03-19T13:54:53 *** mfink_ has quit IRC 2015-03-19T13:55:10 skimmed scrollback. is jenkins04 still in need of help? 2015-03-19T13:55:16 or perhaps those are available workers, so the number is low if they aren't available 2015-03-19T13:55:35 fungi: I'm uncertain 2015-03-19T13:55:52 ahh, looks like it got restarted but its slave list was not evacuated 2015-03-19T13:56:02 *** ddieterly has quit IRC 2015-03-19T13:56:12 so it has many hundreds of offline slaves according to its webui 2015-03-19T13:56:20 awesome 2015-03-19T13:56:30 mordred: what's up 2015-03-19T13:56:41 okay, maybe not many hundreds, but a few hundred 2015-03-19T13:57:10 *** mriedem_away is now known as mriedem 2015-03-19T13:57:19 *** mfink_ has joined #openstack-infra 2015-03-19T13:57:42 mordred: the only special rackspace power I have is internal irc, but if you have a paste or whatever with some details I can link it to people 2015-03-19T13:58:06 *** ddieterly has joined #openstack-infra 2015-03-19T13:58:13 *** aysyd has quit IRC 2015-03-19T13:58:48 Monty Taylor proposed openstack-infra/nodepool: Record interesting info into nova metadata https://review.openstack.org/126621 2015-03-19T13:59:02 *** wuhg has joined #openstack-infra 2015-03-19T13:59:06 mordred: reading again this looks like an incoming rant about uk vs us accounts... I can't help with that but it is being worked on(tm) 2015-03-19T13:59:09 fungi: yes - jenkins04 was completely dead 2015-03-19T13:59:15 *** aysyd has joined #openstack-infra 2015-03-19T13:59:35 jroll: awesome - so, is the best bet just for Ng to poke support and get his account move - or to get a new account created? 2015-03-19T13:59:52 *** koolhead17 has quit IRC 2015-03-19T13:59:54 *** spzala has joined #openstack-infra 2015-03-19T14:00:00 mordred: so fungi appears to be suggesting that evacuating a jenkins slave list prior to a reboot might be something to consider for next time 2015-03-19T14:00:01 jroll: it seems that the UK region has different operational semantics too - as in python-glanceclient does not work there 2015-03-19T14:00:16 anteaya: sure - but the server was completely hung and it was not possible to connect to it 2015-03-19T14:00:20 any more i've taken to killing the jenkins process, deleting all the lines inside from the config.xml, marking everything in the nodepool database assigned to that jenkins as delete, then starting the jenkins process again 2015-03-19T14:00:26 annoying but effective 2015-03-19T14:00:29 fungi: it was not possible to ssh in to the box 2015-03-19T14:00:36 *** mfink__ has quit IRC 2015-03-19T14:00:36 oh marvellous 2015-03-19T14:00:38 yeah. 2015-03-19T14:00:42 this is why I hard rebooted it 2015-03-19T14:00:43 mordred: yeah, it's a huge problem. I hate it. 2015-03-19T14:00:46 it was stuck very badly 2015-03-19T14:00:50 anything interesting on the virtual console, or did you have a chance to check? 2015-03-19T14:00:58 fungi: the virtual console also did not work :) 2015-03-19T14:01:03 mordred: Ng: I'm not sure if poke support or make a new account is the best route, support can tell you :P 2015-03-19T14:01:04 wow awesome 2015-03-19T14:01:18 fungi: yah. it was pleasing 2015-03-19T14:01:20 jroll: thanks. I shall see what my options are with support :) 2015-03-19T14:01:31 *** asselin_ has joined #openstack-infra 2015-03-19T14:01:31 hopefully all of my options involve fanaticism of some kind 2015-03-19T14:01:39 *** zz_dimtruck is now known as dimtruck 2015-03-19T14:01:54 *** yfried is now known as yfried|afk 2015-03-19T14:02:00 i'm going what i can to get as much cleaned up for nodepool as possible, and giving the other jenkins masters a health check 2015-03-19T14:02:07 *** sigmavirus24_awa is now known as sigmavirus24 2015-03-19T14:02:08 fungi: cool, thanks 2015-03-19T14:02:38 Ng: IMO this separate account thin is strictly non-fanatical :) 2015-03-19T14:02:52 jroll: Ng: fantastical! 2015-03-19T14:03:08 *** dkranz has quit IRC 2015-03-19T14:03:26 heh 2015-03-19T14:04:04 jenkins01 is pretty annoyed too 2015-03-19T14:04:13 :( 2015-03-19T14:04:17 but not dead yet 2015-03-19T14:04:22 yay for life 2015-03-19T14:04:37 i'll get it into a graceful restart cycle as soon as possible 2015-03-19T14:05:04 and clean it up while it's restarted for good measure 2015-03-19T14:05:06 fungi: I've got a loop going deleting things from alien-list btw 2015-03-19T14:05:14 * anteaya nods 2015-03-19T14:05:32 mordred: awesome. i hadn't gotten that far yet, cleaning up stale tracked nodes in nodepool first 2015-03-19T14:05:34 *** mfink__ has joined #openstack-infra 2015-03-19T14:06:00 Merged openstack-infra/system-config: Work-around grubby boot failure on Fedora 20 https://review.openstack.org/165681 2015-03-19T14:06:07 since there seemed to be rather a few in ready for far longer than is likely, and no sign of them on the jenkins masters to which they claimed to belong 2015-03-19T14:06:18 that has a tendency to throw off the demand calculations pretty badly 2015-03-19T14:06:54 oh yeah jenkins01 was not much longer for this life. took several minutes just to get logged into the webui 2015-03-19T14:07:00 Was there a mid-cycle infra meeting this time around? If so, where could I look to locate the discussions? I don't see much on the infra wiki page 2015-03-19T14:07:21 *** achuprin has joined #openstack-infra 2015-03-19T14:07:23 fungi: glad you caught it 2015-03-19T14:07:23 pabelanger: we don't usually have mid-cycle meetups 2015-03-19T14:07:52 pabelanger: we prefer to do virtual sprints and similar sorts of activities where more people can participate without needing to travel 2015-03-19T14:08:33 fungi, Oh, perhaps I'm thinking of informal meetings. Didn't something happen in NYC and Australia? 2015-03-19T14:09:01 nyc was in 2013 in june, we had an onboarding 2015-03-19T14:09:02 *** yfried|afk is now known as yfried 2015-03-19T14:09:13 australia was lca and we saw each other there 2015-03-19T14:09:30 Makes sense 2015-03-19T14:09:43 pabelanger: yeah - no mid-cycle this time 2015-03-19T14:09:48 the only thing that took place there that had an etherpad was I locked markmcclain and gus in a room to come up with a direction for nova-net to neutron migration 2015-03-19T14:09:48 yeah, it's more that we have opportunistic gatherings when enough people already expect to be around 2015-03-19T14:09:55 *** radez_g0n3 is now known as radez 2015-03-19T14:09:58 yes that 2015-03-19T14:11:40 the other jenkins masters look like they're doing okay for now. once jenkins01 is back to normal i'll gracefully restart 04 so that i can delete its pile of cruft slaves 2015-03-19T14:12:25 *** mattfarina has joined #openstack-infra 2015-03-19T14:12:33 *** claudiub has joined #openstack-infra 2015-03-19T14:12:55 *** mfink__ has quit IRC 2015-03-19T14:12:58 *** mfink_ has quit IRC 2015-03-19T14:13:09 *** mfink_ has joined #openstack-infra 2015-03-19T14:14:04 *** claudiub_ has quit IRC 2015-03-19T14:14:08 *** ajmiller has joined #openstack-infra 2015-03-19T14:14:51 *** stevemar has joined #openstack-infra 2015-03-19T14:14:54 Claudiu Belu proposed openstack-infra/project-config: Adds compute-hyperv in StackForge https://review.openstack.org/165611 2015-03-19T14:15:37 mordred: wow. we have 555 unassociated floating ips in hpcloud. cleaning those up now 2015-03-19T14:17:03 though these may be from churn... some of them are already coming up not found when deleting 2015-03-19T14:17:36 is jenkins06 having trouble now? 2015-03-19T14:18:17 devananda: i don't think so--seems to be running jobs 2015-03-19T14:18:23 devananda: what makes you ask? 2015-03-19T14:19:03 * devananda is watching https://jenkins06.openstack.org/job/check-heat-dsvm-functional-mysql/662/ 2015-03-19T14:19:32 perhaps just slow ... 2015-03-19T14:19:43 *** dhellmann has quit IRC 2015-03-19T14:19:57 *** dhellmann has joined #openstack-infra 2015-03-19T14:20:06 ah, yep. it's alive still 2015-03-19T14:20:17 mostly dead is still partly alive! 2015-03-19T14:20:22 indeed! 2015-03-19T14:20:29 (and it passed. yay!) 2015-03-19T14:21:02 *** timcline has joined #openstack-infra 2015-03-19T14:21:20 i'm rerunning my fip cruft check with a 5-minute separation instead of 1 2015-03-19T14:21:28 hopefully this will be marginally more accurate 2015-03-19T14:22:05 *** e0ne is now known as e0ne_ 2015-03-19T14:22:37 *** ayoung has quit IRC 2015-03-19T14:23:22 *** e0ne_ is now known as e0ne 2015-03-19T14:24:25 *** _nadya_ has joined #openstack-infra 2015-03-19T14:25:31 *** mrmartin has joined #openstack-infra 2015-03-19T14:26:45 *** erlon has joined #openstack-infra 2015-03-19T14:27:38 *** yamamoto has quit IRC 2015-03-19T14:28:04 Merged openstack-infra/system-config: Remove tripleo-test-cloud-hp1 from nodepool config https://review.openstack.org/156615 2015-03-19T14:28:47 *** sushilkm has joined #openstack-infra 2015-03-19T14:29:02 mordred: something else is wrong with floating ips in hpcloud 2015-03-19T14:29:20 *** jcoufal_ has joined #openstack-infra 2015-03-19T14:29:20 The gating jobs for trove seem to stuck 2015-03-19T14:29:23 https://jenkins04.openstack.org/job/check-grenade-dsvm/28125/console 2015-03-19T14:29:36 I think it needs to be restarted 2015-03-19T14:29:42 Or some fix 2015-03-19T14:29:55 fungi: sweet! 2015-03-19T14:30:09 mordred: `nova floating-ip-delete 01178e82-74c6-4da4-8a60-3315a3a4be7c` returns "ERROR (CommandError): Floating IP 01178e82-74c6-4da4-8a60-3315a3a4be7c not found." but `nova floating-ip-list` says it exists 2015-03-19T14:30:36 Sean Dague proposed openstack-infra/os-loganalyze: extract static methods https://review.openstack.org/165850 2015-03-19T14:30:36 Sean Dague proposed openstack-infra/os-loganalyze: unwind test class multiple inheritance https://review.openstack.org/165851 2015-03-19T14:30:58 mordred: nevermind. looks like nova can't delete floating ips by uuid any longer, only by ip address 2015-03-19T14:32:01 sushilkm: jenkins04 was stuck and has been rebooted 2015-03-19T14:32:03 fungi: neutron can delete them by uuid 2015-03-19T14:32:06 *** sdake_ has joined #openstack-infra 2015-03-19T14:32:06 *** sdake_ has quit IRC 2015-03-19T14:32:11 fungi: and it's more efficient to delete them with neutron, fwiw 2015-03-19T14:32:19 *** jcoufal has quit IRC 2015-03-19T14:32:22 sushilkm: when jenkins comments on your patch you may need to recheck to send it through again 2015-03-19T14:32:41 sushilkm: yes, i think we're probably going to have to restart zuul to clear those up. it doesn't seem to know that the jobs which were running on jenkins04 before it died are no longer running 2015-03-19T14:33:13 *** sdake__ has joined #openstack-infra 2015-03-19T14:33:24 Yes its stuck since 13 hrs 2015-03-19T14:33:25 *** yamamoto has joined #openstack-infra 2015-03-19T14:34:37 *** sdake has quit IRC 2015-03-19T14:34:53 fungi: or push a new patch 2015-03-19T14:35:09 sushilkm: edit the commit message to kick it back into the pool 2015-03-19T14:35:18 sdague: yeah, i was just considering that. there's only 13 in that state 2015-03-19T14:35:20 because they'll need to rerun for new tests anyway 2015-03-19T14:35:26 so maybe better than restarting the whole world 2015-03-19T14:35:29 yeh 2015-03-19T14:35:30 *** achanda has joined #openstack-infra 2015-03-19T14:35:33 yes 2015-03-19T14:35:48 fungi: do you have a list of the 13? I can edit and get them back in 2015-03-19T14:35:56 Wud the recheck not work 2015-03-19T14:36:04 anteaya: i can see them in the top-left corner of the zuul status page 2015-03-19T14:36:05 it will not 2015-03-19T14:36:31 anteaya: the 13 in the check pipeline with times of 13+ hours in queue 2015-03-19T14:36:40 fungi: ah okay I will get them 2015-03-19T14:37:03 i mean, they're not doing any harm being there 2015-03-19T14:37:06 sushilkm: recheck only works if jenkins is finished and has reported back 2015-03-19T14:37:20 sushilkm: it won't work if they are stuck in the queue 2015-03-19T14:37:26 so if people who own those 13 changes care they're also welcome to just push a new patchset, tweak the commit message in the webui, whatever 2015-03-19T14:37:40 oh okay 2015-03-19T14:37:40 it's an independent pipeline, so it's not tying up any other changes 2015-03-19T14:37:48 was just going to be neighbourly 2015-03-19T14:37:51 but I don't have to 2015-03-19T14:38:08 i'm currently more concerned about getting our worker capacity back to sanity 2015-03-19T14:38:08 it might be good to wait on flushing them all until we have more ready nodes 2015-03-19T14:38:13 fungi: ++ 2015-03-19T14:38:26 anteaya: when I want to be neighborly, I tend to give people fluffy large animals 2015-03-19T14:38:41 *** wenlock has joined #openstack-infra 2015-03-19T14:38:47 we're at < 100 in use nodes 2015-03-19T14:38:51 anteaya: although I've been considering adding sea mammals to the mix recently 2015-03-19T14:38:54 mordred: I see that 2015-03-19T14:38:57 which is quite bad for throughput 2015-03-19T14:39:02 mordred: oh like what? 2015-03-19T14:39:03 you're not kidding 2015-03-19T14:39:10 anteaya: your approach is likely more productive than mine 2015-03-19T14:39:31 anteaya: I mean, dolphins are a good start - but manatees could also be fun 2015-03-19T14:39:32 mordred: yours makes more friends though 2015-03-19T14:39:41 mordred: my approach doesn' tend to make friends 2015-03-19T14:40:18 *** tonytan4ever has joined #openstack-infra 2015-03-19T14:40:22 anteaya: maybe you should try distributing mooses? 2015-03-19T14:40:45 not sure that would achieve the goal 2015-03-19T14:40:54 anteaya: probably not 2015-03-19T14:41:03 meese tend to take up quite a bit of space 2015-03-19T14:41:14 got chased by a moose once, that was exciting 2015-03-19T14:41:19 anteaya: but they also tend to keep home invaders at bay 2015-03-19T14:41:28 *** sushilkm has left #openstack-infra 2015-03-19T14:41:56 ah, the openssl vuln is just a DoS attack, not an exploit - https://www.openssl.org/news/vulnerabilities.html 2015-03-19T14:42:04 they consider anyone not them to be a home invader 2015-03-19T14:42:13 hopefully that means no need to take down services for patching 2015-03-19T14:42:40 *** baoli has quit IRC 2015-03-19T14:42:43 sdague: oh good, I wasn't really looking forward to having to do that again 2015-03-19T14:42:51 oh, wait, there are a ton of CVEs in that list today 2015-03-19T14:43:10 *** spzala has quit IRC 2015-03-19T14:43:14 *** baoli has joined #openstack-infra 2015-03-19T14:43:20 *** mfink_ has quit IRC 2015-03-19T14:43:21 *** achanda has quit IRC 2015-03-19T14:43:37 *** mfink_ has joined #openstack-infra 2015-03-19T14:43:43 there are 12 CVEs today actually 2015-03-19T14:43:47 :( 2015-03-19T14:44:43 it looks like they are all DoS or segfault 2015-03-19T14:44:48 *** sarob has joined #openstack-infra 2015-03-19T14:45:10 yeah, it's a fun batch 2015-03-19T14:45:20 i'm still mulling it over 2015-03-19T14:45:25 anteaya: yes - I mean, they also keep you away from your home - but your home will certainly be safe from other people if you're worried about that sort of thing 2015-03-19T14:45:46 right 2015-03-19T14:45:48 *** baoli has quit IRC 2015-03-19T14:46:00 if my home happens to be a muskeg swamp 2015-03-19T14:46:12 it will be nice and safe 2015-03-19T14:46:16 anteaya: I don't want to judge people's home choices 2015-03-19T14:46:27 I did live close for a period of time 2015-03-19T14:46:29 was nice 2015-03-19T14:46:43 hence the experience being chased 2015-03-19T14:47:08 *** baoli has joined #openstack-infra 2015-03-19T14:47:59 *** dkranz has joined #openstack-infra 2015-03-19T14:48:25 *** yfried is now known as yfried|afk 2015-03-19T14:48:28 *** timcline has quit IRC 2015-03-19T14:48:40 fungi: only 5 leaked ports 2015-03-19T14:48:46 morning jeblair 2015-03-19T14:48:56 *** mrmartin has quit IRC 2015-03-19T14:49:04 *** timcline has joined #openstack-infra 2015-03-19T14:49:20 *** sarob has quit IRC 2015-03-19T14:49:28 anteaya: morning 2015-03-19T14:49:36 fungi: are you working on jenkins01? 2015-03-19T14:49:53 *** prad has joined #openstack-infra 2015-03-19T14:50:11 *** flip214 has left #openstack-infra 2015-03-19T14:50:27 ah yeh, it's down to 2 jobs 2015-03-19T14:50:55 *** MarkAtwood has joined #openstack-infra 2015-03-19T14:51:05 *** bhunter71 has joined #openstack-infra 2015-03-19T14:51:34 *** jlanoux has quit IRC 2015-03-19T14:51:37 *** garyh has quit IRC 2015-03-19T14:52:45 *** mrunge has quit IRC 2015-03-19T14:52:58 *** yfried|afk is now known as yfried 2015-03-19T14:54:23 *** carl_baldwin has joined #openstack-infra 2015-03-19T14:54:48 jeblair: yeah waiting for jobs to wrap up there 2015-03-19T14:55:02 fungi: most of the errors i'm seeing are timeouts on server deletion; i'm not actually seeing any build errors 2015-03-19T14:55:03 two more running, almost complete 2015-03-19T14:55:54 i've double checked the ssh timeouts i see, and those are actually just nodes that were in the process of being deleted but got picked up by the periodic check 2015-03-19T14:56:21 interestingly, we have a crazy high number of nodes in a building state proportional to everything else 2015-03-19T14:56:32 *** marcusvrn has joined #openstack-infra 2015-03-19T14:56:49 *** sarob has joined #openstack-infra 2015-03-19T14:57:23 Anthony Michon proposed openstack-infra/project-config: Add cerberus related projects https://review.openstack.org/163876 2015-03-19T14:57:26 we also have quite a few ready nodes on jenkins01 and 04. i suspect those are errors. i will delete them 2015-03-19T14:57:43 *** timcline has quit IRC 2015-03-19T14:57:59 *** timcline has joined #openstack-infra 2015-03-19T14:58:18 sounds like a good idea. i'm also planning to blow away everything assigned to jenkins01 here momentarily, as soon as it's down 2015-03-19T14:58:30 and then circle back around to do the same for jenkins04 2015-03-19T14:59:31 we have 82 nodes in a building state for more than an hour, all in hpcloud 2015-03-19T14:59:48 *** armax has joined #openstack-infra 2015-03-19T15:00:12 320 for more than 30 minutes, also all in hpcloud 2015-03-19T15:00:21 :( 2015-03-19T15:00:25 something's gumming up the works there 2015-03-19T15:00:29 same region? 2015-03-19T15:00:43 we only use one region, but i'll correlate to az 2015-03-19T15:00:55 oh 2015-03-19T15:01:00 sorry 2015-03-19T15:01:08 *** dboik_ has joined #openstack-infra 2015-03-19T15:01:27 anteaya: no need to be sorry - it was the right question 2015-03-19T15:01:34 *** mrunge has joined #openstack-infra 2015-03-19T15:01:42 k 2015-03-19T15:01:50 78% are in az3 2015-03-19T15:01:59 wow 2015-03-19T15:01:59 mordred: jeblair: ^ 2015-03-19T15:02:36 fungi: excellent 2015-03-19T15:02:46 fungi: we target 63% of our nodes to be in az3 2015-03-19T15:02:53 fungi: so that may not be off-mark 2015-03-19T15:02:58 *** yfried is now known as yfried|afk 2015-03-19T15:03:12 yeah, so maybe slightly inflated ratio but not crazy then 2015-03-19T15:03:26 so may just be roughly evenly distributed by assigned quota 2015-03-19T15:03:44 *** garyh has joined #openstack-infra 2015-03-19T15:04:22 *** dboik has quit IRC 2015-03-19T15:04:53 *** pblaho_ has joined #openstack-infra 2015-03-19T15:05:00 *** asselin_ has quit IRC 2015-03-19T15:05:15 *** pblaho has quit IRC 2015-03-19T15:05:31 *** dkranz has quit IRC 2015-03-19T15:06:29 *** mrunge has quit IRC 2015-03-19T15:07:37 *** sputnik13 has quit IRC 2015-03-19T15:07:42 *** sdake has joined #openstack-infra 2015-03-19T15:08:37 *** timcline has quit IRC 2015-03-19T15:09:02 *** sputnik13 has joined #openstack-infra 2015-03-19T15:09:11 *** timcline has joined #openstack-infra 2015-03-19T15:09:23 *** sputnik13 has quit IRC 2015-03-19T15:10:54 *** wayneeseguin has quit IRC 2015-03-19T15:11:22 anteaya: what's the policy on things like this random 'noop' comment - https://review.openstack.org/#/c/165863/1 2015-03-19T15:11:28 looks like a poorly setup test system 2015-03-19T15:11:37 *** spzala has joined #openstack-infra 2015-03-19T15:11:39 *** wayneeseguin has joined #openstack-infra 2015-03-19T15:11:49 *** sdake__ has quit IRC 2015-03-19T15:12:12 sdague: policy is they get disabled 2015-03-19T15:12:20 okay, jenkins01 has finally quiesced so i'm stopping it and cleaning up all its workers now 2015-03-19T15:12:33 anteaya: could you do that please? thanks in advance 2015-03-19T15:12:34 sdague: dims wanted a chance to reach out to them, unless you would like them disabled now 2015-03-19T15:12:37 sdague: sure 2015-03-19T15:15:15 can someone please disable gerrit account id 13431: http://paste.openstack.org/show/193589/ https://review.openstack.org/#/c/165863/1 personal account posting ci comments 2015-03-19T15:15:34 *** mika has quit IRC 2015-03-19T15:16:01 anteaya: done 2015-03-19T15:16:06 jeblair: thank you 2015-03-19T15:16:07 anteaya: sdague: just emailed them 2015-03-19T15:16:19 I'll tell them they are disabled 2015-03-19T15:16:28 anteaya: ack thanks 2015-03-19T15:17:56 *** dkranz has joined #openstack-infra 2015-03-19T15:18:39 *** otter768 has joined #openstack-infra 2015-03-19T15:18:47 can I haz an ETA on https://review.openstack.org/#/c/162146/ because it's kinda "blocking" us to release? 2015-03-19T15:19:20 dims: keep an eye on them for me will you? we might have to clear cache if they have an open connection to prevent them from continuing to post comments 2015-03-19T15:19:46 *** hashar has quit IRC 2015-03-19T15:20:10 jd__: we are dealing with multiple challenges today 2015-03-19T15:20:16 we can't give ETA's 2015-03-19T15:20:18 *** masayukig_ has joined #openstack-infra 2015-03-19T15:20:30 jenkins01 is on its way back up now. all nodepool nodes for it have been marked delete and all slave entries have been deleted from its config 2015-03-19T15:20:39 we are however doing everything we can to help developers and release mangers achieve their goals 2015-03-19T15:21:16 we have a shortage of workers to run tests and are doing what we can to make more available 2015-03-19T15:21:21 fungi: yay 2015-03-19T15:21:26 *** dannywilson has joined #openstack-infra 2015-03-19T15:21:44 renaming a project requires a scheduled gerriy downtime 2015-03-19T15:22:17 jd__ why is that blocking a release? cant you release gnocchi from the stackforge repo? 2015-03-19T15:22:37 *** Hal1 has quit IRC 2015-03-19T15:22:46 *** mika has joined #openstack-infra 2015-03-19T15:23:04 *** otter768 has quit IRC 2015-03-19T15:23:09 *** yfried|afk is now known as yfried 2015-03-19T15:23:33 *** masayukig_ has quit IRC 2015-03-19T15:23:46 clarkb: https://review.openstack.org/#/c/165542/ - I redid your fix of my fix with a test to convince myself that your fix was the right one 2015-03-19T15:24:09 jd__: best guess on the project renames is that we'll batch them up a week from saturday, since we don't want to add potential confusion to this saturday's gerrit os upgrade/ip address change 2015-03-19T15:24:31 sdague thanks 2015-03-19T15:24:40 there is also a stack of test refactorings above that which are not urgent, but while I was in the code I did it anyway. 2015-03-19T15:24:44 jd__: but we'll probably discuss the plan in greater detail in tuesday's meeting 2015-03-19T15:24:44 *** ChuckC has joined #openstack-infra 2015-03-19T15:24:59 *** dannywil_ has joined #openstack-infra 2015-03-19T15:25:00 fungi: my loop of cleaning up all of the aliens has finished, btw 2015-03-19T15:25:34 mordred: awesome. stale floating ip deletion is still going, though jenkins01 is starting to get workers again 2015-03-19T15:25:42 fungi, mordred: i'm seeing a lot of 2015-03-19 15:24:18,583 ERROR nodepool.NodeLauncher: EOFError launching node id: 1365036 in provider: hpcloud-b3 error: 2015-03-19T15:25:52 corvus@fuligin:~$ ssh 15.126.138.93 2015-03-19T15:25:53 Connection closed by 15.126.138.93 2015-03-19T15:25:53 here in a moment i'll set jenkins04 to prepare for shutdown in preparation to clean it up 2015-03-19T15:26:08 *** dannywilson has quit IRC 2015-03-19T15:26:26 clarkb: I can release but I wanted to test things like PyPI etc at the same time as it's our first release; so it's not "blocking" (note the "") it's just less fun ;) 2015-03-19T15:26:28 so - I've got a bunch of things in alien list again 2015-03-19T15:26:39 fungi: ok, thanks :) 2015-03-19T15:26:43 I think something is going south there 2015-03-19T15:26:53 04 appears to have leaked threads according to melody so the ssh-slaves plugin update does not correct that behavior :/ 2015-03-19T15:26:59 jeblair: great. that could be anything from broken filesystem to sshd child processes getting killed to network issues 2015-03-19T15:27:01 mordred: make sure you take 2 snapshots and compare 2015-03-19T15:27:18 mordred: new nodes briefly show up in alien lists during builds 2015-03-19T15:27:39 gotcha 2015-03-19T15:27:41 jeblair: Conflict: Instance 2a478e27-ab9e-48c7-8d14-14190a95b9ed is locked (HTTP 409) (Request-ID: req-e0f296ad-4306-4e6c-a3ba-7df1dd87cc80) 2015-03-19T15:27:44 what does that mean? 2015-03-19T15:27:48 https://jenkins04.openstack.org/monitoring?part=graph&graph=threadCount 2015-03-19T15:27:56 mordred: no idea; last time i asked the noc about it i didn't get a response 2015-03-19T15:28:02 mordred its nova locked 2015-03-19T15:28:11 *** masayukig_ has joined #openstack-infra 2015-03-19T15:28:11 but nova unlock doesnt work on it 2015-03-19T15:28:19 clarkb: :/ for threads 2015-03-19T15:28:23 what does "nova locked" mean? 2015-03-19T15:28:30 and how does something become "nova locked"? 2015-03-19T15:28:37 *** AJaeger has joined #openstack-infra 2015-03-19T15:28:45 jeblair `nova lock` is the command 2015-03-19T15:28:58 clarkb: but we don't run that anywhere, do we? 2015-03-19T15:29:06 fungi, could you review jeblair's irrc check access patch again, please? I rebased it... https://review.openstack.org/#/c/165508/ 2015-03-19T15:29:12 the help text is minimal and I think it says "lock an instance" so the actual semantics are fuzzy there 2015-03-19T15:29:17 mordred no 2015-03-19T15:29:22 *** masayukig has quit IRC 2015-03-19T15:29:22 *** jlanoux has joined #openstack-infra 2015-03-19T15:29:40 yeah, so i'm no closer to knowing what that means 2015-03-19T15:29:55 clarkb, jeblair: I'm asking the noc about 2a478e27-ab9e-48c7-8d14-14190a95b9ed 2015-03-19T15:30:17 which our logs show us as having requested deletion for but is in active state 2015-03-19T15:30:31 *** sdake_ has joined #openstack-infra 2015-03-19T15:30:46 *** Hal1 has joined #openstack-infra 2015-03-19T15:31:32 fungi, thanks! 2015-03-19T15:31:53 *** masayukig_ has quit IRC 2015-03-19T15:33:28 *** pblaho_ has quit IRC 2015-03-19T15:33:37 *** pblaho_ has joined #openstack-infra 2015-03-19T15:34:02 anteaya: ack 2015-03-19T15:34:03 *** sdake has quit IRC 2015-03-19T15:34:03 *** masayukig has joined #openstack-infra 2015-03-19T15:34:41 fungi: i have found a console log difference between a host with EOFError and one without 2015-03-19T15:35:24 oh! i've not had tons of luck with nova console yet 2015-03-19T15:35:52 fungi: http://paste.openstack.org/show/193593/ 2015-03-19T15:36:01 fungi: working node on top, failing node on bottom 2015-03-19T15:36:20 jeblair: nice! theory is lack of entropy at boot, no hostkey generated 2015-03-19T15:36:57 *** ayoung has joined #openstack-infra 2015-03-19T15:37:02 also why is /etc/sysconfig/network-scripts/ifcfg-* missing? 2015-03-19T15:37:12 oh, it does that on both 2015-03-19T15:37:23 *** gyee has joined #openstack-infra 2015-03-19T15:37:24 right, not red hat 2015-03-19T15:37:29 *** reed has joined #openstack-infra 2015-03-19T15:37:49 jeblair, fungi, anteaya: My understanding of a discussion here on IRC some days ago (don't remember when exactly) is that we prefer if projects have a single lint target. In that spirit, I merged for project-config the two targets bashate and pep8 in patch https://review.openstack.org/164531 - and confused anteaya with it. Could you confirm whether this is the right direction, please? 2015-03-19T15:38:41 AJaeger: i'll have to look into that later 2015-03-19T15:38:45 jeblair: you know how to open hp support tickets, yeah? 2015-03-19T15:38:55 mordred: through the web ui? 2015-03-19T15:39:14 k 2015-03-19T15:39:22 jeblair, ok, thanks. There's no urgency - just like to get direction 2015-03-19T15:39:31 mordred: and then go check infra-root's mailbox because they send the response there and forgot to copy-paste it into the support ticket tracker? 2015-03-19T15:39:47 *** Bsony has quit IRC 2015-03-19T15:39:59 AJaeger: yeah we are trying to get more workers available for jobs right now 2015-03-19T15:40:00 I prefered to email for that reason 2015-03-19T15:40:09 at least, that's the most recent procedure 2015-03-19T15:40:11 support@hpcloud.com iirc 2015-03-19T15:40:12 Merged openstack-infra/project-config: Fix failure detection in irc access check https://review.openstack.org/165508 2015-03-19T15:40:16 AJaeger: perhaps we can discuss it tomorrow and thanks for bring it up 2015-03-19T15:40:44 anteaya, feel free to discuss when you have time and give me a pointer if I'm not around ;) I can read the logs on eavesdrop then 2015-03-19T15:41:15 you can bring it up, it is your issue, just another time is better 2015-03-19T15:41:54 *** unicell1 has quit IRC 2015-03-19T15:42:30 *** unicell has joined #openstack-infra 2015-03-19T15:43:37 fungi: i've confirmed the eoferror and working nodes i looked at were built from the same image in the same region, and within less than 1 second of each other 2015-03-19T15:43:51 jeblair: was there any console output earlier during boot about host key generation breaking? 2015-03-19T15:46:50 *** Trozz has quit IRC 2015-03-19T15:46:57 *** bhuvan has joined #openstack-infra 2015-03-19T15:47:04 *** Trozz has joined #openstack-infra 2015-03-19T15:47:16 fungi: it's missing more things like Your identification has been saved in /etc/ssh/ssh_host_rsa_key. 2015-03-19T15:47:19 Generating public/private rsa key pair. 2015-03-19T15:47:31 fungi: check ~corvus/okaylog and ~corvus/eoflog on puppetmaster 2015-03-19T15:47:36 *** dangers_away is now known as dangers 2015-03-19T15:47:40 jeblair: awesome, thanks 2015-03-19T15:47:56 i wonder if it's a filesystem problem 2015-03-19T15:48:55 fungi: 2015-03-19 14:57:14,832 - util.py[WARNING]: Failed fetching metadata from url http://169.254.169.254/2009-04-04/meta-data 2015-03-19T15:49:02 fungi: that seems to be where it starts to diverge 2015-03-19T15:49:16 we rely on the metadata server after all? 2015-03-19T15:49:20 *** marun has joined #openstack-infra 2015-03-19T15:49:27 *** bdpayne has joined #openstack-infra 2015-03-19T15:49:35 for some reason i thought we'd managed to not need it 2015-03-19T15:49:55 me too, but it seems somehow tied into cloud-init's setting up of ssh keys? 2015-03-19T15:50:19 mordred, SpamapS: ^ can you shed light on this? 2015-03-19T15:50:21 this is hp or rax? 2015-03-19T15:50:24 hp 2015-03-19T15:50:30 yeah - that's how keypair extension works 2015-03-19T15:50:41 this is why I want to not install cloud-init 2015-03-19T15:50:44 and just bake in keys 2015-03-19T15:51:01 I am very sorry that is taking so long 2015-03-19T15:51:23 well, it seems to be the underlying reason for a majority of boot failures in hpcloud 2015-03-19T15:51:23 *** ashleighfarnham has joined #openstack-infra 2015-03-19T15:51:44 *** tjones1 has joined #openstack-infra 2015-03-19T15:51:45 397 in the past few hours 2015-03-19T15:51:49 can we config drive insyead? 2015-03-19T15:51:50 as a stop-gap we could switch to config-drive and we could change our base image cloud-init config to ignore ec2 metadata service 2015-03-19T15:52:02 i wonder if cloud-init couldn't just be coaxed into retrying that for longer as a stopgap? 2015-03-19T15:52:10 or that 2015-03-19T15:52:11 or - since it's hp - we _could_ just uninstall cloud-init in the base image and bake a nodepool key ni to it 2015-03-19T15:52:17 fungi metadata is likely notnroutable 2015-03-19T15:52:25 fungi so timeouts wont help 2015-03-19T15:52:34 what changed, something changed about 24 hours ago to result in these errors 2015-03-19T15:52:41 we don't have to fix the problem for real yet - uninstalling cloud-init will not break rax 2015-03-19T15:52:47 clarkb: it's linklocal. that's never routable. you mean it's not responding? 2015-03-19T15:53:05 you want I should make a patch to do that real quick? 2015-03-19T15:53:15 fungi no the way the metadata service works is it gets nated amd must be routable 2015-03-19T15:53:30 fungi my experience is this is fragile and breaks a lot 2015-03-19T15:53:53 mordred does nodepool support that? 2015-03-19T15:54:00 clarkb: it doesn't need to 2015-03-19T15:54:05 clarkb: where do those nat rules live? something has to arp reply for that linklocal address on the local broadcast segment obviously 2015-03-19T15:54:08 mordred initial boot relies on cloud init iirc 2015-03-19T15:54:11 *** dhritishikhar has joined #openstack-infra 2015-03-19T15:54:12 clarkb: no 2015-03-19T15:54:22 fungi nova/neutron set them up 2015-03-19T15:54:25 I mean, image build step will wonky 2015-03-19T15:54:33 but that doesn't affect node throughput 2015-03-19T15:54:37 fungi it lives on the router device iirc 2015-03-19T15:54:49 lemme make a patch - it'll be 5 minutes 2015-03-19T15:55:04 clarkb: so nova/neutron is failing to present that interface or a nat rule attached to it into the bridge to which the instance gets attached? 2015-03-19T15:55:07 mordred if image build doesnt work then you get no images with your fix... 2015-03-19T15:55:16 I am experiencing an apparent race condition between nodepool and jenkins. Job #91 finishes up and nodepool receives the onCompleted and onFinalized notification, before nodepool has removed the node from the jenkins pool, the node gets picked up by Jenkins for a new job (#92). #92 quickly fails because the test can't grab the FC devices (the same node is still holding on to them). This all repeats for job #93. 2015-03-19T15:55:18 clarkb, mordred: i don't think zmq from jenkins04 is working 2015-03-19T15:55:27 jeblair: awesome 2015-03-19T15:55:39 clarkb: we can boot images _some_ times 2015-03-19T15:55:45 fungi my experience was that it was presented ok but packet return path is easily borked because they only DNAT 2015-03-19T15:55:48 clarkb: all we need to do boot is one node during image build 2015-03-19T15:55:51 for snapshot images 2015-03-19T15:56:05 once we do that, all the rest of the nodes booted from taht one will be fine 2015-03-19T15:56:11 oh lovely, and now freenode is falling over on me 2015-03-19T15:56:12 I think it's worth a try 2015-03-19T15:56:34 mordred ok 2015-03-19T15:56:36 fungi: :( 2015-03-19T15:56:55 *** gampel has quit IRC 2015-03-19T15:56:55 does anyone have any ideas what might be going on? I am about to dig into nodepool and see if I can close the race window. 2015-03-19T15:57:19 rhe00: at the moment we are all focused on getting working nodes for our zuul queues 2015-03-19T15:57:20 Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the Nofifications plugin to support 1.9 https://review.openstack.org/165785 2015-03-19T15:57:37 rhe00: we might have more time to help in a few hours 2015-03-19T15:58:02 rhe00: we set our jobs to pass an OFFLINE_NODE_WHEN_COMPLETE parameter for the job with a value of 1 so that the gearman plugin offlines the node in jenkins before it can pick up a new job 2015-03-19T15:58:04 anteaya: ok, I can wait. I will dig some more on my own. thanks! 2015-03-19T15:58:24 rhe00: thanks for understanding 2015-03-19T15:58:32 fungi: ok, I will look into that. 2015-03-19T15:59:04 i've got jenkins04 in prepare for shutdown, and will clean up its slave cruft once i can safely stop it 2015-03-19T15:59:26 morning 2015-03-19T15:59:29 Jerry Zhao proposed openstack-infra/nodepool: add option to use ipv6 for image update and node launching https://review.openstack.org/156178 2015-03-19T15:59:30 rhe00: we use a zuul parameter function to add that to our jobs by default 2015-03-19T15:59:32 morning zaro 2015-03-19T15:59:49 jeblair its likely the sockets werent closed on the other ends so they never reconnected 2015-03-19T15:59:58 clarkb: i think so 2015-03-19T16:00:05 we will probably need to restart nodepool and logstash client 2015-03-19T16:00:08 fungi, clarkb, mordred: i suspect nodepool wil need to be restarted to correct the zmq error with 04 2015-03-19T16:00:09 ya 2015-03-19T16:00:14 mordred: I don't have magical powers, but I do know an array of people to prod if something is wrong 2015-03-19T16:00:41 04 is in shutdown mode now, it should prevent the problem from getting worse. i will delete more 04 nodes 2015-03-19T16:00:54 *** esker has joined #openstack-infra 2015-03-19T16:01:03 jeblair: i can restart nodepoold once i have jenkins04 stopped 2015-03-19T16:01:29 mordred: nevermind, I see jroll "resolved" your issue 2015-03-19T16:01:51 *** rhsu has joined #openstack-infra 2015-03-19T16:02:44 *** baoli has quit IRC 2015-03-19T16:03:02 we can try killing the tcp connection on the clients but iirc that never works well 2015-03-19T16:03:03 *** baoli has joined #openstack-infra 2015-03-19T16:03:12 Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 2015-03-19T16:03:21 clarkb, fungi, jeblair: ^^ 2015-03-19T16:03:36 that's what I'm thinking to deal with the metadata service instability 2015-03-19T16:03:55 *** koolhead17 has joined #openstack-infra 2015-03-19T16:04:27 gertty has started dumping urllib3 warnings this morning. making it somewhat non functional ... 2015-03-19T16:04:30 mordred that will break because the distros want you to use not root 2015-03-19T16:04:42 *** LinuxJed_ has joined #openstack-infra 2015-03-19T16:04:44 mordred I thibk we ssh as jenkins anyways so maybe unnecessary 2015-03-19T16:04:45 clarkb: oh right. so I need to add it to each of the distro users 2015-03-19T16:05:01 clarkb: when we boot, don't we do an ssh in to test things? 2015-03-19T16:05:04 as the nodepool user 2015-03-19T16:05:04 mordred or maybe we dont need it at all if nodepool sshs in as jenkins 2015-03-19T16:05:06 to the distro user 2015-03-19T16:05:09 it does not 2015-03-19T16:05:17 it ssh's in as the nodepool user to the appropriate distro user 2015-03-19T16:05:22 modified patch coming 2015-03-19T16:05:33 then ya it needs to be the distro user 2015-03-19T16:05:41 i'm deleting all nodes on 04 which are not running a job 2015-03-19T16:06:01 thanks 2015-03-19T16:06:10 that should help 2015-03-19T16:06:43 *** ashleighfarnham has quit IRC 2015-03-19T16:07:07 *** tnovacik has joined #openstack-infra 2015-03-19T16:07:43 *** Somay has joined #openstack-infra 2015-03-19T16:08:00 *** jcoufal_ has quit IRC 2015-03-19T16:08:06 *** LinuxJedi has quit IRC 2015-03-19T16:08:32 taht's ubuntu on ubuntu, cloud-user on centos and fedora on fedora? 2015-03-19T16:09:00 sounds right, there is a list in nodepool itself 2015-03-19T16:09:09 it's not mapped - we just try each of them :) 2015-03-19T16:09:37 oh - centos is root 2015-03-19T16:10:00 Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 2015-03-19T16:10:13 fungi: question about this ^ 2015-03-19T16:10:26 Lars Kellogg-Stedman proposed openstack-infra/project-config: Add "heat-coe-templates" project https://review.openstack.org/164806 2015-03-19T16:11:15 Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 2015-03-19T16:11:16 *** _nadya_ has quit IRC 2015-03-19T16:11:17 fungi: do you think the same code to manage my.cnf should also be duplicated in gerrit::mysql ?? 2015-03-19T16:11:20 clarkb: ^^ that should fix it 2015-03-19T16:12:28 *** belmoreira has quit IRC 2015-03-19T16:12:36 zaro: well, i'm not sure that ! defined(Class['gerrit::mysql']) will actually ensure that gerrit::mysql isn't used, just that it hasn't been defined before that gets tested. but someone with better puppet mojo should confirm 2015-03-19T16:12:37 mordred the linter may not like the extra whitespace but other than that it looks good 2015-03-19T16:12:47 jeblair: just now sat down. Reading backscroll. 2015-03-19T16:12:56 Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 2015-03-19T16:13:03 fungi, jeblair, clarkb: ^^ whitespace fixed 2015-03-19T16:13:31 I mean, given the state of the queue, we may need ninja-merge that 2015-03-19T16:13:37 nibalizer: I've adding refreshonly and subscribe to this https://review.openstack.org/#/c/165145/6/modules/openstack_project/manifests/gerrit.pp 2015-03-19T16:13:39 mordred ya 2015-03-19T16:13:39 so I'd love to get eyes on it 2015-03-19T16:13:45 zaro: as far as doing something similar in the gerrit module's mysql.pp, if you want to do that you probably need to implement it via the mysqldb::config class (i think that's what it's called) 2015-03-19T16:13:55 nibalizer: was wondering if i should remove some of the requires? 2015-03-19T16:14:19 FYI I believe the reason HP's ec2 metadata service fails has something to do with conntrack filling up on compute hosts or neutron-metadata-agent nodes. I forget which. 2015-03-19T16:14:19 mordred the package is the same name on centos and fedora too? 2015-03-19T16:14:24 clarkb: checking 2015-03-19T16:14:29 mordred: you don't need commas after 'fedora' 'ubuntu' and 'root'? 2015-03-19T16:14:50 anteaya: nope - it's a case statement, not a parameter list ... but good eye 2015-03-19T16:14:56 okay 2015-03-19T16:15:06 clarkb: it's the same on centos - I'm not sure I've got a fedora lying aroudn ... 2015-03-19T16:15:11 oh - I can check yum in a docker image 2015-03-19T16:15:12 *** scheuran has quit IRC 2015-03-19T16:15:15 Kiall: ^^ would you concur, that the reason HP Cloud's ec2 metadata stops responding is conntrack related? 2015-03-19T16:15:23 devananda: known issue. you can either roll back to an earlier urllib3, install requests[security] or apply the proposed gertty change to silence that warning 2015-03-19T16:15:23 fungi: hmm, is that an existing class? i can't seem to find a config.pp 2015-03-19T16:15:32 *** ildikov has quit IRC 2015-03-19T16:15:34 zaro: it's not our module 2015-03-19T16:15:49 *** ildikov has joined #openstack-infra 2015-03-19T16:15:53 SpamapS: I think there might be another reason behind that .. 2015-03-19T16:16:02 fungi: great. thanks 2015-03-19T16:16:04 *** unicell has quit IRC 2015-03-19T16:16:17 SpamapS: enjoy my most recent patch :) 2015-03-19T16:16:36 clarkb: yes - also cloud-init on fedora 2015-03-19T16:16:51 zaro: aha, it's called mysql::server::config but probably the parameter you want is exposed through mysql::server https://forge.puppetlabs.com/puppetlabs/mysql#customizing-server-options 2015-03-19T16:18:17 *** mpavone has quit IRC 2015-03-19T16:18:40 mordred: to which thing? 2015-03-19T16:18:53 clarkb: I got the public key value via: 2015-03-19T16:18:54 nodepool@nodepool:~$ ssh-keygen -y -f ~/.ssh/id_rsa 2015-03-19T16:18:57 clarkb: fwiw 2015-03-19T16:19:04 SpamapS: https://review.openstack.org/#/c/165889/ 2015-03-19T16:19:34 mordred: oh, so why aren't we just switching to config drive? 2015-03-19T16:19:52 Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add support for 'raw' to inject direct xml to job configs. https://review.openstack.org/160158 2015-03-19T16:20:01 SpamapS: because that would involve adding features to nodepool, and we'd STILL have to make a system-cofig change to configure cloud-init to skip ec2 metadata 2015-03-19T16:20:18 mordred: skipping ec2 metadata would just speed things up a bit 2015-03-19T16:20:18 SpamapS: whereas, the only thing we use it for on these hosts is getting a known ssh key on them 2015-03-19T16:20:27 mordred: it would work .. just would have a 2 minute boot delay 2015-03-19T16:20:29 SpamapS: so, really, that patch above solves everything :) 2015-03-19T16:20:42 and removes one more piece of fragility 2015-03-19T16:20:46 *** rhsu has quit IRC 2015-03-19T16:21:23 mordred: sure, I think it's fine. It does mean our images are now _OUR_ images. But that was always true really, nobody's using them stock right? 2015-03-19T16:21:24 fungi: from what i've gathered defined('gerrit::mysql') check that the class has been defined. while defined(Class['gerrit::mysql']) checks whether it has been declared. 2015-03-19T16:21:32 https://docs.puppetlabs.com/references/latest/function.html#defined 2015-03-19T16:21:43 *** dboik_ has quit IRC 2015-03-19T16:21:45 Building in SSH keys is now things worked on EC2 for a lot of people too before cloud-init. 2015-03-19T16:21:46 SpamapS: that's right 2015-03-19T16:21:54 Because fetching bits from ec2 metadata was poorly understood. 2015-03-19T16:22:04 *** pblaho__ has joined #openstack-infra 2015-03-19T16:22:04 *** amotoki has quit IRC 2015-03-19T16:22:07 *** dboik has joined #openstack-infra 2015-03-19T16:22:07 *** ddieterly has quit IRC 2015-03-19T16:22:07 yup 2015-03-19T16:22:13 SpamapS I use them stock 2015-03-19T16:22:25 but I have the benefit of having my key baked in :) 2015-03-19T16:22:26 clarkb: jeblair is busy, but says he's ++ on 86ing cloud-init ... fungi - opinion? 2015-03-19T16:22:30 clarkb: :) 2015-03-19T16:22:34 *** ddieterly has joined #openstack-infra 2015-03-19T16:22:45 clarkb: honestly, right now, I care more about us and less about people who might reuse this - we can do better later 2015-03-19T16:22:53 mordred ya 2015-03-19T16:22:59 *** ihrachyshka has quit IRC 2015-03-19T16:23:00 ya, sorry :( 2015-03-19T16:23:04 yep, wfm 2015-03-19T16:23:10 great. I'm going to ninja approve it 2015-03-19T16:23:45 Hello all. I'd like to add a project to stackforge that will run a commit check using tox, but *not* pep8 (because it's not python). What's the correct way to set this up? 2015-03-19T16:23:52 mordred: what's with the require => File['/root/.ssh'] there? 2015-03-19T16:24:16 larsks: IIRC pep8 is in python-jobs 2015-03-19T16:24:19 fungi: THANK YOU 2015-03-19T16:24:20 larsks: so don't list python-jobs 2015-03-19T16:24:31 SpamapS: Right. But I'm not entirely clear how to add a *new* job. 2015-03-19T16:25:22 *** radez is now known as radez_g0n3 2015-03-19T16:25:37 *** pblaho_ has quit IRC 2015-03-19T16:26:02 Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 2015-03-19T16:26:06 fungi, clarkb: ^^ 2015-03-19T16:26:21 *** emagana has joined #openstack-infra 2015-03-19T16:26:25 *** teran_ has quit IRC 2015-03-19T16:27:05 larsks: maybe look at how devstack.yaml is done in project-config/jenkins/jobs 2015-03-19T16:27:17 SpamapS: I'll take a look...thanks. 2015-03-19T16:27:28 mordred lgtm 2015-03-19T16:27:46 Merged openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 2015-03-19T16:28:01 ok. now to kick off a new image ... I'm going to start with one in hp 2015-03-19T16:28:03 just to see 2015-03-19T16:28:17 do you have to put the private key in the secret place for infra root only? 2015-03-19T16:28:36 SpamapS: that is the public key for the already-existing private key for the nodepool user 2015-03-19T16:28:41 I assume there's puppet code that just references variables that aren't set in the normal hiera? 2015-03-19T16:28:47 * SpamapS is using words he's not entirely comfortable with 2015-03-19T16:28:53 mordred: OH 2015-03-19T16:29:00 SpamapS: all of the puppet code that references things in the secret hiera is in the hiera() calls 2015-03-19T16:29:03 but that is still true yes? 2015-03-19T16:29:05 yes 2015-03-19T16:29:07 ok 2015-03-19T16:29:11 * SpamapS stops sweating 2015-03-19T16:29:24 *** ghostpl_ has quit IRC 2015-03-19T16:29:44 SpamapS: if you look in manifest/site.pp, you'll see: 2015-03-19T16:29:46 nodepool_ssh_private_key => hiera('jenkins_ssh_private_key_contents', 'XXX'), 2015-03-19T16:30:00 clarkb: heh. ^^ you were half-right before 2015-03-19T16:30:05 it is the jenkins KEY that gets used 2015-03-19T16:30:07 :) 2015-03-19T16:30:11 fungi: is there a way to remove the stuck jobs from zuul check queue display ? 2015-03-19T16:30:27 ttx by uploading a new commit message 2015-03-19T16:30:31 we discussed that 2015-03-19T16:30:42 ttx: yeah, new patchset to them. are they targeted for the freeze? 2015-03-19T16:30:54 or just unsightly? 2015-03-19T16:30:56 but figured gettin gmore workers was a higher priority than kicking them back in and having them use resources 2015-03-19T16:31:06 clarkb: root@nodepool:~# nodepool image-update hpcloud-b5 nodepool-trusty 2015-03-19T16:31:10 is what I'm doing right now 2015-03-19T16:31:17 fungi: not necessarily, it's just that people tend to scream "OMG please tag K3 Monday check queue is BORK" 2015-03-19T16:31:29 even if it isn't 2015-03-19T16:31:38 it is very slow 2015-03-19T16:31:48 which we are working on but getting new workers 2015-03-19T16:31:56 but k-3 isn't a surprise 2015-03-19T16:32:04 clarkb: I lied: root@nodepool:~# nodepool image-update hpcloud-b5 devstack-trusty 2015-03-19T16:32:18 and some ptls were willing to merge stuff on the weekend to avoid problems 2015-03-19T16:32:30 mikal mentioned it at last week's nova meeting 2015-03-19T16:32:42 *** esker has quit IRC 2015-03-19T16:32:49 *** dmorita has joined #openstack-infra 2015-03-19T16:32:56 *** Bsony has joined #openstack-infra 2015-03-19T16:32:57 mordred ok that should do dib build then upload, if it works dont image-update that image again, instead just image-upload it to the other providers 2015-03-19T16:33:05 clarkb: ++ 2015-03-19T16:33:52 clarkb: nice thing - since this is the dib nodes, we don't even have to wait on a template node to be able to spin up against the metadata borking 2015-03-19T16:33:53 *** tsg_ has joined #openstack-infra 2015-03-19T16:34:06 *** tqtran has joined #openstack-infra 2015-03-19T16:34:08 *** sarob has quit IRC 2015-03-19T16:34:26 Lars Kellogg-Stedman proposed openstack-infra/project-config: Add "heat-coe-templates" project https://review.openstack.org/164806 2015-03-19T16:34:56 *** ghostpl_ has joined #openstack-infra 2015-03-19T16:35:59 I'm going to be afk for a bit 2015-03-19T16:36:02 back later 2015-03-19T16:36:46 good morning 2015-03-19T16:37:22 *** _nadya_ has joined #openstack-infra 2015-03-19T16:37:26 mordred: "nodepool-trusty" or "devstack-trusty" ? 2015-03-19T16:37:35 jeblair: the second 2015-03-19T16:37:43 *** yfried has quit IRC 2015-03-19T16:37:44 cool, that'll be better 2015-03-19T16:37:48 jeblair: turns out, we don't have any images called nodepool-trusty 2015-03-19T16:38:00 jeblair: luckily - nodepool refused to operate :) 2015-03-19T16:39:00 zaro: shrug 2015-03-19T16:39:02 clarkb: did you see I got https://review.openstack.org/#/c/165682/ worked out 2015-03-19T16:39:02 clarkb: STAB STAB STAB 2015-03-19T16:39:07 requires never hurt anybody 2015-03-19T16:39:09 clarkb: for doing monkeypatching of fakes 2015-03-19T16:39:16 ended up working nicely 2015-03-19T16:39:31 *** ayoung has quit IRC 2015-03-19T16:39:35 deleting another round of jenkins04 nodes 2015-03-19T16:39:40 clarkb: in our dib images, we don't have an ubuntu user, apparently 2015-03-19T16:40:02 nibalizer: cool, i'll take that :) 2015-03-19T16:40:02 SpamapS: is that expected in images built from dib on the ubuntu element? 2015-03-19T16:40:10 mordred lol 2015-03-19T16:40:31 mordred: i thought that was a desired feature of dib images? 2015-03-19T16:40:37 *** sandywalsh has joined #openstack-infra 2015-03-19T16:40:45 jeblair: yes, I believe so - it just makes this puppet mildly harder 2015-03-19T16:41:09 greghaynes I noted you pushed a lot of code while I watched day9 play grim fandango then I fell asleep 2015-03-19T16:41:16 mordred: well, only until we're all-dib there ;) 2015-03-19T16:41:19 mordred: yes 2015-03-19T16:41:24 mordred: wierd, I thought we got that for free with ubuntu cloudimg 2015-03-19T16:41:25 then it makes the puppet much, much easier 2015-03-19T16:41:25 mordred: cloud-init makes the user 2015-03-19T16:41:25 fungi: yes. :) 2015-03-19T16:41:29 oh! 2015-03-19T16:41:30 mordred: should be true on all OS's 2015-03-19T16:41:31 lolol 2015-03-19T16:41:32 SpamapS: hahahahaahahah 2015-03-19T16:41:36 SpamapS: orly 2015-03-19T16:41:38 ? 2015-03-19T16:41:39 SpamapS wait 2015-03-19T16:41:42 since cloud-init 0.7 2015-03-19T16:41:47 SpamapS: so - does the "you can't log in with the root user" come from cloud-init too? 2015-03-19T16:41:48 WAT 2015-03-19T16:41:49 this isnt a distro sillyness? 2015-03-19T16:41:53 its cloud init? 2015-03-19T16:41:58 what are they smoking 2015-03-19T16:42:00 SpamapS: like, if I remove cloud-init - can I just put the key in /root/ ? 2015-03-19T16:42:04 and have it work? 2015-03-19T16:42:06 mordred: No I think they just smash that into sshd's config 2015-03-19T16:42:06 mordred: i believe so. i patch that out in my cloudinit config 2015-03-19T16:42:15 during image build 2015-03-19T16:42:15 wow 2015-03-19T16:42:17 SpamapS: it has an option for it at least 2015-03-19T16:42:17 I dont 2015-03-19T16:42:19 even 2015-03-19T16:42:20 ok. follow up patch coming 2015-03-19T16:42:21 or did they add that too? 2015-03-19T16:42:21 wat 2015-03-19T16:42:34 clarkb: see - this is why we're getting out of the cloud-init business 2015-03-19T16:42:41 Wouldn't surprise me if they put all cloud image intelligence into cloudinit 2015-03-19T16:42:44 STOP TRYING TO BE CLEVER 2015-03-19T16:43:00 Why dont youall bake in a user you know the username of? 2015-03-19T16:43:03 we're installing sshd_config files with puppet already aren't we? 2015-03-19T16:43:03 I love suggesting this 2015-03-19T16:43:03 mordred: I believe this makes the cloud images more consumable in different environments. 2015-03-19T16:43:08 greghaynes: it's called "root" 2015-03-19T16:43:10 but everyone hates doing that for some reason 2015-03-19T16:43:11 greghaynes: we could call it... root 2015-03-19T16:43:15 it's amazing 2015-03-19T16:43:18 it's a user that has root access 2015-03-19T16:43:20 SpamapS: rackspace installs a /etc/cloud/cloud.cfg.d/10_rackspace.cfg which turns off passwordauthentication and permitrootlogin for example 2015-03-19T16:43:21 *** jordanP has joined #openstack-infra 2015-03-19T16:43:22 hah, fine 2015-03-19T16:43:29 mordred: don't hate on something people do actually like. You're just made because it doesn't work well on hpcloud. ;) 2015-03-19T16:43:30 *** unicell has joined #openstack-infra 2015-03-19T16:43:31 that works too :) 2015-03-19T16:43:34 or rax 2015-03-19T16:43:36 ;) 2015-03-19T16:43:42 SpamapS: or, well, for any of our use cases 2015-03-19T16:43:45 SpamapS: but I heard you 2015-03-19T16:43:47 hear 2015-03-19T16:44:14 EC2 is the 800lb. gorilla, and most of what cloud-init does is at the request of EC2. The rest is at the request of harlowja_away 2015-03-19T16:44:19 *** BobBall is now known as BobBall_AWOL 2015-03-19T16:44:44 why would someone who isn't ec2... nevermind, i'm falling into the same trap 2015-03-19T16:44:50 Douglas Mendizábal proposed openstack-infra/project-config: Add Python 2.6 jobs to Castellan https://review.openstack.org/165913 2015-03-19T16:44:52 yes its amazing that every install comes with a user preinstalled to do exactly this 2015-03-19T16:44:57 hi guys. I am in charge of a third party CI. I am using jenkins + gerrit trigger (not zuuul). I have a formating issue with how to report back the results. Can someone help me ? Do you know if someone is using successfully gerrit trigger ? 2015-03-19T16:45:01 clarkb: hah about day9, I think I was actually netsplit from openstackbot when I was pushing that and assume no one was seeing me sneak in all those patches 2015-03-19T16:45:03 then for some reason everyone has decided to not use that user 2015-03-19T16:46:05 I'll end my cloud-init sermon with this: The dream is that one root FS can be used on any cloud, for any user, with userdata being the only thing that customizes the image. 2015-03-19T16:46:09 clarkb: because being able to log in as root is bad. because it's a predictable name which eases brute-forcing. so replace it with an equally-guessable name 2015-03-19T16:46:15 The reality is nobody cares about that except distro vendors. :) 2015-03-19T16:46:34 anyone here have some historical knowledge about openstackid? was messing with it the other day to get logins going on zanata 2015-03-19T16:46:37 Monty Taylor proposed openstack-infra/system-config: Just install the key into the root user https://review.openstack.org/165914 2015-03-19T16:46:42 SpamapS: using root as the default user does nothing to impede that goal 2015-03-19T16:46:45 fungi, SpamapS, clarkb, jeblair: ^^ 2015-03-19T16:47:02 SpamapS: actually i care a lot about that. the problem is that the methods for getting the userdata to the instances is, well, not straightforward 2015-03-19T16:47:06 clarkb: using root as the default user removes all chance at audit logging via sudo. 2015-03-19T16:47:13 It's a sane general default. 2015-03-19T16:47:15 mordred: sorry that my initial comment sent us down this path 2015-03-19T16:47:18 clarkb: it's ok 2015-03-19T16:47:37 SpamapS: I hear that - but in my experience so far - userdata via cloud init is WAY less useful than userdata via ssh post boot 2015-03-19T16:47:45 SpamapS: enabling sudo removes all chance at audit logging via sudo 2015-03-19T16:47:49 all one needs is one very simple method to get an ssh key onto the host 2015-03-19T16:47:53 hi, if you have some time please take a look at https://review.openstack.org/#/c/163344/ it's adding a gate check for openstack/ironic-python-agent (non-voting) to test PXE drivers using the IPA ramdisk 2015-03-19T16:47:53 mordred: becuase your userdata delivery methods suck. :) 2015-03-19T16:47:55 mordred, clarkb: so if this runs on a snapshot image, won't we still have an ubuntu user? 2015-03-19T16:48:04 clarkb: eh? 2015-03-19T16:48:12 jeblair: yes, but hrm gah 2015-03-19T16:48:24 SpamapS: once you have root via sudo its game over regardless 2015-03-19T16:48:26 jeblair: but if we install the key onto the root user 2015-03-19T16:48:37 jeblair: then the nodepool user cycle should still find it 2015-03-19T16:48:46 regardless of whetehr the ubuntu user exists 2015-03-19T16:48:47 clarkb: the Ubuntu security team's thinking has always been that the system should be setup to ship auth/security logs elsewhere. So you should, in theory, at least get a 'sudo rm /etc/syslog/....' before it goes dark 2015-03-19T16:48:50 mordred: except that the initial cloud init would have broken our ability to ssh as root 2015-03-19T16:48:56 mordred: right, because we try root first 2015-03-19T16:49:04 clarkb: but we install an sshd_config 2015-03-19T16:49:06 with puppet 2015-03-19T16:49:09 so we will ahve fixed it 2015-03-19T16:49:13 mordred: gotcha 2015-03-19T16:49:42 *** hdd has joined #openstack-infra 2015-03-19T16:49:50 clarkb: let me verify that we do install an sshd_config 2015-03-19T16:49:58 we better :) 2015-03-19T16:49:59 clarkb: so that's not exactly "game over" , that is "first battle goes to the person who gained access to the passwordless sudo user".. but you at least have a breadcrumb to that moment. 2015-03-19T16:50:04 I know we do on our long-lived hosts 2015-03-19T16:50:07 *** thingee has joined #openstack-infra 2015-03-19T16:50:14 For the purposes of single use nodes, yes.. just root in baby. 2015-03-19T16:50:24 mordred: ya I think you are correct, the sshd config is applied universally 2015-03-19T16:50:47 yolanda.robla proposed openstack-infra/puppet-zuul: Add fetch_url and poll parameters to zuul.conf https://review.openstack.org/162591 2015-03-19T16:50:47 SpamapS: (as an aside, if you want the breadcrumb with root logins, just have ssh log the key id of incoming users) 2015-03-19T16:51:41 clarkb: I cannot find where we do it 2015-03-19T16:51:48 *** sandywalsh has quit IRC 2015-03-19T16:51:50 mordred: let me find it 2015-03-19T16:51:54 jeblair: the theory is that you want regular interactive users to be trained to never shell as root, so they're used to sudo'ing. 2015-03-19T16:52:03 jeblair: It's all theory. 2015-03-19T16:52:11 SpamapS: sure. and I appluad that theory 2015-03-19T16:52:24 I don't want it to turn into a security theory debate, just trying to explain why I think it's fine to force people through the sudo funnel. 2015-03-19T16:52:26 I just think that a second shared full-root user like the ubuntu user is actually a bad idea 2015-03-19T16:52:29 *** dustins_ has joined #openstack-infra 2015-03-19T16:52:37 mordred: include ssh in template.pp I think 2015-03-19T16:52:49 mordred: openstack_project::template 2015-03-19T16:52:55 awesome 2015-03-19T16:52:57 *** yamahata has joined #openstack-infra 2015-03-19T16:53:06 mordred: well in theory 90% of what you want to do as a sysadmin is fine as a regular user. top, ps, ls, netstat, etc. 2015-03-19T16:53:15 SpamapS: maybe cloud init should have created a cloud-init user if that was the goal? 2015-03-19T16:53:25 clarkb: oh - actually - we're going to need ... 2015-03-19T16:53:28 blast 2015-03-19T16:53:28 SpamapS: using distro specific users provides zero benefit and a lot of work 2015-03-19T16:53:32 clarkb: it did, but each distro decided it should be a marketing opportunity. :) 2015-03-19T16:53:40 mordred: i say "we better" install an sshd_config, because that was literally change 1 in gerrit ;) 2015-03-19T16:53:52 I agree 100% that having the user be the distro name was short sighted. 2015-03-19T16:53:57 https://review.openstack.org/1 2015-03-19T16:54:04 nice! 2015-03-19T16:54:05 even worse was Fedora making it 'ec2-user' 2015-03-19T16:54:08 *** baoli has quit IRC 2015-03-19T16:54:08 jeblair: we do have that with PermitRootLogin off 2015-03-19T16:54:10 and 2015-03-19T16:54:12 Match host puppetmaster.openstack.org 2015-03-19T16:54:14 PermitRootLogin without-password 2015-03-19T16:54:30 mordred: ha! 2015-03-19T16:54:33 :) 2015-03-19T16:54:38 one sec ... 2015-03-19T16:54:42 so, yeah, we'll need some tweaking 2015-03-19T16:55:13 *** baoli has joined #openstack-infra 2015-03-19T16:55:13 does anybody know if a per-key ssh option will override that? 2015-03-19T16:55:16 how hard would it be to just config drive instead? 2015-03-19T16:55:20 harder 2015-03-19T16:55:32 mordred: we can't just set the image metadata in nodepool config? 2015-03-19T16:55:36 *** dustins has quit IRC 2015-03-19T16:55:41 *** ildikov has quit IRC 2015-03-19T16:55:43 mordred: iirc that was all we needed to do right (and one reason we added image metadata) 2015-03-19T16:55:52 clarkb: it will be at LEAST as much work as this 2015-03-19T16:56:07 because we'll ahve to configure the nodes to mount config-drive 2015-03-19T16:56:18 mordred: yes but thats a one line update to nodepool.yaml 2015-03-19T16:56:25 I'm going to keep working on this 2015-03-19T16:56:29 because cloud-init can suck it 2015-03-19T16:56:30 then cloud init does the rest? 2015-03-19T16:56:37 no it doesn't 2015-03-19T16:56:39 *** Ryan_Lane has joined #openstack-infra 2015-03-19T16:56:43 cloud-init does not mount config-drive 2015-03-19T16:56:48 because BONGHITS 2015-03-19T16:56:51 *** skolekonov has quit IRC 2015-03-19T16:56:58 *** gyee has quit IRC 2015-03-19T16:57:01 I thought it did 2015-03-19T16:57:03 but Ubuntu does? 2015-03-19T16:57:04 yeah - me too 2015-03-19T16:57:08 it dose not 2015-03-19T16:57:12 hrm 2015-03-19T16:57:23 mordred: silly question time -- are you focused on dib or snapshot images? 2015-03-19T16:57:24 I promise 2015-03-19T16:57:37 jeblair: both - I'm just doing this in puppet so it should hit both of them 2015-03-19T16:57:53 mordred: so nodepool does not ssh into anything other than jenkins on the dib path 2015-03-19T16:57:53 AJaeger: if you're still about, could use some confirmation about where we need zanata client installed, I suspect it's just our translations slave but I want to be sure 2015-03-19T16:58:30 jeblair: oh - ok - so I can make this simpler then 2015-03-19T16:58:52 AJaeger: translations/proposals - it auto-submits changes to gerrit, like 75%+ translations pulled from transifex 2015-03-19T16:59:26 *** hyakuhei has joined #openstack-infra 2015-03-19T17:00:04 *** ihrachyshka has joined #openstack-infra 2015-03-19T17:00:17 okay, jenkins04 just achieved quiescence, so stopping the jenkins service now and cleaning it up, then i'll restart nodepoold before i start jenkins again 2015-03-19T17:00:32 mordred: and on the snapshot path -- it only does that when creating a snapshot, before puppet is run (that ssh connection is so that it can run puppet) 2015-03-19T17:00:52 fungi: let me know when you do that and I will restart the logstash client too 2015-03-19T17:01:02 clarkb: you bet 2015-03-19T17:01:27 *** Swami has joined #openstack-infra 2015-03-19T17:01:41 jeblair: really? ok, then I'm confused as to why the metadata server is giving us grief 2015-03-19T17:01:55 *** arxcruz has quit IRC 2015-03-19T17:02:03 Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 2015-03-19T17:02:05 mordred: because cloud init is still enabled 2015-03-19T17:02:07 mordred: well, i mean cloud-init is run 2015-03-19T17:02:23 how does disabling cloud init affect ssh host key generation? 2015-03-19T17:02:30 mordred: i believe it's because cloud init blanks the ssh host key and wants to recreate it so that it's unique, but goes sideways when it can't talk to the metadata server 2015-03-19T17:02:32 will nodepool stop working because too many hosts with the same key? 2015-03-19T17:02:49 *** derekh has quit IRC 2015-03-19T17:02:52 would be good to verify though 2015-03-19T17:03:07 fungi: that seems reasonable 2015-03-19T17:03:17 so, actually then, all we need to do is remove cloud-init 2015-03-19T17:03:24 because we already have the key baked in that we care about 2015-03-19T17:03:29 mordred: correct 2015-03-19T17:03:32 and it literally does nothing 2015-03-19T17:03:33 awesome 2015-03-19T17:03:37 that's the easiest patch ever 2015-03-19T17:03:43 that sounds likely anyway. likely enough to test 2015-03-19T17:03:53 *** unicell has quit IRC 2015-03-19T17:03:58 *** unicell1 has joined #openstack-infra 2015-03-19T17:03:59 *** lucasagomes has quit IRC 2015-03-19T17:04:12 Honestly, if you're not going to replace the key that is in the fetchable image .. telnet ftw? ;) 2015-03-19T17:04:15 Monty Taylor proposed openstack-infra/system-config: Just remove cloud-init https://review.openstack.org/165914 2015-03-19T17:04:24 SpamapS: what? 2015-03-19T17:04:28 SpamapS: that makes no sense at all 2015-03-19T17:04:34 SpamapS: the key is a public key 2015-03-19T17:04:45 the key that is not being replaced is the private host key 2015-03-19T17:05:12 oh - the one that provides no security at all because of how clouds work? 2015-03-19T17:05:13 yeah 2015-03-19T17:05:16 important to replace that 2015-03-19T17:05:17 SpamapS: don't most sshd initscripts create the key at start if it's missing? 2015-03-19T17:05:18 Exactly> :) 2015-03-19T17:05:20 yolanda.robla proposed openstack-infra/project-config: Add stackforge/puppet-nscld https://review.openstack.org/165922 2015-03-19T17:05:21 since I have no secure path to verify it 2015-03-19T17:05:22 SpamapS is right -- we will have identical private keys (which can be retrieved by a job that issues 'cat /etc/ssh/id_rsa...') 2015-03-19T17:05:23 fungi: yes 2015-03-19T17:05:35 so we could just delete the host keys in our images 2015-03-19T17:05:38 i am not sure what someone would do with that, but still... 2015-03-19T17:05:39 *** pcaruana has quit IRC 2015-03-19T17:05:52 as part of the image build process 2015-03-19T17:05:53 fungi: that sounds like it would work. 2015-03-19T17:06:04 with that one can mitm and decrypt passively sniffed packets with a full ssh session. 2015-03-19T17:06:13 fungi: want me to put that into the puppet change above? 2015-03-19T17:06:15 *** sputnik13 has joined #openstack-infra 2015-03-19T17:06:17 ergo: telnet ftw :) 2015-03-19T17:06:26 I'm half serious. 2015-03-19T17:06:55 mordred: yeah, that can't hurt. at least we'll find out where it works and where we need to add a boot-time shim to call ssh-keygen to do it 2015-03-19T17:07:00 yeah, this is probably not important for most of our single-use nodes today, but eventually, we will want to run important things on single-use nodes, so, we should probably fix it 2015-03-19T17:07:13 mordred: there's a secure way to verify host keys, but it involves passing one in that you generate, and then throwing away that one and generating a new one. 2015-03-19T17:07:16 *** jistr has quit IRC 2015-03-19T17:07:17 jeblair: you want to fix it with this patch or as a follow on 2015-03-19T17:07:31 and that assumes nobody has rooted the box before you get to it to replace the one you passed in. 2015-03-19T17:07:35 mordred: this one 2015-03-19T17:07:39 jeblair: k on it 2015-03-19T17:07:53 SpamapS: I think the issue is still anyone can make a job to cat it right? 2015-03-19T17:08:09 *** amitgandhinz has quit IRC 2015-03-19T17:08:15 greghaynes: they'd be catting their own host's private key. It would be useless after that node was destroyed. 2015-03-19T17:08:38 SpamapS: and mostly useless in the interim since they already have access to that node 2015-03-19T17:08:51 SpamapS: No, because you could still use it to mitm since itll verify fine 2015-03-19T17:08:56 clarkb: i'm restarting nodepoold now 2015-03-19T17:09:06 greghaynes: you generate a small (1024bit) RSA key, pass it in via userdata, and let the box install that. Once you SSH in with that one, you rm that key, and regenerate the private host key, noting the public key. 2015-03-19T17:09:07 SpamapS: you have to also revoke it and then its complicated 2015-03-19T17:09:21 *** harlowja_away is now known as harlowja_ 2015-03-19T17:09:25 clarkb: right 2015-03-19T17:09:43 greghaynes: You wouldn't put it in your normal known_hosts file. :) 2015-03-19T17:09:45 okay, nodepoold is running again 2015-03-19T17:09:50 SpamapS lol (just saw cloud-init comment) 2015-03-19T17:09:56 and i've marked all the jenkins04 nodes delete 2015-03-19T17:09:58 greghaynes: that initial SSH would be in known_hosts_for_this_one_time_only .. 2015-03-19T17:10:08 and deleted them from the jenkins config 2015-03-19T17:10:16 ah, yea, that works 2015-03-19T17:10:22 so the window for compromise relies on somebody intercepting that private key between the time you fed it to nova and the time you SSH in. 2015-03-19T17:10:28 jenkins service is coming back online on jenkins04 now 2015-03-19T17:10:43 Monty Taylor proposed openstack-infra/project-config: Remove ssh host keys during image build https://review.openstack.org/165924 2015-03-19T17:10:49 jeblair, fungi: ^^ 2015-03-19T17:10:51 fungi: ok restarting the gearman client now 2015-03-19T17:10:56 this prevents mitm w/ arbitrary key. it does not prevent #allthethings. 2015-03-19T17:11:39 clarkb, SpamapS: ^^ you too 2015-03-19T17:11:51 mordred: both lgtm 2015-03-19T17:11:51 nice 2015-03-19T17:11:57 mordred: +1'd 2015-03-19T17:12:28 *** sandywalsh has joined #openstack-infra 2015-03-19T17:12:29 mordred: see comment 2015-03-19T17:12:46 mordred: I don't think the dib element needs sudo but pretty sure snapshot does 2015-03-19T17:12:49 *** timcline has quit IRC 2015-03-19T17:12:56 ++ 2015-03-19T17:13:00 *** nilasae has quit IRC 2015-03-19T17:13:10 pleia2: i meant to ask you if we should try to move to the later wildfly 2015-03-19T17:13:35 Monty Taylor proposed openstack-infra/project-config: Remove ssh host keys during image build https://review.openstack.org/165924 2015-03-19T17:13:39 clarkb: nice catch 2015-03-19T17:13:50 mordred: I have hit that enough times myself :) 2015-03-19T17:14:04 *** Hal2 has joined #openstack-infra 2015-03-19T17:14:11 mordred: also reading cloud init docs and config drive docs you are correct 2015-03-19T17:14:14 *** Hal1 has quit IRC 2015-03-19T17:14:19 mordred: it appears you must mount the config drive yourself 2015-03-19T17:14:26 which makes me wonder why cloud init even exists 2015-03-19T17:14:29 yes 2015-03-19T17:14:37 cinerama: we'll need to do some testing, not all of the wildfly version work just yet 2015-03-19T17:14:44 clarkb: every. single. use. of cloud init is broken in some little tiny way 2015-03-19T17:14:47 clarkb: drives me batty 2015-03-19T17:14:56 Merged openstack-infra/system-config: Just remove cloud-init https://review.openstack.org/165914 2015-03-19T17:15:00 Merged openstack-infra/project-config: Remove ssh host keys during image build https://review.openstack.org/165924 2015-03-19T17:15:00 no mount yourself, lol 2015-03-19T17:15:03 mordred: harlowja_ suggests that configdrive should be mounted by clout-init if Ubuntu's cloud image has it enabled. 2015-03-19T17:15:03 liesss 2015-03-19T17:15:08 cinerama: we had been stuck on a particular one because of compatibility problems, but with the latest zanata release now is a good time to check 2015-03-19T17:15:20 ^ ya, not a lie what SpamapS says, lol, lie that u need to mount it yourself :-P 2015-03-19T17:15:27 pleia2: i mean as long as we haven't rolled out yet it might be worth a shot 2015-03-19T17:15:29 I have checked jenkins thread counts on our masters, no master appears to currently link threads 2015-03-19T17:15:29 harlowja_: sure. except that it does not work 2015-03-19T17:15:39 s/link/leak/ 2015-03-19T17:15:41 harlowja_: empirical testing has shown me that it does not 2015-03-19T17:15:46 cinerama: also fyi - like gerrit, we'll have a zanata-dev server as well once we go into production, so we'll have a space to work on upgrade testing in the future :) 2015-03-19T17:15:52 mordred hmmm, works on yahoo cloud for a long time 2015-03-19T17:16:02 so, I don't really know or care about the theory - in teh existing images on the public clouds I touch, it does not work 2015-03-19T17:16:10 pleia2: party time in upgrade city 2015-03-19T17:16:13 I'm sure it coudl be configured to 2015-03-19T17:16:15 but 04 did have latest ssh-slaves plugin and still leaked so that may not be the fix, but 02, 05, and 07 all have that plugin version to and haven't leaked since so it is possible that this helps somewhat 2015-03-19T17:16:15 maybe public clouds disabled it 2015-03-19T17:16:19 but why it would ever be configured NOT to is insane 2015-03-19T17:16:21 cinerama: (as much as I *love* our hacky vagrant/cloud testing right now ;)) 2015-03-19T17:16:21 *** tonytan4ever has quit IRC 2015-03-19T17:16:26 *** dimtruck is now known as zz_dimtruck 2015-03-19T17:16:27 and once again, a good idea being made useless by vendors 2015-03-19T17:16:39 *** timcline has joined #openstack-infra 2015-03-19T17:16:39 ./cloud.cfg.d/90_dpkg.cfg:datasource_list: [ NoCloud, ConfigDrive, OpenNebula, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, Ec2, CloudStack, None ] 2015-03-19T17:16:40 mordred do u need a hug 2015-03-19T17:16:43 mordred: ^^ 2015-03-19T17:17:20 harlowja_: I would accept working cloud 2015-03-19T17:17:27 lol 2015-03-19T17:17:28 *** claudiub has quit IRC 2015-03-19T17:17:31 just a hug, jeez 2015-03-19T17:17:32 I'm rebuilding devstack-trusty on hcloud-b5 again 2015-03-19T17:17:34 mordred: thats from my instance on HPCloud East btw. 2015-03-19T17:17:36 stop being so greedy, lol 2015-03-19T17:17:45 *** Bsony has quit IRC 2015-03-19T17:17:51 i do not expect working cloud so will cut my losses and accept a hug 2015-03-19T17:17:53 SpamapS: have you tried booting an instance in hpcloud with config drive enabled and then logging in to see if it's mounted? 2015-03-19T17:17:56 jeblair: ++ 2015-03-19T17:17:57 so then u may need to look @ /var//log/cloud-init.log 2015-03-19T17:18:13 mordred: I ran the 'nova boot' command for that about 30s ago. ;) 2015-03-19T17:18:13 harlowja_: that's hard to do if cloud-init has an error and you can't log in :) 2015-03-19T17:18:23 SpamapS: and config drive is mounted? 2015-03-19T17:18:33 mordred: it's not booted yet 2015-03-19T17:18:34 mordred agreed 2015-03-19T17:18:35 oh 2015-03-19T17:18:48 *** Hal2 has quit IRC 2015-03-19T17:18:52 SpamapS: if it works for you, you and I are going to sit in a dark room and figure out why 2015-03-19T17:18:57 because it's never worked for me :) 2015-03-19T17:19:16 always worked for me, in fact all yahoo vms on openstack are using it, lol 2015-03-19T17:19:26 but i can help trouble shoot it (since i'm one of the creators, lol) 2015-03-19T17:19:30 *** otter768 has joined #openstack-infra 2015-03-19T17:19:35 heh 2015-03-19T17:19:40 *** Bsony has joined #openstack-infra 2015-03-19T17:19:43 http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L73 (the magical mounting code) 2015-03-19T17:20:07 harlowja_: what version is that first in? 2015-03-19T17:20:17 harlowja_: i wonder if ubuntu 12.04 cloud-init is too old 2015-03-19T17:20:20 hmmm 2015-03-19T17:20:28 there should be a version of it in that 2015-03-19T17:20:39 0.6.3 2015-03-19T17:20:45 * SpamapS drums fingers while instance schedules 2015-03-19T17:20:52 SpamapS: good luck with that 2015-03-19T17:20:55 let me download 0.6.3; forgot what was in it 2015-03-19T17:21:25 at this point nova boot hasn't even come back 2015-03-19T17:21:31 ahh, only 90s to respond 2015-03-19T17:21:37 *** ashleighfarnham has joined #openstack-infra 2015-03-19T17:21:48 *** patrickeast has joined #openstack-infra 2015-03-19T17:21:54 *** xyang1 has joined #openstack-infra 2015-03-19T17:22:05 SpamapS: welcome to hpcloud! 90s is a good response time from what i've seen 2015-03-19T17:22:28 so 0.6.3 is pretty damn old 2015-03-19T17:23:03 it may/may not work correctly, i don't think it has support for what nova is writing into it (but i'm not 100% sure) 2015-03-19T17:23:09 it has the old config-drive v1 code 2015-03-19T17:23:13 harlowja_: ubuntu 14.04 (the other ubuntu we run) has 0.7.5 at least 2015-03-19T17:23:17 yup 2015-03-19T17:23:53 *** otter768 has quit IRC 2015-03-19T17:24:05 Cloud-init v. 0.7.5 finished at Thu, 19 Mar 2015 17:22:21 +0000. Datasource DataSourceConfigDriveNet [net,ver=2][source=/dev/sr0]. Up 50.63 seconds 2015-03-19T17:24:08 mordred: ^ 2015-03-19T17:24:20 0.6.3 was before my huge refactor 2015-03-19T17:24:30 0.7.0 include josh super sauce 2015-03-19T17:24:37 0.7.0+ 2015-03-19T17:24:53 (which is why there is no 0.6.4, lol) 2015-03-19T17:24:55 * SpamapS watches harlowja_ flex 2015-03-19T17:24:55 harlowja_: ok so its likely will not work for us 2015-03-19T17:24:58 *** bhuvan has quit IRC 2015-03-19T17:25:00 SpamapS oh ya 2015-03-19T17:25:00 lol 2015-03-19T17:25:22 i've seen people ask for it to be backported; but i don't think thats how canonical works 2015-03-19T17:25:49 mordred: 2015-03-19 17:21:40,798 - util.py[DEBUG]: Running command ['mount', '-o', 'ro,sync', '/dev/sr0', '/tmp/tmpVPaVgX'] with allowed return codes [0] (shell=False, capture=True) 2015-03-19T17:25:55 amazing :-P 2015-03-19T17:25:56 JayF: and centos 6 has cloud-init 0.7.5 as well (or 0.7.4 in epel) 2015-03-19T17:26:03 er, harlowja_ ^ 2015-03-19T17:26:08 sorry JayF 2015-03-19T17:26:11 ya 2015-03-19T17:26:15 It's fine, It's good for me to know too 2015-03-19T17:26:24 probably means I should bump our downstream patched versino of cloud-init as well, heh 2015-03-19T17:26:40 mordred: so what you may not like, is that it just mounts it _for its purposes_ 2015-03-19T17:26:59 mordred: but it leaves everything it found in /var/lib/cloud/instance 2015-03-19T17:27:04 SpamapS: HEADDESK 2015-03-19T17:27:17 SpamapS: STAB STAB STAB 2015-03-19T17:27:24 lol 2015-03-19T17:27:26 so you have the entirety of the data there. 2015-03-19T17:27:26 *** mwagner_lap has quit IRC 2015-03-19T17:27:26 http://cloudinit.readthedocs.org/en/latest/topics/dir_layout.html 2015-03-19T17:27:29 Valeriy Ponomaryov proposed openstack-infra/project-config: Add tempest + postgres job for manila https://review.openstack.org/165935 2015-03-19T17:27:31 and more.... 2015-03-19T17:28:00 *** achanda has joined #openstack-infra 2015-03-19T17:28:01 mordred: pretty sure you just want to hate. Perhaps you should go to Belgium and help them hate the Flemmish instead? 2015-03-19T17:28:05 okay, so cloud-init mounts it, copies the data to a known stable location in the root filesystem and then unmounts it again. that's not so bad 2015-03-19T17:28:09 Valeriy Ponomaryov proposed openstack-infra/project-config: Add tempest + postgres job for manila https://review.openstack.org/165935 2015-03-19T17:28:11 SpamapS: but I don't have it where config-drive 2015-03-19T17:28:13 puts it 2015-03-19T17:28:15 anyway 2015-03-19T17:28:16 you do 2015-03-19T17:28:24 mordred hug, lol 2015-03-19T17:28:29 its still there on /dev/sr0 2015-03-19T17:28:31 SpamapS: that was a trusty node right? 2015-03-19T17:28:34 *** achanda has quit IRC 2015-03-19T17:28:35 clarkb: aye 2015-03-19T17:28:41 SpamapS: we will need to check precise as well 2015-03-19T17:28:41 *** ominakov_ has quit IRC 2015-03-19T17:28:43 Description: Ubuntu 14.04.1 LTS 2015-03-19T17:28:52 *** xyang1 has quit IRC 2015-03-19T17:28:55 http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L228 (not so magical find rountine) 2015-03-19T17:28:56 maybe once we're able to stop supporting precise we can revisit 2015-03-19T17:28:58 since precise is older cloud init and we need precise 2015-03-19T17:28:59 *** achanda has joined #openstack-infra 2015-03-19T17:28:59 nova boot --config-drive 1 --key-name default --image 9d25fe2d-cf31-4b05-8c58-f238ec78e633 test-cdrive --flavor 100 2015-03-19T17:29:02 fungi: ya 2015-03-19T17:29:03 in useast 2015-03-19T17:29:14 *** xyang1 has joined #openstack-infra 2015-03-19T17:29:14 SpamapS: how did you get that to work without network info? 2015-03-19T17:29:26 clarkb: magic? 2015-03-19T17:29:29 *** wuhg has quit IRC 2015-03-19T17:29:31 SpamapS: unrelated its just my biggest annoyance booting nodes in hpcloud 2015-03-19T17:29:55 I have a network already setup of course. 2015-03-19T17:30:00 which is like o_O when you have to set it up 2015-03-19T17:30:01 *** e0ne has quit IRC 2015-03-19T17:30:01 *** dtantsur is now known as dtantsur|afk 2015-03-19T17:30:08 SpamapS: yes but even then you still have to provide a network to boot on 2015-03-19T17:30:09 but once I made it, I only have one, so hpcloud attaches it 2015-03-19T17:30:14 SpamapS: then you have to attach floating ip to get into the node 2015-03-19T17:30:18 *** ominakov has joined #openstack-infra 2015-03-19T17:30:21 clarkb: No it will attach all the networks you have access to 2015-03-19T17:30:21 oh ou have only one 2015-03-19T17:30:35 or did they screw that up now? 2015-03-19T17:30:39 SpamapS: it errors for us an our many networks if you don't supply a specific network 2015-03-19T17:30:56 is there a reason you have many networks? 2015-03-19T17:31:12 *** mestery_ has joined #openstack-infra 2015-03-19T17:31:12 SpamapS: yes, the router/networks can only handle about 100 nodes a piece 2015-03-19T17:31:20 lurvely 2015-03-19T17:31:21 SpamapS: neutron router instances don't scale well to our nova instance count 2015-03-19T17:31:31 also even if they did, they're provisioned as a /24 2015-03-19T17:31:32 DVR should solve that 2015-03-19T17:31:38 *** andreykurilin_ has joined #openstack-infra 2015-03-19T17:31:39 DVR won't solve that 2015-03-19T17:31:42 so we couldn't put more than ~253 nodes on one 2015-03-19T17:31:52 fungi: ya we can make bigger networks it just wasn't required since we were told ~100 nodes per router 2015-03-19T17:31:58 but you can make it a /8 if you want 2015-03-19T17:32:07 oh, if we have control to push that lower than /24 then cool 2015-03-19T17:32:21 I don't know if they rolled out DVR in the public cloud tho 2015-03-19T17:32:25 SpamapS: why would DVR solve the scaling issue? 2015-03-19T17:32:29 thought they'd imposed a hard limit there but i must have misremembered 2015-03-19T17:32:34 SpamapS: aiui DVR is more similar to nova net multihost 2015-03-19T17:32:46 *** _nadya_ has quit IRC 2015-03-19T17:32:46 SpamapS: which is more about redundancy than performance 2015-03-19T17:32:57 (though you get some performance improvment because you are partitioned) 2015-03-19T17:33:00 jenkins04 is now looking good and happy 2015-03-19T17:33:07 *** mestery has quit IRC 2015-03-19T17:33:30 *** _nadya_ has joined #openstack-infra 2015-03-19T17:33:38 *** coolsvap is now known as coolsvap|afk 2015-03-19T17:33:50 clarkb: IIRC, east/west gets faster, north/south still has to go through l3 agent. 2015-03-19T17:33:58 ugh people need to stop using those terms 2015-03-19T17:34:10 because you don't like them or because they're wrong? 2015-03-19T17:34:22 SpamapS: because the are non descriptive and tell me little 2015-03-19T17:34:31 networking has nothing to do with compass direction 2015-03-19T17:34:50 clarkb: it has to do with the most common network layout diagram 2015-03-19T17:34:58 east is east of west right? though i guess east is also west of west. damn you spherical coordinate system 2015-03-19T17:35:11 north == out, south == in, east/west == amongst peers 2015-03-19T17:35:15 *** packet has joined #openstack-infra 2015-03-19T17:35:38 SpamapS: and for our use case that won't help anyting since we re prodominently pushing packets off network 2015-03-19T17:35:44 that's such a marvellous example of cargo culting terminology. almost as great as the origin of "cloud" 2015-03-19T17:35:56 jenkins <-> slave <-> mirrors 2015-03-19T17:36:00 fungi: In this particular diagram east of east thar be dragons. 2015-03-19T17:36:13 I need a gif of 1s and 0s flying in random directions 2015-03-19T17:36:21 SpamapS: ooh! hplcoud is deployed on a flat earth! 2015-03-19T17:36:30 clarkb: yeah, agree. 2015-03-19T17:36:34 *** ildikov has joined #openstack-infra 2015-03-19T17:36:40 *** markus_z has quit IRC 2015-03-19T17:36:44 fungi: explains a lot 2015-03-19T17:36:45 now i have proof for my spherical fallacy conspiracy 2015-03-19T17:36:54 *** ominakov_ has joined #openstack-infra 2015-03-19T17:36:56 *** ominakov has quit IRC 2015-03-19T17:37:03 *** claudiub has joined #openstack-infra 2015-03-19T17:37:05 *** amitgandhinz has joined #openstack-infra 2015-03-19T17:37:10 *** andreykurilin__ has joined #openstack-infra 2015-03-19T17:37:10 *** andreykurilin_ has quit IRC 2015-03-19T17:37:12 *** xyang1 has quit IRC 2015-03-19T17:37:52 The real question is how the east/west map into toroid networks 2015-03-19T17:38:40 greghaynes: mostly with a hammer to your head and some shrooms 2015-03-19T17:38:54 SpamapS: also why not describe it as l2 vs l3 communication? 2015-03-19T17:39:05 clarkb: its not 2015-03-19T17:39:06 in any case I should go back to poking at jenkinses 2015-03-19T17:39:12 its about moving up the tree 2015-03-19T17:39:12 clarkb: because east/west can describe L3 segment hops too. 2015-03-19T17:39:17 *** garyh has quit IRC 2015-03-19T17:39:36 *** marcusvrn has quit IRC 2015-03-19T17:39:39 clarkb: in the DVR case, if you're just sending to another 10.x range DVR has that route and doesn't need l3 agent, it just sends. 2015-03-19T17:39:40 SpamapS: what about the term "east/west" describes any of that? 2015-03-19T17:39:51 *** mestery_ is now known as mestery 2015-03-19T17:40:16 anyways I am just grumping, I find the terms to be silly as they do not describe what is going on in a meaningful way 2015-03-19T17:40:41 *** ominakov_ has quit IRC 2015-03-19T17:40:46 fungi: jenkins01 was restarted too right? 2015-03-19T17:41:03 *** bhuvan has joined #openstack-infra 2015-03-19T17:41:12 clarkb: yep 2015-03-19T17:41:35 clarkb: east/west is generally used to mean "doesn't go through the thing that connects north/south" .. In a big network, you'll have L2 segments behind L3 routers linked together as peers and then north of those the uplinks to the internet/WAN. 2015-03-19T17:42:11 clarkb: I don't know a better short hand for this, but I'm open to suggestions. 2015-03-19T17:42:27 fungi: https://jenkins01.openstack.org/monitoring?part=graph&graph=threadCount it also leaked threads but isn't running the latest ssh-slaves plugin 2015-03-19T17:42:49 *** garyh has joined #openstack-infra 2015-03-19T17:42:52 *** amitgandhinz has quit IRC 2015-03-19T17:43:11 *** amitgandhinz has joined #openstack-infra 2015-03-19T17:43:17 *** ivar-lazzaro has joined #openstack-infra 2015-03-19T17:43:47 fungi: it did not leak as badly as 04 which is probably why you were able to more gracefully restart it 2015-03-19T17:44:06 *** yamamoto has quit IRC 2015-03-19T17:44:53 clarkb: well, i caught 01 by manually looking for misbehaving servers. in the wake of 04 which mordred caught once it had fallen over to the point where it wasn't even responding to ssh any longer 2015-03-19T17:45:15 so i have a feeling 01 would have gotten there itself in due time 2015-03-19T17:45:40 so, we can update ssh-slaves plugin everywhere simply because it appears to help but not fix the problem (05, 02 and 07 haven't leaked yet) but not sure what causes the leaks so that may just be wishful thinking 2015-03-19T17:45:47 *** ivar-lazzaro has quit IRC 2015-03-19T17:45:49 fungi: ya, the leak does not seem to be cleaned up 2015-03-19T17:46:05 fungi: I tested that on 05 by leaving it in shutdown mode for a day a few weeks back and the thread count did not fall 2015-03-19T17:46:26 fun 2015-03-19T17:46:29 mordred: how is that image build going? 2015-03-19T17:46:41 *** andreykurilin__ has quit IRC 2015-03-19T17:46:49 zaro: ^ any other ideas on the thread leak? 2015-03-19T17:46:56 zaro: that patch to the ssh-slaves plugin looked so promising 2015-03-19T17:47:00 *** tonytan4ever has joined #openstack-infra 2015-03-19T17:47:01 *** ivar-lazzaro has joined #openstack-infra 2015-03-19T17:47:54 *** crc32 has joined #openstack-infra 2015-03-19T17:49:22 *** hyakuhei has quit IRC 2015-03-19T17:50:41 clarkb: reading logs 2015-03-19T17:50:48 *** ayoung has joined #openstack-infra 2015-03-19T17:51:08 Exception: DIB failed creating /opt/nodepool_dib/devstack-trusty-1426785436.qcow2 2015-03-19T17:51:45 *** tonytan4ever has quit IRC 2015-03-19T17:52:08 *** dustins_ has quit IRC 2015-03-19T17:52:41 *** zz_dimtruck is now known as dimtruck 2015-03-19T17:53:07 *** Somay has quit IRC 2015-03-19T17:53:30 *** ghostpl_ has quit IRC 2015-03-19T17:53:57 *** Somay has joined #openstack-infra 2015-03-19T17:54:53 clarkb: except it seems to have succeeded 2015-03-19T17:55:25 mordred: that doesn't end up in the image.log for me to read right? because it was a command not the daemon 2015-03-19T17:55:42 *** SumitNaiksatam has joined #openstack-infra 2015-03-19T17:56:30 *** HeOS has quit IRC 2015-03-19T17:56:46 yah 2015-03-19T17:56:47 2015-03-19 17:41:15,623 INFO nodepool.DiskImageUpdater: Image devstack-trusty-1426786672.template.openstack.org in hpcloud-b5 is ready 2015-03-19T17:57:35 *** tonytan4ever has joined #openstack-infra 2015-03-19T17:58:22 clarkb: http://paste.openstack.org/show/193643/ 2015-03-19T17:58:58 fyi, the ubuntu ssl package updates seem to have just surfaced in the mirrors 2015-03-19T17:59:11 mordred: so it uploaded the previous image I think 2015-03-19T17:59:18 mordred: I would do image-build not image-update now 2015-03-19T17:59:23 mordred: but look further back for where dib failed 2015-03-19T17:59:56 *** reed has quit IRC 2015-03-19T18:00:13 clarkb: look at the top of that 2015-03-19T18:00:29 mordred: that looks like dib cleanup at the top 2015-03-19T18:00:29 clarkb: oh - wait - no, not the rm -rf 2015-03-19T18:00:46 2015-03-19 17:37:51,310 INFO nodepool.image.build.devstack-trusty: qemu-img: error while compressing sector 13184128: No space left on device 2015-03-19T18:00:47 there it is 2015-03-19T18:00:49 mordred: you need to go back, this is one problem with dib, something will fail about 1000 log lines earlier before you get the exit one 2015-03-19T18:00:51 mordred: :( 2015-03-19T18:00:53 thar ya go 2015-03-19T18:00:54 *** jlanoux has quit IRC 2015-03-19T18:01:01 /dev/xvde2 72G 65G 3.2G 96% /opt 2015-03-19T18:02:01 greghaynes: I havea bunch of devstack-fedora21-dib-1426774571.d directories 2015-03-19T18:02:10 those are all leftovers from failed buidls, right? 2015-03-19T18:02:13 mordred: those are fallout from f21 image builds failing Ithink 2015-03-19T18:02:19 dib doesn't cleanup those dirs after failure 2015-03-19T18:02:24 *** bhunter71 has quit IRC 2015-03-19T18:02:30 *** boris-42 has quit IRC 2015-03-19T18:02:34 in what locatoin? 2015-03-19T18:02:34 but they are small 2015-03-19T18:02:50 greghaynes: in the image output dir 2015-03-19T18:03:15 mordred: I think adding the f21 image pushed our little disk over the limit 2015-03-19T18:03:15 sdague: thanks for the heads up. i poured over the list of vulnerabilities today and it's at worst remote denial of service for some ssl-enabled services. i'm okay with letting them get applied through our usual automatic channels 2015-03-19T18:03:16 ah, youre making unique image names 2015-03-19T18:03:21 *** ZZelle_ has joined #openstack-infra 2015-03-19T18:03:22 greghaynes: yes 2015-03-19T18:03:23 yes, those arent cleaned 2015-03-19T18:03:28 python needs a mode where it ignores any .pyc's that don't have their supporting .py 2015-03-19T18:03:30 *** _nadya_ has quit IRC 2015-03-19T18:03:34 *** enikanorov has joined #openstack-infra 2015-03-19T18:03:34 I dont see it because I tend to reuse the same image name 2015-03-19T18:03:49 I think its just manifests 2015-03-19T18:03:54 greghaynes: yup 2015-03-19T18:03:56 The amount of time I've spent this week chasing ghosts then running 'find . -name '*.pyc' -print0 | xargs -0 rm' is stupid 2015-03-19T18:04:00 its not a significant amount of disk space 2015-03-19T18:04:44 mordred: we can probably get rid of the precise-dib and trusty-dib images 2015-03-19T18:05:12 huh, looks like for some reason the dpkg element depends on manifests 2015-03-19T18:05:18 clarkb: kk 2015-03-19T18:05:19 that should probably just be removed 2015-03-19T18:05:25 mordred: I will nodepool image-delete the two older ones of that image 2015-03-19T18:05:31 mordred: that should give you enough to get a new image for now 2015-03-19T18:06:01 mordred: also you didn't sudo -H -u nodepool so you will need to fix the dib cache and all that 2015-03-19T18:06:05 *** enikanorov__ has quit IRC 2015-03-19T18:06:18 greghaynes: probably just a bug that got missed. 2015-03-19T18:06:19 I will now sudo -H -u nodepool 2015-03-19T18:06:25 mordred: first you need to fix it 2015-03-19T18:06:33 *** gyee has joined #openstack-infra 2015-03-19T18:06:33 mordred: I need to remember what the fix is 2015-03-19T18:06:50 clarkb: its chown -R nodepool:nodepool /opt/dib_cache now? 2015-03-19T18:06:52 no? 2015-03-19T18:07:15 mordred: ya that looks right 2015-03-19T18:07:18 doing that 2015-03-19T18:07:32 mordred: also chown the -new or maybe rm that image in /opt/nodepool_dib 2015-03-19T18:07:36 mordred: I think that is your failed image 2015-03-19T18:08:04 greghaynes: 4c467964 .. probably just need to drop a file in environment.d for manifests, and then make dpkg inspect whether it is set or not. 2015-03-19T18:08:48 SpamapS: yep 2015-03-19T18:08:57 looks like source-repositories also deps on it but could use that method too 2015-03-19T18:09:22 * SpamapS was never really fully onboard with the manifests plan but wanted to get out of their way because it seemed important to the people working on it. 2015-03-19T18:09:23 mordred: you have 16GB of disk, should be enough to build that image 2015-03-19T18:09:46 Yea, its hard to tell a user with a clear use case no 2015-03-19T18:09:50 mordred: I ran sudo -H -u nodepool nodepool dib-image-delete devstack-*-dib essentially 2015-03-19T18:09:55 *** sdake_ has quit IRC 2015-03-19T18:11:19 *** sdake has joined #openstack-infra 2015-03-19T18:11:33 *** coolsvap|afk has quit IRC 2015-03-19T18:11:36 *** crc32 has quit IRC 2015-03-19T18:12:42 we've had a steady (and steep) climb in building nodes corresponding to clearing up tons of delete nodes. not sure if this is something we've triggered or something improving in hpcloud 2015-03-19T18:13:07 fungi: I don't think we have any of the fixes in yet other than having unstuck some jenkins 2015-03-19T18:14:02 hrm, or things getting worse in hpcloud? we have 125 nodes building for >1 hour 2015-03-19T18:14:48 looks like maybe this is still the replenishment from the nodepool restart 2015-03-19T18:15:07 clarkb, fungi: unsticking jenkins actually helped a bit -- i believe when jenkins is very slow to respond, it slows down the state conversions of servers enough to have an effect 2015-03-19T18:15:14 clarkb: rebuilding 2015-03-19T18:15:24 jeblair: ahh, yep that would make sense 2015-03-19T18:15:27 clarkb, fungi: also, if it's slow enough to error out, that can actually kill a build that otherwise worked 2015-03-19T18:15:29 *** dustins has joined #openstack-infra 2015-03-19T18:15:50 clarkb, fungi: so, for example, once jenkins01 was shut down, there was an immediate uptick of about 50 ready nodes 2015-03-19T18:16:07 jeblair: gotcha 2015-03-19T18:16:18 it does look like the current building hill is peaking, so maybe we'll see lots more used nodes here shortly 2015-03-19T18:16:53 *** hodos has joined #openstack-infra 2015-03-19T18:17:04 anteaya: still see comments from that id we banned this AM - https://review.openstack.org/#/c/165947/ 2015-03-19T18:17:08 *** bhunter71 has joined #openstack-infra 2015-03-19T18:17:21 Joe Gordon proposed openstack-infra/project-config: Remove check-tempest-dsvm-f20 https://review.openstack.org/165532 2015-03-19T18:17:25 dims: its cached, you have to wait for that to get invalidated 2015-03-19T18:17:36 clarkb: ack thanks 2015-03-19T18:17:42 *** ddieterly has quit IRC 2015-03-19T18:17:59 i don't recall if anyone got around to disabling that account yet 2015-03-19T18:18:08 fungi: I think jeblair did it 2015-03-19T18:18:11 k 2015-03-19T18:18:16 but the cache affects disabled accounts too 2015-03-19T18:18:19 yep 2015-03-19T18:19:55 *** ddieterly has joined #openstack-infra 2015-03-19T18:19:59 *** cdent has quit IRC 2015-03-19T18:19:59 *** johnthetubaguy is now known as zz_johnthetubagu 2015-03-19T18:20:02 anyone know how to get the apikey forthe jenkin's user credintials-id? 2015-03-19T18:21:03 asselin: login as that user, go to settings page click button to get api key 2015-03-19T18:21:26 asselin: I think you have to do that at least once but from that point on the key is available on disk in $JENKINSHOME/users/username/config.xml or something like that 2015-03-19T18:22:09 clarkb, yeah...in that file it's ecrypted...but then there's no credentails-id 2015-03-19T18:22:34 2015-03-19 18:21:08,826 INFO nodepool.image.build.devstack-trusty: Downloading pandas-0.15.2.tar.gz (4.6MB) 2015-03-19T18:22:34 *** jordanP has quit IRC 2015-03-19T18:22:40 I just think that's a funny line 2015-03-19T18:23:04 asselin: not sure then 2015-03-19T18:23:25 asselin: but I think you need to request credentials before they have n id 2015-03-19T18:23:37 asselin: so try getting the api key if you haven't yet 2015-03-19T18:23:49 *** Somay has quit IRC 2015-03-19T18:24:27 clarkb, yeah I crated a new user, got the api key, but the're no credentials-id....seems they moved stuff around in the latest version of jenkins....I will poke a bit more 2015-03-19T18:24:42 dims: thank you 2015-03-19T18:25:01 clarkb: did we want to clear that cache? 2015-03-19T18:25:09 mordred: It's been years since I downloaded pandas 2015-03-19T18:25:11 anteaya: it clears on its own 2015-03-19T18:25:18 how long does it take? 2015-03-19T18:25:27 anteaya: have you banned "Evgeny Antyshev" yet? 2015-03-19T18:25:31 anteaya: I am unsure 2015-03-19T18:25:45 https://review.openstack.org/#/c/165953/ - they are commenting on *every* nova patch atm 2015-03-19T18:25:49 sdague: yes, but with open connections we have to either clear the cache or wait for it to expire 2015-03-19T18:26:29 ldap groups has a cache of 1 hour, web sessions has a cache of 12 hours 2015-03-19T18:26:36 any idea who they work for? 2015-03-19T18:26:38 I don't think this will be in ldap groups 2015-03-19T18:26:51 we disalbed them more than an hour ago 2015-03-19T18:27:11 sdague: parallels.com? 2015-03-19T18:27:22 *** ghostpl_ has joined #openstack-infra 2015-03-19T18:27:35 I don't think its a web session cache either since comments should come via ssh? 2015-03-19T18:27:48 we disabled them 3 hours ago 2015-03-19T18:28:42 *** AJaeger_ has joined #openstack-infra 2015-03-19T18:29:04 *** reed has joined #openstack-infra 2015-03-19T18:29:24 I'm not seeing that they created a new account 2015-03-19T18:30:00 pleia2: We need the zanata client installed on systems that run the post translation jobs and the translation proposal 2015-03-19T18:30:19 *** spzala has quit IRC 2015-03-19T18:30:48 clarkb, I got it thanks 2015-03-19T18:30:49 and nodes in building state just went up over 450 2015-03-19T18:30:51 show-caches makes me think it is the accounts cache but I can't find docs on the timeout for that 2015-03-19T18:30:55 fungi: You just answered my question on https://review.openstack.org/165661 - but I'm still confused 2015-03-19T18:31:03 *** emagana has quit IRC 2015-03-19T18:31:06 clarkb: can we clear the cache? 2015-03-19T18:31:17 Wouldn't a normal merge have already updated the content and the release job do it again? 2015-03-19T18:31:36 anteaya: we can, is this super urgent and letting the cache invalidate itself isn't cutting it? 2015-03-19T18:31:52 (I don't know how often this user is commenting) 2015-03-19T18:32:02 sdague said it is every nova patch 2015-03-19T18:32:06 AJaeger_: AJaeger_ so translations proposal is the one that queries transifex/zanata to look for 75%+ translations and proposes them to gerrit, what are the post translations jobs? 2015-03-19T18:32:07 AJaeger_: updating the _version_ if embedded in the document 2015-03-19T18:32:20 *** emagana has joined #openstack-infra 2015-03-19T18:32:24 AJaeger_: as in release version number of the repo 2015-03-19T18:32:50 clarkb: it's just really misleading as we are now getting "This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset." 2015-03-19T18:33:09 fungi: Ah, so we would have an update after the merge, than one with new version after a tag - and another one with the next change. 2015-03-19T18:33:32 from their poorly configured zuul 2015-03-19T18:33:32 So, if merges are rare, it's worth doing it both times - or what is the reason here? 2015-03-19T18:34:31 the flush-caches command is running now 2015-03-19T18:34:36 clarkb: thank you 2015-03-19T18:34:41 pleia2: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n385 and http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n392 - the upstream jobs 2015-03-19T18:34:43 AJaeger_: it totally depends on the merge and release cadence of a given project. since i'm not intimately familiar with this one i simply proposed it do both. the extra run when the repo is tagged is not a lot of overhead 2015-03-19T18:35:04 pleia2: so, this is the push to transifex/zanata 2015-03-19T18:35:16 pleia2: while the proposal is the other way round 2015-03-19T18:35:30 fungi: Ok, understood. Thanks for the explanation 2015-03-19T18:35:31 mordred: can the following files in /opt/nodepool_dib be deleted: devstack-fedora21-dib-1426774571.qcow2-new devstack-trusty-1426785436.qcow2-new devstack-trusty-dib-1426774571.qcow2-new ? 2015-03-19T18:35:33 AJaeger_: ah, gotcha, I'll dig into which slaves these are 2015-03-19T18:36:14 /opt/nodepool_dib/devstack-trusty-1426788873.qcow2-new is the current in-flight one 2015-03-19T18:36:18 clarkb: ^^ 2015-03-19T18:36:29 sdague: anteaya dims caches are flushed, if you see comments after now then I would worry 2015-03-19T18:36:37 clarkb: thank you 2015-03-19T18:36:39 clarkb: thanks 2015-03-19T18:37:29 Merged openstack-infra/system-config: Revert "Temporarily stop adding devstack-f20 nodes in rax" https://review.openstack.org/165621 2015-03-19T18:37:46 clarkb: hi, Phillip who you have helped in the past http://lists.openstack.org/pipermail/openstack-dev/2015-February/057585.html is claiming that his CI is waiting on some changes from you. I'm just verifying if you have any idea what he's talking about so I can take into consideration with Cinder's CI deadline for drivers. 2015-03-19T18:37:47 fungi: oh, did rax fix the problem ^ ? 2015-03-19T18:37:59 thingee: uh 2015-03-19T18:38:01 i'll keep an eye on ^ and make sure the new images there are working with ianw's patch in place 2015-03-19T18:38:05 jeblair: in theory, yes 2015-03-19T18:38:14 thingee: if there is a change up to propose the job to run on one node then I guess they would be waiting on us 2015-03-19T18:38:21 thingee: otherwise no I don't think there is anything preventing ^ 2015-03-19T18:38:43 clarkb: ok, is that up for review? Sorry would check myself but context switching a bit at the moment. 2015-03-19T18:38:49 jeblair: patch to install_puppet.sh at https://review.openstack.org/165681 2015-03-19T18:38:50 thingee: my suggestion was to work on making the simple thing go before worrying about multi node and all that 2015-03-19T18:39:01 thingee: I don't think the simple thing has been proposed but I haven't checked recently 2015-03-19T18:39:16 clarkb: ok, thank you 2015-03-19T18:39:18 jeblair: hopefully we can clean that back up as soon as we stop using f20 2015-03-19T18:39:19 thingee: I would ask them 2015-03-19T18:39:23 fungi: oh ok 2015-03-19T18:39:42 *** ominakov has joined #openstack-infra 2015-03-19T18:40:18 which i assume is not too far away given the testing going on with f21 currently 2015-03-19T18:41:14 thingee: see also http://lists.openstack.org/pipermail/openstack-dev/2015-February/057588.html 2015-03-19T18:41:39 clarkb: I believe it worked 2015-03-19T18:41:41 2015-03-19 18:40:28,630 INFO nodepool.DiskImageUpdater: Image devstack-trusty-1426790224.template.openstack.org in hpcloud-b5 is ready 2015-03-19T18:41:43 thingee: jeblair has a note on the simplest and quickest way to move forward 2015-03-19T18:41:47 2015-03-19 18:37:04,903 INFO nodepool.DiskImageUpdater: Uploading dib image id: 11617 from /opt/nodepool_dib/devstack-trusty-1426788873.qcow2 for devstack-trusty-1426790224.template.openstack.org in hpcloud-b5 2015-03-19T18:41:54 thingee: that is what I think has not happened 2015-03-19T18:41:57 *** hashar has joined #openstack-infra 2015-03-19T18:42:08 mordred: yay now we wait for one of those to boot successfully 2015-03-19T18:42:16 *** hdd has quit IRC 2015-03-19T18:42:16 so - hpcloud-b5 is region to watch 2015-03-19T18:42:27 mordred: and if so you want to do `sudo -H -u nodepool nodepool image-upload hpcloud-bX devstack-trusty` 2015-03-19T18:42:38 *** dimtruck is now known as zz_dimtruck 2015-03-19T18:42:46 thingee: i don't see a pending project config change from someone named Philip 2015-03-19T18:43:29 yup 2015-03-19T18:43:32 clarkb: thanks! 2015-03-19T18:44:25 mordred, clarkb, fungi: fyi up to 625 instances of eoferror, so it's still been happening (so hopefully the new cloudinitless image will still help current situation) 2015-03-19T18:44:37 *** yamamoto has joined #openstack-infra 2015-03-19T18:44:43 fingers crossed 2015-03-19T18:44:44 jeblair: it's good that it's still happening, so that we can see if the patch helped 2015-03-19T18:45:33 mordred: I have cleaned up those -new images that are not the one you just built 2015-03-19T18:45:52 mordred: that gives us a total of 22GB free disk after I also removed the older precise-dib, trusty-dib and centos7-dib images 2015-03-19T18:45:57 mordred: but basically we need more disk 2015-03-19T18:46:01 mordred: or fewer images 2015-03-19T18:46:21 * fungi is trying to make traction on the fewer images idea 2015-03-19T18:46:21 clarkb: fewer images 2015-03-19T18:46:22 *** claudiub has quit IRC 2015-03-19T18:46:27 or smaller images 2015-03-19T18:46:31 or all of the above :) 2015-03-19T18:47:09 speaking of fewer images, nova py27 job on devstack-trusty node using my experimental job which prepped the db and installed the distro packages needed http://logs.openstack.org/44/164444/1/experimental/experimental-nova-dsvm-python27/f4d9841/console.html 2015-03-19T18:47:25 log confirms migration tests ran 2015-03-19T18:47:47 nice 2015-03-19T18:48:14 unfortunately since we use snapshots for all bare-* nodes ths doesn't quite help us :) 2015-03-19T18:48:16 some eyeballs on that for obvious issues would be appreciated once things cool down 2015-03-19T18:48:25 at least not with the ENODISK problem 2015-03-19T18:48:35 *** tkelsey has quit IRC 2015-03-19T18:48:51 *** yfried has joined #openstack-infra 2015-03-19T18:49:21 *** yamamoto has quit IRC 2015-03-19T18:49:21 well, if we switched the tox builder to always run the database prep builders, we could drop mysql and postgres setup from our image build scripts and switch bare-.* to dib sooner 2015-03-19T18:49:22 the back button doesn't work on jenkins' jira 2015-03-19T18:50:02 fungi: yup 2015-03-19T18:50:10 *** SotK has quit IRC 2015-03-19T18:50:51 which is still a reasonable stepping-stone to where i want to get with bindep used for defining the additional packages to cache on the devstack images and install at the start of jobs 2015-03-19T18:50:56 "If this is a CloudBees customer we have two hotfixes that seem to work around the deadlock with the side-effect of degrading UI performance" 2015-03-19T18:51:09 clarkb: open core! 2015-03-19T18:51:11 * greghaynes is curious if anyone is up for explaining how bindep gets us a single image vs many (that is the goal, right?) 2015-03-19T18:51:13 I am not going to login to respond to that but figured I should point out that this goes on on their tracker 2015-03-19T18:51:30 greghaynes: single image per distro because every job can describe its system deps 2015-03-19T18:51:50 oh! so the bindep will be at runtime 2015-03-19T18:52:01 yes 2015-03-19T18:52:06 and then we can remove those deps out from buildtime 2015-03-19T18:52:08 make sense 2015-03-19T18:52:11 *** spzala has joined #openstack-infra 2015-03-19T18:52:16 greghaynes: it _eventually_ enables us to start putting lists of required distro packages in individual repos, allowing us to self-test whether they're sufficient too 2015-03-19T18:52:20 with lots of caching in the image build still 2015-03-19T18:52:40 yea, that makes a ton of sense all of a sudden ;) 2015-03-19T18:52:44 right, we'd iterate over all branches of all repos in gerrit and pre-cache the union of all their bindep files 2015-03-19T18:52:50 *** radez_g0n3 is now known as radez 2015-03-19T18:53:13 conveniently, we already clone those repos onto the images anyway 2015-03-19T18:53:24 Is there any next to dos on that? 2015-03-19T18:53:48 greghaynes: you can start by reviewing my changes to bindep and testing it out if you haven't already 2015-03-19T18:54:01 fungi, isn't there a backup-as-a-service for the machines managed by infra? (re: the question on the community list about askbot) 2015-03-19T18:54:02 sounds good 2015-03-19T18:54:34 *** yolanda has quit IRC 2015-03-19T18:54:41 reed: there is, but what evgeny's talking about is dumping copies of critical files and database contents as a snapshot onto the local filesystem 2015-03-19T18:55:05 reed: then we can back that up to a remote server like we do for other critical systems, but the local dump is convenient for other reasons 2015-03-19T18:56:02 I am not finding any bugs on the jenkins tracker that matcht the behavior we see 2015-03-19T18:56:08 greghaynes: i also have a bindep list mostly assembled that effectively turns our current devstack-.* workers into bare-.* workers https://review.openstack.org/164447 though that also needs some rigorous testing 2015-03-19T18:56:33 I will try to file one, pretty sure I have an account I can use 2015-03-19T18:56:39 Adam Gandelman proposed openstack/requirements: Relax pysendfile requirement https://review.openstack.org/165969 2015-03-19T18:56:45 greghaynes: if nothing else, it's a good example of a complex-ish list 2015-03-19T18:57:14 awesome, ill try it out 2015-03-19T18:57:44 fungi, ok 2015-03-19T18:58:01 *** SotK has joined #openstack-infra 2015-03-19T18:58:07 hrm doesn't look like the heap stack error dumps that I collected the first time around have stack traces 2015-03-19T18:58:08 fungi, I think Marton overlooked those 2015-03-19T18:58:20 well I will file what I have and give it a go 2015-03-19T18:58:47 reed: they were non-obvious, but thankfully also something which can be implemented in parallel with testing the rest since they're non-service-impacting to add 2015-03-19T18:59:16 clarkb: we still don't have a GREAT way of answering "what image did this instance boot from" do we? 2015-03-19T18:59:41 mordred: we do not 2015-03-19T19:00:00 mordred: what you can do is nova show the instance 2015-03-19T19:00:08 mordred: then take the image id from there and cross check against nodepoo 2015-03-19T19:00:27 nod 2015-03-19T19:00:29 mordred: so its doable, but you have to query nova not just nodepool 2015-03-19T19:00:50 clarkb, mordred: i did an image-list, looked at the age of the most recent image, then found instances in build whose ages were less than that 2015-03-19T19:00:59 clarkb, mordred: that got me, among a few others, 1370425 2015-03-19T19:01:05 which i'm watching 2015-03-19T19:01:10 and it's in error state. 2015-03-19T19:01:12 mordred: or if you can ssh into it, you can see the image logged in /etc/nodepool but if the instance is erroring then not so much 2015-03-19T19:01:15 | fault | {"message": "No valid host was found. ", "code": 500, "created": "2015-03-19T18:56:46Z"} | 2015-03-19T19:01:30 *** timcline has quit IRC 2015-03-19T19:01:48 next is 1370426 2015-03-19T19:02:02 | image | devstack-trusty-1426790224.template.openstack.org (026f353e-0365-4769-a5a8-910e57d454a8) | 2015-03-19T19:02:09 mordred: is that right ^ ? 2015-03-19T19:02:27 yes. that's the one we want 2015-03-19T19:02:47 no floatig ip yet 2015-03-19T19:03:53 b5 has a task queue 85 deep, could be a few mins 2015-03-19T19:06:26 David Shrewsbury proposed openstack-infra/shade: Add design and skeleton for an object interface https://review.openstack.org/164506 2015-03-19T19:07:17 hrm. we should invert what the provider manager does -- so that the calls happen in the worker threads, but the provider manager serializes and paces them. 2015-03-19T19:07:44 that way when a deletion call takes 30 seconds, everything isn't waiting on it, but you still don't issue more than the target rate... 2015-03-19T19:07:46 *** pelix has left #openstack-infra 2015-03-19T19:08:03 *** Hal1 has joined #openstack-infra 2015-03-19T19:08:09 i wonder if openstack calculates rate based on when the request is received or completed? 2015-03-19T19:08:45 *** claudiub has joined #openstack-infra 2015-03-19T19:08:45 *** Hal1 has quit IRC 2015-03-19T19:08:55 i wonder where that rate limiting is implemented 2015-03-19T19:08:58 *** sushilkm has joined #openstack-infra 2015-03-19T19:08:58 nova? keystone? 2015-03-19T19:09:18 *** Hal1 has joined #openstack-infra 2015-03-19T19:09:35 *** marun has quit IRC 2015-03-19T19:10:01 is there a way to rerun the tests in between ... because out of multiple jenkins job the smalles job has failed .... and bigger jobs would continue running and come back later saying for a recheck 2015-03-19T19:10:30 sushilkm: why have the jobs failed? 2015-03-19T19:10:39 sushilkm: there may be a problem with the patch 2015-03-19T19:10:53 in which case the jobs are working as expected 2015-03-19T19:11:10 no problem is not with patch while verifying it worked, now there are same tests running for merge 2015-03-19T19:11:20 and smallest test has failed 2015-03-19T19:11:27 its a known bug 2015-03-19T19:11:27 which patch? 2015-03-19T19:11:33 which comes up sometimes 2015-03-19T19:11:41 its regarding a patch in trove 2015-03-19T19:11:54 well if your patch is triggering a known bug your patch may be making the bug worse 2015-03-19T19:12:13 https://jenkins02.openstack.org/job/gate-trove-python27/2302/console 2015-03-19T19:12:24 *** Hal2 has joined #openstack-infra 2015-03-19T19:13:22 *** Hal1 has quit IRC 2015-03-19T19:13:43 *** zz_dimtruck is now known as dimtruck 2015-03-19T19:14:26 yeah patchset 2 failed 5 times, patchset 5 failed 3 times 2015-03-19T19:14:49 I think you need to consider the possiblity your patch may be increasing the tendency of a bug to appear 2015-03-19T19:15:12 https://issues.jenkins-ci.org/browse/JENKINS-27514 2015-03-19T19:15:21 clarkb: now you're famous 2015-03-19T19:16:03 ok anteaya .. i wud have a look 2015-03-19T19:16:09 thanks for the suggestion 2015-03-19T19:16:10 fungi: filing a bug is all it took to be famous? I should've done this ealier (I really thought that change I linked in the bug was the fix :( ) 2015-03-19T19:17:15 you're in the who's who of jenkins jira bugs 2015-03-19T19:17:37 hehe 2015-03-19T19:18:13 maybe cloudbees will offer you an open-core hotfix 2015-03-19T19:18:24 ha ha ha 2015-03-19T19:19:36 *** marun has joined #openstack-infra 2015-03-19T19:20:20 *** otter768 has joined #openstack-infra 2015-03-19T19:20:34 Khai Do proposed openstack-infra/gerritlib: Update project docs and setup for tests https://review.openstack.org/165979 2015-03-19T19:20:51 Matthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server https://review.openstack.org/149719 2015-03-19T19:21:51 *** HeOS has joined #openstack-infra 2015-03-19T19:22:48 Merged openstack-infra/project-config: Start using neutron for periodic stress job https://review.openstack.org/164705 2015-03-19T19:24:08 mordred: any idea if your new image is working yet? 2015-03-19T19:24:23 *** bdpayne has quit IRC 2015-03-19T19:24:43 *** garyh has quit IRC 2015-03-19T19:25:18 *** otter768 has quit IRC 2015-03-19T19:25:25 James E. Blair proposed openstack-infra/nodepool: Run tasks in worker threads https://review.openstack.org/165982 2015-03-19T19:25:43 trying the new warfile in the puppet module 2015-03-19T19:25:44 clarkb, mordred: the test node i looked at is still in the nodepool queue 2015-03-19T19:25:49 SlickNik: there are a couple changes in the gate for trove consistently failing python 2.7 unit tests. known issue? 2015-03-19T19:25:55 clarkb, mordred, fungi: which prompted me to write https://review.openstack.org/165982 2015-03-19T19:26:09 *** xyang1 has joined #openstack-infra 2015-03-19T19:26:09 cinerama: fingers crossed :) 2015-03-19T19:26:24 "we've secretly replaced the coffee here with folger's crystals. let's watch" 2015-03-19T19:26:25 clarkb, mordred, fungi: which is either going to be a _massive_ improvement to our throughput and efficiency, radically reducing the amount of servers we have sitting in build and delete 2015-03-19T19:26:34 *** hdd has joined #openstack-infra 2015-03-19T19:26:37 jeblair: looking at it now 2015-03-19T19:26:38 clarkb, mordred, fungi: or it could make everything blow up all at once 2015-03-19T19:26:46 jeblair: thankfully we have some tests now 2015-03-19T19:26:50 clarkb, mordred, fungi: frankly, a little hard to tell from where i'm sitting 2015-03-19T19:26:52 fungi: funny you should mention that — that's what I'm looking at right now. 2015-03-19T19:26:57 clarkb: actually it passes tests :) 2015-03-19T19:27:06 oh pants, i forgot to update the template 2015-03-19T19:27:08 SlickNik: okay, cool. just making sure it was on your radar 2015-03-19T19:27:08 jeblair: thats a start then :) 2015-03-19T19:27:20 *** asettle has joined #openstack-infra 2015-03-19T19:27:32 cinerama: I haven't done a diff on the new config file, but I expect it to be unpleasant 2015-03-19T19:27:32 clarkb: i think the failure mode is likely to be that we find that "start of api request" is not how rate limits are calculated 2015-03-19T19:28:01 clarkb: i think the next most probably failure is "nodepool runs closer to specified api rate limit which is wrong and needs to be adjusted to slow it down a bit" 2015-03-19T19:28:06 that's pretty tractable 2015-03-19T19:28:17 jeblair: ya we should be able to handle that easily 2015-03-19T19:28:20 *** bhunter71 has quit IRC 2015-03-19T19:28:28 there's a knob for that 2015-03-19T19:28:35 *** asettle has quit IRC 2015-03-19T19:28:44 i'm cool with that risk 2015-03-19T19:29:18 i need to grab food; if people like that change, i can hand-apply it on nodepool and restart when i get back 2015-03-19T19:29:40 *** bhunter71 has joined #openstack-infra 2015-03-19T19:29:54 fix the world or blow it up, what's not to like 2015-03-19T19:30:14 jeblair: can you explain the changes to error handling? 2015-03-19T19:30:16 the world's partway to blown up already, so not sure anyone will notice 2015-03-19T19:30:34 pleia2: it looks like we only need the infinispan bit (though the version is different). the other bit in the release notes refers to a one-off thing needed for migration from an older version so if we start on the new version we should be good 2015-03-19T19:30:37 clarkb: old error handling was to capture the exception in provider manager thread, and transfer it to calling thread 2015-03-19T19:30:49 now for some spaghetti testing = throw it against the wall & see if it sticks 2015-03-19T19:30:53 jeblair: I see 2015-03-19T19:30:55 clarkb: (i thought that was a neat trick ;) 2015-03-19T19:31:13 jeblair: but now with run happening in the calling thread it can bubble straight up 2015-03-19T19:31:17 clarkb: new change runs actual work in calling thread, so is simpler -- task.main() will just raise the exception in the right context 2015-03-19T19:31:18 yep 2015-03-19T19:31:53 *** ddieterly has quit IRC 2015-03-19T19:31:53 *** ddieterl_ has joined #openstack-infra 2015-03-19T19:32:04 fungi: good point 2015-03-19T19:32:18 clarkb: it could before I think 2015-03-19T19:32:36 *** pc_m has quit IRC 2015-03-19T19:32:48 greghaynes: ya they bubbled up before via the transfer thing jeblair described 2015-03-19T19:32:56 but thats no longer necessary so we can rip it out 2015-03-19T19:33:32 ah, sorry, I thought you were pointing that out as an issue not a statement of fact ;) 2015-03-19T19:33:48 greghaynes: ah, mostly just wrapping my head around the change 2015-03-19T19:33:56 jeblair: it lgtm I say go for it when you are done with lunch 2015-03-19T19:34:23 I wonder if it passes my monkeypatch tests 2015-03-19T19:34:32 I had to poke around a lot in providermanager for that 2015-03-19T19:36:18 clarkb: jeblair: looks good to me too. wanting to hold off approving it until it's proven in production? 2015-03-19T19:36:54 test in prod! 2015-03-19T19:37:26 *** tjones1 has quit IRC 2015-03-19T19:37:29 greghaynes: well, nodepool _is_ already tested 2015-03-19T19:39:21 *** SumitNaiksatam has quit IRC 2015-03-19T19:39:31 it is afternoon already 2015-03-19T19:40:04 yep, it's about time to rename the board ml 2015-03-19T19:40:18 clarkb: sorry, haven't been paying attention. scp plugin is still leaking memory? 2015-03-19T19:40:38 zaro: no, the ssh-slaves plugin is leaking threads see https://issues.jenkins-ci.org/browse/JENKINS-27514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel 2015-03-19T19:40:49 zaro: this is the same issue from a few weeks ago, updating the ssh-slaves plugin does not fix it 2015-03-19T19:41:14 Lars Kellogg-Stedman proposed openstack-infra/project-config: Add "heat-coe-templates" project https://review.openstack.org/164806 2015-03-19T19:41:51 *** sushilkm has left #openstack-infra 2015-03-19T19:41:58 *** garyh has joined #openstack-infra 2015-03-19T19:42:58 *** _nadya_ has joined #openstack-infra 2015-03-19T19:43:24 *** _nadya_ has quit IRC 2015-03-19T19:43:39 SlickNik: looks like trove is still using probuscis 2015-03-19T19:45:00 SlickNik: looks like something database related though? 2015-03-19T19:45:08 clarkb: yes for its legacy fake tests. We've moved all of the new unit tests to run under testr. 2015-03-19T19:45:28 It's one of the new testr unit tests that is causing the issue. 2015-03-19T19:45:42 SlickNik: its failing in the old tests too 2015-03-19T19:46:06 Specifically: FAIL - mgmt.test_models.TestMgmtInstanceDeleted.test_show_deleted_mgmt_instances 2015-03-19T19:46:52 oh wait those tracebacks from run_tests are normal? 2015-03-19T19:46:58 I should just stop looking 2015-03-19T19:47:43 *** dprince has quit IRC 2015-03-19T19:47:45 clarkb: Yeah it sucks that the old tests puke stacktraces for the negative tests that are supposed to result in exceptions. There's a bug to fix that in the works. 2015-03-19T19:48:04 *** sdake_ has joined #openstack-infra 2015-03-19T19:48:10 SlickNik: you should be able to just wrap in expected exception context managers or asserts 2015-03-19T19:50:03 fungi: clarkb: When are the infra meetings held? 2015-03-19T19:50:16 tuesdays 1900 2015-03-19T19:50:23 thx 2015-03-19T19:50:23 in #openstack-meeting 2015-03-19T19:50:31 clarkb: so is it happening only on some servers and not others? 2015-03-19T19:50:34 thank you for asking, be nice to see you there 2015-03-19T19:50:40 clarkb: yes we do. There was a change to the way that we patch eventlet for some of the fake tests that caused this to happen. I haven't dug deep enough into it yet to figure out why that was the case. 2015-03-19T19:50:46 anteaya: do you have a posted agenda? 2015-03-19T19:50:50 yes 2015-03-19T19:50:53 * anteaya fetches 2015-03-19T19:50:58 zaro: so far I think it has hit 01, 07, 02, 05, 04 2015-03-19T19:51:15 can I get a chance to get a +A on https://review.openstack.org/#/c/164451/ ? - it's about "zuul: add check-grenade-dsvm-neutron to tempest checks" 2015-03-19T19:51:19 Shrews: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 2015-03-19T19:51:37 anteaya: thanks again 2015-03-19T19:51:40 *** sdake has quit IRC 2015-03-19T19:51:45 welcome 2015-03-19T19:52:15 EmilienM: not from me today, anything that isn't required for ff gets put off to tomorrow for me 2015-03-19T19:52:31 EmilienM: we are having difficulty getting resources for patches in check 2015-03-19T19:52:34 clarkb: ooh, 5 out of 9? i thought it might be something different between the machines, like the underlying ssh library? 2015-03-19T19:52:38 don't want to add to that 2015-03-19T19:52:39 Shrews: the agenda is usually in a mild state of limbo between things which were discussed in the previous meeting which need to be removed and things which have been added to discuss in the next meeting, but add whatever you want to talk about 2015-03-19T19:52:47 zaro: 5 out of 7 2015-03-19T19:52:48 anteaya: ack 2015-03-19T19:53:17 EmilienM: thanks 2015-03-19T19:53:23 *** sdake has joined #openstack-infra 2015-03-19T19:53:57 *** [HeOS] has joined #openstack-infra 2015-03-19T19:55:29 *** sdake_ has quit IRC 2015-03-19T19:56:08 *** kgiusti has left #openstack-infra 2015-03-19T19:56:49 Hi infra folks! Looks like this patch (https://review.openstack.org/155470) has been stuck in the gate for 20 hours or so, any ideas what may be happening? /cc otherwiseguy 2015-03-19T19:56:52 *** HeOS has quit IRC 2015-03-19T19:57:05 clarkb: gearman only offlines the node correct? not actually disconnects them. 2015-03-19T19:57:10 the check-tempest-neutron-dsvm-dvr job appears to be just hung there 2015-03-19T19:57:15 zaro: correct, then nodepool deletes them 2015-03-19T19:57:30 mestery: that job was running on a jenkins master that fell over, you can push a new patchset to unstick it 2015-03-19T19:57:32 armax: The stuck patch is yours, FYI 2015-03-19T19:57:41 clarkb: Ack, armax can you do that? 2015-03-19T19:57:45 clarkb: Thanks! 2015-03-19T19:58:13 a small number of changes were affected so rather than restart the world we decided that patch authors that are affected could address their specific changes 2015-03-19T19:58:24 clarkb: Makes sense to me 2015-03-19T20:00:02 *** eharney has quit IRC 2015-03-19T20:00:48 mestery armax changing the commit message should be enough for zuul to consider that an event 2015-03-19T20:01:42 *** teran has joined #openstack-infra 2015-03-19T20:02:11 thanks anteaya 2015-03-19T20:02:22 *** timcline has joined #openstack-infra 2015-03-19T20:03:10 mestery: you're welcome 2015-03-19T20:03:46 mordred: http://paste.openstack.org/show/193676/ still shows up in the console log so I don't think removing that package completely cleans out cloud init 2015-03-19T20:04:02 mordred: but it doesn't appear to look for metadata so at least that bit seems to work 2015-03-19T20:04:23 clarkb: can it find the keys? 2015-03-19T20:04:43 *** mmedvede has joined #openstack-infra 2015-03-19T20:04:46 anteaya: nodepool hasn't tried to do that yet 2015-03-19T20:04:47 *** kgiusti has joined #openstack-infra 2015-03-19T20:04:49 ah 2015-03-19T20:04:54 which is what jeblair's change from before should address 2015-03-19T20:05:58 clarkb: does it have an ip address yet? 2015-03-19T20:06:08 *** amuller has joined #openstack-infra 2015-03-19T20:06:19 ah having the requests performed in the calling thread 2015-03-19T20:06:19 mordred: only on the 10 net 2015-03-19T20:06:24 *** sambetts has quit IRC 2015-03-19T20:06:24 sigh 2015-03-19T20:06:38 *** timcline has quit IRC 2015-03-19T20:06:42 clarkb: I really want to know why it's trying to do something with cloud-init 2015-03-19T20:06:49 mordred: I think thats an init script 2015-03-19T20:06:54 *** sambetts has joined #openstack-infra 2015-03-19T20:07:02 mordred: its lilely that init script isn't removed by the ensure => absent 2015-03-19T20:07:08 clarkb: looks like there's a known issue with memory leak if too many slaves. https://issues.jenkins-ci.org/browse/JENKINS-20620 2015-03-19T20:07:25 *** shardy has quit IRC 2015-03-19T20:07:29 *** AJaeger_ has quit IRC 2015-03-19T20:07:33 how many slaves do we have setup for max? 2015-03-19T20:07:51 clarkb: probably should have done => purged - does puppet support that? 2015-03-19T20:07:54 zaro: its about 200 per max 2015-03-19T20:07:58 mordred: I do not know 2015-03-19T20:08:06 zaro: but its not really a memory leak. The threads leak 2015-03-19T20:08:14 zaro: the threads are relatively light on memory 2015-03-19T20:08:25 zaro: but we only have like 100MB for threds to live in 2015-03-19T20:08:29 clarkb: yp - that's in an init script 2015-03-19T20:08:31 so once we hit that we lose 2015-03-19T20:09:20 *** e0ne has joined #openstack-infra 2015-03-19T20:10:01 *** amuller has left #openstack-infra 2015-03-19T20:10:30 *** dprince has joined #openstack-infra 2015-03-19T20:10:38 mordred: 8b8f8f1b-e1bf-48f4-aae6-3b0e91d13978 is the node I looked at fwiw 2015-03-19T20:11:03 mordred: yes you can use purged 2015-03-19T20:11:08 mordred: https://docs.puppetlabs.com/references/latest/type.html#package-attribute-ensure 2015-03-19T20:11:18 Monty Taylor proposed openstack-infra/system-config: Purge cloud-init https://review.openstack.org/165996 2015-03-19T20:11:21 clarkb: I agree ^^ :) 2015-03-19T20:12:21 *** kgiusti has left #openstack-infra 2015-03-19T20:13:06 console log shows it started ssh too but without mention of generating new host keys 2015-03-19T20:13:35 lunch now, back in a bit 2015-03-19T20:13:53 enjoy food 2015-03-19T20:14:20 eek lunch, it's late! 2015-03-19T20:14:29 *** e0ne has quit IRC 2015-03-19T20:14:58 happy food to you too 2015-03-19T20:15:18 thanks :) 2015-03-19T20:15:34 mordred: also http://admintweets.com/regenerate-ssh-host-keys-on-boot-on-ubuntu/ (looking now to confirm) 2015-03-19T20:16:11 fungi: sigh 2015-03-19T20:16:32 fungi: so I guess I probably want to do that then 2015-03-19T20:16:59 it can't hurt, but it's got me curious because i thought most platforms took care of that 2015-03-19T20:17:44 *** mrda-away is now known as mrda 2015-03-19T20:18:28 i'm testing with a held bare-trusty worker now 2015-03-19T20:19:29 *** Hal2 has quit IRC 2015-03-19T20:19:37 *** Hal1 has joined #openstack-infra 2015-03-19T20:20:15 *** emagana has quit IRC 2015-03-19T20:20:15 mordred: yeah, at least for the bare-trusty worker i tested with in rax, it comes back up with sshd running but erroring on connect 2015-03-19T20:20:23 fungi: k. I've got a patch coming 2015-03-19T20:20:50 SpamapS: theory disproven :/ 2015-03-19T20:21:59 Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 2015-03-19T20:22:24 fungi: ^^ we can do that better - with a proper upstart script that goes after openssh server rather than in rc.local 2015-03-19T20:22:31 fungi: but we have an rc.local we're installing anyway 2015-03-19T20:22:39 mordred: nice round number on that patch 2015-03-19T20:22:46 woohoo! 2015-03-19T20:23:43 mordred: extra line in https://review.openstack.org/#/c/166000/1/nodepool/scripts/prepare_node.sh 2015-03-19T20:23:43 * greghaynes points at his patch 100000 2015-03-19T20:23:54 greghaynes: I was just thinking of that 2015-03-19T20:23:59 :) 2015-03-19T20:24:03 I remember the race for that number 2015-03-19T20:24:17 I had forgotten it was you that won the lottery 2015-03-19T20:24:24 Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 2015-03-19T20:24:24 Also, seems like 99-unbound is becoming the 'do everything at the end' script rather than 99-unbound 2015-03-19T20:24:29 yeah 2015-03-19T20:24:36 that whole stack needs some serious refactoring 2015-03-19T20:24:41 *** _nadya_ has joined #openstack-infra 2015-03-19T20:24:57 *** _nadya_ has quit IRC 2015-03-19T20:24:59 mordred: testing that rc.local patch with another held trusty machine 2015-03-19T20:25:05 fungi: woo 2015-03-19T20:25:23 should know in a second when it finishes booting 2015-03-19T20:25:31 *** emagana has joined #openstack-infra 2015-03-19T20:25:55 no dice. i'll try to step through what's wrong there 2015-03-19T20:26:27 *** Swami has quit IRC 2015-03-19T20:27:05 *** salv-orlando has quit IRC 2015-03-19T20:27:26 *** tkelsey has joined #openstack-infra 2015-03-19T20:27:31 *** tjones1 has joined #openstack-infra 2015-03-19T20:27:35 mordred: i think your conditional is wrong 2015-03-19T20:27:45 clarkb: have you read this? https://wiki.jenkins-ci.org/display/JENKINS/Remoting+issue 2015-03-19T20:27:59 `type dpkg-reconfigure >/dev/null 2>&1 ! test -f /etc/ssh/ssh_host_rsa_keg` (checking for a nonexistent file) returns 1 for me there 2015-03-19T20:28:03 fungi, mordred: just to confirm -- you found a node with mordred's change, but it didn't work? 2015-03-19T20:28:38 jeblair: no, i manually tried adding mordred's rc.local change to regenerate host keys to a held trusty node and then deleted its host keys and rebooted it 2015-03-19T20:28:54 "functional test" 2015-03-19T20:29:07 *** pritesh has joined #openstack-infra 2015-03-19T20:29:09 fungi: oh ok. so you're pretty sure it wouldn't work. the node i was looking at failed the ssh test 2015-03-19T20:29:12 2015-03-19 19:40:47,007 ERROR nodepool.NodeLauncher: EOFError launching node id: 1370426 in provider: hpcloud-b5 error: 2015-03-19T20:29:22 fungi: which i think may coroborrate your experiment 2015-03-19T20:29:36 yah 2015-03-19T20:29:52 okay, so while mordred iterates, i'm going to apply nodepool patch and restart 2015-03-19T20:29:53 wow 2015-03-19T20:29:59 mordred: what is the type dpkg-reconfigure trying to do? just see if dpkg-reconfigure exists? 2015-03-19T20:30:03 jeblair: fungi did I raelly do keg? 2015-03-19T20:30:09 oh noes! my patch failed pep8! 2015-03-19T20:30:11 mordred: why not just go for it and || true, if it fails then it doesnt exist ;) 2015-03-19T20:30:16 zaro: I had not but the problem isn't connection issues 2015-03-19T20:30:18 jeblair: no 2015-03-19T20:30:23 greghaynes: could do that too 2015-03-19T20:30:25 *** marcusvrn1 has quit IRC 2015-03-19T20:30:29 zaro: the problem is that jenkins leaks the threads when trying to disconnect nodes 2015-03-19T20:30:37 mordred: no, that was an intentional typo on my part to simulate a missing host key 2015-03-19T20:30:38 (I think the type line is wrong and thats easier than my thinking of how to fix it) 2015-03-19T20:30:42 zaro: but the nodes go away and all that 2015-03-19T20:31:06 mordred: as in test that conditional yourself, i think it doesn't do what you think it does 2015-03-19T20:31:29 *** tkelsey has quit IRC 2015-03-19T20:31:33 fungi: I did - but I left out the && 2015-03-19T20:31:36 copypasta problem 2015-03-19T20:31:56 aha, yep. thought that might be what you were going for 2015-03-19T20:31:57 ah, that would explain my confusion for what youre trying to do ;) 2015-03-19T20:32:03 *** wenlock has quit IRC 2015-03-19T20:32:06 James E. Blair proposed openstack-infra/nodepool: Run tasks in worker threads https://review.openstack.org/165982 2015-03-19T20:32:19 mordred: inserting && it definitely does what i think we want 2015-03-19T20:32:25 mordred: your shade refactor proposal makes me yearn for python class templates 2015-03-19T20:32:31 Shrews: ++ 2015-03-19T20:32:47 Shrews: ABC kinda does that 2015-03-19T20:32:49 i'll give that another round of functional testing 2015-03-19T20:32:59 clarkb: go away because we don't see it the reference in the gerrit ui? 2015-03-19T20:33:13 zaro: ya 2015-03-19T20:33:25 zaro: we will only have a couple hundred nodes but maybe 8k threads 2015-03-19T20:33:27 Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 2015-03-19T20:33:30 fungi: this time with && 2015-03-19T20:33:30 clarkb, fungi, mordred: i'm going to restart nodepool with my change now 2015-03-19T20:33:38 jeblair: awesome 2015-03-19T20:33:41 clarkb: yeah - I used abc ... but it's still not exactly what we want 2015-03-19T20:33:48 clarkb: because C++ class tempaltes are AWESOME 2015-03-19T20:33:58 wondering if maybe the thread for the connection is still around, just not reported on jenkins ui? 2015-03-19T20:34:08 mordred: you don't need them in python... 2015-03-19T20:34:13 *** dims has quit IRC 2015-03-19T20:34:35 clarkb: yeah you do - because you also need typing in python - because it's the biggest problem with python programming 2015-03-19T20:34:40 it is doing things very very quickly 2015-03-19T20:34:47 jeblair: woot! 2015-03-19T20:34:47 jeblair: yay 2015-03-19T20:34:53 mordred: one other question. why the duplicate check for test -f /etc/ssh/ssh_host_rsa_key 2015-03-19T20:34:54 jeblair: is it doing good things very very quickly? 2015-03-19T20:35:09 fungi: because my fingers move too fast 2015-03-19T20:35:10 mordred: 3.5 is adding that or is it 3.6? but I think many pythonistas are cranky about it 2015-03-19T20:35:17 mordred: it's getting 500 unknown errors from hpcloud faster than ever before 2015-03-19T20:35:29 mordred: i'm going to test without the redundant inner one 2015-03-19T20:35:38 Has there been a Zuul change that would effect Fedora 20 TripleO nodes today? 2015-03-19T20:35:44 Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 2015-03-19T20:35:51 Timeout: HTTPSConnectionPool(host='region-b.geo-1.compute.hpcloudsvc.com', port=443): Read timed out. 2015-03-19T20:35:53 I've not seen Fedora 20 CI jobs running for several hours... 2015-03-19T20:35:54 mordred: it isn't blowing up, the choices were doing things quickly or blowing up 2015-03-19T20:35:55 that is new 2015-03-19T20:36:07 jeblair: ooh. that is new 2015-03-19T20:36:15 does infra host a build copy of scp.hpi anywhere? 2015-03-19T20:36:23 asselin: on tarballs.openstack.org 2015-03-19T20:37:55 dprince: well, we've been having one of those days 2015-03-19T20:38:21 mordred: f20 is different 2015-03-19T20:38:30 mordred: sorry to hear that. I figured I'd ask because I'm perplexed by this. 2015-03-19T20:38:40 f20 broke because rax did something weird in the base image or something 2015-03-19T20:38:57 fwiw I am not sure that install_puppet.sh is where we should put the workaround for these rpobems 2015-03-19T20:39:04 should go in prepare_node.sh 2015-03-19T20:39:12 Would that effect tripleo f20 though? 2015-03-19T20:39:12 *** ssam2 has quit IRC 2015-03-19T20:39:16 which workaroudn is in install_puppet? 2015-03-19T20:39:18 clarkb: ?? 2015-03-19T20:39:29 mordred: the f20 make grub work workaround 2015-03-19T20:39:33 ah. gotcha 2015-03-19T20:39:35 yeah 2015-03-19T20:39:37 I agree 2015-03-19T20:39:42 greghaynes: oh probably not 2015-03-19T20:39:49 *** dkranz has quit IRC 2015-03-19T20:39:50 Yea, that would be wierd if it did 2015-03-19T20:39:53 greghaynes: unless f20 is broken across the board and tripleo has same bug as rax? 2015-03-19T20:39:53 clarkb: is it possible taht the f20 make grub work workaround broke the tripleo cloud? 2015-03-19T20:39:57 FWIW, I see Fedora 20 nodes getting created in our cloud. But nothing is running actualy CI jobs for a few hours now 2015-03-19T20:39:58 mordred: maybe 2015-03-19T20:40:00 clarkb, fungi, mordred: i've lowered the rate limit to see if that affects the read time outs 2015-03-19T20:40:08 jeblair: cool 2015-03-19T20:40:23 jeblair: that's a good sign though, i think 2015-03-19T20:40:24 I wonder if those nodes got lost when 04 fell over 2015-03-19T20:40:28 I can look at nodepool to check 2015-03-19T20:40:52 dprince: all of the nodes are either building or deleting 2015-03-19T20:41:09 let me try manually deleting the deleting nodes 2015-03-19T20:41:12 fungi: any luck with that latest patch? 2015-03-19T20:41:26 mordred: looks like it worked 2015-03-19T20:41:31 yay 2015-03-19T20:41:47 clarkb: okay, I saw some ACTIVE instances from a Nova prospective 2015-03-19T20:41:48 mordred: yep, new host keys on boot now 2015-03-19T20:42:01 *** yfried is now known as yfried|afk 2015-03-19T20:42:03 *** timcline has joined #openstack-infra 2015-03-19T20:42:03 fungi: woot. then I'm going to push in those two patches and try rebuilding b5 again 2015-03-19T20:42:06 clarkb: perhaps they are stuck in one of the prepare scripts? 2015-03-19T20:42:07 *** timcline has quit IRC 2015-03-19T20:42:15 mordred: doing one last boot without deleting the keys just to make sure it's also not regenerating them when they already exist 2015-03-19T20:42:45 dprince: maybe, hopefully this foreground delete will shed some light on what may be happening 2015-03-19T20:42:50 fungi: excellent point 2015-03-19T20:42:53 *** timcline has joined #openstack-infra 2015-03-19T20:42:54 mordred: also fine 2015-03-19T20:43:01 Merged openstack-infra/system-config: Purge cloud-init https://review.openstack.org/165996 2015-03-19T20:43:02 mordred: good to go, at least for trusty 2015-03-19T20:43:17 *** radez is now known as radez_g0n3 2015-03-19T20:43:25 Merged openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 2015-03-19T20:44:01 sudo rm -f /etc/ssh/ssh_host_* ? 2015-03-19T20:44:10 Are we sure ^^^ that doesn't hose everything? 2015-03-19T20:44:27 * dprince isn't sure what is deployed... just checking history 2015-03-19T20:44:32 dprince: well, we are fixing that right now ... 2015-03-19T20:44:46 mordred: okay, I'll leave you to it and check back in a bit 2015-03-19T20:44:56 and it wasn't deployed to tripleo 2015-03-19T20:45:01 dprince: although that's very new - so it's unlikely to be in your tripleo nodes 2015-03-19T20:45:09 and it should work just fine on fedora because supposedly fedora does the right thing unlike ubuntu 2015-03-19T20:46:17 rebuilding b5 2015-03-19T20:46:48 ClientException: Unknown Error (HTTP 502) 2015-03-19T20:46:52 also seeing a bunch of those 2015-03-19T20:47:00 the read timeouts have subsided though 2015-03-19T20:47:24 jeblair: UnknownError are my favorite 2015-03-19T20:47:27 pleia2: okay i think we got it going with the new zanata 2015-03-19T20:47:30 Khai Do proposed openstack-infra/gerritlib: Update project docs and setup for tests https://review.openstack.org/165979 2015-03-19T20:47:34 cinerama: woo 2015-03-19T20:47:50 pleia2: oh wait i need more coffee :) 2015-03-19T20:48:26 Khai Do proposed openstack-infra/gerritlib: Update project docs https://review.openstack.org/165979 2015-03-19T20:50:04 i reverted one of my manifests but it also reverted the version change :) 2015-03-19T20:50:14 but the new template doesn't fail? :) 2015-03-19T20:50:41 *** jaypipes has quit IRC 2015-03-19T20:50:52 nibalizer: 2015-03-19 20:50:42,147 INFO nodepool.image.build.devstack-trusty: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations 2015-03-19T20:51:02 nibalizer: apparently puppet really wants us to know that 2015-03-19T20:51:06 *** ibiris is now known as ibiris_away 2015-03-19T20:51:33 ttx is waiting on https://review.openstack.org/#/c/150267/ to cut nova, just thought I would let you know 2015-03-19T20:52:03 it is in check and almost finished running tests 2015-03-19T20:52:09 Exception: Timeout waiting for server 22243675-f398-4779-9650-d3e7da24b782 deletion in tripleo-test-cloud-rh1 2015-03-19T20:52:25 clarkb: wow. they're having the delete server problem too? 2015-03-19T20:52:55 mordred: I guess? I am trying a different node to see if it is consistent 2015-03-19T20:53:06 Shrews: btw - I figure it'll take us a few passes at one of the resources to get the pattern right 2015-03-19T20:53:09 clarkb: wut 2015-03-19T20:53:10 Khai Do proposed stackforge/python-jenkins: Add info on how to contribute. https://review.openstack.org/166006 2015-03-19T20:53:16 Shrews: I'm starting to think we actually don't want ResourceRequest after all 2015-03-19T20:53:17 mordred: it DOES 2015-03-19T20:53:23 clarkb: IT NEEDS YOU TO KNOW 2015-03-19T20:53:27 er mordred 2015-03-19T20:53:36 Shrews: since you coudl also "get" a resource and have that resource be still in a building state 2015-03-19T20:53:41 clarkb: so what did I break? 2015-03-19T20:53:49 nibalizer: I do not know what you are referring to 2015-03-19T20:53:57 Shrews: so I think I may have been wrong about the ResourceRequest class 2015-03-19T20:53:58 *** emagana has quit IRC 2015-03-19T20:54:11 mordred: I'm working through modeling a scenario now to see what this would actually look like 2015-03-19T20:54:29 ++ 2015-03-19T20:54:40 I started going down that path and then got distracted 2015-03-19T20:54:47 time for more wackypants java errors 2015-03-19T20:54:53 cinerama: o/ 2015-03-19T20:54:59 *** emagana has joined #openstack-infra 2015-03-19T20:55:32 mordred: i'll post something up that's depedent on your review, when i have it 2015-03-19T20:55:39 this has really been old home week for languages i haven't done in a while 2015-03-19T20:55:42 Khai Do proposed openstack-infra/jenkins-job-builder: Add standard openstack info on how to contribute. https://review.openstack.org/166008 2015-03-19T20:56:00 *** Hal1 has quit IRC 2015-03-19T20:56:18 i'm going to have to do perl soon aren't i? 2015-03-19T20:56:34 cinerama: algol 2015-03-19T20:57:18 fungi: i have an algol book somewhere around here 2015-03-19T20:57:28 *** ominakov has quit IRC 2015-03-19T20:57:32 don't we all? 2015-03-19T20:57:55 *** Sukhdev has joined #openstack-infra 2015-03-19T20:58:15 clarkb: blast - today has been so fun I forgot to turn on the day of sportsball 2015-03-19T20:58:31 mordred: any upsets yet? I haven't really followed it either 2015-03-19T20:58:33 i can see it from here actually. it's next to 'code complete' and 'common lisp' 2015-03-19T20:58:35 mordred: it has been exciting 2015-03-19T20:58:42 notre dame really tried to lose 2015-03-19T20:58:45 maybe I should go downstairs and turn on the tv 2015-03-19T20:59:22 *** bdpayne has joined #openstack-infra 2015-03-19T20:59:34 clarkb: need a quick favor - hope you can help 2015-03-19T20:59:38 clarkb: yes - iowa state lost 2015-03-19T20:59:41 Sukhdev: maybe 2015-03-19T20:59:44 Sukhdev: just ask 2015-03-19T20:59:44 cinerama: you need "fortran programming: a spiral approach" i can't recommend it enough 2015-03-19T20:59:48 clarkb: to uab 2015-03-19T20:59:58 clarkb: and Georgia State beat Baylor 2015-03-19T21:00:11 If I look at these instructions, http://docs.openstack.org/infra/manual/drivers.html#end-of-release 2015-03-19T21:00:14 I had both of those in the 8 - so that's not great for the old bracket 2015-03-19T21:00:19 but go GA State 2015-03-19T21:00:29 It says to delete a release select the release and delete it 2015-03-19T21:00:46 the eastern wa game is one I want to se 2015-03-19T21:00:50 fungi: sweet cover but i'm disappointed it's not spiral bound 2015-03-19T21:00:52 but, when I go here - https://review.openstack.org/#/admin/projects/stackforge/networking-arista,branches 2015-03-19T21:01:02 there is no way to select the release - 2015-03-19T21:01:03 *** asselin_ has joined #openstack-infra 2015-03-19T21:01:15 eg. I want to delete proposed/kilo 2015-03-19T21:01:17 clarkb: I have georgetown there- but I'll be rooting for ewa 2015-03-19T21:01:20 Sukhdev: have you tagged your release? 2015-03-19T21:01:29 not yet 2015-03-19T21:01:39 mordred: at some point the big sky is going to win a tournament game 2015-03-19T21:01:42 you have to tag before you delete 2015-03-19T21:01:45 mordred: and this is a good shot for them to do it 2015-03-19T21:01:56 Sukhdev: This step should be performed by the OpenStack Release Manager after the release is tagged. 2015-03-19T21:02:05 Sukhdev: see where that section starts with "This step should be performed by the OpenStack Release Manager..." 2015-03-19T21:02:12 *** yfried|afk is now known as yfried 2015-03-19T21:02:29 clarkb, fungi, mordred: i'm continuing to tweak tunables to see if i can get this to perform better 2015-03-19T21:02:35 jeblair: cool 2015-03-19T21:02:36 Sukhdev: you'll need to get a release manager or infra admin to delete branches for you. it's not a gerrit permission we can safely grant 2015-03-19T21:02:49 clarkb: yeah - I don' tbelieve in georgetown at all 2015-03-19T21:02:56 *** patrickeast_ has joined #openstack-infra 2015-03-19T21:02:58 fungi: I think we can grant it to stackforge projecs? 2015-03-19T21:03:06 clarkb: didn't we try? 2015-03-19T21:03:06 *** tsg_ has quit IRC 2015-03-19T21:03:07 fungi: I am trying to remember if that was possible, they have to be branch owners right? 2015-03-19T21:03:08 fungi: Ah ha - that is the part I was missing - thanks for clarification 2015-03-19T21:03:22 how do we prevent them from deleteing prior to tagging 2015-03-19T21:03:32 that will be a fun one to try to fix 2015-03-19T21:03:33 clarkb: i think they had to be project owners. zaro: do you recall? 2015-03-19T21:03:34 *** dannywil_ has quit IRC 2015-03-19T21:03:43 *** patrickeast has quit IRC 2015-03-19T21:03:43 *** patrickeast_ is now known as patrickeast 2015-03-19T21:03:43 jeblair: anything we can do to help? 2015-03-19T21:03:49 fungi: ya thats what it was 2015-03-19T21:03:59 fungi: which imprated a bunch of other prems that werent great 2015-03-19T21:04:04 anteaya: we don't prevent anything. we just load the gun and they can tell us if they want it aimed at their foot 2015-03-19T21:04:10 ok second node in rh1 failed to delete due to a timeout as well 2015-03-19T21:04:15 so all the clouds are broken 2015-03-19T21:04:16 fungi: yes 2015-03-19T21:04:28 *** dannywilson has joined #openstack-infra 2015-03-19T21:04:33 picturing infra become a waiting room with bleeding feet next release 2015-03-19T21:04:34 *** dangers is now known as dangers_away 2015-03-19T21:04:42 *** ivar-laz_ has joined #openstack-infra 2015-03-19T21:04:43 clarkb: have I mentioned that I don't think deleting things should be hard? 2015-03-19T21:04:51 fungi anteaya: Actually, I first called it proposed/kilo, and then later decided, I should really call it kilo - so, end up creating two branches - will tag the kilo one, but, wanted to get rid of proposed/kilo - 2015-03-19T21:05:22 Sukhdev: glad you are thinking about your own feet 2015-03-19T21:05:31 *** asselin_ has quit IRC 2015-03-19T21:05:31 *** rfolco has quit IRC 2015-03-19T21:05:35 *** ivar-laz_ has quit IRC 2015-03-19T21:06:01 Sukhdev: if you're ready for that branch to be deleted, i can do so. but having a branch named "kilo" instead of "stable/kilo" is going to cause integration testing problems 2015-03-19T21:06:02 ClientException: Unknown Error (HTTP 503) 2015-03-19T21:06:06 wow! 2015-03-19T21:06:10 Merged openstack-infra/jenkins-job-builder: Add missing ' https://review.openstack.org/165786 2015-03-19T21:06:12 so many unknown errors! 2015-03-19T21:06:13 BadRequest: Error. Unable to associate floating ip (HTTP 400) (Request-ID: req-ae2cf0db-28bc-4081-b09b-79a60e5495f2) is the rh1 issue I think 2015-03-19T21:06:19 *** ihrachyshka has quit IRC 2015-03-19T21:06:19 *** ivar-laz_ has joined #openstack-infra 2015-03-19T21:06:24 500, 502, 503... what, no 501? 2015-03-19T21:06:32 clarkb: also - texas just lost - so that's 3 losses for the B12 2015-03-19T21:06:37 jeblair: I'm sure there's a 501 coming ... 2015-03-19T21:06:38 you are missing the full series 2015-03-19T21:06:48 501 is the collector's item apparently 2015-03-19T21:06:52 dprince: so if you want ot look into something I think ^ is the thing 2015-03-19T21:06:56 fungi: Thanks for pointing this out - in that case, can you please do me a favor and delete them both - I will create stable/kilo and tag that one. 2015-03-19T21:07:01 *** ivar-laz_ has quit IRC 2015-03-19T21:07:32 *** bhuvan has quit IRC 2015-03-19T21:07:50 *** ivar-laz_ has joined #openstack-infra 2015-03-19T21:08:07 fungi: Thank you - that did it…appreciate it... 2015-03-19T21:08:08 Sukhdev: done. i've deleted them both, they each had the same commit sha as the tip of master, 467aa2a8c0f5cc1f52eeef07e36fb6894f8641c3 2015-03-19T21:08:35 *** ivar-lazzaro has quit IRC 2015-03-19T21:08:35 *** masayukig has quit IRC 2015-03-19T21:08:58 *** masayukig has joined #openstack-infra 2015-03-19T21:09:36 clarkb: are you getting 501 errors from Nova API? 2015-03-19T21:09:48 dprince: no that is a 400 error 2015-03-19T21:10:15 *** bswartz has quit IRC 2015-03-19T21:10:31 hpcloud-b5 image built - uploading 2015-03-19T21:10:42 2015-03-19 21:09:40,919 INFO nodepool.DiskImageBuilderThread: DIB image 118717 with file devstack-trusty is built 2015-03-19T21:11:20 clarkb: I'll check the logs, thanks 2015-03-19T21:12:01 *** andreykurilin_ has joined #openstack-infra 2015-03-19T21:12:13 *** yfried is now known as yfried|afk 2015-03-19T21:12:17 *** Somay has joined #openstack-infra 2015-03-19T21:13:50 wow we upload logs for every instance booted by tempest? 2015-03-19T21:15:18 clarkb: well the qemu logs, we added that a while ago 2015-03-19T21:15:25 *** ldnunes has quit IRC 2015-03-19T21:15:33 *** ivar-laz_ has quit IRC 2015-03-19T21:16:00 dprince: fwiw I can keystone catalog but operations like nova list and neutron floatingip-list hang 2015-03-19T21:16:09 *** ominakov has joined #openstack-infra 2015-03-19T21:16:19 *** yamamoto has joined #openstack-infra 2015-03-19T21:17:22 *** radez_g0n3 is now known as radez 2015-03-19T21:17:45 fungi: Is there a way to delete the tags? 2015-03-19T21:19:01 Sukhdev: not really, no. tag deletion propagates to push mirror but not via pull mechanisms like fetch all or remote update 2015-03-19T21:19:20 Sukhdev: so lots of places will have the old tags cached after deletion 2015-03-19T21:19:21 dprince: ERROR (ConnectionRefused): Unable to establish connection to http://10.1.8.37:5000/v2.0/tokens the plot thickens 2015-03-19T21:19:40 dprince: seems like a broken proxy or something 2015-03-19T21:19:49 clarkb, fungi, mordred: i'm continuing to slow the rate and increase timeouts on hpcloud, but i'm also continuing to see problems 2015-03-19T21:19:51 fungi: cool - thanks 2015-03-19T21:20:28 *** andreykurilin_ has quit IRC 2015-03-19T21:20:31 *** dboik_ has joined #openstack-infra 2015-03-19T21:20:43 jeblair, clarkb, fungi, pleia2: hpcloud seems to be having some systemic issues, fwiw 2015-03-19T21:20:52 *** andreykurilin_ has joined #openstack-infra 2015-03-19T21:20:52 *** dboik_ has quit IRC 2015-03-19T21:21:14 *** otter768 has joined #openstack-infra 2015-03-19T21:21:16 *** yamamoto has quit IRC 2015-03-19T21:21:17 *** davideagnello has joined #openstack-infra 2015-03-19T21:21:20 *** dboik_ has joined #openstack-infra 2015-03-19T21:21:25 mordred: yes, ec2 meta service is unwork :) 2015-03-19T21:21:36 i hope we didn't cause them. afaik, we never exceeded our rate limit on the calling side 2015-03-19T21:21:53 clarkb: nope- bigger 2015-03-19T21:22:04 clarkb: an incident has started 2015-03-19T21:22:15 no 2015-03-19T21:22:19 not an incident 2015-03-19T21:22:22 clarkb: not just "metadata service is yuck" 2015-03-19T21:22:22 mordred: jeblair I would not be surprised if this has been part of the larger underlying issue we have had with hpcloud over the last week or two 2015-03-19T21:22:29 clarkb: ++ 2015-03-19T21:22:40 *** ominakov has quit IRC 2015-03-19T21:23:27 *** dboik has quit IRC 2015-03-19T21:23:29 we're currently at around one api call every 0.8 seconds (across all 5 providers) 2015-03-19T21:24:07 *** davideagnello has quit IRC 2015-03-19T21:24:44 considering that my 'nova show' command i've been trying periodically for the past 10 minutes still is returning 502 every time i try, i'm starting to think something has fallen over 2015-03-19T21:24:45 *** ivar-lazzaro has joined #openstack-infra 2015-03-19T21:24:55 *** dkliban is now known as dkliban_afk 2015-03-19T21:25:39 *** davideagnello has joined #openstack-infra 2015-03-19T21:25:39 *** xyang1 has quit IRC 2015-03-19T21:26:00 *** otter768 has quit IRC 2015-03-19T21:27:12 and hrm, we've got basically nothing from hpcloud in use 2015-03-19T21:27:37 *** salv-orlando has joined #openstack-infra 2015-03-19T21:28:31 *** achanda has quit IRC 2015-03-19T21:28:47 oi 2015-03-19T21:30:07 maybe we should take this as an opportunity to patch apache against openssl problems >_> 2015-03-19T21:30:16 *** andreykurilin_ has quit IRC 2015-03-19T21:30:24 *** andreykurilin_ has joined #openstack-infra 2015-03-19T21:30:57 I see the Evgeny Antyshev user is still spamming 2015-03-19T21:31:21 jogo: link? 2015-03-19T21:31:35 clarkb: https://review.openstack.org/#/c/165767/ 2015-03-19T21:31:43 hmm 2015-03-19T21:31:48 one moment 2015-03-19T21:32:02 clarkb: ahh yup https://review.openstack.org/#/c/165767/ 2015-03-19T21:32:37 that happened just a few minutes ago 2015-03-19T21:32:53 same email 2015-03-19T21:33:06 *** dkranz has joined #openstack-infra 2015-03-19T21:33:13 is there more that one user with the same full name and email address in the gerrit db? 2015-03-19T21:33:51 no, and I have confirmed that user is marked inactive 2015-03-19T21:33:59 and I did flush the caches 2015-03-19T21:34:06 so maybe if they have a connection open it stays open? 2015-03-19T21:34:14 I get one account id for that email 2015-03-19T21:34:33 anteaya: yes, 13431 2015-03-19T21:34:44 any idea how we id which connection is theirs and kill it? 2015-03-19T21:34:50 clarkb: confirmed 2015-03-19T21:34:57 anteaya: no, not sure 2015-03-19T21:35:23 its possible we could find PTR records that belonged to parallels.com and shoot those tcp connections? 2015-03-19T21:35:30 I'm for that 2015-03-19T21:35:36 how can I help? 2015-03-19T21:35:39 *** carl_baldwin has quit IRC 2015-03-19T21:35:42 *** andreykurilin_ has quit IRC 2015-03-19T21:35:44 anteaya: well I am not really for it :) 2015-03-19T21:35:51 okay 2015-03-19T21:35:54 what are you for? 2015-03-19T21:35:55 its imprecise and even then the tools for it are not great 2015-03-19T21:35:59 *** mattfarina has quit IRC 2015-03-19T21:36:07 anyone try contacting that email address? 2015-03-19T21:36:10 so first question, has anyone sent email to that address? 2015-03-19T21:36:12 yes 2015-03-19T21:36:16 *** andreykurilin_ has joined #openstack-infra 2015-03-19T21:36:23 jogo: dims did this morning and cc'd me, no reply 2015-03-19T21:36:46 *** ChuckC has quit IRC 2015-03-19T21:37:05 *** achanda has joined #openstack-infra 2015-03-19T21:37:06 11:15am eastern time 2015-03-19T21:38:46 oh sweet gerrit show-connections will show you uids too 2015-03-19T21:38:53 awesome 2015-03-19T21:39:16 now to figure out if there is a gerrit command to kill a connection 2015-03-19T21:39:28 *** alexpilotti has joined #openstack-infra 2015-03-19T21:39:30 *** dims has joined #openstack-infra 2015-03-19T21:39:54 zaro: do you know if gerrit kill can be fed a session id? 2015-03-19T21:40:18 good news is I only count about 40 comments from him today 2015-03-19T21:40:25 so this is bad but not terribad 2015-03-19T21:40:37 *** salv-orlando has quit IRC 2015-03-19T21:40:45 jogo: well we needed to know if flushing caches solved the problem 2015-03-19T21:40:53 anteaya: right 2015-03-19T21:40:58 and it didn't, so thanks for letting us know 2015-03-19T21:41:00 https://groups.google.com/forum/#!topic/repo-discuss/_xWxgvEryVg 2015-03-19T21:41:37 2013 2015-03-19T21:41:40 anteaya: do you have a email for CloudByte CI ? 2015-03-19T21:42:01 so, last-ditch option is restart gerrit 2015-03-19T21:42:05 hmm never mind may not be related 2015-03-19T21:42:10 openstack-dev@cloudbyte.com 2015-03-19T21:42:16 *** salv-orlando has joined #openstack-infra 2015-03-19T21:43:18 *** garyh has quit IRC 2015-03-19T21:43:24 anteaya: it looks like he was testing open patches in the sandbox and there are a lot of cloudbyte ones 2015-03-19T21:43:31 jogo: ah 2015-03-19T21:43:49 jeblair: just looking at the graphs for nodes it doesn't appear that rax has fallen over due to your chnage 2015-03-19T21:43:51 was hoping he was working on a CI system that had a contact ... :/ 2015-03-19T21:44:07 jeblair: seems likely that your change is a good one considering that 2015-03-19T21:44:07 clarkb: i agree 2015-03-19T21:44:18 *** dboik_ has quit IRC 2015-03-19T21:44:31 clarkb: well, i don't know if it has caused hpcloud to fall over :/ 2015-03-19T21:44:36 german company going by parallels.com 2015-03-19T21:44:47 *** rlandy has quit IRC 2015-03-19T21:44:51 jogo, I have some e-mails from cloudbyte 2015-03-19T21:44:53 jeblair: ya I guess that it may still have caused that :) 2015-03-19T21:45:14 *** dboik has joined #openstack-infra 2015-03-19T21:45:21 *** timcline has quit IRC 2015-03-19T21:45:50 jogo, let me konw if you want more names e-mail addresses 2015-03-19T21:46:13 *** peristeri has quit IRC 2015-03-19T21:46:22 I'm more concerned we don't have a way of closing an open connection for a disabled account 2015-03-19T21:46:36 I'm surprised this never came up before 2015-03-19T21:46:48 *** dboik_ has joined #openstack-infra 2015-03-19T21:48:01 *** dhritishikhar has quit IRC 2015-03-19T21:48:40 *** eharney has joined #openstack-infra 2015-03-19T21:48:59 *** mtanino has quit IRC 2015-03-19T21:49:10 *** mriedem is now known as mriedem_away 2015-03-19T21:49:21 anteaya: ya not sure what gerrit is doing internally there 2015-03-19T21:49:26 *** tsg has joined #openstack-infra 2015-03-19T21:49:35 anteaya: its possible that the session timeout of 12 hours is what will clean it up 2015-03-19T21:49:53 darn. I see a 'gerrit close-connection' command in gerrit 2.11. 2015-03-19T21:50:23 there's a 2.11 already? 2015-03-19T21:50:31 *** dboik has quit IRC 2015-03-19T21:50:32 *** eharney has quit IRC 2015-03-19T21:50:43 asselin: that was a red herring, thanks anyway 2015-03-19T21:50:56 yes I'm looking and so far your unanswered question from 2013 is the best lead 2015-03-19T21:51:05 modred: perhaps not, google just took me to 2.11-rc1 2015-03-19T21:51:13 one thing kind of funny about depends-on is that I *could* use the same change-id in 2 projects... e.g., same changeid in keystone as in devstack. 2015-03-19T21:51:24 *** dboik_ has quit IRC 2015-03-19T21:51:31 and I would prefer to since it makes the other one easy to find. 2015-03-19T21:52:29 pleia2: this time for real :) 2015-03-19T21:53:36 *** achanda has quit IRC 2015-03-19T21:53:40 since things are sort of derpy right now anyone want to review https://review.openstack.org/#/c/165542/ to fix html rendering for console.html files? 2015-03-19T21:53:59 cinerama: how's it going? :) 2015-03-19T21:54:02 2015-03-19 21:28:40,286 INFO nodepool.DiskImageUpdater: Image devstack-trusty-1426799380.template.openstack.org in hpcloud-b5 is ready 2015-03-19T21:54:15 Stephanie Miller proposed openstack-infra/puppet-zanata: Initial commit of Zanata-specific sections https://review.openstack.org/147947 2015-03-19T21:54:34 pleia2: actually works now. i've just pushed the template changes 2015-03-19T21:54:55 cinerama: nice! I'll test 2015-03-19T21:55:03 *** andreykurilin_ has quit IRC 2015-03-19T21:55:09 pleia2: there were a few more changes in the standalone template that weren't advertised in the release notes 2015-03-19T21:55:11 clarkb: yes the kill command takes a task id, but I don't see any returned values from show-connections that could be construed as a task id 2015-03-19T21:55:19 *** andreykurilin_ has joined #openstack-infra 2015-03-19T21:55:26 clarkb: could we try username though or ip? 2015-03-19T21:55:26 cinerama: I'm not surprised 2015-03-19T21:55:27 clarkb: it looks like you kill with an ID. 2015-03-19T21:55:37 from the docs anyways 2015-03-19T21:55:45 zaro: ya but session ids are not task ids from what I can tell 2015-03-19T21:56:01 pleia2: i added the infinispan stuff they mentioned, which curiously is not in the prebaked 3.6.0 standalone xml 2015-03-19T21:56:08 clarkb: can we try? 2015-03-19T21:56:21 cinerama: yeah, I don't even know what that is 2015-03-19T21:56:24 clarkb: worst thing that can happen is it doesn't work, yeah? 2015-03-19T21:56:50 anteaya: no worst thing is we kill a real task that is doing something else with the same id number 2015-03-19T21:56:58 clarkb: oh 2015-03-19T21:57:05 clarkb: yeah okay let's not that 2015-03-19T21:57:07 sigh 2015-03-19T21:57:21 clarkb: you could probably just do suexec to do anything you want 2015-03-19T21:58:05 zaro: well we need to disconnect an inactivated account, I don't think suexec will do that 2015-03-19T21:58:08 *** dustins has quit IRC 2015-03-19T21:58:38 *** yfried|afk has quit IRC 2015-03-19T21:58:40 zaro: account 13431 was disabled about 6 hours ago 2015-03-19T21:58:44 and we flushed the caches 2015-03-19T21:58:50 *** yfried|afk has joined #openstack-infra 2015-03-19T21:59:00 but it is still spamming patches with comments from a misconfigured ci 2015-03-19T21:59:03 *** hdd has quit IRC 2015-03-19T21:59:19 most recent was about 30 minutes ago 2015-03-19T21:59:27 *** oomichi has joined #openstack-infra 2015-03-19T21:59:57 not a big deal but we would like the ability to be able to stop disabled accounts from posting patches after they are disabled 2015-03-19T22:00:08 clarkb, mordred, fungi: hpcloud is effectively offline for us 2015-03-19T22:00:13 posting comments to patches, sorry 2015-03-19T22:00:19 jeblair: :( 2015-03-19T22:00:27 no new nodes in the past couple of hours 2015-03-19T22:00:33 *** eharney has joined #openstack-infra 2015-03-19T22:00:37 sdague: comment on one of your os-loganalyze test cleanups, I think we can make it even simpler 2015-03-19T22:00:49 clarkb, anteaya : you then you probably can do with close-connection command? 2015-03-19T22:01:02 jeblair: my node upload just wokred 2015-03-19T22:01:10 zaro: that doesn't seem to exist in our gerrit 2015-03-19T22:01:10 zaro: do tell us about the close-connection command, does it have some docs? 2015-03-19T22:01:28 mordred: image upload? probably beause nova != glance? 2015-03-19T22:01:31 oops must be new, https://gerrit-review.googlesource.com/Documentation/cmd-close-connection.html 2015-03-19T22:01:39 clarkb: it had been failing for the last $time 2015-03-19T22:01:43 clarkb: becaue the glance apis are broken too 2015-03-19T22:01:47 *** achanda has joined #openstack-infra 2015-03-19T22:01:51 mordred: my 'nova show' still fails 2015-03-19T22:01:54 mordred: fun 2015-03-19T22:01:57 https://gerrit-review.googlesource.com/Documentation/cmd-close-connection.html 2015-03-19T22:02:05 jeblair: sigh 2015-03-19T22:02:13 clarkb, anteaya maybe i can cherry pick that into our gerit? 2015-03-19T22:02:13 zaro: 2.11-rc1 2015-03-19T22:02:19 zaro: oh please 2015-03-19T22:02:23 yes please 2015-03-19T22:02:27 Morning 2015-03-19T22:02:30 well that won't fix anything immediately 2015-03-19T22:02:32 jhesketh: good morning 2015-03-19T22:02:38 morning jhesketh 2015-03-19T22:02:40 clarkb: true 2015-03-19T22:02:49 clarkb: do we have any options for what will? 2015-03-19T22:03:05 *** dannywilson has quit IRC 2015-03-19T22:03:06 *** virmitio has joined #openstack-infra 2015-03-19T22:03:11 btw the heat functional test takes almost 2 hours 2015-03-19T22:04:05 wonderful 2015-03-19T22:04:12 good thing ttx went to bed 2015-03-19T22:04:14 *** hashar has quit IRC 2015-03-19T22:04:59 wow, so much for the 1 hour budget 2015-03-19T22:05:32 jeblair: yah - nova command also no worky for me 2015-03-19T22:06:13 jhesketh: basically hpcloud and rh1 tripleo cloud are both borked 2015-03-19T22:06:22 jhesketh: so its been a fun day :) https://review.openstack.org/#/c/165542/ may interest you 2015-03-19T22:07:20 *** adalbas has quit IRC 2015-03-19T22:07:23 anteaya, clarkb : looks promising, no conflicts. now just need to test it. 2015-03-19T22:07:35 clarkb: ouch, sounds fun.. Let me know if I can help 2015-03-19T22:07:48 Will take a look, thanks 2015-03-19T22:07:48 *** dannywilson has joined #openstack-infra 2015-03-19T22:07:57 just got a few more spams from the parallels guy ... :( 2015-03-19T22:08:38 zaro: k, I see that command has a wait option, so if the wait option is not invoked it kills an open command immediately? 2015-03-19T22:08:48 zaro: that is the functionality I'm looking for 2015-03-19T22:09:15 jogo: dang, so far I don't know of a good way to kill that connection 2015-03-19T22:09:37 jogo: right now the best option I have heard is wait on the expiration of some other kind of cache 2015-03-19T22:09:55 at this point I say we ride it out to the 12 hour session invalidation 2015-03-19T22:09:56 *** jamielennox|away is now known as jamielennox 2015-03-19T22:10:02 if its still trouble then we restart gerrit 2015-03-19T22:10:04 zaro: open connection, not open command 2015-03-19T22:10:22 clarkb: if the only option is to restart gerrit then yes 2015-03-19T22:10:48 *** dboik has joined #openstack-infra 2015-03-19T22:11:06 *** e0ne has joined #openstack-infra 2015-03-19T22:11:25 sounds like this is a feature we want for future gerrit versions 2015-03-19T22:11:34 *** dboik_ has joined #openstack-infra 2015-03-19T22:11:47 jogo: yes 2015-03-19T22:12:10 so a session is invalid 12 hours after it begins? 2015-03-19T22:12:32 clarkb: do we have any ability to search that account in open-connections and see when the session began? 2015-03-19T22:12:36 anteaya: 12 hours is when the cache should check if the account is still valid 2015-03-19T22:12:49 gerrit username: eantyshev 2015-03-19T22:12:50 anteaya: we do, but I am not sure its from when the session began 2015-03-19T22:12:59 hmmm 2015-03-19T22:13:01 its from whenver the cache was last updated for that entry most likely 2015-03-19T22:13:21 at worst it should be no later than 12 hours after we marked the account inactive 2015-03-19T22:13:33 anteaya: yeah, this one right? https://gerrit-review.googlesource.com/#/c/63100/ 2015-03-19T22:13:40 from the docs it should show-connections should show a start time 2015-03-19T22:13:48 oh 2015-03-19T22:14:23 zaro: that looks like the one, let me look at the patch 2015-03-19T22:15:13 *** dboik has quit IRC 2015-03-19T22:15:46 *** packet has quit IRC 2015-03-19T22:16:11 *** timcline has joined #openstack-infra 2015-03-19T22:16:18 looks like you can feed it a list 2015-03-19T22:16:50 and looks like the wait option is if you want the command to wait for the connection to close before exiting 2015-03-19T22:16:57 *** VijayTripathi has joined #openstack-infra 2015-03-19T22:16:57 I had read that wrong 2015-03-19T22:17:29 you feed it a session id 2015-03-19T22:17:30 *** yamamoto has joined #openstack-infra 2015-03-19T22:17:41 clarkb: show-connections gives you session ids? 2015-03-19T22:18:06 zaro: yes please, I would like taht functionality 2015-03-19T22:18:08 *** aysyd has quit IRC 2015-03-19T22:18:15 *** e0ne has quit IRC 2015-03-19T22:18:57 *** AJaeger has quit IRC 2015-03-19T22:19:46 *** dims has quit IRC 2015-03-19T22:20:42 Is there any place where Jenkins job run times are collected and stats for specific jobs can be viewed? 2015-03-19T22:20:52 *** aysyd has joined #openstack-infra 2015-03-19T22:20:55 http://status.openstack.org/zuul/ 2015-03-19T22:20:58 *** timcline has quit IRC 2015-03-19T22:21:11 stats? what kind of stats? 2015-03-19T22:21:15 vagrant upgrade, time to put the kettle on 2015-03-19T22:21:22 we have graphs at the bottom 2015-03-19T22:21:23 *** ghostpl_ has quit IRC 2015-03-19T22:21:50 *** yamamoto has quit IRC 2015-03-19T22:22:04 YorikSar: graphite.openstack.org 2015-03-19T22:22:10 anteaya: I want to see if some specific jobs became faster or changes pace in any way. 2015-03-19T22:22:23 ah yes as fungi says http://graphite.openstack.org/ 2015-03-19T22:22:43 *** ashleighfarnham has quit IRC 2015-03-19T22:23:06 *** dprince has quit IRC 2015-03-19T22:23:43 YorikSar: also we have a new service tracking individual tests by interpreting the subunit data and extracting failure rates and timing for them, though it's currently only exposed as a read-only mysql query socket 2015-03-19T22:23:46 fungi: Oh, cool. Will try to find it there... 2015-03-19T22:23:58 but that may be more granular than you need 2015-03-19T22:24:03 *** ChuckC has joined #openstack-infra 2015-03-19T22:24:36 Merged openstack-dev/pbr: Don't avoid egg-info quite so often https://review.openstack.org/165511 2015-03-19T22:24:48 fungi: Wow... MySQL socket exposed to Internet?.. 2015-03-19T22:24:57 fungi: Sounds dangerous 2015-03-19T22:25:06 YorikSar: yep. we're brave like that 2015-03-19T22:25:27 *** xyang1 has joined #openstack-infra 2015-03-19T22:25:47 YorikSar: the docs for the db service are here: http://docs.openstack.org/developer/subunit2sql/ 2015-03-19T22:25:59 trust me we talked through all the security implications therein ;) 2015-03-19T22:26:27 fungi: OpenStack Infra - bunch of cool brave people :) 2015-03-19T22:26:48 fungi: heh, didn't we just say something like meh we'll throw it away if there's an issue :) 2015-03-19T22:26:52 YorikSar: it's ... aggregated test data - if you hack it - we will have lost aggregated test run timing data - it's probably ok :) 2015-03-19T22:26:54 yah 2015-03-19T22:26:58 mtreinish: yep! 2015-03-19T22:27:05 *** bknudson has quit IRC 2015-03-19T22:27:12 fungi: Oh, I don't need to trust you. It's you who needs to trust you ;) 2015-03-19T22:27:21 YorikSar: well, I stopped doing thatyears ago 2015-03-19T22:27:27 i try not to trust me 2015-03-19T22:27:55 mordred: I'd prefer to mine some *coins on your MySQL cluster, why kill it? ;) 2015-03-19T22:28:06 *** xyang1 has quit IRC 2015-03-19T22:28:32 *** xyang1 has joined #openstack-infra 2015-03-19T22:28:41 YorikSar: just cut us in on the take 2015-03-19T22:29:10 anteaya: yes session ids are returned 2015-03-19T22:29:17 *** mtanino has joined #openstack-infra 2015-03-19T22:29:30 *** amitgandhinz has quit IRC 2015-03-19T22:29:34 we should just create infracoin 2015-03-19T22:29:35 anteaya, clarkb : https://review.openstack.org/166025 2015-03-19T22:29:51 cinerama: for all you know, we already have 2015-03-19T22:29:56 anteaya: you need to be an admin to close connections though. 2015-03-19T22:30:17 cinerama: your patch cannot land until it has mined a coin 2015-03-19T22:30:25 clarkb: great 2015-03-19T22:30:35 anteaya, clarkb : tested and worked like a charm. 2015-03-19T22:30:36 jhesketh: fwiw I think we should move forward are using swift logs only for not dsvm jobs, and also add swift logs to the dsvm jobs but still copy to disk 2015-03-19T22:30:40 fungi: nice digital currency you've got there...it'd be a real shame if someone busted it up 2015-03-19T22:30:42 zaro: understood, but not at least an admin can close a connection, right now they can't 2015-03-19T22:30:44 jhesketh: then the last task will be sorting out the help footer 2015-03-19T22:30:49 zaro: awesome thank you 2015-03-19T22:30:54 *** AJaeger has joined #openstack-infra 2015-03-19T22:30:54 *** AJaeger has quit IRC 2015-03-19T22:30:54 *** AJaeger has joined #openstack-infra 2015-03-19T22:30:57 Oh, it looks (from the first Google page) that noone thought about mining coins inside MySQL cluster. 2015-03-19T22:30:58 jhesketh: but everything continues to work with swift so keep pushing on it imo 2015-03-19T22:31:08 zaro: what is your position on getting this into gerrit on saturday? 2015-03-19T22:31:08 *** xyang1 has quit IRC 2015-03-19T22:31:09 we have this new digital currency whereby service providers donate virtual machine resources and in return they get tested software they can use 2015-03-19T22:31:43 YorikSar: it would probably just be easier to push up a self replicating patch to do that inside of gate jobs... :) 2015-03-19T22:31:47 clarkb: sounds good to me. I need to get onto the footer stuff, but will push some more jobs to flick over soon 2015-03-19T22:32:08 anteaya: ohh, wait up that was for 2.9 wasn't it? let me give it a try for 2.8 and see 2015-03-19T22:32:13 mtreinish: Naaah, too easy :) 2015-03-19T22:32:35 zaro: I'm fine with either but if we can have it for 2.8 on saturday I'd love to try 2015-03-19T22:32:45 zaro: do you still have a 2.8 up for testing? 2015-03-19T22:33:04 hrm is this saturday that saturday? 2015-03-19T22:33:08 *** patrickeast has quit IRC 2015-03-19T22:33:11 * clarkb may have made a scheduling fubar 2015-03-19T22:33:39 yep this is that 2015-03-19T22:33:48 this saturday is trusty 2015-03-19T22:33:55 clarkb: this saturday is upgrade gerrit os to trusty yes 2015-03-19T22:34:00 *** mjturek1 has quit IRC 2015-03-19T22:34:00 clarkb: starting at 1500 2015-03-19T22:34:02 derp, I am supposed to go learn to not kill babies on saturday 2015-03-19T22:34:06 clarkb: you were on errands 2015-03-19T22:34:13 clarkb: well that is important too 2015-03-19T22:34:21 Zane Bitter proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1433555 https://review.openstack.org/166026 2015-03-19T22:34:22 bug 1433555 in heat "functional tests are taking too long and cause false failures" [Critical,In progress] https://launchpad.net/bugs/1433555 - Assigned to Angus Salkeld (asalkeld) 2015-03-19T22:34:26 clarkb: here's what you need to know: "don't kill them" 2015-03-19T22:34:31 clarkb: considering you have impending babies, I'd go with that as the higher priority 2015-03-19T22:34:42 clarkb: we have almost everyone else I think 2015-03-19T22:34:50 yeah, we've got a full house 2015-03-19T22:34:55 I might be drunk 2015-03-19T22:35:03 mordred: did duke already lose? 2015-03-19T22:35:07 >_> 2015-03-19T22:35:21 * mordred throws an agitated emu at clarkb 2015-03-19T22:35:43 how can you tell an emu is agitated 2015-03-19T22:35:47 I've looked at one 2015-03-19T22:35:50 how do you know? 2015-03-19T22:36:00 *** boris-42 has joined #openstack-infra 2015-03-19T22:36:02 anteaya: more rapid hollow thumping noises 2015-03-19T22:36:09 ah 2015-03-19T22:36:17 this one made zero thumping noises 2015-03-19T22:36:22 I believe emu's win the prize for one of the weirdest noises 2015-03-19T22:36:36 *** VijayTripathi has quit IRC 2015-03-19T22:37:14 clarkb: if it's that cpr thing, i fogot it all like the next day 2015-03-19T22:37:25 kookaburras sound like random jungle noises from every movie ever 2015-03-19T22:37:26 zaro: its like a full day of everything 2015-03-19T22:37:42 yeah, forgot it all 2015-03-19T22:37:49 mordred: really 2015-03-19T22:37:50 Thanh Ha proposed openstack-infra/jenkins-job-builder: Add FindBugs support to Publishers https://review.openstack.org/166028 2015-03-19T22:38:03 mordred: for me that is won by partriges 2015-03-19T22:38:23 clarkb: the biggest thing you learn is who to call and when 2015-03-19T22:38:33 clarkb: so you don't have to think about it when you have to call 2015-03-19T22:38:49 *** gordc has quit IRC 2015-03-19T22:38:50 cpr...there's some mnemonic...Airway Be Closing? 2015-03-19T22:39:16 airways always be closing 2015-03-19T22:39:23 https://review.openstack.org/#/c/165629/3 2015-03-19T22:39:30 This change depends on a change that failed to merge. 2015-03-19T22:39:35 not sure what is failing there 2015-03-19T22:40:09 https://review.openstack.org/#/c/135768/ this isn't approved 2015-03-19T22:40:14 is it because it is on top of a patch that has a depends-on in it 2015-03-19T22:40:37 anteaya: right, not trying to merge anything per se, just have the tests run 2015-03-19T22:40:37 cinerama: when you're testing, is your local.pp still the one I pastebined a while back? 2015-03-19T22:41:00 jhesketh: oh - do you have powers to do magical things to people's accounts? 2015-03-19T22:41:08 jogo: oh depends-on and dependencies shouldn't prevent check jobs running on patches that need check jobs to run 2015-03-19T22:41:11 jogo: oh, you got that in the check pipeline? 2015-03-19T22:41:13 pleia2: lemme get what i currently have for you...i split out the mysql on request 2015-03-19T22:41:16 jeblair: yup 2015-03-19T22:41:20 *** erlon has quit IRC 2015-03-19T22:41:27 cinerama: right, thank you 2015-03-19T22:41:29 jogo: cool, we've been waiting for that to appear again since we turned on extra debugging 2015-03-19T22:41:30 jhesketh: because Ng made a rackspace account, but they put him in the LON region which apparently has nothing to do with the rest of rackspace 2015-03-19T22:41:31 *** patrickeast has joined #openstack-infra 2015-03-19T22:41:55 jeblair: o_O so your saying it may not be me doing something silly 2015-03-19T22:42:10 jeblair: I don't know if you caught that this morning before you crisis-moded - but apparently rackspace LON does not work like DFW, IAD and ORD 2015-03-19T22:42:24 jeblair: and it is not possible to use python-glanceclient with it 2015-03-19T22:42:30 jogo: yeah, is a bug somewhere i think; you can recheck for now, and i'll go see if we have logs that tell us what happened 2015-03-19T22:42:41 Thanh Ha proposed openstack-infra/jenkins-job-builder: Add FindBugs support to Publishers https://review.openstack.org/166028 2015-03-19T22:42:45 mordred: i guess we shouldn't ask for quota there then? :) 2015-03-19T22:42:49 mordred: or presumably syd which is where i split half of my personal systems for global redundancy purposes 2015-03-19T22:42:58 *** VijayTripathi has joined #openstack-infra 2015-03-19T22:43:10 syd and iad seem to work very similarly from what i've seen anyway 2015-03-19T22:43:20 jeblair: thanks 2015-03-19T22:43:24 pleia2: something like http://paste.openstack.org/show/193725/ 2015-03-19T22:43:45 cinerama: thanks 2015-03-19T22:44:04 simple, I like it :) 2015-03-19T22:44:29 pleia2: the mysql module does funky things at the moment if you specify a blank password for the zanata db user 2015-03-19T22:44:42 Ok, I'm lost... I'm trying to find job run time somewhere in "stats.timers.zuul.pipeline.check.job.check-tempest-dsvm-neutron-full.SUCCESS.*", but I don't understand those meters. 2015-03-19T22:44:50 *** radez is now known as radez_g0n3 2015-03-19T22:45:05 cinerama: yeah, I saw your comment in the review a couple days back, nice find 2015-03-19T22:45:29 i think i got the foundation-board{,-confidential} ml split done correctly. after the list admin gets a chance to confirm, the foundation-board ml goes public with an (initially) empty archive 2015-03-19T22:45:37 pleia2: i mention it again because i don't want it to eat your afternoon :) 2015-03-19T22:45:48 cinerama: much appreciated :) 2015-03-19T22:45:54 fungi: splitting mailing lists, was it fun? 2015-03-19T22:46:12 anteaya: as fun as splitting firewood, but minus the blisters 2015-03-19T22:46:28 oh I like splitting firewood 2015-03-19T22:46:34 and yay for no blisters 2015-03-19T22:47:19 i'm okay with splitting firewood except when you get a really dense hardwood like locust where the blade just bounces right back at you 2015-03-19T22:47:28 ah yes 2015-03-19T22:47:35 never swung at locust 2015-03-19T22:47:45 but ironwood bounces 2015-03-19T22:48:37 fungi: I find the best way to split wood is to go to your neighborhood artisinal log store 2015-03-19T22:48:46 fungi: you can get some lovely heirloom organic logs 2015-03-19T22:48:51 anteaya: not so easy to get close-connection onto gerrit 2.8 2015-03-19T22:48:55 with your tight pants 2015-03-19T22:49:02 zaro: not to worry 2015-03-19T22:49:05 mordred: sadly no sorry. I'd be talking to support, likely like you are 2015-03-19T22:49:06 anteaya: build errors, will continue to look at it. 2015-03-19T22:49:09 mordred: manhattan never ceases to amaze me 2015-03-19T22:49:10 zaro: 2.9 is not far away 2015-03-19T22:49:15 And yeah, lon is a little special 2015-03-19T22:49:16 jhesketh: yah - that's already in the works 2015-03-19T22:49:26 zaro: don't hurt yourself, we can get by for another month or so 2015-03-19T22:49:29 jhesketh: I just figured I never know what you have passwords to 2015-03-19T22:49:35 zaro: glad 2.9 was an easy cherry pick 2015-03-19T22:49:39 zaro: thank you 2015-03-19T22:49:47 anteaya: ok, i won't bother then. 2015-03-19T22:49:49 fungi: :) 2015-03-19T22:49:57 mordred: what are you londoning? 2015-03-19T22:49:59 zaro: yeah, don't bother 2015-03-19T22:50:04 last I looked into it I needed a new account entirel 2015-03-19T22:50:04 mordred: yep fair question. I'm also happy to help bat for things as needed too 2015-03-19T22:50:12 Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support to CFP in builders sections https://review.openstack.org/164504 2015-03-19T22:51:18 jhesketh: thanks! I'll let you know if we run in to any roadblocks - as soon as we get the account working, Ng can finish hacking on the systemd stuff for the nodepool-dib things for rax 2015-03-19T22:51:32 clarkb: i think the idea was not to london anything 2015-03-19T22:51:45 clarkb: but Ng got inadvertently londoned and can't break out 2015-03-19T22:51:59 *** Somay has quit IRC 2015-03-19T22:52:02 I've heard that about that place 2015-03-19T22:52:06 something about a tower 2015-03-19T22:52:07 Ng can't use our fine artisinal MURICAN VMs? 2015-03-19T22:52:32 not even the organic ones 2015-03-19T22:52:36 clarkb: lol! 2015-03-19T22:53:11 *** asettle has joined #openstack-infra 2015-03-19T22:53:48 mordred: sounds good 2015-03-19T22:54:18 *** sputnik13 has quit IRC 2015-03-19T22:54:21 clarkb: the MURICAN VMs have the ssh port on the opposite side 2015-03-19T22:54:37 *** mmedvede has quit IRC 2015-03-19T22:54:39 *** gordc has joined #openstack-infra 2015-03-19T22:54:52 clarkb: the ohio state vcu game is pretty good 2015-03-19T22:55:05 mordred: and you don't "boot" murican vms, you trunk them 2015-03-19T22:55:15 fungi: ++ 2015-03-19T22:55:17 *** sputnik13 has joined #openstack-infra 2015-03-19T22:55:18 * jeblair groans 2015-03-19T22:55:18 ++ 2015-03-19T22:55:26 that was good I laughed 2015-03-19T22:55:35 fungi wins today 2015-03-19T22:55:37 that was good i groaned 2015-03-19T22:55:49 * fungi fires off a groaner 2015-03-19T22:56:02 Khai Do proposed stackforge/python-jenkins: Add info on how to contribute https://review.openstack.org/166006 2015-03-19T22:56:10 looks like the current top of stack heat test is giong to cuase a reset 2015-03-19T22:56:22 I am tempted to just shoot that test in the head early 2015-03-19T22:56:33 so what happened to our one hour budget anyway? 2015-03-19T22:56:38 jeblair: I have no idea 2015-03-19T22:56:41 clarkb: no 2015-03-19T22:56:52 rargh 2015-03-19T22:56:58 Min Pae proposed openstack-infra/system-config: adding openstack-cue to eavesdrop https://review.openstack.org/166033 2015-03-19T22:57:12 ttx needs the nova patch next in line, which has completed successfully 2015-03-19T22:57:12 Khai Do proposed openstack-infra/jenkins-job-builder: Add standard openstack info on how to contribute https://review.openstack.org/166008 2015-03-19T22:57:27 sdague: was there a conscious decision to abandon the 1-hour budget? 2015-03-19T22:57:40 *** tiswanso has joined #openstack-infra 2015-03-19T22:57:41 boo 2015-03-19T22:57:45 https://review.openstack.org/#/c/154325/ bumped it 2015-03-19T22:57:49 *** Sukhdev has quit IRC 2015-03-19T22:58:06 I reviewed it and didn't catchthe affect it would have 2015-03-19T22:58:15 merely pointed out how to make the timeout more better :/ 2015-03-19T22:58:33 well, i mean we just approved a 1.5 hour trove change yesterday too 2015-03-19T22:58:50 *** dannywilson has quit IRC 2015-03-19T22:58:59 *** gordc has quit IRC 2015-03-19T22:59:03 *** arxcruz has joined #openstack-infra 2015-03-19T22:59:25 *** hodos has quit IRC 2015-03-19T22:59:31 greghaynes: btw one good thing that did happen today is we deployed your nodepool fix 2015-03-19T22:59:52 I did not get around to landing the nodepool image build logging change though and would rather avoid that right now simply because hair on fire 2015-03-19T23:00:14 *** bswartz has joined #openstack-infra 2015-03-19T23:00:24 Min Pae proposed openstack-infra/project-config: Add openstack-cue to gerritbot https://review.openstack.org/158011 2015-03-19T23:00:26 *** vipul is now known as vipuls 2015-03-19T23:01:08 anteaya: to make that test failure even more awesome it failed due to an internal timeout 2015-03-19T23:01:10 jeblair: yep, though that trove job was already at 1.5 hours and we bumped it up an extra 10 minutes or something 2015-03-19T23:01:14 *** ddieterl_ has quit IRC 2015-03-19T23:01:14 anteaya: took longer than 1200 seconds 2015-03-19T23:01:19 clarkb: wooooo 2015-03-19T23:01:24 love that 2015-03-19T23:01:39 which I think is what the bug above that zaneb's change points out is about 2015-03-19T23:02:09 stevebaker: hi, what would it take to get the heat functional job down to about an hour? 2015-03-19T23:02:27 *** oomichi has quit IRC 2015-03-19T23:02:52 stevebaker: is there something that could be more efficient, or more paralellized, or can we split it into two jobs? 2015-03-19T23:03:28 looks like they're trying to address it via https://review.openstack.org/166030 2015-03-19T23:03:47 refrain refrain! 2015-03-19T23:03:47 lol 2015-03-19T23:04:03 fungi, stevebaker: oh cool :) 2015-03-19T23:04:06 clarkb: overtime 2015-03-19T23:04:17 mordred: what channel do I need to turn on? 2015-03-19T23:04:26 *** bhunter71 has quit IRC 2015-03-19T23:04:39 fungi: that change hasn't run that job yet but should tell us soonish I guess 2015-03-19T23:04:39 clarkb: tnt 2015-03-19T23:04:48 indeed 2015-03-19T23:04:58 hrm no more dprince 2015-03-19T23:05:04 *** andreykurilin_ has quit IRC 2015-03-19T23:05:05 Doug Wiegley proposed openstack-infra/project-config: For neutron and neutron-lbaas, skip more wasted jobs https://review.openstack.org/166035 2015-03-19T23:05:10 *** pritesh has quit IRC 2015-03-19T23:05:15 I would sggest just turning off the rh1 region but there is no fallback so meh 2015-03-19T23:05:16 *** dimtruck is now known as zz_dimtruck 2015-03-19T23:05:17 clarkb: np; the config-validation jobs are also ready to go i think 2015-03-19T23:05:35 harlowja_: I OK kook i rigor w assessment of ofiii 2015-03-19T23:05:40 lol 2015-03-19T23:05:43 *** asettle has quit IRC 2015-03-19T23:05:45 what u say about my mom 2015-03-19T23:05:48 how dare u 2015-03-19T23:05:52 i tried to run it through rot-13 but to no avail 2015-03-19T23:06:00 *** asettle has joined #openstack-infra 2015-03-19T23:06:04 fungi: I prefer rot26 2015-03-19T23:06:10 easier to read 2015-03-19T23:06:25 clarkb: yeah, i think that e-mail was drunken keyboard mashing run through rot-26 2015-03-19T23:06:45 *** arxcruz has quit IRC 2015-03-19T23:07:10 clarkb: did my change to turn off tripleo-hp1 finally merge? 2015-03-19T23:07:37 ahh, yep, earlier today looks like\ 2015-03-19T23:07:55 fungi: yup 2015-03-19T23:08:07 *** jamespage has quit IRC 2015-03-19T23:08:07 *** gnuoy has quit IRC 2015-03-19T23:08:18 fungi: turns out nodepool alien-list with it still in there was UNHAPPY 2015-03-19T23:08:40 *** gnuoy has joined #openstack-infra 2015-03-19T23:08:41 *** jamespage has joined #openstack-infra 2015-03-19T23:09:13 mordred: yep, that was one of the reasons i submitted that change 2015-03-19T23:09:41 fungi: I also submitted a nodepool patch to not be unhappy on that command if a cloud is unhappy 2015-03-19T23:09:52 fungi: but it's been a day - so I have not followed up on it 2015-03-19T23:09:54 oh, great idea 2015-03-19T23:10:10 add a test please :) 2015-03-19T23:10:24 yes test please 2015-03-19T23:10:30 *** bookwar has left #openstack-infra 2015-03-19T23:10:33 we actually have some decent testing going on now 2015-03-19T23:10:41 yay testing 2015-03-19T23:11:07 oh this game is at the rose garden 2015-03-19T23:11:23 clarkb: yah. you have rounds there in town 2015-03-19T23:11:42 i've been forcing myself to add tests for each of the new bindep features i committed too, since it unsurprisingly started out with a decent testing framework and i didn't want to regress it 2015-03-19T23:12:14 nice work 2015-03-19T23:12:36 clarkb: so the real question - is harvard going to beat unc? 2015-03-19T23:12:50 that heat test is still running 2015-03-19T23:13:04 okay, so i've confirmed that a gerrit query for a change issued immediately after a patchset is created may lack depends-on information 2015-03-19T23:13:24 started at 21:08 2015-03-19T23:13:25 jeblair huh 2015-03-19T23:13:29 OH 2015-03-19T23:13:45 finished 2015-03-19T23:13:52 jeblair: aroo? 2015-03-19T23:13:52 so the error that jogo (and sdague before him) reported about changes in check being ejected because they depend on a change that failed to merge is because later on someone comments on a change and zuul updates information for it, and the second time sees the dependency that was not originally there 2015-03-19T23:14:05 Build timed out (after 125 minutes). Marking the build as failed. 2015-03-19T23:14:46 jeblair: that must have been fascinting to track 2015-03-19T23:15:03 *** tiswanso_ has joined #openstack-infra 2015-03-19T23:15:22 *** tiswanso has quit IRC 2015-03-19T23:15:39 mordred, clarkb: it may happen when a patch series is being updated simultaneously 2015-03-19T23:16:34 jeblair does it see the old patch of the dep or no dep at all? 2015-03-19T23:16:42 jeblair: so you were able to diagnose that with the additional logging then? 2015-03-19T23:16:56 clarkb: the dependsOn section is entirely missing from the returned data 2015-03-19T23:17:09 fungi: yeah, logging all of the traffic to/from gerrit 2015-03-19T23:17:26 *** aysyd has quit IRC 2015-03-19T23:18:01 huh, so gerrit's omitting that from the event sometimes? 2015-03-19T23:18:22 yeah, i suspect it either hasn't received the other commit, or processed it, or something. 2015-03-19T23:18:47 that seems like it's going to be weird to work around 2015-03-19T23:19:18 i can think of a thorough-and-slow way and a good-enough-and-fast way... 2015-03-19T23:19:52 the thorough way would be to query for the parent commit sha, and if it is not a merged change, keep waiting for a depends on to show up. 2015-03-19T23:20:04 the good-enough way would be to sleep for a few seconds :) 2015-03-19T23:20:10 ha ha ha 2015-03-19T23:20:50 this is thr gerrit internal depends on ya? 2015-03-19T23:21:11 that's how i interpreted it 2015-03-19T23:21:19 not our commit message header 2015-03-19T23:21:35 correct, only talking about git-depends here 2015-03-19T23:21:37 so, yeah, the sleep doesn't seem all that bad 2015-03-19T23:21:46 *** ghostpl_ has joined #openstack-infra 2015-03-19T23:22:07 *** otter768 has joined #openstack-infra 2015-03-19T23:22:24 *** tjones1 has quit IRC 2015-03-19T23:22:30 it's already a rare enough occurrence 2015-03-19T23:22:48 i've only seen it reported a few times in the past couple weeks 2015-03-19T23:23:12 so if that gets rid of most of the incidents, then it'll be rare enough we just don't care 2015-03-19T23:23:55 i wouldn't be surprised if it only happens when gerrit's really busy 2015-03-19T23:23:58 *** stevemar has quit IRC 2015-03-19T23:24:07 Doug Wiegley proposed openstack-infra/project-config: Promote lbaasv2 api tempest tests to voting for neutron and neutron-lbaas https://review.openstack.org/166038 2015-03-19T23:24:35 *** kutija has quit IRC 2015-03-19T23:25:32 *** _nadya_ has joined #openstack-infra 2015-03-19T23:26:54 *** ghostpl_ has quit IRC 2015-03-19T23:27:06 *** otter768 has quit IRC 2015-03-19T23:27:14 do we have issues with symlinks in repos? 2015-03-19T23:27:22 *** chlong has joined #openstack-infra 2015-03-19T23:27:29 ianw: packaging them is problematic 2015-03-19T23:27:53 particularly i'm wondering if symlinking top-level HACKING.rst into the docs directory is ok 2015-03-19T23:29:02 ianw: you can already tell sphinx to include files from the top-level directory. we have at least some packages which do that for their readme 2015-03-19T23:29:17 ahh, yeah, that's a better path to take 2015-03-19T23:29:18 i don't know an example off the top of my head, but someone else might 2015-03-19T23:30:11 *** bdpayne has quit IRC 2015-03-19T23:30:15 *** _nadya_ has quit IRC 2015-03-19T23:32:31 *** camunoz has quit IRC 2015-03-19T23:33:36 jeblair: ah - I totally thought you were talking about Depends-On footers 2015-03-19T23:33:38 *** stevemar has joined #openstack-infra 2015-03-19T23:33:43 ianw: we have people doing symlinks 2015-03-19T23:34:03 Merged openstack-infra/jenkins-job-builder: Add 'Job Log Logger' wrapper https://review.openstack.org/164338 2015-03-19T23:34:10 openstack-infra/infra-specs:doc/source is a 2015-03-19T23:34:14 great place to look 2015-03-19T23:34:41 *** MarkAtwood has quit IRC 2015-03-19T23:34:49 Merged openstack-infra/jenkins-job-builder: Remove nonexistent servername and cleanup build publisher https://review.openstack.org/156091 2015-03-19T23:34:51 *** garyh has joined #openstack-infra 2015-03-19T23:35:26 mordred: is it mostly symlinking directories and/or python files which gets hairy for packaging? 2015-03-19T23:35:45 *** dannywilson has joined #openstack-infra 2015-03-19T23:36:54 fungi: I think it's mostly the python files 2015-03-19T23:38:25 ianw: though poking around, you can apparently do: 2015-03-19T23:38:36 .. include:: ../../HACKING.rst 2015-03-19T23:38:54 *** tonytan4ever has quit IRC 2015-03-19T23:39:04 which will cause it to just slurp that content into your doc/source/hacking.rst or whatever 2015-03-19T23:39:07 *** tjones1 has joined #openstack-infra 2015-03-19T23:40:31 *** stevemar has quit IRC 2015-03-19T23:40:32 fungi: yeah that's what I think everyone does. Some projects used to use symlinks but there was some reason we stopped 2015-03-19T23:40:38 I can't remember exactly why though 2015-03-19T23:40:40 which is apparently slightly more friendly to our comrades on filesystems which lack symlinks 2015-03-19T23:41:17 fungi: bah, I think we should just insert arbitrary symlinks into all repos 2015-03-19T23:41:37 symlinks break distutils or setuptools or both 2015-03-19T23:42:27 *** garyh has quit IRC 2015-03-19T23:43:31 fungi, jeblair, clarkb: we may be about to get a request to turn off HP for a bit to help them recover 2015-03-19T23:44:01 *** tjones1 has quit IRC 2015-03-19T23:44:27 wow! 2015-03-19T23:44:58 *** camunoz has joined #openstack-infra 2015-03-19T23:45:33 hah 2015-03-19T23:45:34 mordred it is effectively off 2015-03-19T23:45:35 yup. they would like us to stop hitting their API for a bit 2015-03-19T23:45:37 clarkb: no it's not 2015-03-19T23:45:43 *** radez_g0n3 is now known as radez 2015-03-19T23:45:48 it's continually hitting their api asking for things 2015-03-19T23:46:12 well ya but its not actually doing anything 2015-03-19T23:46:14 hpcloud is effectively off for us, we're not effectively off for hpcloud 2015-03-19T23:46:21 right. what fungi said 2015-03-19T23:46:28 clarkb: many things can disturb the distutils and setuptools gods. i bet someone didn't make their sacrifice and now they've smote hp cloud 2015-03-19T23:46:30 they're having an API layer disaster at the moment 2015-03-19T23:46:35 hpcloud may _also_ be effectively off because of us. who knows 2015-03-19T23:46:50 *** ajmiller has quit IRC 2015-03-19T23:46:52 ianw: I haven't seen a smoting in centuries 2015-03-19T23:48:23 Monty Taylor proposed openstack-infra/system-config: Turn off HP Public Cloud https://review.openstack.org/166043 2015-03-19T23:48:41 fungi, clarkb, jeblair, pleia2, jhesketh: ^^ 2015-03-19T23:49:08 reviewed 2015-03-19T23:49:13 sad trombone 2015-03-19T23:49:36 jeblair: do you have local changes applied? 2015-03-19T23:50:27 *** derekh has joined #openstack-infra 2015-03-19T23:51:16 *** mfink_ has quit IRC 2015-03-19T23:52:06 Merged openstack-infra/system-config: Turn off HP Public Cloud https://review.openstack.org/166043 2015-03-19T23:52:14 might also be worth pointing out that they don't need us to do that for them... 2015-03-19T23:52:24 but maybe firewalling off nodepool.o.o isn't a nice thing to do? 2015-03-19T23:52:52 *** derekh has quit IRC 2015-03-19T23:53:00 well on the bright side if there's no more changes to make hpcloud work, that's more time we can spend making the new cloud ;) 2015-03-19T23:53:06 well when I deal with third party ci, I always consider it polite if they turn themselves off 2015-03-19T23:53:23 rather than forcing me to shut them down, as I am slow to turn them back on 2015-03-19T23:53:38 anteaya: sure, happy to do it (and have done it many times before) 2015-03-19T23:53:43 yup 2015-03-19T23:53:51 for all the clouds too 2015-03-19T23:54:22 so hoping they sort this out 2015-03-19T23:54:26 jeblair: ok - puppet is disabled on nodepool I believe because you have local changes 2015-03-19T23:54:27 Matthew Treinish proposed openstack/requirements: Bump tempest-lib min version https://review.openstack.org/166044 2015-03-19T23:54:34 *** mtanino has quit IRC 2015-03-19T23:54:50 mordred: jeblair probably the change to nodepool to do the requests at the rate limit? 2015-03-19T23:55:31 clarkb, jeblair: I applied the quota changes to nodepool.yaml by hand 2015-03-19T23:55:56 clarkb: it's been a while - is there a better way to get nodepool to pick up nodepool.yaml changes? 2015-03-19T23:56:22 mordred: no, next time through the run loop it will get them 2015-03-19T23:56:24 mordred: it automagically sees them 2015-03-19T23:56:27 clarkb: awesome 2015-03-19T23:56:28 thanks 2015-03-19T23:58:45 mordred: maybe we should be looking at spreading our across east and west? can we ask about using west again? 2015-03-19T23:59:07 *** garyh has joined #openstack-infra 2015-03-19T23:59:11 clarkb: I will ask about that post-crisis 2015-03-19T23:59:21 s/our/out/