| clarkb | keystoneclient deprecated their cli interface so the package is still isntalled but not the command. You are supposed to use openstackclient now | 00:00 |
|---|---|---|
| osnexus | Ah, thank you | 00:00 |
| *** rossella_s has joined #openstack-infra | 00:00 | |
| osnexus | command not found | 00:00 |
| osnexus | openstackclient service-create | 00:01 |
| osnexus | something like that right? | 00:01 |
| *** EricGonczer_ has joined #openstack-infra | 00:01 | |
| *** salv-orlando has joined #openstack-infra | 00:01 | |
| *** Swami has quit IRC | 00:02 | |
| *** rbuzatu has joined #openstack-infra | 00:02 | |
| clarkb | just openstack whatever | 00:02 |
| *** ihrachys has quit IRC | 00:02 | |
| *** thorst_ has joined #openstack-infra | 00:02 | |
| clarkb | I don't know what the keystone commands are but there is built in help | 00:03 |
| clarkb | eg openstack help | 00:03 |
| osnexus | aight thanks | 00:03 |
| *** asettle has quit IRC | 00:05 | |
| *** salv-orlando has quit IRC | 00:06 | |
| *** rbuzatu has quit IRC | 00:07 | |
| ianw | pabelanger: ok, so that host is shut-off ... i'm not sure how devstack managed that. kernel oops / watchdog? | 00:07 |
| clarkb | hrm why does the layout job fail for me but tox -ezuul works locally? | 00:09 |
| fungi | ianw: i mean, you can get the server into that state with `poweroff` or `halt -p` | 00:09 |
| fungi | though i have doubts devstack would try to do such a thing | 00:09 |
| anteaya | clarkb: I don't know | 00:09 |
| anteaya | clarkb: but you have stevedore installed successfully it sounds like | 00:10 |
| *** thorst_ has quit IRC | 00:10 | |
| fungi | clarkb: make sure you totally blow away any stale .test directory in your checkout | 00:10 |
| ianw | fungi: yeah, it's an odd state ... rebooting and will see if logs help... | 00:10 |
| clarkb | fungi: ya I am testing that now. | 00:10 |
| *** elo has joined #openstack-infra | 00:10 | |
| clarkb | ran a git clean -xfd for good measure | 00:10 |
| fungi | clarkb: right, though that won't remove directories with git clones under them | 00:11 |
| clarkb | fungi: it won't? | 00:11 |
| clarkb | I thought the -f was for that | 00:11 |
| fungi | clarkb: nope | 00:11 |
| fungi | as far as i can find, there is no git clean option that will force it to remove an untracked directory that contains another git clone | 00:11 |
| clarkb | huh it wont | 00:11 |
| clarkb | neat TIL | 00:12 |
| fungi | and i have in the past gotten strange results from `tox -e zuul` et al which boiled down to stale checkouts of repos under .test | 00:12 |
| clarkb | not sure I am a fan of the -f that doesn't -f | 00:13 |
| *** rbuzatu has joined #openstack-infra | 00:13 | |
| fungi | --totally-f | 00:14 |
| fungi | (if only) | 00:14 |
| ianw | greghaynes: one thing i notice is that greghaynes is getting called for every tap/qr that neutron is fiddling with and failing (IOError: [Errno 2] No such file or directory: '/sys/class/net/qr/54208b40/f7/addr_assign_type') | 00:14 |
| ianw | bah, replace that second greghaynes with glean :) | 00:15 |
| ianw | pabelanger: and the other thing that i notice is http://paste.openstack.org/show/542719/, that big bunch-o-null suggests that it was not shut down nicely | 00:17 |
| *** rbuzatu has quit IRC | 00:17 | |
| greghaynes | ianw: Ya, theres a glean bug. It should be fixed by the needed enhancement for glean to only assert networking once on a host (rather than once per interface) | 00:18 |
| greghaynes | once someone writes it... | 00:18 |
| jesusaur | clarkb: fungi: you need to give git-clean another -f for best effect | 00:18 |
| ianw | greghaynes: hmm, that sounds like something systemd can do, if you know how | 00:19 |
| *** xarses has quit IRC | 00:19 | |
| greghaynes | ianw: I mean once per host ever, even on reboot - basically we need to cache the machine-id | 00:19 |
| greghaynes | I think systemd cant store that kind of state beyond a reboot? | 00:20 |
| greghaynes | actually, I probably shouldnt say things systemd cant do | 00:20 |
| greghaynes | since that list is forever shrinking | 00:22 |
| ianw | greghaynes: well, you simply need to write bindings for a d-bus connector to reflect an api that marshals arguments into an RPC over a fifo to a unix socket | 00:22 |
| ianw | or maybe we could drop a file in /var/lib :) | 00:22 |
| greghaynes | nice | 00:22 |
| fungi | greghaynes: it slices, it dices, it makes julienne fries of service logs right before your eyes | 00:23 |
| greghaynes | :) | 00:24 |
| *** rbuzatu has joined #openstack-infra | 00:24 | |
| clarkb | fungi: I can indeed reproduce the fails if I rm -rf .test (that -f works properly) | 00:25 |
| fungi | indeed. a proper -f | 00:25 |
| *** sarob has quit IRC | 00:26 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: add coverage, translation and releasenotes https://review.openstack.org/346274 | 00:26 |
| *** sputnik13 has quit IRC | 00:26 | |
| *** david-lyle_ has joined #openstack-infra | 00:28 | |
| *** rbuzatu has quit IRC | 00:29 | |
| ianw | pabelanger: ok, so i'll try a run and see if i can capture the console log for it. if that doesn't work, my next thing would be to try and send kernel console over net with http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n409 | 00:31 |
| *** roxanaghe has quit IRC | 00:31 | |
| *** rbuzatu has joined #openstack-infra | 00:35 | |
| pabelanger | ianw: sounds good | 00:36 |
| *** pt_15 has quit IRC | 00:36 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Use ovs-interface-nondefault instead of ovs-native job https://review.openstack.org/338944 | 00:36 |
| pabelanger | just waiting for centos-7 DIB to finish up so I can start uploads | 00:36 |
| anteaya | clarkb: how are your tests installing stevedore properly and my do not? | 00:38 |
| clarkb | anteaya: I don't know, I think it just installs into the virtualenv | 00:39 |
| clarkb | actually no there is no stevedor in the pip freeze output | 00:39 |
| anteaya | huh | 00:40 |
| anteaya | thanks for checking that | 00:40 |
| anteaya | I'll keep trying | 00:40 |
| clarkb | anteaya: I am just running tox -ezuul | 00:40 |
| clarkb | is that what is breaking for you? | 00:40 |
| * anteaya has first executed rm -rf .test | 00:40 | |
| anteaya | that was last night | 00:40 |
| anteaya | trying again now | 00:40 |
| *** rbuzatu has quit IRC | 00:41 | |
| *** kushal has joined #openstack-infra | 00:41 | |
| *** thorst_ has joined #openstack-infra | 00:43 | |
| *** rdx_matt has joined #openstack-infra | 00:43 | |
| *** rdx_matt has quit IRC | 00:43 | |
| *** thorst_ has quit IRC | 00:44 | |
| *** rdx_matt has joined #openstack-infra | 00:44 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run integrated gate on xenial https://review.openstack.org/348078 | 00:44 |
| clarkb | now that actually passes testing | 00:44 |
| *** thorst_ has joined #openstack-infra | 00:44 | |
| clarkb | and time to find dinner | 00:45 |
| *** rbuzatu_ has joined #openstack-infra | 00:47 | |
| anteaya | enjoy dinner | 00:47 |
| *** pvaneck has quit IRC | 00:47 | |
| *** xarses has joined #openstack-infra | 00:50 | |
| anteaya | and my zuul tests also are happy and noone wanted stevedore now apparently | 00:51 |
| *** rbuzatu_ has quit IRC | 00:51 | |
| anteaya | yay rm -rf | 00:51 |
| *** thorst_ has quit IRC | 00:52 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Fix CI status report to show all job results on 1 line https://review.openstack.org/348093 | 00:53 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Fix CI status report to show all job results on 1 line https://review.openstack.org/348093 | 00:54 |
| *** mtanino has joined #openstack-infra | 00:55 | |
| bodepd | is this the right channle for glean questions? | 00:56 |
| bodepd | I actually have some questions about this patch: https://review.openstack.org/#/c/279818/ | 00:57 |
| bodepd | I'll just comment inline | 00:57 |
| *** rbuzatu has joined #openstack-infra | 00:57 | |
| *** rbuzatu has joined #openstack-infra | 00:58 | |
| *** hurgleburgler has quit IRC | 00:59 | |
| *** tqtran has quit IRC | 00:59 | |
| *** apetrich has quit IRC | 01:00 | |
| *** apetrich has joined #openstack-infra | 01:01 | |
| *** gyee has quit IRC | 01:03 | |
| *** chlong has joined #openstack-infra | 01:06 | |
| anteaya | yes this is the right channel to ask glean questions | 01:07 |
| anteaya | I personally don't have any glean answers | 01:07 |
| anteaya | so be patient and hopefully someone who does will wander by | 01:07 |
| bodepd | I just commented inline on the patch that I had a question about, I think I got it all sorted out, it turns out all of my answers were on the internet | 01:08 |
| *** gouthamr has joined #openstack-infra | 01:08 | |
| bodepd | (well, almost all of them) | 01:09 |
| *** yamamoto_ has joined #openstack-infra | 01:12 | |
| *** david-lyle_ has quit IRC | 01:14 | |
| *** rdx_matt has quit IRC | 01:17 | |
| *** Sukhdev has quit IRC | 01:18 | |
| bkero | funny how that happens | 01:20 |
| bkero | tucked away in an achived mailing list post in a locked filing cabinet in a disused lavatory in the basement with a sign on the door that says beware of the leopard. | 01:21 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Fix CI status report to show all job results on 1 line https://review.openstack.org/348093 | 01:22 |
| *** Apoorva has joined #openstack-infra | 01:24 | |
| *** esberglu has joined #openstack-infra | 01:24 | |
| *** zhurong has joined #openstack-infra | 01:25 | |
| *** thorst_ has joined #openstack-infra | 01:25 | |
| *** sdague has quit IRC | 01:29 | |
| *** shashank_hegde has quit IRC | 01:30 | |
| *** roxanaghe has joined #openstack-infra | 01:32 | |
| *** kien-ha has joined #openstack-infra | 01:33 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for enabling empty results https://review.openstack.org/244588 | 01:33 |
| *** thorst_ has quit IRC | 01:34 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for enabling empty results https://review.openstack.org/244588 | 01:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Use "arp -n" when finding gateway MAC for netconsole https://review.openstack.org/348100 | 01:36 |
| *** Apoorva has quit IRC | 01:36 | |
| ianw | pabelanger: ^ got it running, capturing the kernel console ... see what happens | 01:36 |
| *** roxanaghe has quit IRC | 01:36 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for CodeCover plugin https://review.openstack.org/346974 | 01:39 |
| *** esberglu has quit IRC | 01:45 | |
| *** yanyanhu has joined #openstack-infra | 01:48 | |
| *** mriedem has quit IRC | 01:49 | |
| *** zz_dimtruck is now known as dimtruck | 01:52 | |
| *** esberglu has joined #openstack-infra | 01:55 | |
| *** nwkarsten has joined #openstack-infra | 01:56 | |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Add integration test job for Senlin https://review.openstack.org/347160 | 01:56 |
| *** EricGonczer_ has quit IRC | 01:57 | |
| *** EricGonczer_ has joined #openstack-infra | 01:59 | |
| *** woodster_ has quit IRC | 01:59 | |
| *** ddieterly has quit IRC | 01:59 | |
| *** coolsvap has joined #openstack-infra | 01:59 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: add coverage, translation and releasenotes https://review.openstack.org/346274 | 02:00 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Delivery Pipeline plugin to use convert xml https://review.openstack.org/346988 | 02:01 |
| *** esberglu has quit IRC | 02:02 | |
| ianw | pabelanger: hmm, shut itself off again. got nothing on the net console (no oops or anything) ... open to ideas :) | 02:03 |
| *** thorst_ has joined #openstack-infra | 02:04 | |
| *** baoli has quit IRC | 02:06 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for FTP-Publisher plugin https://review.openstack.org/347589 | 02:09 |
| *** amotoki has quit IRC | 02:10 | |
| *** thorst_ has quit IRC | 02:10 | |
| *** tphummel has quit IRC | 02:10 | |
| *** yamahata has quit IRC | 02:13 | |
| *** EricGonczer_ has quit IRC | 02:13 | |
| *** esberglu has joined #openstack-infra | 02:13 | |
| *** ajmiller has joined #openstack-infra | 02:13 | |
| *** rdx_matt has joined #openstack-infra | 02:15 | |
| *** dimtruck is now known as zz_dimtruck | 02:15 | |
| *** mmandel has joined #openstack-infra | 02:15 | |
| *** itisha has quit IRC | 02:20 | |
| *** ajmiller has quit IRC | 02:22 | |
| *** esberglu has quit IRC | 02:24 | |
| *** ayoung has quit IRC | 02:25 | |
| *** ayoung has joined #openstack-infra | 02:27 | |
| *** rwsu has quit IRC | 02:28 | |
| *** yamamoto_ has quit IRC | 02:31 | |
| *** amotoki has joined #openstack-infra | 02:32 | |
| *** ayoung has quit IRC | 02:32 | |
| *** sflanigan is now known as sflanigan_afk | 02:33 | |
| *** roxanaghe has joined #openstack-infra | 02:33 | |
| *** nwkarsten has quit IRC | 02:34 | |
| *** nwkarsten has joined #openstack-infra | 02:35 | |
| *** amotoki has quit IRC | 02:36 | |
| *** sarob has joined #openstack-infra | 02:36 | |
| *** ayoung has joined #openstack-infra | 02:36 | |
| *** eeiden has quit IRC | 02:36 | |
| *** skipp_ has joined #openstack-infra | 02:37 | |
| *** roxanaghe has quit IRC | 02:37 | |
| *** stew925 has joined #openstack-infra | 02:37 | |
| *** stewie925 has quit IRC | 02:37 | |
| *** skipp has quit IRC | 02:38 | |
| *** wcriswelc has quit IRC | 02:38 | |
| *** nwkarsten has quit IRC | 02:39 | |
| *** rwsu has joined #openstack-infra | 02:40 | |
| *** thorst_ has joined #openstack-infra | 02:40 | |
| *** amotoki has joined #openstack-infra | 02:40 | |
| *** sarob has quit IRC | 02:41 | |
| *** thorst_ has quit IRC | 02:41 | |
| *** sflanigan_afk is now known as sflanigan | 02:46 | |
| *** yuanying has quit IRC | 02:47 | |
| *** rdx_matt has quit IRC | 02:48 | |
| *** eeiden has joined #openstack-infra | 02:51 | |
| *** mmandel has quit IRC | 02:55 | |
| *** jrist has quit IRC | 02:55 | |
| *** mmandel has joined #openstack-infra | 02:55 | |
| *** gouthamr_ has joined #openstack-infra | 02:55 | |
| *** zhurong has quit IRC | 02:56 | |
| *** tqtran has joined #openstack-infra | 02:57 | |
| *** rlandy has quit IRC | 02:58 | |
| *** gouthamr has quit IRC | 02:59 | |
| *** amotoki has quit IRC | 03:01 | |
| *** tqtran has quit IRC | 03:01 | |
| *** mmandel has quit IRC | 03:03 | |
| *** zhurong has joined #openstack-infra | 03:03 | |
| fungi | ianw: only happening on rax regions, or everywhere? | 03:03 |
| *** kien-ha has quit IRC | 03:04 | |
| openstackgerrit | Xiang Chen proposed openstack/diskimage-builder: check APT_SOURCES earlier https://review.openstack.org/346605 | 03:09 |
| *** amotoki has joined #openstack-infra | 03:10 | |
| *** gouthamr_ has quit IRC | 03:10 | |
| *** esberglu has joined #openstack-infra | 03:11 | |
| *** gouthamr has joined #openstack-infra | 03:14 | |
| *** ayoung has quit IRC | 03:14 | |
| *** sdake has joined #openstack-infra | 03:15 | |
| *** tonytan4ever has joined #openstack-infra | 03:17 | |
| *** salv-orlando has joined #openstack-infra | 03:20 | |
| *** nwkarsten has joined #openstack-infra | 03:21 | |
| *** sputnik13 has joined #openstack-infra | 03:22 | |
| *** sdake_ has joined #openstack-infra | 03:25 | |
| *** Liuqing has joined #openstack-infra | 03:26 | |
| *** sdake has quit IRC | 03:27 | |
| *** salv-orlando has quit IRC | 03:30 | |
| *** sputnik13 has quit IRC | 03:33 | |
| *** esberglu has quit IRC | 03:34 | |
| ianw | fungi: haven't got it going on a non-rax yet | 03:37 |
| *** mtanino has quit IRC | 03:39 | |
| ianw | all i can think is that maybe a shutdown command intended for a cirros guest has leaked out? far fetched... | 03:40 |
| *** sandanar has joined #openstack-infra | 03:40 | |
| *** nwkarsten has quit IRC | 03:42 | |
| *** rdx_matt has joined #openstack-infra | 03:44 | |
| *** david-lyle_ has joined #openstack-infra | 03:45 | |
| *** sdake_ has quit IRC | 03:46 | |
| *** sdake has joined #openstack-infra | 03:46 | |
| *** yuanying has joined #openstack-infra | 03:50 | |
| *** tphummel has joined #openstack-infra | 03:50 | |
| *** david-lyle_ is now known as david-lyle | 03:54 | |
| *** liusheng has joined #openstack-infra | 03:58 | |
| *** nwkarsten has joined #openstack-infra | 03:59 | |
| *** julim has quit IRC | 03:59 | |
| *** julim has joined #openstack-infra | 04:02 | |
| *** julim has quit IRC | 04:02 | |
| *** links has joined #openstack-infra | 04:03 | |
| *** david-lyle has quit IRC | 04:07 | |
| *** tphummel has quit IRC | 04:08 | |
| *** mctaylor has joined #openstack-infra | 04:11 | |
| *** david-lyle has joined #openstack-infra | 04:11 | |
| *** sree has joined #openstack-infra | 04:11 | |
| *** mctaylor has quit IRC | 04:12 | |
| *** mctaylor has joined #openstack-infra | 04:12 | |
| *** Sukhdev has joined #openstack-infra | 04:15 | |
| *** rdx_matt has quit IRC | 04:18 | |
| *** IlyaG has joined #openstack-infra | 04:20 | |
| *** tphummel has joined #openstack-infra | 04:27 | |
| *** harlowja_at_home has joined #openstack-infra | 04:30 | |
| *** psachin has joined #openstack-infra | 04:30 | |
| *** gouthamr has quit IRC | 04:32 | |
| *** javeriak has joined #openstack-infra | 04:36 | |
| *** javeriak has quit IRC | 04:38 | |
| *** pgadiya has joined #openstack-infra | 04:43 | |
| *** javeriak has joined #openstack-infra | 04:48 | |
| *** shashank_hegde has joined #openstack-infra | 04:51 | |
| *** javeriak has quit IRC | 04:57 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Use "arp -n" when finding gateway MAC for netconsole https://review.openstack.org/348100 | 04:57 |
| *** Sukhdev has quit IRC | 04:58 | |
| *** IlyaG has quit IRC | 04:58 | |
| *** csomerville has joined #openstack-infra | 04:59 | |
| *** cody-somerville has quit IRC | 04:59 | |
| *** tqtran has joined #openstack-infra | 04:59 | |
| *** bhavik has joined #openstack-infra | 05:01 | |
| *** Apoorva has joined #openstack-infra | 05:03 | |
| *** tqtran has quit IRC | 05:04 | |
| *** Apoorva has quit IRC | 05:06 | |
| *** asettle has joined #openstack-infra | 05:10 | |
| *** shashank_hegde has quit IRC | 05:11 | |
| clarkb | ianw: is it crashing during tempest? | 05:12 |
| *** asettle has quit IRC | 05:14 | |
| *** rdx_matt has joined #openstack-infra | 05:15 | |
| *** sandanar has quit IRC | 05:16 | |
| *** sflanigan is now known as sflanigan_afk | 05:19 | |
| *** pt_15 has joined #openstack-infra | 05:20 | |
| *** ramishra_ has quit IRC | 05:22 | |
| *** watersoul has quit IRC | 05:23 | |
| *** watersoul has joined #openstack-infra | 05:23 | |
| *** ramishra has joined #openstack-infra | 05:24 | |
| openstackgerrit | garyk proposed openstack-infra/project-config: Networking-l2gw: enable python 3 voting https://review.openstack.org/348123 | 05:24 |
| *** bhavik has quit IRC | 05:24 | |
| *** javeriak has joined #openstack-infra | 05:25 | |
| *** javeriak has joined #openstack-infra | 05:25 | |
| *** oanson has joined #openstack-infra | 05:27 | |
| *** nwkarsten has quit IRC | 05:28 | |
| *** rdx_matt_ has joined #openstack-infra | 05:28 | |
| *** watersoul has joined #openstack-infra | 05:28 | |
| *** rdx_matt has quit IRC | 05:32 | |
| *** abregman_ has joined #openstack-infra | 05:33 | |
| *** watersoul has joined #openstack-infra | 05:34 | |
| *** bhavik has joined #openstack-infra | 05:35 | |
| *** rbuzatu has quit IRC | 05:35 | |
| *** javeriak_ has joined #openstack-infra | 05:37 | |
| *** javeriak has quit IRC | 05:38 | |
| *** watersoul has joined #openstack-infra | 05:39 | |
| *** tphummel has quit IRC | 05:40 | |
| *** gongysh has joined #openstack-infra | 05:41 | |
| *** rbuzatu has joined #openstack-infra | 05:42 | |
| *** watersoul has joined #openstack-infra | 05:45 | |
| openstackgerrit | garyk proposed openstack-infra/project-config: Networking-l2gw: enable python 3 voting https://review.openstack.org/348123 | 05:45 |
| *** rbuzatu has quit IRC | 05:46 | |
| *** rdx_matt_ has quit IRC | 05:48 | |
| *** rbuzatu has joined #openstack-infra | 05:48 | |
| *** amotoki_ has joined #openstack-infra | 05:48 | |
| *** sarob has joined #openstack-infra | 05:49 | |
| *** rbuzatu has quit IRC | 05:49 | |
| *** watersoul has joined #openstack-infra | 05:50 | |
| *** amotoki has quit IRC | 05:51 | |
| *** sarob has quit IRC | 05:53 | |
| ianw | clarkb: yep | 05:54 |
| *** yamahata has joined #openstack-infra | 05:55 | |
| *** rbuzatu has joined #openstack-infra | 05:57 | |
| *** gongysh has quit IRC | 05:58 | |
| *** rbuzatu_ has joined #openstack-infra | 05:59 | |
| *** AJaeger has quit IRC | 06:01 | |
| *** rbuzatu has quit IRC | 06:02 | |
| *** david-lyle has quit IRC | 06:02 | |
| *** AJaeger has joined #openstack-infra | 06:03 | |
| *** asettle has joined #openstack-infra | 06:03 | |
| *** mctaylor has quit IRC | 06:03 | |
| *** rbuzatu_ has quit IRC | 06:03 | |
| *** aeng has quit IRC | 06:06 | |
| *** asettle has quit IRC | 06:08 | |
| ianw | pabelanger: hmm, i can't replicate this dropping off locally ATM (bit of a hack effort, need to deploy a full node). giving up for now, will think on it a bit | 06:08 |
| *** Srinu has quit IRC | 06:08 | |
| *** rbuzatu has joined #openstack-infra | 06:10 | |
| *** AJaeger has quit IRC | 06:11 | |
| *** AJaeger has joined #openstack-infra | 06:11 | |
| *** roxanaghe has joined #openstack-infra | 06:12 | |
| *** javeriak_ has quit IRC | 06:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Test api-ref theming with openstackdocstheme https://review.openstack.org/345441 | 06:14 |
| *** rbuzatu has quit IRC | 06:15 | |
| *** harlowja_at_home has quit IRC | 06:16 | |
| *** roxanaghe has quit IRC | 06:16 | |
| *** Srinu has joined #openstack-infra | 06:17 | |
| *** pcaruana has joined #openstack-infra | 06:17 | |
| *** gongysh has joined #openstack-infra | 06:19 | |
| *** rbuzatu has joined #openstack-infra | 06:22 | |
| openstackgerrit | Merged openstack-infra/project-config: [sahara-dashboard] add py27dj110 testenv https://review.openstack.org/346457 | 06:22 |
| *** sflanigan_afk is now known as sflanigan | 06:23 | |
| *** rbuzatu has quit IRC | 06:26 | |
| *** yamamoto_ has joined #openstack-infra | 06:27 | |
| *** Daisy has joined #openstack-infra | 06:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix Monasca Log API tempest gate config https://review.openstack.org/347813 | 06:31 |
| *** rbuzatu has joined #openstack-infra | 06:31 | |
| *** gongysh has quit IRC | 06:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: setup voting for py35 jobs in sahara projects https://review.openstack.org/347291 | 06:32 |
| *** rbuzatu_ has joined #openstack-infra | 06:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/347683 | 06:33 |
| *** sandanar has joined #openstack-infra | 06:33 | |
| *** rbuzatu has quit IRC | 06:36 | |
| *** rbuzatu_ has quit IRC | 06:37 | |
| *** gildub has quit IRC | 06:38 | |
| *** tonytan4ever has quit IRC | 06:38 | |
| *** rbuzatu has joined #openstack-infra | 06:39 | |
| *** abregman_ is now known as abregman | 06:41 | |
| *** hferenc has quit IRC | 06:41 | |
| *** Daisy has quit IRC | 06:42 | |
| *** tesseract- has joined #openstack-infra | 06:42 | |
| *** sdake has quit IRC | 06:42 | |
| *** rdx_matt has joined #openstack-infra | 06:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:46 | |
| *** salv-orlando has joined #openstack-infra | 06:48 | |
| *** Daisy has joined #openstack-infra | 06:49 | |
| openstackgerrit | Merged openstack-infra/project-config: fix the issue that networking-sfc repo is not installed in $BASE/new https://review.openstack.org/347979 | 06:52 |
| openstackgerrit | Merged openstack-infra/project-config: Switch all xstatic projects to the correct publish job https://review.openstack.org/347550 | 06:53 |
| AJaeger | project-config cores, could you review https://review.openstack.org/#/c/347663/ - to help with testing constraints, please? | 06:54 |
| *** rdx_matt has quit IRC | 06:55 | |
| *** rdx_matt has joined #openstack-infra | 06:55 | |
| *** jlanoux has joined #openstack-infra | 06:56 | |
| *** flepied1 has joined #openstack-infra | 06:57 | |
| *** asettle has joined #openstack-infra | 06:57 | |
| *** javeriak has joined #openstack-infra | 06:57 | |
| *** jlanoux has quit IRC | 06:58 | |
| *** jlanoux has joined #openstack-infra | 06:59 | |
| openstackgerrit | Merged openstack-infra/project-config: setup voting for py35 jobs in sahara projects https://review.openstack.org/347291 | 06:59 |
| *** nherciu has joined #openstack-infra | 07:00 | |
| *** flepied has quit IRC | 07:01 | |
| *** asettle has quit IRC | 07:01 | |
| *** Daisy has quit IRC | 07:02 | |
| *** jtomasek has joined #openstack-infra | 07:02 | |
| *** Daisy has joined #openstack-infra | 07:02 | |
| *** atom_ has joined #openstack-infra | 07:06 | |
| *** Daisy has quit IRC | 07:07 | |
| yolanda | good morning | 07:08 |
| Srinu | hi | 07:10 |
| Srinu | i have two slaves added to jenkin master and if one of the job finished then nodepool deleting the completed job slave and also deleting the slave.jar process running on the different slave on which tests are going on. did anyone face this issue | 07:12 |
| *** Daisy has joined #openstack-infra | 07:12 | |
| *** roxanaghe has joined #openstack-infra | 07:13 | |
| *** permalac has joined #openstack-infra | 07:13 | |
| *** rbuzatu has quit IRC | 07:13 | |
| *** rbuzatu has joined #openstack-infra | 07:14 | |
| AJaeger | morning, yolanda ! | 07:14 |
| *** Na3iL has joined #openstack-infra | 07:15 | |
| *** flepied1 is now known as flepied | 07:15 | |
| *** ifarkas_away is now known as ifarkas | 07:15 | |
| *** rbuzatu has quit IRC | 07:17 | |
| *** roxanaghe has quit IRC | 07:17 | |
| Srinu | good morning all | 07:19 |
| *** kzaitsev_mb has quit IRC | 07:20 | |
| *** ihrachys has joined #openstack-infra | 07:21 | |
| *** nmagnezi has joined #openstack-infra | 07:22 | |
| *** gongysh has joined #openstack-infra | 07:25 | |
| *** rbuzatu has joined #openstack-infra | 07:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:25 | |
| AJaeger | yolanda, could you review https://review.openstack.org/#/c/347663/ - to help with testing constraints , please? | 07:27 |
| *** rbuzatu_ has joined #openstack-infra | 07:27 | |
| *** tonytan4ever has joined #openstack-infra | 07:29 | |
| *** rbuzatu has quit IRC | 07:29 | |
| *** mikelk has joined #openstack-infra | 07:30 | |
| *** kzaitsev_mb has quit IRC | 07:31 | |
| *** rbuzatu_ has quit IRC | 07:31 | |
| *** asettle has joined #openstack-infra | 07:33 | |
| *** Daisy has quit IRC | 07:34 | |
| *** lezbar has joined #openstack-infra | 07:34 | |
| *** abregman_ has joined #openstack-infra | 07:36 | |
| *** abregman has quit IRC | 07:37 | |
| *** rbuzatu has joined #openstack-infra | 07:38 | |
| *** asettle has quit IRC | 07:38 | |
| *** ihrachys has quit IRC | 07:40 | |
| *** vincentll has joined #openstack-infra | 07:41 | |
| *** mikelk has quit IRC | 07:42 | |
| *** rbuzatu has quit IRC | 07:42 | |
| *** Daisy has joined #openstack-infra | 07:43 | |
| *** pt_15 has quit IRC | 07:46 | |
| *** acruz has joined #openstack-infra | 07:46 | |
| *** rbuzatu has joined #openstack-infra | 07:46 | |
| *** chlong has quit IRC | 07:47 | |
| *** sarob has joined #openstack-infra | 07:49 | |
| *** armax has quit IRC | 07:50 | |
| *** hashar has joined #openstack-infra | 07:51 | |
| *** mikelk has joined #openstack-infra | 07:51 | |
| *** andymaier has joined #openstack-infra | 07:51 | |
| *** abregman_ is now known as abregman_|mtg | 07:54 | |
| *** sarob has quit IRC | 07:54 | |
| *** jpich has joined #openstack-infra | 07:56 | |
| *** asettle has joined #openstack-infra | 07:56 | |
| ianw | AJaeger: swap you for a review of https://review.openstack.org/#/c/345816/ :) | 07:57 |
| *** kzaitsev_mb has joined #openstack-infra | 07:57 | |
| *** esikachev has joined #openstack-infra | 07:58 | |
| *** rossella_s has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** rossella_s has joined #openstack-infra | 08:00 | |
| *** shardy has joined #openstack-infra | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** dizquierdo has joined #openstack-infra | 08:00 | |
| *** kushal has quit IRC | 08:00 | |
| *** asettle has quit IRC | 08:01 | |
| *** e0ne has joined #openstack-infra | 08:01 | |
| *** matrohon has joined #openstack-infra | 08:01 | |
| *** salv-orlando has quit IRC | 08:01 | |
| *** abregman_|mtg has quit IRC | 08:03 | |
| *** andymaier has quit IRC | 08:05 | |
| *** kzaitsev_mb has quit IRC | 08:08 | |
| *** BobBall_AWOL is now known as BobBall | 08:10 | |
| *** hparekh has quit IRC | 08:11 | |
| *** derekh has joined #openstack-infra | 08:14 | |
| *** Daisy has quit IRC | 08:14 | |
| *** vincentll has quit IRC | 08:16 | |
| *** Daisy has joined #openstack-infra | 08:16 | |
| *** kushal has joined #openstack-infra | 08:17 | |
| *** vincentll has joined #openstack-infra | 08:17 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:18 | |
| *** dizquierdo has quit IRC | 08:19 | |
| *** lucas-dinner is now known as lucasagomes | 08:21 | |
| *** hparekh has joined #openstack-infra | 08:24 | |
| hashar | ianw: hello good evening! I had a review for Nodepool to let one configure the delay before deleting an instance I have replied to it https://review.openstack.org/#/c/245220/4/nodepool/config.py | 08:26 |
| *** permalac has quit IRC | 08:26 | |
| hashar | if you get some spare time, I am wondering how to express the new delete_delay setting in the yaml layout | 08:26 |
| *** abregman has joined #openstack-infra | 08:26 | |
| *** permalac has joined #openstack-infra | 08:26 | |
| *** hashar has quit IRC | 08:29 | |
| sulo | hi, anyone around to help review this please: https://review.openstack.org/#/c/345453/ | 08:29 |
| *** hashar has joined #openstack-infra | 08:29 | |
| sulo | AJaeger: ^ when you have a min please | 08:30 |
| *** r-mibu has quit IRC | 08:31 | |
| *** kzaitsev_mb has quit IRC | 08:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:32 | |
| *** otherwiseguy has quit IRC | 08:34 | |
| Srinu | asselin_ asselin: hi | 08:36 |
| yolanda | AJaeger, starting with reviews | 08:36 |
| *** mhickey has joined #openstack-infra | 08:39 | |
| *** Liuqing_ has joined #openstack-infra | 08:39 | |
| *** Liuqing has quit IRC | 08:40 | |
| *** r-mibu has joined #openstack-infra | 08:41 | |
| yolanda | AJaeger, approved | 08:42 |
| *** kzaitsev_mb has quit IRC | 08:42 | |
| *** otherwiseguy has joined #openstack-infra | 08:43 | |
| *** ramishra has quit IRC | 08:43 | |
| *** asettle has joined #openstack-infra | 08:44 | |
| *** asettle has quit IRC | 08:44 | |
| *** asettle has joined #openstack-infra | 08:44 | |
| *** ramishra has joined #openstack-infra | 08:44 | |
| ianw | hashar: i guess i don't feel strongly about a [nodepoold] section if the plan is to put more than one thing in there | 08:46 |
| *** rbuzatu has quit IRC | 08:48 | |
| hashar | ianw: there might be a better idea / way to handle it such as a nodepool.ini | 08:49 |
| hashar | zuul uses that for the daemons settings. Might be better than cluttering the layout.yaml | 08:49 |
| *** amitgandhinz has quit IRC | 08:51 | |
| *** tonytan4ever has quit IRC | 08:51 | |
| *** amitgandhinz has joined #openstack-infra | 08:52 | |
| *** acoles_ is now known as acoles | 08:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Add some debug output for release-note test, add further job https://review.openstack.org/347663 | 08:53 |
| *** andymaier has joined #openstack-infra | 08:53 | |
| tlbr | mordred, hi, we need +1 from PTL https://review.openstack.org/307742 https://review.openstack.org/346095 Could you please review it ? | 08:54 |
| *** rbuzatu has joined #openstack-infra | 08:57 | |
| *** kzaitsev_mb has quit IRC | 08:58 | |
| *** rbuzatu_ has joined #openstack-infra | 08:59 | |
| *** sflanigan has quit IRC | 08:59 | |
| *** roxanaghe has joined #openstack-infra | 09:00 | |
| *** rbuzatu has quit IRC | 09:01 | |
| *** rbuzatu_ has quit IRC | 09:02 | |
| *** Daisy has quit IRC | 09:02 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Don't spam postci with deployment time loop https://review.openstack.org/346990 | 09:02 |
| *** rbuzatu has joined #openstack-infra | 09:02 | |
| *** dizquierdo has joined #openstack-infra | 09:02 | |
| *** daemontool has joined #openstack-infra | 09:03 | |
| *** gouthamr has joined #openstack-infra | 09:04 | |
| *** roxanaghe has quit IRC | 09:04 | |
| *** bhavik has quit IRC | 09:04 | |
| *** asettle has quit IRC | 09:05 | |
| *** asettle has joined #openstack-infra | 09:06 | |
| *** javeriak has quit IRC | 09:07 | |
| *** javeriak has joined #openstack-infra | 09:08 | |
| *** bhavik has joined #openstack-infra | 09:09 | |
| *** jordanP has joined #openstack-infra | 09:10 | |
| *** Daisy has joined #openstack-infra | 09:12 | |
| *** claudiub|2 has joined #openstack-infra | 09:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Run Gnocchi MySQL jobs on Ubuntu Xenial https://review.openstack.org/347295 | 09:14 |
| *** e0ne has quit IRC | 09:17 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories https://review.openstack.org/312666 | 09:19 |
| *** mhickey has quit IRC | 09:20 | |
| *** mhickey has joined #openstack-infra | 09:21 | |
| *** permalac has quit IRC | 09:25 | |
| *** electrofelix has joined #openstack-infra | 09:26 | |
| *** javeriak has quit IRC | 09:29 | |
| *** e0ne has joined #openstack-infra | 09:32 | |
| *** cdent has joined #openstack-infra | 09:33 | |
| *** rbuzatu has quit IRC | 09:37 | |
| *** rbuzatu has joined #openstack-infra | 09:38 | |
| *** electrofelix has quit IRC | 09:39 | |
| *** rbuzatu has quit IRC | 09:39 | |
| *** rbuzatu has joined #openstack-infra | 09:43 | |
| *** yamamoto_ has quit IRC | 09:43 | |
| *** claudiub|2 has quit IRC | 09:45 | |
| *** rbuzatu has quit IRC | 09:48 | |
| *** ihrachys has joined #openstack-infra | 09:54 | |
| *** jordanP_ has joined #openstack-infra | 09:56 | |
| *** _degorenko|afk is now known as degorenko | 09:56 | |
| *** yamahata has quit IRC | 09:58 | |
| *** jordanP has quit IRC | 10:00 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** electrofelix has joined #openstack-infra | 10:05 | |
| *** zhurong has quit IRC | 10:07 | |
| *** salv-orlando has joined #openstack-infra | 10:08 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add back rh1 https://review.openstack.org/348212 | 10:09 |
| *** Liuqing_ has quit IRC | 10:09 | |
| *** salv-orlando has quit IRC | 10:15 | |
| *** sree has quit IRC | 10:18 | |
| *** hwcomcn has joined #openstack-infra | 10:21 | |
| *** yanyanhu has quit IRC | 10:23 | |
| *** Daisy has quit IRC | 10:27 | |
| *** Daisy has joined #openstack-infra | 10:28 | |
| *** gildub has joined #openstack-infra | 10:29 | |
| *** tosky has joined #openstack-infra | 10:31 | |
| *** john-davidge has joined #openstack-infra | 10:33 | |
| *** rdx_matt has quit IRC | 10:33 | |
| *** rdx_matt has joined #openstack-infra | 10:34 | |
| *** john-dav_ has joined #openstack-infra | 10:34 | |
| *** john-davidge has quit IRC | 10:37 | |
| *** sdague has joined #openstack-infra | 10:39 | |
| *** ams__ has joined #openstack-infra | 10:42 | |
| *** rbuzatu has joined #openstack-infra | 10:44 | |
| *** yamamoto has joined #openstack-infra | 10:46 | |
| *** roxanaghe has joined #openstack-infra | 10:48 | |
| *** rbuzatu has quit IRC | 10:49 | |
| *** yamamoto has quit IRC | 10:49 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: WIP - Merge same patchset jobs in cistatus https://review.openstack.org/348231 | 10:49 |
| *** roxanaghe has quit IRC | 10:53 | |
| *** dizquierdo has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:54 | |
| *** rbuzatu has joined #openstack-infra | 10:55 | |
| *** thorst has joined #openstack-infra | 10:56 | |
| *** rbuzatu has quit IRC | 10:58 | |
| *** rbuzatu has joined #openstack-infra | 10:59 | |
| *** Daisy has quit IRC | 11:07 | |
| *** Daisy has joined #openstack-infra | 11:08 | |
| *** ramishra has quit IRC | 11:08 | |
| *** jlanoux has quit IRC | 11:08 | |
| *** weshay has joined #openstack-infra | 11:09 | |
| *** dizquierdo has joined #openstack-infra | 11:09 | |
| *** ramishra has joined #openstack-infra | 11:12 | |
| *** Daisy has quit IRC | 11:12 | |
| *** hashar has quit IRC | 11:14 | |
| odyssey4me | AJaeger / yolanda When you have a moment, please review https://review.openstack.org/347952 | 11:16 |
| *** jordan__ has joined #openstack-infra | 11:20 | |
| AJaeger | thanks, yolanda ! | 11:21 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Merge same patchset jobs in cistatus https://review.openstack.org/348231 | 11:21 |
| *** jordanP_ has quit IRC | 11:24 | |
| *** ldnunes has joined #openstack-infra | 11:24 | |
| *** acoles is now known as acoles_ | 11:25 | |
| *** acoles_ is now known as acoles | 11:25 | |
| *** asettle has quit IRC | 11:27 | |
| *** john-dav_ is now known as john-davidge | 11:29 | |
| *** Daisy has joined #openstack-infra | 11:30 | |
| *** dizquierdo has quit IRC | 11:32 | |
| *** jlanoux has joined #openstack-infra | 11:32 | |
| *** EricGonczer_ has joined #openstack-infra | 11:32 | |
| *** Daisy has quit IRC | 11:32 | |
| *** Daisy has joined #openstack-infra | 11:33 | |
| *** salv-orlando has joined #openstack-infra | 11:33 | |
| *** jkilpatr has joined #openstack-infra | 11:33 | |
| odyssey4me | thanks AJaeger | 11:33 |
| *** hashar has joined #openstack-infra | 11:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Tighten the trusty job filter for OpenStack-Ansible properly https://review.openstack.org/347952 | 11:36 |
| *** gongysh has quit IRC | 11:37 | |
| *** hrubi has quit IRC | 11:37 | |
| *** Daisy has quit IRC | 11:37 | |
| *** rhallisey has joined #openstack-infra | 11:37 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add Ironic multinode job https://review.openstack.org/332490 | 11:38 |
| *** hrubi has joined #openstack-infra | 11:39 | |
| *** rdx_matt_ has joined #openstack-infra | 11:39 | |
| *** rdx_matt_ has quit IRC | 11:39 | |
| *** rdx_matt_ has joined #openstack-infra | 11:40 | |
| *** ccamacho is now known as ccamacho|lunch | 11:41 | |
| *** asettle has joined #openstack-infra | 11:41 | |
| *** rdx_matt has quit IRC | 11:42 | |
| *** ramishra has quit IRC | 11:42 | |
| *** EricGonczer_ has quit IRC | 11:43 | |
| azvyagintsev | Hello, is the-are any best-practice example\job for flow: configure and install devstack => run tox test on it ? | 11:44 |
| *** gouthamr has quit IRC | 11:44 | |
| *** rdx_matt_ has quit IRC | 11:44 | |
| *** dizquierdo has joined #openstack-infra | 11:45 | |
| *** EricGonczer_ has joined #openstack-infra | 11:45 | |
| *** sarob has joined #openstack-infra | 11:50 | |
| *** shardy has quit IRC | 11:50 | |
| *** EricGonczer_ has quit IRC | 11:51 | |
| sdague | how long after pypi release do we have access to packages in our mirror? | 11:51 |
| *** tonytan4ever has joined #openstack-infra | 11:53 | |
| mordred | tlbr: +1 | 11:54 |
| *** EricGonczer_ has joined #openstack-infra | 11:54 | |
| *** lucasagomes is now known as lucas-hungry | 11:54 | |
| *** EricGonczer_ has quit IRC | 11:54 | |
| *** rbuzatu has quit IRC | 11:54 | |
| *** sarob has quit IRC | 11:55 | |
| *** chlong has joined #openstack-infra | 11:55 | |
| *** rfolco has joined #openstack-infra | 11:55 | |
| sdague | mordred: any ideas on the mirror timing? | 11:55 |
| *** Liuqing has joined #openstack-infra | 11:56 | |
| *** tonytan4ever has quit IRC | 11:57 | |
| tlbr | mordred, thanks, and this one please https://review.openstack.org/#/c/346095/ | 11:58 |
| *** rbuzatu has joined #openstack-infra | 12:00 | |
| sulo | AJaeger: yolanda: if you have a moment, would appreciate some eyes on this https://review.openstack.org/#/c/345453/ | 12:00 |
| *** ramishra has joined #openstack-infra | 12:02 | |
| *** rbuzatu has quit IRC | 12:04 | |
| *** amotoki_ has quit IRC | 12:06 | |
| *** aysyd has joined #openstack-infra | 12:07 | |
| dims_ | Hi, anyone around to peek at bandersnatch logs? | 12:10 |
| dims_ | oslo.rootwrap 5.0.0 does not seem to have made it to the mirrors | 12:10 |
| *** EricGonczer_ has joined #openstack-infra | 12:10 | |
| dims_ | AJaeger : yolanda : one more request :) ^ | 12:11 |
| *** mtanino has joined #openstack-infra | 12:12 | |
| *** amotoki has joined #openstack-infra | 12:13 | |
| azvyagintsev | mmedvede fungi: folks, is there any best-practice example\job for flow: configure and install devstack => run tox test on it ? (since i don't really think that its good idea to run tox -e in job-template ) | 12:14 |
| *** rbuzatu has joined #openstack-infra | 12:16 | |
| *** maishsk has joined #openstack-infra | 12:17 | |
| *** maishsk has quit IRC | 12:18 | |
| *** maishsk has joined #openstack-infra | 12:18 | |
| AJaeger | dims_: sorry, can't help further, just let me give links: https://pypi.python.org/pypi/oslo.rootwrap has 5.0.0 but http://mirror.iad.rax.openstack.org/pypi/simple/oslo.rootwrap/ missed it. | 12:19 |
| AJaeger | infra-root, could one of you help dims_, please? ^ | 12:20 |
| tlbr | AJaeger, could you please merge this request https://review.openstack.org/307742 ? After mordred will review https://review.openstack.org/#/c/346095/ it'd be really great to merge it too :) | 12:21 |
| AJaeger | sdague: could you import stable translations for nova, please? https://review.openstack.org/#/c/344070/ | 12:21 |
| AJaeger | tlbr: I'm busy rest of the day, will do it once I do the next big review round if nobody beats me to it... | 12:21 |
| AJaeger | tlbr: wait - 307742 I can do now... | 12:22 |
| tlbr | yep :) | 12:22 |
| dims_ | thanks AJaeger | 12:23 |
| AJaeger | tlbr: do you want to import the repo with all branches and tags? | 12:24 |
| AJaeger | http://logs.openstack.org/42/307742/11/check/gate-project-config-gerrit/30a16f7/console.html#_2016-07-27_19_12_34_022555 is the list of them. | 12:24 |
| tlbr | AJaeger, yes, it would be really nice | 12:24 |
| AJaeger | tlbr: deletion of tags and branches is expensive, so please don't ask in a few months for it. Now is the time to cleanup. | 12:24 |
| *** dizquierdo has quit IRC | 12:24 | |
| *** yaume has joined #openstack-infra | 12:25 | |
| AJaeger | tlbr: we import with all of them by default, just want your confirmation on it to go ahead as is. | 12:25 |
| *** EricGonc_ has joined #openstack-infra | 12:25 | |
| tlbr | AJaeger, ok, I understand that | 12:25 |
| *** EricGonczer_ has quit IRC | 12:25 | |
| AJaeger | tlbr: +2A - and now I'm off again... | 12:26 |
| *** dizquierdo has joined #openstack-infra | 12:26 | |
| tlbr | AJaeger, thank you! | 12:26 |
| *** gildub has quit IRC | 12:26 | |
| *** sarob has joined #openstack-infra | 12:27 | |
| russellb | i just used telnet to watch the log for a running job for the first time <3 <3 <3 love it | 12:30 |
| *** nmagnezi_ has joined #openstack-infra | 12:31 | |
| *** nmagnezi has quit IRC | 12:31 | |
| *** hwcomcn has quit IRC | 12:31 | |
| *** sarob has quit IRC | 12:32 | |
| *** esikachev has quit IRC | 12:33 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 12:33 |
| *** rbuzatu has quit IRC | 12:33 | |
| *** tpsilva has joined #openstack-infra | 12:34 | |
| *** amotoki has quit IRC | 12:34 | |
| *** salv-orlando has quit IRC | 12:36 | |
| *** roxanaghe has joined #openstack-infra | 12:36 | |
| *** EricGonczer_ has joined #openstack-infra | 12:38 | |
| *** kgiusti has joined #openstack-infra | 12:38 | |
| *** amotoki has joined #openstack-infra | 12:39 | |
| *** EricGonc_ has quit IRC | 12:39 | |
| *** links has quit IRC | 12:40 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:40 | |
| *** roxanaghe has quit IRC | 12:41 | |
| *** julim has joined #openstack-infra | 12:43 | |
| *** apetrich has quit IRC | 12:43 | |
| *** chem is now known as chem|lunch | 12:43 | |
| *** trown|outtypewww is now known as trown | 12:43 | |
| *** apetrich has joined #openstack-infra | 12:43 | |
| *** deadnull has joined #openstack-infra | 12:43 | |
| *** gordc has joined #openstack-infra | 12:46 | |
| *** amotoki has quit IRC | 12:47 | |
| *** kushal has quit IRC | 12:47 | |
| *** psilvad has joined #openstack-infra | 12:48 | |
| *** esikachev has joined #openstack-infra | 12:49 | |
| *** thiagop has joined #openstack-infra | 12:50 | |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Add swift job tox-func-in-process-encryption https://review.openstack.org/348292 | 12:50 |
| *** rbuzatu has joined #openstack-infra | 12:51 | |
| *** baoli has joined #openstack-infra | 12:53 | |
| *** baoli_ has joined #openstack-infra | 12:54 | |
| *** ccamacho|lunch is now known as ccamacho | 12:55 | |
| *** kushal has joined #openstack-infra | 12:55 | |
| *** woodster_ has joined #openstack-infra | 12:55 | |
| *** jsavak has joined #openstack-infra | 12:56 | |
| *** pradk has quit IRC | 12:58 | |
| *** baoli has quit IRC | 12:58 | |
| *** amitgandhinz has quit IRC | 12:59 | |
| *** amitgandhinz has joined #openstack-infra | 13:00 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio https://review.openstack.org/342481 | 13:02 |
| *** tqtran has joined #openstack-infra | 13:02 | |
| *** lucas-hungry is now known as lucasagomes | 13:03 | |
| *** rdx_matt has joined #openstack-infra | 13:03 | |
| mtreinish | yolanda: on https://review.openstack.org/#/c/347924 it's defining 2 ports to listen on. You can configure mosquitto to listen on as many ports as you want | 13:06 |
| *** edmondsw has joined #openstack-infra | 13:07 | |
| *** tqtran has quit IRC | 13:07 | |
| *** ramishra has quit IRC | 13:07 | |
| *** edmondsw has quit IRC | 13:08 | |
| mtreinish | yolanda: in a follow on WIP patch I added 2 mroe: https://review.openstack.org/#/c/348019/1/templates/mosquitto.conf.erb | 13:08 |
| *** nmagnezi_ is now known as nmagnezi | 13:09 | |
| *** mdrabe has joined #openstack-infra | 13:09 | |
| yolanda | mtreinish, thanks for clarifying. I actually thought about it, but the fact that was only exposed on that change confused me... going to change the review | 13:10 |
| sdague | yolanda: oh, hey, and infra root person :) | 13:10 |
| sdague | any ability to look into why oslo.rootwrap 5.0.0 is not in our mirrors? | 13:10 |
| sdague | 3 hours after it hit pypi | 13:11 |
| mtreinish | yolanda: yeah, prior to adding the websocket interface I was relying on the default port being setup. But setting an unbound port in the config turns off the default so I had to manually add it | 13:11 |
| craige | Can we blame pypi? sdague ;-) | 13:12 |
| *** amotoki has joined #openstack-infra | 13:12 | |
| sdague | craige: not as far as I can tell | 13:12 |
| craige | :-( | 13:13 |
| *** edmondsw has joined #openstack-infra | 13:14 | |
| sdague | http://logs.openstack.org/50/348250/2/check/gate-grenade-dsvm-multinode/faa4bba/logs/grenade.sh.txt.gz#_2016-07-28_12_55_18_910 as an example failure | 13:15 |
| *** gongysh has joined #openstack-infra | 13:16 | |
| *** chem|lunch has quit IRC | 13:16 | |
| *** chem|lunch has joined #openstack-infra | 13:16 | |
| sdague | I'm walking away from my computer for a bit, if this could be looked into, that would be great. There is a stack of work in os-brick, os-vif, cinder, nova, neutron all stacked up behind having this out there | 13:16 |
| *** oanson has quit IRC | 13:16 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mosquitto: Remove trailing whitespace from mosquitto.conf template https://review.openstack.org/348307 | 13:16 |
| mtreinish | sdague: is it isolated to a specific region/deployer or across the board? | 13:17 |
| *** nwkarsten has joined #openstack-infra | 13:17 | |
| *** akshai has joined #openstack-infra | 13:18 | |
| *** zz_dimtruck is now known as dimtruck | 13:19 | |
| *** rbuzatu has quit IRC | 13:20 | |
| *** matt-borland has joined #openstack-infra | 13:20 | |
| *** jcoufal has joined #openstack-infra | 13:21 | |
| *** daemontool has quit IRC | 13:21 | |
| *** gouthamr has joined #openstack-infra | 13:22 | |
| *** ramishra has joined #openstack-infra | 13:23 | |
| *** maishsk has quit IRC | 13:25 | |
| *** rbuzatu has joined #openstack-infra | 13:26 | |
| *** markvoelker has joined #openstack-infra | 13:30 | |
| *** rbuzatu has quit IRC | 13:30 | |
| *** mmandel has joined #openstack-infra | 13:32 | |
| *** rbuzatu has joined #openstack-infra | 13:33 | |
| *** markvoelker has quit IRC | 13:34 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement basic undercloud sanity checks https://review.openstack.org/346551 | 13:34 |
| clarkb | its across the board since all mirrors are backed by the same content. last modified file says we havent updated since 09:35UTC | 13:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement a new job for undercloud testing only https://review.openstack.org/346147 | 13:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 13:34 |
| clarkb | yesterday | 13:35 |
| trown | can someone check on a simple reviewstats patch? https://review.openstack.org/#/c/339572/ | 13:35 |
| *** esberglu has joined #openstack-infra | 13:35 | |
| *** gouthamr has quit IRC | 13:36 | |
| *** sdake has joined #openstack-infra | 13:36 | |
| clarkb | (I was worried we got a contraints update through without package in mirror but the failrd job is for the update change) | 13:37 |
| *** maishsk has joined #openstack-infra | 13:37 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fix CI status report to show all job results on 1 line https://review.openstack.org/348093 | 13:38 |
| *** yamamoto has joined #openstack-infra | 13:41 | |
| phschwartz | Is there a way to see errors from gertty? I keep getting an Error while sync'ing and need to figure out the cause. | 13:41 |
| *** rdx_matt has quit IRC | 13:42 | |
| *** mriedem has joined #openstack-infra | 13:42 | |
| clarkb | phschwartz: run with -d and look at the debug log in $HOME | 13:42 |
| *** ayoung has joined #openstack-infra | 13:43 | |
| *** rbuzatu has quit IRC | 13:44 | |
| clarkb | I see bandersnatch is running right now. The logs don't have the "bandersnatch completed successfully" messages implying bandersnatch is failing to update for some reason. I don't see anything logged about the bandersnatch run itself | 13:44 |
| *** shardy has joined #openstack-infra | 13:44 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add reno jobs for ironic-lib https://review.openstack.org/348327 | 13:44 |
| *** yamamoto has quit IRC | 13:46 | |
| clarkb | hrm is that because we consume the output of bandersnatch and parse it to know when to PURGE objects? | 13:46 |
| *** jcoufal has quit IRC | 13:46 | |
| *** itisha has joined #openstack-infra | 13:46 | |
| *** jcoufal has joined #openstack-infra | 13:47 | |
| *** bhavik1 has joined #openstack-infra | 13:48 | |
| *** bhavik has quit IRC | 13:48 | |
| *** bhavik1 is now known as bhavik | 13:48 | |
| *** bhavik has quit IRC | 13:48 | |
| derekh | Hi, rh1 should be ready to go back into rotation again, I'd first like to run a small number of jobs on it | 13:48 |
| derekh | if anybody has a chance would you mind taking a look at https://review.openstack.org/#/c/348212/ | 13:48 |
| derekh | also is it possible to get a release of gear? | 13:49 |
| *** PollyZ has joined #openstack-infra | 13:51 | |
| *** pgadiya has quit IRC | 13:51 | |
| *** Liuqing has quit IRC | 13:51 | |
| *** gouthamr has joined #openstack-infra | 13:51 | |
| pabelanger | morning | 13:52 |
| *** thiagop is now known as thiagop-afk | 13:52 | |
| *** Liuqing has joined #openstack-infra | 13:52 | |
| *** EricGonczer_ has quit IRC | 13:52 | |
| clarkb | pabelanger: do you know how to get the logs from bandersnatch (its currently failing) | 13:52 |
| *** rbuzatu has joined #openstack-infra | 13:52 | |
| *** tonytan4ever has joined #openstack-infra | 13:52 | |
| clarkb | derekh: I think jeblair had intended on doing a release but not sure if mroe tested was desired | 13:52 |
| pabelanger | clarkb: not bandersnatch, I've just looked at apache logs on mirrors | 13:53 |
| derekh | clarkb: ack, thanks | 13:53 |
| clarkb | pabelanger: reading our scripts I think we may be bitbucketting them when we parse them to determine if we need to PURGE pypy objects | 13:54 |
| clarkb | its actually a small patch to correct that, let me try to write that really quick | 13:54 |
| *** rbuzatu_ has joined #openstack-infra | 13:54 | |
| mmedvede | azvyagintsev: would devstack-gate not work for your case of configuring devstack? You can find a few examples of its use in https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml | 13:55 |
| pabelanger | clarkb: ack | 13:55 |
| clarkb | oh wait no there is already a print(line) in there. So no I am confused | 13:55 |
| clarkb | maybe new bandersnatch doesn't write to stdout/stderr like the old version did? | 13:56 |
| *** rbrndt has joined #openstack-infra | 13:56 | |
| *** rbuzatu has quit IRC | 13:56 | |
| *** ramishra has quit IRC | 13:57 | |
| pabelanger | I'd have to look at the code myself, I've never run it locally before | 13:57 |
| *** dimtruck is now known as zz_dimtruck | 13:57 | |
| *** markusry has joined #openstack-infra | 13:57 | |
| *** ddieterly has joined #openstack-infra | 13:58 | |
| tlbr | pabelanger, could you please set w+1 https://review.openstack.org/#/c/307742/ ? | 13:59 |
| clarkb | ya looking at slightly older log files it was logging up until the point where we started publishing | 13:59 |
| clarkb | which was yesterday at 0935 | 13:59 |
| *** rbuzatu_ has quit IRC | 13:59 | |
| *** xarses has quit IRC | 14:00 | |
| clarkb | and reading those logs it apepars that we are hitting the 30 minute timeout each time around | 14:01 |
| *** pradk has joined #openstack-infra | 14:02 | |
| *** ramishra has joined #openstack-infra | 14:02 | |
| *** matt-borland has quit IRC | 14:04 | |
| *** mmandel has quit IRC | 14:05 | |
| clarkb | strace shows what appears to be syncing of lots of data | 14:05 |
| *** rbuzatu has joined #openstack-infra | 14:05 | |
| *** pt_15 has joined #openstack-infra | 14:06 | |
| *** matt-borland has joined #openstack-infra | 14:06 | |
| *** mctaylor has joined #openstack-infra | 14:06 | |
| clarkb | It almost looks like bandersnatch has decided it needs to do a full sync | 14:07 |
| *** mctaylor has quit IRC | 14:07 | |
| *** mctaylor has joined #openstack-infra | 14:08 | |
| *** rbuzatu has quit IRC | 14:10 | |
| *** ajmiller has joined #openstack-infra | 14:11 | |
| *** amitgandhinz has quit IRC | 14:11 | |
| *** amitgandhinz has joined #openstack-infra | 14:12 | |
| pabelanger | oh, that's not good | 14:12 |
| clarkb | I have taken the bandersnatch lock in screen on mirror-update, will start a manual bandersnatch update in another window in that screen | 14:12 |
| *** zz_dimtruck is now known as dimtruck | 14:13 | |
| *** Liuqing has quit IRC | 14:14 | |
| *** Liuqing has joined #openstack-infra | 14:14 | |
| clarkb | pabelanger: based on strace output I think we had managed to sync into the 'c's | 14:15 |
| *** rbuzatu has joined #openstack-infra | 14:15 | |
| *** gongysh has quit IRC | 14:15 | |
| clarkb | crawl_something and couchdb_something had gone by | 14:15 |
| clarkb | pabelanger: do you know if k5start is going to give me a token that will live long enough for this? | 14:15 |
| pabelanger | clarkb: make sure you run it in screen | 14:16 |
| *** rbuzatu_ has joined #openstack-infra | 14:16 | |
| clarkb | pabelanger: yup have that, more worried about token expiring | 14:17 |
| pabelanger | clarkb: but don't do vos release on that server, we'll have to do it on afsdb01.o.o using --localauth | 14:17 |
| *** chem|lunch has quit IRC | 14:17 | |
| clarkb | oh right its the vos release not the writing to the rw volume that is troublesome | 14:17 |
| tlbr | could please someone set w+1, i've rebased patch and lost w+1 from Andreas =\ https://review.openstack.org/#/c/307742/ | 14:17 |
| clarkb | ok I will just be running run-bandersnatch under k5start for now | 14:17 |
| * clarkb spies a fungi | 14:19 | |
| *** rbuzatu has quit IRC | 14:19 | |
| clarkb | fungi: ^ do you want me to wait or go ahead on running it manually in foreground? | 14:19 |
| fungi | clarkb: go for it--i just started catching up | 14:20 |
| *** piet has joined #openstack-infra | 14:20 | |
| *** rbuzatu_ has quit IRC | 14:21 | |
| clarkb | that is running now, still no output so guessing bandersnatch changed how it does that | 14:21 |
| fungi | did we update bandersnatch yesterday maybe? | 14:22 |
| *** roxanaghe has joined #openstack-infra | 14:22 | |
| fungi | presumably not--last release was in may | 14:23 |
| clarkb | last week I approved the change to stop using our fork of bandersnatch | 14:23 |
| clarkb | so I would've expected it to update about a week ago | 14:24 |
| fungi | that would be a pretty delayed reaction, agreed | 14:24 |
| corvus | clarkb, phschwartz: gertty writes error tracebacks to .gertty.log with or without -d | 14:24 |
| fungi | dstufft: was there any new major schema change on pypi yesterday that might cause bandersnatch to want to initiate a full rescan? | 14:25 |
| corvus | fungi: speaking of bandersnatch... https://review.openstack.org/319449 | 14:25 |
| openstackgerrit | Jonas Sticha proposed openstack-infra/nodepool: Add documentation for user-home https://review.openstack.org/348383 | 14:25 |
| fungi | corvus: did you want to reply to ianw's question on that change? | 14:26 |
| *** amitgandhinz has quit IRC | 14:26 | |
| clarkb | ya looks like we are back in the 'A's again so likely has decided it wants to resync from scratch | 14:26 |
| clarkb | (again using strace to infer the packages being synced) | 14:26 |
| *** thiagolib has joined #openstack-infra | 14:27 | |
| *** roxanaghe has quit IRC | 14:27 | |
| *** hurgleburgler has joined #openstack-infra | 14:27 | |
| *** xyang1 has joined #openstack-infra | 14:27 | |
| *** mdrabe has quit IRC | 14:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Enable config-drive for tripleo-test-cloud-rh2 images https://review.openstack.org/348389 | 14:30 |
| *** mmandel has joined #openstack-infra | 14:30 | |
| corvus | fungi, ianw: done | 14:30 |
| *** deadnull has quit IRC | 14:30 | |
| clarkb | pabelanger: derekh oh we probably need config-drive on https://review.openstack.org/#/c/348212/1/nodepool/nodepool.yaml too | 14:30 |
| *** amitgandhinz has joined #openstack-infra | 14:30 | |
| pabelanger | clarkb: fungi: derekh: I've noticed tripleo-test-cloud-rh2 was not mounting /mnt/config properly, turns out config-drive was not enabled in nodepool^, which stopped glean from working | 14:30 |
| derekh | clarkb: will add | 14:31 |
| *** shardy has quit IRC | 14:31 | |
| pabelanger | clarkb: I'm pretty sure, we're not passing SSH keys with glean today in nodepool? | 14:32 |
| clarkb | pabelanger: correct, we rely on the baked in keys | 14:32 |
| pabelanger | Ya, wasn't sure if it changed recently or not | 14:32 |
| thiagolib | Hi someone already did make configuration of pacemake to cinder-volume in mirantis openstack 8 | 14:33 |
| tlbr | infra-root, sorry for annoying, but could please someone set w+1 https://review.openstack.org/#/c/307742/ ? | 14:33 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add back rh1 https://review.openstack.org/348212 | 14:33 |
| derekh | clarkb: AJaeger rebased on top of the config drive patch from pabelanger ^ | 14:34 |
| *** shardy has joined #openstack-infra | 14:34 | |
| pabelanger | tlbr: +3 | 14:34 |
| *** mdrabe has joined #openstack-infra | 14:34 | |
| tlbr | pabelanger, thank you! | 14:34 |
| *** jistr is now known as jistr|call | 14:35 | |
| clarkb | we should probably consider makign config-drive: true the default in nodepool since all clouds support it and if you are gleaning you need it | 14:35 |
| clarkb | won't be a strictly backward compat change if people are using a cloud that doesn't supprot it or need to have working live migration | 14:36 |
| *** zhurong has joined #openstack-infra | 14:36 | |
| *** matrohon has quit IRC | 14:36 | |
| *** ramishra has quit IRC | 14:36 | |
| *** ramishra has joined #openstack-infra | 14:37 | |
| fungi | corvus: pet peeve of mine, using os.path.join() instead of '/'.join() on a url (os.path is for handling platform-specific path separators and urls are non-platform-specific), but that's not really relevant since nobody is likely to run this from windows or something so i just approved it | 14:38 |
| *** nmagnezi has quit IRC | 14:39 | |
| clarkb | just saw DAWG-python go by so its doing stuff | 14:39 |
| fungi | DAWG! | 14:39 |
| *** chem has joined #openstack-infra | 14:40 | |
| corvus | fungi: granted, though '/'.join would be insufficient. | 14:40 |
| corvus | fungi: it doesn't add/remove extra '/' as necessary | 14:40 |
| fungi | corvus: oh, good point, that can be a useful side effect | 14:41 |
| clarkb | corvus: ianw in a path like /afs/.openstack.org/mirror/pypi/web/packages/10/00/2e5253a593906aecc0086819cc5b8b0ac2c9eb329d915e4ae27a430171f3 where are the 10 and 00 coming from? They don't seem to be derived from the end of that path | 14:41 |
| phschwartz | corvus: nice to know, ty | 14:41 |
| pabelanger | clarkb: I'd be okay with making config-drive: true by default | 14:41 |
| *** clenimar has joined #openstack-infra | 14:42 | |
| corvus | clarkb: i believe that's straight from pypi. iirc it's a sha256sum and it's actually concatenated, so the orig value is 10002e5253a593906aecc0086819cc5b8b0ac2c9eb329d915e4ae27a430171f3 | 14:42 |
| clarkb | gotcha | 14:42 |
| fungi | corvus: though urlparse.urljoin() does seem to eat duplicate separators | 14:42 |
| corvus | fungi: yeah, that's probably the right thing to use | 14:43 |
| fungi | (also in stdlib) | 14:43 |
| pabelanger | clarkb: And we zuulv3 coming, I think we can sneak the change in | 14:43 |
| *** asettle has quit IRC | 14:43 | |
| clarkb | Django-MobilityHelpers. I think this is goign to take a while. Probably a good time for breakfast | 14:44 |
| corvus | fungi: so many extra characters to type! [mock despair] | 14:44 |
| fungi | heh | 14:44 |
| sulo | hi, can someone help with https://review.openstack.org/#/c/345453/ please :) | 14:44 |
| fungi | <thecount>5! 5 keystrokes! ha, ha, ha</thecount> | 14:44 |
| *** piet has quit IRC | 14:45 | |
| fungi | or maybe 4. i miscounted | 14:45 |
| clarkb | sulo: no docs for craton? | 14:45 |
| *** Teresita-Warrior has joined #openstack-infra | 14:46 | |
| *** timl0101 has joined #openstack-infra | 14:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 14:47 |
| *** jistr|call is now known as jistr | 14:49 | |
| fungi | our wikispammers much be mostly in the americas... there's hardly ever any new spam showing up overnight while i'm asleep; most of it appears later in my day | 14:50 |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Make gate-rally-dsvm-networking-ovn voting https://review.openstack.org/348402 | 14:50 |
| *** asettle has joined #openstack-infra | 14:51 | |
| clarkb | fungi: thats when the botnet horde awakens from its slumber | 14:51 |
| *** xarses has joined #openstack-infra | 14:52 | |
| *** nmagnezi has joined #openstack-infra | 14:52 | |
| *** maishsk has quit IRC | 14:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Created jobs for nodejs version 6 https://review.openstack.org/344501 | 14:53 |
| fungi | ...and begins to feed | 14:53 |
| krotscheck | nodejs 6 jobs landed. Wooooo doggy, let's see what happens! | 14:55 |
| *** Swami has joined #openstack-infra | 14:55 | |
| *** Swami_ has joined #openstack-infra | 14:55 | |
| *** maishsk has joined #openstack-infra | 14:56 | |
| *** ifarkas is now known as ifarkas_afk | 14:58 | |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Purge normalized URLs https://review.openstack.org/319449 | 14:58 |
| *** maishsk has quit IRC | 14:59 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add gate-rally-dsvm-networking-ovn to grafana https://review.openstack.org/348413 | 15:00 |
| *** Swami__ has joined #openstack-infra | 15:01 | |
| *** amitgandhinz has quit IRC | 15:02 | |
| *** amitgandhinz has joined #openstack-infra | 15:03 | |
| *** Swami has quit IRC | 15:05 | |
| *** yamamoto has joined #openstack-infra | 15:06 | |
| *** PollyZ has quit IRC | 15:06 | |
| *** piet has joined #openstack-infra | 15:07 | |
| *** Swami has joined #openstack-infra | 15:08 | |
| *** ccamacho is now known as ccamacho|afk | 15:08 | |
| pabelanger | clarkb: are we okay with moving our infra jobs directly ubuntu-xenial? | 15:08 |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: WIP for CLI for python client https://review.openstack.org/348418 | 15:08 |
| *** armax has joined #openstack-infra | 15:09 | |
| clarkb | pabelanger: I think we shouod be for the ones that arent platform apecific like beaker rspec and noop apply | 15:09 |
| clarkb | infra-root may have opinions though | 15:10 |
| *** dizquierdo has quit IRC | 15:10 | |
| pabelanger | clarkb: agreed | 15:10 |
| *** yamamoto has quit IRC | 15:10 | |
| pabelanger | gate-project-config-layout will still be ubuntu-precise | 15:11 |
| *** Swami__ has quit IRC | 15:11 | |
| fungi | we should actually make that one run on trusty now | 15:11 |
| fungi | we were (incorrectly) running it on trusty when zuul.o.o was still running precise | 15:12 |
| pabelanger | okay, let me fix that now | 15:12 |
| fungi | and then got bitten by an ordering issue between the python on trusty and precise that took zuul down | 15:12 |
| clarkb | any docs or lint type jobs can be xenialed I think | 15:12 |
| fungi | so at that point we switched it back to trusty | 15:12 |
| fungi | er back to precise | 15:12 |
| fungi | but now that zuul.o.o is running trusty we should test the layout on trusty | 15:13 |
| fungi | for similar safety reasons | 15:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Move gate-project-config-layout to ubuntu-trusty https://review.openstack.org/348420 | 15:13 |
| pabelanger | ack | 15:13 |
| pabelanger | changed | 15:13 |
| pabelanger | that means only system-config is using ubuntu-precise now | 15:13 |
| pabelanger | for puppet-apply | 15:14 |
| pabelanger | we still need that until control servers are finally migrated | 15:14 |
| *** Liuqing has quit IRC | 15:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate infra jobs to ubuntu-xenial https://review.openstack.org/348424 | 15:16 |
| *** Liuqing has joined #openstack-infra | 15:16 | |
| *** EricGonczer_ has joined #openstack-infra | 15:17 | |
| *** openstackgerrit has quit IRC | 15:18 | |
| *** openstackgerrit has joined #openstack-infra | 15:18 | |
| *** yuanying_ has joined #openstack-infra | 15:20 | |
| *** pcaruana has quit IRC | 15:22 | |
| *** yuanying has quit IRC | 15:22 | |
| fungi | pabelanger: yep... wiki, lists, puppetdb, paste, planet, static, zuul, ci-backup-rs-ord and proposal.slave are still being tested against precise, though more than half of those can be switched to trusty now | 15:24 |
| pabelanger | fungi: Oh, I thought we did static | 15:25 |
| pabelanger | and zuul | 15:25 |
| pabelanger | maybe we didn't update site.pp | 15:26 |
| *** zhurong has quit IRC | 15:27 | |
| fungi | pabelanger: yep, there is a fair amount of cleanup that can be done there | 15:28 |
| pabelanger | fungi: Yup, just pushed up a patch | 15:28 |
| pabelanger | https://review.openstack.org/#/c/348434/ | 15:28 |
| phschwartz | This is the gertty error I am getting http://paste.openstack.org/show/543295/ not very helpful O.o | 15:28 |
| *** amitgandhinz has quit IRC | 15:29 | |
| *** amitgandhinz has joined #openstack-infra | 15:29 | |
| fungi | pabelanger: my checking suggests that paste, static, zuul and proposal.slave are running trusty now | 15:30 |
| *** PollyZ has joined #openstack-infra | 15:30 | |
| *** megm has quit IRC | 15:30 | |
| pabelanger | fungi: okay, let me get those 4 | 15:30 |
| *** elo has quit IRC | 15:30 | |
| fungi | pabelanger: while wiki, lists, puppetdb, planet and ci-backup-rs-ord haven't been upgraded off precise yet | 15:31 |
| *** megm has joined #openstack-infra | 15:31 | |
| *** elo has joined #openstack-infra | 15:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Clean up puppet-apply testing on precise https://review.openstack.org/348434 | 15:33 |
| *** Guest51388 has quit IRC | 15:33 | |
| fungi | pabelanger: also we have two dummy entries in there for single-use slaves (a bare version and a devstack version) tested on precise, which looks like a holdover from the old days of having separate node types. we could stand to collapse/dedup those accordingly | 15:34 |
| *** mikelk has quit IRC | 15:34 | |
| *** ddieterly is now known as ddieterly[away] | 15:35 | |
| dstufft | fungi: no | 15:36 |
| fungi | dstufft: thanks for confirming. likely some sort of mysterious failure mode in that case | 15:36 |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: WIP for CLI for python client https://review.openstack.org/348418 | 15:37 |
| *** Swami__ has joined #openstack-infra | 15:37 | |
| *** acruz has quit IRC | 15:37 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova/neutron dns mitaka bug 1607412 https://review.openstack.org/348445 | 15:37 |
| openstack | bug 1607412 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode fails server build on newton side with "Unrecognized attribute(s) 'dns_name'"" [Undecided,New] https://launchpad.net/bugs/1607412 | 15:37 |
| *** mhickey has quit IRC | 15:38 | |
| *** roxanaghe has joined #openstack-infra | 15:38 | |
| pabelanger | fungi: Agreed | 15:38 |
| *** Swami__ has quit IRC | 15:38 | |
| *** Swami__ has joined #openstack-infra | 15:38 | |
| *** gibi has quit IRC | 15:39 | |
| *** vincentll has quit IRC | 15:39 | |
| *** ddieterly[away] is now known as ddieterly | 15:40 | |
| *** Swami has quit IRC | 15:40 | |
| *** Swami_ has quit IRC | 15:40 | |
| *** jistr is now known as jistr|afk | 15:42 | |
| *** roxanaghe has quit IRC | 15:42 | |
| pabelanger | fungi: I could use your input on 347839 too, adds the jessie-newton-backports repo that zigo needs | 15:43 |
| *** gibi has joined #openstack-infra | 15:43 | |
| *** shardy has quit IRC | 15:43 | |
| sdague | clarkb et al: how are the mirrors looking? | 15:45 |
| *** e0ne has quit IRC | 15:45 | |
| clarkb | sdague: bandersnatch is still going. Pollster just scrolled by in my strace | 15:45 |
| *** e0ne has joined #openstack-infra | 15:46 | |
| fungi | pabelanger: did the change that adds the mechanism for building that merge yet? i skimmed it but abstained from voting on it since i'm still not convinced it's the right solution and am hoping some other reviewer thinks of a better alternative | 15:46 |
| fungi | but i didn't -1 because i didn't have a better suggestion for implementation (or at least don't have time to think up one) | 15:47 |
| pabelanger | fungi: we haven't added that to JJB yet, but our existing template should be straightforward to upload to the new repo | 15:47 |
| *** piet has quit IRC | 15:48 | |
| *** hashar has quit IRC | 15:49 | |
| sdague | clarkb: ok, cool, any guess on an eta? really hoping to move this privsep work forward | 15:49 |
| *** woodster_ has quit IRC | 15:49 | |
| *** hashar has joined #openstack-infra | 15:49 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement a new job for undercloud testing only https://review.openstack.org/346147 | 15:50 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 15:50 |
| clarkb | sdague: assuming the afs publishing bit doens't take forever probably a couple hours | 15:50 |
| *** Liuqing has quit IRC | 15:50 | |
| *** PollyZ has quit IRC | 15:52 | |
| sdague | clarkb: ok, any idea why things broken? | 15:53 |
| *** atom_ has quit IRC | 15:54 | |
| *** mtanino has quit IRC | 15:54 | |
| clarkb | no I do not know why bandersnatch has decided it needs a full resync | 15:54 |
| *** comstud has quit IRC | 15:55 | |
| clarkb | but thats why we aren't updating because the 30 minute timeout is not long enough for a full resync | 15:55 |
| *** rossella_s has quit IRC | 16:00 | |
| *** rossella_s has joined #openstack-infra | 16:00 | |
| *** Swami__ has quit IRC | 16:00 | |
| clarkb | are we ready to take the plunge on https://review.openstack.org/#/c/348078/2 ? | 16:03 |
| clarkb | AJaeger left it for when more people were awake | 16:03 |
| *** tosky has quit IRC | 16:03 | |
| pabelanger | clarkb: sure, I can help | 16:04 |
| corvus | i'm semi-afk again today at ansiblefest | 16:04 |
| *** psachin has quit IRC | 16:04 | |
| anteaya | corvus: have a happy fest | 16:04 |
| clarkb | ok going to approve it | 16:06 |
| *** PollyZ has joined #openstack-infra | 16:06 | |
| clarkb | there are a lot of Py.* packages on pypy | 16:06 |
| *** david-lyle has joined #openstack-infra | 16:07 | |
| clarkb | er pypi | 16:07 |
| *** e0ne has quit IRC | 16:07 | |
| pabelanger | corvus: when you have time to catch up, had a comment about elapsed time estimates in status.o.o/zuul: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-07-26.log.html#t2016-07-26T23:47:43 | 16:07 |
| *** maishsk has joined #openstack-infra | 16:09 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories https://review.openstack.org/312666 | 16:13 |
| clarkb | PytSite now | 16:13 |
| *** cdent has quit IRC | 16:13 | |
| * rcarrillocruz envy corvus , Shrews and nibalizer for being at AnsibleFest, assumes mordred is also there :( | 16:17 | |
| * rcarrillocruz sobs | 16:17 | |
| Shrews | rcarrillocruz: nibalizer is evil, do not envy us | 16:18 |
| rcarrillocruz | :D | 16:18 |
| rcarrillocruz | but he's funny to hang out with tho :D | 16:18 |
| rcarrillocruz | clarkb , fungi : yesterday i was thinking 'wow, this firehose/mqtt thing would be cool to report ansible runs into logstash' | 16:19 |
| pabelanger | clarkb: just did some clean-up on nodepool.o.o, there was some hung processes relating to DIBs. EG: git clones. I stopped nodepool-builder, checked up processes and restarted it again. FYI | 16:19 |
| rcarrillocruz | there's an ansible mqtt module to send stuff | 16:19 |
| *** baoli_ has quit IRC | 16:19 | |
| rcarrillocruz | and some input plugin for logstash mqtt around, but it's been idlish in terms of development | 16:19 |
| rcarrillocruz | i wonder if that would be a good use case to make use of firehose? | 16:20 |
| rcarrillocruz | mtreinish: ^ | 16:20 |
| clarkb | pabelanger: ok | 16:20 |
| mtreinish | rcarrillocruz: it sounds like it would be | 16:20 |
| *** woodster_ has joined #openstack-infra | 16:20 | |
| mtreinish | that was the basic idea of it get every event from an infra service to report to it | 16:20 |
| clarkb | rcarrillocruz: mtreinish yes though I think our logstash setup is still really cranky for some reason I haven't been able to determine | 16:20 |
| rcarrillocruz | yep | 16:20 |
| clarkb | elasticsearch looks happy but we aren't really indexing as quickly as I would like | 16:20 |
| rcarrillocruz | clarkb: k, just some random idea to hack on | 16:21 |
| *** ddieterly is now known as ddieterly[away] | 16:21 | |
| *** esikachev has quit IRC | 16:21 | |
| rcarrillocruz | the log-to-central-syslog on ansible-role-puppet is still on my queue tho | 16:21 |
| rcarrillocruz | i need to rebase now that i check... | 16:21 |
| clarkb | ya just warning that we need to fix things beforehand | 16:21 |
| rcarrillocruz | we can summon jesusaur to fix everything | 16:22 |
| rcarrillocruz | blink blink | 16:22 |
| clarkb | looking at elasticsearch-head I am not really seeing the docs count climb | 16:22 |
| *** nwkarsten has quit IRC | 16:22 | |
| mtreinish | rcarrillocruz: were you referring to this: https://github.com/elastic/logstash/issues/4487 ? | 16:22 |
| pabelanger | mtreinish: you hoping to get firehose.o.o going before barcelona? | 16:23 |
| openstackgerrit | Merged openstack-infra/project-config: Run integrated gate on xenial https://review.openstack.org/348078 | 16:23 |
| rcarrillocruz | clarkb , fungi , pabelanger : are there any stuff that needs attention ? i'm 'starting' to work *now* | 16:23 |
| mtreinish | pabelanger: I was hoping to get it done pretty soon :) | 16:23 |
| fungi | pabelanger: yeah, at least with the gerrit event stream in it as an initial poc | 16:23 |
| rcarrillocruz | been at child hospital , wife and son sick, sigh | 16:23 |
| rcarrillocruz | mtreinish: https://github.com/widarlein/logstash-input-mqtt rather | 16:23 |
| rcarrillocruz | if what i understand by input and output in logstash context is correct | 16:23 |
| rcarrillocruz | input = things logstash can consume | 16:24 |
| mtreinish | rcarrillocruz: ah, in the other direction | 16:24 |
| rcarrillocruz | output = things logstash can emit on | 16:24 |
| fungi | pabelanger: mtreinish and i have a talk proposed to present it in the upstream dev track at the conference | 16:24 |
| rcarrillocruz | yeah | 16:24 |
| rcarrillocruz | i'm intrigued by ^ | 16:24 |
| *** jlanoux has quit IRC | 16:24 | |
| rcarrillocruz | hope you get it thru folks :-) | 16:24 |
| rcarrillocruz | random googling it seems mqtt is quite big in IoT, raspberry pi, home assistant... hakerish things | 16:25 |
| pabelanger | fungi: mtreinish: ya, seen that. Got my votes, if you need help standing up the server, happy to offer up some time | 16:25 |
| pabelanger | rcarrillocruz: moving doing reviews and waiting for patches to land | 16:25 |
| *** maishsk has quit IRC | 16:26 | |
| fungi | pabelanger: that would be excellent! i'm trying to make time for it, but only so many hours in the day | 16:26 |
| rcarrillocruz | in other news, if you wondered why os_server always return 'changed' (i.e. not being idempotent at least from the state returned) , I pushed https://github.com/ansible/ansible-modules-core/pull/4261 | 16:26 |
| *** ddieterly[away] is now known as ddieterly | 16:26 | |
| rcarrillocruz | hope to get it reviewed by Shrews and mordred when they are not doing conf things | 16:26 |
| rcarrillocruz | jlk: ^ | 16:26 |
| pabelanger | fungi: sure, once you're ready, feel free to ping me if you don't have the bandwidth | 16:26 |
| pabelanger | fungi: mtreinish: looking forward to testing it | 16:27 |
| *** Apoorva has joined #openstack-infra | 16:27 | |
| nibalizer | rcarrillocruz: haha | 16:27 |
| rcarrillocruz | once that lands, i'll work on adding pluggable pre/post provision/unprovision tasks on the launcher | 16:27 |
| rcarrillocruz | nibalizer: ;-) | 16:27 |
| nibalizer | rcarrillocruz: next time you too can have a sticker stuck to your face! | 16:27 |
| jlk | rcarrillocruz: will be testing soon | 16:27 |
| jlk | and the floating IP change too in shade | 16:27 |
| rcarrillocruz | lulz, saw the picture on twitter :D | 16:27 |
| rcarrillocruz | jlk: ++ | 16:27 |
| jlk | serious FOMO for not being at AnsibleFest. | 16:28 |
| rcarrillocruz | +++++++++++++++++++++ ^ | 16:28 |
| EmilienM | pabelanger: we have serious image cache issue I think | 16:28 |
| pabelanger | EmilienM: any info about the issue? | 16:29 |
| *** maishsk has joined #openstack-infra | 16:29 | |
| EmilienM | pabelanger: can you loging on #puppet-openstack? | 16:29 |
| clarkb | looking at strace of logstash worker and indexer processes the workers are all blocked on sends to the indexer | 16:29 |
| clarkb | the indexer is running get time of day a lot but I don't actually see it writing anything to elasticsearch | 16:30 |
| anteaya | clarkb: :( | 16:30 |
| *** jistr|afk is now known as jistr | 16:30 | |
| *** tphummel has joined #openstack-infra | 16:30 | |
| rcarrillocruz | the idea for that would be something like http://paste.openstack.org/show/543312/ | 16:31 |
| rcarrillocruz | so we could plugin any tasks to a given server, on each lifecycle stage | 16:31 |
| rcarrillocruz | that would be handy for doing automated deploys of servers | 16:31 |
| rcarrillocruz | but | 16:32 |
| rcarrillocruz | 1. need to fix os_server to be idempotent, as changed is shown always, regardless if it created the server or nothing happened | 16:32 |
| *** tesseract- has quit IRC | 16:32 | |
| clarkb | no recent change to our logstash rules from what I can see | 16:32 |
| rcarrillocruz | 2. need to return something more useful to the task result than just changed. Something like result=created, result=deleted | 16:32 |
| *** maishsk has quit IRC | 16:33 | |
| osnexus | So I'm receiving an error that i have no public endpoint for network service, and I'm trying to find what I'm missing. Here are my lists of services and endpoints | 16:33 |
| osnexus | http://paste.openstack.org/show/543314/ | 16:33 |
| pabelanger | fungi: clarkb: need some help with git and zuul-cloner, EmilienM is having an issue | 16:34 |
| *** coolsvap is now known as coolsvap_ | 16:34 | |
| pabelanger | fungi: clarkb: basically, http://logs.openstack.org/06/339806/44/check/gate-puppet-openstack-integration-3-scenario002-tempest-centos-7/4bf1d45/console.html#_2016-07-28_12_49_16_315962 is not checking out the latest commit of master, and trying to figure out why that is | 16:34 |
| rcarrillocruz | osnexus: you're better of on #openstack | 16:35 |
| rcarrillocruz | but that could mean your cloud doesn't have neutron | 16:35 |
| rcarrillocruz | just nova-network | 16:35 |
| pabelanger | fungi: clarkb: I think we made a change to zuul-cloner recently? | 16:35 |
| rcarrillocruz | is that a devstack instance ? | 16:35 |
| osnexus | rcarrillocruz it is | 16:36 |
| rcarrillocruz | thttps://wiki.openstack.org/wiki/NeutronDevstack | 16:36 |
| clarkb | https://github.com/elastic/logstash/issues/3276 looks similar though we are logstash 2.0 | 16:36 |
| rcarrillocruz | create a local.conf | 16:36 |
| rcarrillocruz | put that contents | 16:36 |
| rcarrillocruz | then ./unstack.sh && ./stack.sh | 16:36 |
| *** EricGonczer_ has quit IRC | 16:36 | |
| rcarrillocruz | that would reconfigure the devstack cloud with neutron | 16:36 |
| rcarrillocruz | and list the net endpoint | 16:37 |
| pabelanger | clarkb: fungi: or is it possible that is the latest version of master on our git servers | 16:37 |
| EmilienM | pabelanger: maybe https://github.com/openstack-infra/zuul/commit/9f16d522a976fc32d841e2d2bd92d6a4216c92c5 ? | 16:37 |
| osnexus | ty rcarrillocruz I'll get on it | 16:37 |
| rcarrillocruz | yw | 16:38 |
| clarkb | pabelanger: maybe there was a zuul ref for that repo? | 16:38 |
| clarkb | hrm no I think the falling back to master means there wasnt one | 16:38 |
| pabelanger | clarkb: https://github.com/openstack/puppet-openstack-integration/blob/master/functions#L37-L42 is how EmilienM uses zuul-cloner for the puppet-jobs | 16:39 |
| *** derekh has quit IRC | 16:39 | |
| clarkb | pabelanger: it appears to have found master ~3 | 16:39 |
| pabelanger | ya, that commit is from 7 days ago, with 2 more a head of it | 16:40 |
| pabelanger | At first I thought it was using /opt/git cache version but the images were rebuilt last night | 16:40 |
| pabelanger | and I remember fungi saying something about git.o.o | 16:41 |
| *** jpich has quit IRC | 16:42 | |
| fungi | pabelanger: yeah, git04 had a corrupt copy of puppet-openstackid that i fixed (master branch was missing entirely). i checked all teh repos on all our servers and that was the only one in that particular state | 16:43 |
| *** gyee has joined #openstack-infra | 16:43 | |
| *** jordan__ has quit IRC | 16:44 | |
| pabelanger | fungi: what is the easier way for me to check openstack/puppet-tempest on our git farm? I guess logging into each server and running something? | 16:44 |
| clarkb | https://github.com/elastic/logstash/blob/2.1/CHANGELOG.md#input is possibly related since we have multiple tcp connections. I am going to restart logstash on worker01 and only start one worker to see if it stays alive in that case | 16:45 |
| *** mtanino has joined #openstack-infra | 16:45 | |
| *** rbuzatu has joined #openstack-infra | 16:47 | |
| *** burgerk has joined #openstack-infra | 16:47 | |
| *** hashar has quit IRC | 16:47 | |
| *** rbuzatu has quit IRC | 16:47 | |
| clarkb | if anyone is interested in learning how to test a new version of logstash I am happy to work through that with them (I haven't dont it in a few months though so likely will laern things myself :) ) | 16:47 |
| *** gyee has quit IRC | 16:48 | |
| clarkb | pabelanger: its pretty easy to hit the cgit web ui across the baord to check something liek a ref | 16:48 |
| clarkb | pabelanger: http://git01.openstack.org:8080/cgit/openstack/puppet-tempest | 16:48 |
| pabelanger | clarkb: ah, nice. Didn't know that | 16:48 |
| *** PollyZ has quit IRC | 16:48 | |
| *** jsavak has quit IRC | 16:48 | |
| *** jsavak has joined #openstack-infra | 16:49 | |
| jesusaur | rcarrillocruz: clarkb: what's going on with elasticsearch? | 16:49 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add details for rh1 deployments https://review.openstack.org/347797 | 16:49 |
| clarkb | jesusaur: it looks like logstash is falling over. It doesn't respond to sigterm either | 16:49 |
| clarkb | jesusaur: one theory is we are hitting the issues fixed in https://github.com/elastic/logstash/blob/2.1/CHANGELOG.md#input | 16:49 |
| clarkb | jesusaur: I have restarted things on worker01 and will only run one worker daemon there so that we have only one tcp connection | 16:49 |
| *** Swami has joined #openstack-infra | 16:50 | |
| *** coreyob has quit IRC | 16:50 | |
| jesusaur | clarkb: what version of logstash do we currently have running? | 16:51 |
| *** maishsk has joined #openstack-infra | 16:51 | |
| clarkb | jesusaur: 2.0 | 16:51 |
| clarkb | 2.3.something is current which we could jump to likely | 16:52 |
| clarkb | also I find it funny logstash is going from version 2.x to 5.x | 16:53 |
| *** asettle has quit IRC | 16:54 | |
| clarkb | actually easy mode to test this would be to just pull 01 out of puppet, install 2.3.4 and see if it works | 16:54 |
| *** amotoki has quit IRC | 16:54 | |
| *** Daisy has joined #openstack-infra | 16:55 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add Ironic multinode job https://review.openstack.org/332490 | 16:56 |
| *** alee has joined #openstack-infra | 16:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable config-drive for tripleo-test-cloud-rh2 images https://review.openstack.org/348389 | 16:58 |
| *** tkelsey has joined #openstack-infra | 16:59 | |
| jesusaur | clarkb: were you seeing the logstash process running with no open port? or was something else happening? | 17:00 |
| krotscheck | clarkb: If I run devstack-gate, will the various OS_AUTH_URL etc. parameters be set before running the test hooks? | 17:00 |
| *** amotoki has joined #openstack-infra | 17:00 | |
| jesusaur | clarkb: and did you find anything in your strace? | 17:00 |
| clarkb | jesusaur: its listening on the tcp port its just not doing anything | 17:00 |
| clarkb | jesusaur: strace is just a lot of get time of day but no reads or writes | 17:00 |
| *** Daisy has quit IRC | 17:00 | |
| *** yamahata has joined #openstack-infra | 17:00 | |
| clarkb | krotscheck: no you'll have to source those out of the devstack openrc | 17:00 |
| clarkb | krotscheck: or use the clouds.yaml file it writes | 17:00 |
| *** chem is now known as chem|away | 17:01 | |
| fungi | pabelanger: yeah, i loop through them. something like... for n in `seq 8`;do ssh git0$n.openstack.org git --git-dir=/var/lib/git/openstack/puppet-tempest.git show-ref master;done | 17:01 |
| mordred | krotscheck: there are two entries in the clouds.yaml - one with admin and one without, fwiw | 17:01 |
| *** ddieterly is now known as ddieterly[away] | 17:01 | |
| *** BobBall is now known as BobBall_AWOL | 17:01 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add specs dir https://review.openstack.org/336109 | 17:01 |
| *** PollyZ has joined #openstack-infra | 17:01 | |
| fungi | pabelanger: which claims that the master branch on all of them has 03d1e97 at this point | 17:01 |
| krotscheck | mordred: Where is that file generated? | 17:02 |
| pabelanger | fungi: EmilienM: The only thing I can think of is maybe the commit wasn't replicated to our git servers when the job ran? | 17:02 |
| pabelanger | EmilienM: have you rechecked yet? | 17:02 |
| pabelanger | if I run the code locally, I get the right sha1 | 17:02 |
| mordred | krotscheck: in functions-common - function "write_clouds_yaml" | 17:03 |
| EmilienM | pabelanger: 10 times | 17:03 |
| *** david-lyle has quit IRC | 17:03 | |
| clarkb | pabelanger: fungi maybe its using the zuul cloner's master version? | 17:03 |
| mordred | krotscheck: oh, unless you were asking where it's written ... /etc/openstack/clouds.yaml | 17:03 |
| pabelanger | EmilienM: ack | 17:03 |
| fungi | pabelanger: yes, a simple explanation would be that one of the git servers was lagging behind replicating that repo when the job ran a little over 4 hours ago and then caught up | 17:03 |
| *** amotoki has quit IRC | 17:03 | |
| clarkb | I am going to go ahead and test a newer version of logstash really quickly just to rule that in or out | 17:04 |
| krotscheck | mordred: Urm, devstack-gate doesn't have a write_clouds_yaml method.... | 17:04 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Spec for adding diskimage_builder module in-chroot https://review.openstack.org/336664 | 17:04 |
| pabelanger | clarkb: not sure I understand | 17:04 |
| *** nwkarsten has joined #openstack-infra | 17:04 | |
| *** nwkarsten has quit IRC | 17:04 | |
| *** e0ne has joined #openstack-infra | 17:04 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Spec for adding diskimage_builder module in-chroot https://review.openstack.org/336664 | 17:04 |
| fungi | clarkb: that would be weird since zuul-cloner is being told to use git://git.openstack.org/openstack/puppet-tempest (unless it failed to remote update for some reason and silently continued?) | 17:04 |
| pabelanger | EmilienM: when is the most recent job? | 17:04 |
| *** nwkarsten has joined #openstack-infra | 17:05 | |
| krotscheck | Oh, that's in devstack | 17:05 |
| fungi | clarkb: so maybe that's the cached master branch state on centos-7 images in rax-ord? | 17:05 |
| rcarrillocruz | yep | 17:05 |
| *** coreyob has joined #openstack-infra | 17:05 | |
| jesusaur | clarkb: possibly similar to https://github.com/elastic/logstash/issues/5695 ? | 17:05 |
| rcarrillocruz | you'll get 'devstack-admin' and 'devstack' | 17:05 |
| rcarrillocruz | clouds | 17:05 |
| rcarrillocruz | on clouds.yaml | 17:05 |
| *** abregman has quit IRC | 17:05 | |
| EmilienM | pabelanger: a few hours ago | 17:05 |
| *** nmagnezi has quit IRC | 17:06 | |
| *** ddieterly[away] is now known as ddieterly | 17:06 | |
| pabelanger | EmilienM: lets try again now, see if it matches my local test | 17:06 |
| EmilienM | pabelanger: you want me to recheck? | 17:06 |
| pabelanger | fungi: ya, I was thinking that too it was using /opt/git version | 17:06 |
| pabelanger | EmilienM: please | 17:06 |
| clarkb | jesusaur: maybe? I can't even get it to shutdown cleanly though I have to kill -9 | 17:06 |
| EmilienM | pabelanger: sure | 17:07 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Make map-packages use dib-python https://review.openstack.org/339065 | 17:07 |
| pabelanger | EmilienM: I still think adding Depends-On would be a good thing to do, since you need that specific commit. But lets move forward without doing that for now | 17:07 |
| EmilienM | pabelanger: https://review.openstack.org/#/c/344916/8 is a good example | 17:07 |
| EmilienM | it worked for some jobs | 17:07 |
| fungi | pabelanger: also on the composed integration ref theory, you can see in http://logs.openstack.org/06/339806/44/check/gate-puppet-openstack-integration-3-scenario002-tempest-centos-7/4bf1d45/_zuul_ansible/playbook that it was not tested in combination with any puppet-tempest changes | 17:07 |
| *** trown is now known as trown|lunch | 17:08 | |
| openstackgerrit | Ben Kero proposed openstack-infra/project-config: Move chef jobs on xenial https://review.openstack.org/347633 | 17:08 |
| *** Kelso has joined #openstack-infra | 17:08 | |
| *** rbuzatu has joined #openstack-infra | 17:08 | |
| *** maishsk has quit IRC | 17:08 | |
| pabelanger | fungi: ya, so I think in that case, adding the Depends-On might have been the way to go for puppet-tempest? | 17:09 |
| fungi | pabelanger: i have no idea what you're suggesting with that | 17:10 |
| *** akshai has quit IRC | 17:10 | |
| *** cdent has joined #openstack-infra | 17:11 | |
| pabelanger | fungi: maybe I am confused, if we are saying git.o.o was lagged behind the job run, and since it merged so close, adding a Depends-On header to the puppet-tempest commit might have ensured the right version when checked out | 17:11 |
| clarkb | ok logstash 2.3.4 is running on logstash-worker20 with all 4 worker daemons as well. If this works well for the next hour or so then I think the immediate issue has been corrected by newer logstash | 17:12 |
| *** akshai has joined #openstack-infra | 17:12 | |
| krotscheck | mordred, clarkb: One last question. When tyring to run a local devstack slave, I'm running into this problem: /opt/stack/new/devstack-gate/devstack-vm-gate.sh: No such file or directory - How/where is devstack cloned onto the slave? | 17:12 |
| fungi | pabelanger: doubtful since that would have to be done by the zuul-merger which would see in its version of the repo that the depended-on change was already merged | 17:12 |
| clarkb | krotscheck: zuul-cloner is run to update the devstack-gate repo | 17:14 |
| *** jsavak has quit IRC | 17:14 | |
| clarkb | krotscheck: there is a builder for it in jjb | 17:14 |
| rajinir | Slave node went offline. ERROR: Disconnected by anonymous : Offline due to Gearman request. What does this mean? | 17:14 |
| *** jcoufal_ has joined #openstack-infra | 17:14 | |
| krotscheck | clarkb: Yes, I ran that. | 17:14 |
| *** maishsk has joined #openstack-infra | 17:14 | |
| clarkb | rajinir: it means that the slave was put offline due to a request made by gearman to do that. This is useful to prevent running >1 job on a single slave | 17:15 |
| krotscheck | clarkb: devstack-checkout, right? | 17:15 |
| fungi | pabelanger: clarkb: as for the cached state on disk theory, http://paste.openstack.org/show/543412/ | 17:15 |
| clarkb | krotscheck: yes I think that is the one | 17:15 |
| *** cloudtrainme has joined #openstack-infra | 17:16 | |
| fungi | pabelanger: clarkb: looks like that's what got checked out | 17:16 |
| *** rbuzatu has quit IRC | 17:16 | |
| krotscheck | clarkb: Righto, so that works and all, but devstack itself doesn't seem to get installed when I run ./safe-devstack-vm-gate-wrap.sh | 17:16 |
| *** jcoufal has quit IRC | 17:17 | |
| EmilienM | pabelanger: failing on rax, working on ovh, osic | 17:17 |
| EmilienM | I think its an image issue or I don't know | 17:17 |
| fungi | EmilienM: i think that means that ovh and osic have newer images, but doesn't explain why we're not managing to remote update properly | 17:18 |
| clarkb | krotscheck: devstack-gate will install devstack for you | 17:18 |
| *** sputnik13 has joined #openstack-infra | 17:18 | |
| EmilienM | fungi: right. | 17:18 |
| pabelanger | rax images are 15 hours old | 17:18 |
| *** ddieterly is now known as ddieterly[away] | 17:18 | |
| krotscheck | clarkb: Which brings me to my original question. Where does it do that? because /opt/stack is empty | 17:18 |
| clarkb | krotscheck: maybe there are file permissions issues reading/executing /opt/stack/new/devstack-gate/devstack-vm-gate.sh | 17:18 |
| pabelanger | ovh and osic are 14 hours, so they should be the same DIB | 17:19 |
| clarkb | krotscheck: the setup workspace function does all of that | 17:19 |
| fungi | EmilienM: not sure if you saw, but i confirmed that a random centos-7 node booted in rax-ord has that puppet-tempest master state as its initial on-disk state, so something is preventing it from correctly doing a remote update from the origin i think | 17:19 |
| openstackgerrit | Merged openstack-infra/project-config: Add back rh1 https://review.openstack.org/348212 | 17:19 |
| *** nherciu has quit IRC | 17:19 | |
| fungi | EmilienM: which, as pabelanger mentioned, could be a recent regression in zuul-cloner | 17:19 |
| *** sabari has quit IRC | 17:20 | |
| *** jsavak has joined #openstack-infra | 17:20 | |
| rajinir | Is there a login/password for the image of the ubuntu build vm? My node went offline during the build and I'm trying to login | 17:20 |
| *** PollyZ has quit IRC | 17:21 | |
| *** baoli has joined #openstack-infra | 17:22 | |
| clarkb | rajinir: assuming this is a nodepool test image no you will need to configure it with an ssh key via config-drive | 17:22 |
| pabelanger | fungi: EmilienM: maybe enable debugs and see what is happening in zuul-cloner | 17:23 |
| clarkb | jesusaur: testing indicates that having a single worker doesn't help. Neither does using logstash 2.3.4. The worker daemons are just stuck sending data according to strace | 17:23 |
| clarkb | jesusaur: and looking at a strace of logstash its just futex and gettimeofday. No reads or writes that I can see | 17:24 |
| rajinir | clarkb: we tried to using flat networking. There was an outage on the connection when it was moving the interface to the ovs bridge. so the node is lost. | 17:24 |
| jesusaur | clarkb: strange | 17:24 |
| EmilienM | pabelanger: how? | 17:24 |
| clarkb | rajinir: do you need to log into it or just delete it? | 17:24 |
| *** mctaylor has quit IRC | 17:24 | |
| rajinir | clarkb: I can delete it using nodepool. Just wanted to see how the bridge was setup. But looks like it was not going to work anyway | 17:25 |
| rajinir | clarkb: the main issue I'm seeing is dhcp packets are not forwarded to the tap with default network settings | 17:26 |
| *** roxanagh_ has joined #openstack-infra | 17:26 | |
| jesusaur | clarkb: did anything change in your elk environment recently? | 17:26 |
| clarkb | jesusaur: nope, the last big changes were the upgrades to logstash 2.0 and ES 1.7.5 a few months back | 17:26 |
| clarkb | I am going to look at ES again and make sure it isn't stopping the works | 17:27 |
| jesusaur | fwiw, I upgraded to 2.0.0 about a month ago and havent noticed this issue | 17:27 |
| *** sabari has joined #openstack-infra | 17:27 | |
| clarkb | ya we were fine on that version for a while too, so not sure what is going on, but also 2.3.4 seems no better so likely not related to that issue I pointed out | 17:28 |
| *** rdx_matt has joined #openstack-infra | 17:28 | |
| *** david-lyle has joined #openstack-infra | 17:28 | |
| clarkb | everything I can see on the ES clsuter looks fine though. Cluster is green, all nodes are present, lpenty of disk space, only things in logs are GC statistics | 17:29 |
| *** cloudtrainme has quit IRC | 17:29 | |
| fungi | rajinir: clarkb: or ssh into it with the same account/keys zuul is using | 17:30 |
| *** roxanagh_ has quit IRC | 17:31 | |
| clarkb | I can tcp from worker20 to all ES nodes | 17:31 |
| fungi | (or s/zuul/jenkins/ i guess) | 17:31 |
| *** tqtran has joined #openstack-infra | 17:33 | |
| *** bryan_att has quit IRC | 17:33 | |
| *** john-davidge has quit IRC | 17:35 | |
| openstackgerrit | Sirushti Murugesan proposed openstack-infra/project-config: Add experimental tempest-dsvm xenial py3 job to devstack https://review.openstack.org/339778 | 17:35 |
| *** Swami has quit IRC | 17:36 | |
| krotscheck | clarkb, mordred: This thing's still failing. Is there some presumption about what directory the workspace is in? | 17:37 |
| krotscheck | http://paste.openstack.org/show/543413/ | 17:37 |
| *** hurgleburgler1 has joined #openstack-infra | 17:37 | |
| *** Swami has joined #openstack-infra | 17:37 | |
| *** tqtran has quit IRC | 17:38 | |
| *** hurgleburgler has quit IRC | 17:39 | |
| *** akshai has quit IRC | 17:39 | |
| clarkb | krotscheck: no it should build that dir from scratch | 17:39 |
| clarkb | krotscheck: look in the setup workspace log file which should be in $PWD/logs I think | 17:40 |
| krotscheck | clarkb: Hhrm. mkdir: missing operand | 17:42 |
| *** yaume has quit IRC | 17:43 | |
| *** Kelso has quit IRC | 17:44 | |
| krotscheck | clarkb: Tracing, gimme a bit... | 17:44 |
| *** maishsk has quit IRC | 17:44 | |
| *** andymaier has quit IRC | 17:45 | |
| *** Kelso has joined #openstack-infra | 17:45 | |
| *** maishsk has joined #openstack-infra | 17:45 | |
| clarkb | jesusaur: http://paste.openstack.org/show/543416/ progress, have you seen that before? | 17:45 |
| jesusaur | clarkb: no, but i think ive seen similar issues back in the day of es 0.9x | 17:47 |
| *** akshai has joined #openstack-infra | 17:47 | |
| jesusaur | does logstash join the cluster? if so, does the node count reflect that? | 17:47 |
| clarkb | https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/304 | 17:47 |
| jesusaur | I'm still running local elasticsearch nodes on the indexers for routing | 17:47 |
| clarkb | jesusaur: it doesn't join the cluster as its using http | 17:48 |
| clarkb | fungi: pabelanger ^ it almost looks like this could be inter dfw networking troubles | 17:48 |
| clarkb | which would explain why we were good for a while if its a new thing rax side | 17:49 |
| fungi | clarkb: yeah, wouldn't surprise me in the least. we see this all the time with job nodes in dfw failing to reach the mirror server there or our git farm | 17:49 |
| *** ihrachys has quit IRC | 17:50 | |
| fungi | as well as nodes in ord | 17:50 |
| jesusaur | clarkb: im surprised that manticore doesn't establish a new connection | 17:50 |
| fungi | less so in iad, but our quota there is also pretty low | 17:50 |
| clarkb | jesusaur: oh it does :) I have a bunch of those errors logged now | 17:50 |
| *** Kelso has quit IRC | 17:50 | |
| fungi | and we hardly ever see it for, e.g., nodes in other providers communicating with the git farm | 17:50 |
| jesusaur | ahh | 17:51 |
| clarkb | (I had to set the timeout value in the logstash output config, before it just waited forever, but at least now I know we are going 60 seconds without any traffic) | 17:51 |
| pabelanger | clarkb: thanks for the heads up, EmilienM^ | 17:51 |
| fungi | which suggests it's an east-west problem not north-south (i _so_ hate that terminology) | 17:51 |
| clarkb | fungi: ++ to hating that terminology | 17:51 |
| *** asettle has joined #openstack-infra | 17:51 | |
| *** rdx_matt has quit IRC | 17:51 | |
| jesusaur | clarkb: so next step is to try setting tcp keepalive? | 17:51 |
| *** rdx_matt has joined #openstack-infra | 17:52 | |
| EmilienM | pabelanger: ok good to know. /me afk lunch now, will catch up later | 17:52 |
| *** thiagop-afk is now known as thiagop | 17:53 | |
| clarkb | jesusaur: maybe? its actually failing really quickly | 17:53 |
| clarkb | jesusaur: eg I think we would have to have some creazy keep alive settings to notice a difference | 17:53 |
| jesusaur | hmm | 17:53 |
| *** raildo has joined #openstack-infra | 17:53 | |
| *** EricGonczer_ has joined #openstack-infra | 17:54 | |
| clarkb | fungi: I wonder if they really want us to use the private addrs | 17:55 |
| clarkb | jesusaur: ^ is the other thing that may be worth trying | 17:55 |
| jesusaur | oh, that could be | 17:55 |
| jesusaur | you can try adding the private addrs to /etc/hosts on worker20 to test | 17:55 |
| clarkb | ya I will have to update firewall rules too | 17:56 |
| jesusaur | oh wait, you need to modify the publish address in the es config | 17:56 |
| jesusaur | or maybe thats only needed when joining the cluster | 17:56 |
| krotscheck | clarkb: Turns out I didn't have ZUUL_BRANCH defined. It seems to be moving farther now. | 17:56 |
| clarkb | jesusaur: its listening on :::9200 so I think it should be fine to use private addr | 17:57 |
| *** maishsk has quit IRC | 17:57 | |
| clarkb | fungi: any opposition to really quickly trying that? have better ideas? | 17:57 |
| *** mctaylor has joined #openstack-infra | 17:58 | |
| *** cloudtrainme has joined #openstack-infra | 17:58 | |
| *** maishsk has joined #openstack-infra | 17:59 | |
| *** akshai has quit IRC | 18:00 | |
| *** akshai has joined #openstack-infra | 18:01 | |
| *** tkelsey has quit IRC | 18:02 | |
| jesusaur | clarkb: yeah, i was definitely thinking of the join-cluster handshake where you get re-routed to the publish address | 18:03 |
| mtreinish | pabelanger: cool, all the patches for the basic setup are up for review: https://review.openstack.org/#/q/status:open++branch:master+topic:firehose | 18:03 |
| mtreinish | but after all those land comes the fun part and seeing whether it works or not :) | 18:04 |
| *** cloudtra_ has joined #openstack-infra | 18:05 | |
| *** cloudtra_ has quit IRC | 18:05 | |
| clarkb | ok I quickly did hax to update /etc/hosts on es02-07 and logstash-worker20 so that the fireawll will let logstash worker talk to es via private net | 18:05 |
| fungi | clarkb: no concern from me. it's unlikely to make it not-work even worse, and if it does you'll probably notice | 18:05 |
| fungi | krotscheck: did we miss adding that to the devstack-gate readme? | 18:06 |
| clarkb | right its basically not working at all right now :( | 18:06 |
| *** Sukhdev has joined #openstack-infra | 18:07 | |
| *** mctaylor has quit IRC | 18:07 | |
| *** trown|lunch is now known as trown | 18:07 | |
| *** mctaylor has joined #openstack-infra | 18:07 | |
| *** cloudtrainme has quit IRC | 18:08 | |
| *** asettle has quit IRC | 18:08 | |
| sdague | clarkb: checking in on the package mirrors, to know if it's worth rechecking yet | 18:08 |
| clarkb | sdague: we haven;t published the volume yet. Last I checked bandersnatch was still syncing | 18:09 |
| sdague | ok | 18:09 |
| *** sandanar has quit IRC | 18:09 | |
| clarkb | looks like we are at 'cipher' I guess it wrapped around with the caps and non caps sorting? | 18:09 |
| *** cloudtrainme has joined #openstack-infra | 18:10 | |
| *** Sukhdev has quit IRC | 18:10 | |
| pleia2 | enjoy cameos by clarkb, anteaya, greghaynes, freerunner, therve (chat snippit) and fungi, corvus (etherpad screenshot) here https://opensource.com/life/16/7/5-remote-sysadmin-tools | 18:12 |
| fungi | ooh! | 18:12 |
| nibalizer | noice | 18:13 |
| * fungi is famousz now | 18:13 | |
| *** PollyZ has joined #openstack-infra | 18:13 | |
| prometheanfire | ok, think I have everything back to puppet, which is doing some odd things, but at least it's known odd things :D | 18:13 |
| *** ociuhandu has quit IRC | 18:14 | |
| clarkb | jesusaur: the other thing to try is going back to local ES daemon on the workers since ES is a lot better about logging issues | 18:14 |
| nibalizer | puppet doing odd things is a feature ! | 18:14 |
| clarkb | like exiting non zero on success | 18:14 |
| *** ajmiller has quit IRC | 18:14 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Generalize logic for skipping final image generation https://review.openstack.org/334042 | 18:15 |
| anteaya | pleia2: nice article | 18:15 |
| pleia2 | anteaya: thanks | 18:15 |
| anteaya | I forget some folks still haven't figured out remote team work | 18:15 |
| pleia2 | anteaya: busy week for publishing, I wrote everything weeks ago ;) | 18:16 |
| anteaya | pleia2: thanks for preaching the gospel | 18:16 |
| *** csomerville has quit IRC | 18:16 | |
| pleia2 | it's what I do! | 18:16 |
| *** Kelso has joined #openstack-infra | 18:16 | |
| * anteaya takes a look at mercury | 18:16 | |
| *** Sukhdev has joined #openstack-infra | 18:17 | |
| *** siva_krish has joined #openstack-infra | 18:17 | |
| clarkb | there is definite a lack of traffic between the worker and the es hosts according to tcpdump (which I have decided is easier to read than strace at this point) | 18:17 |
| *** rbrndt has quit IRC | 18:18 | |
| *** ddieterly[away] has quit IRC | 18:18 | |
| *** tqtran has joined #openstack-infra | 18:19 | |
| *** piet has joined #openstack-infra | 18:19 | |
| siva_krish | Hi guys! | 18:19 |
| clarkb | and my 600 second timeout just timed out | 18:20 |
| clarkb | ok | 18:20 |
| anteaya | pleia2: looks like a good time to publish | 18:20 |
| anteaya | siva_krish: sigh, women work here too | 18:20 |
| siva_krish | Need help with gerrit. How to create a patch which depends on two existing patchses on review. can soemone help me with this ? | 18:20 |
| clarkb | I need to take a break and rest my eyeballs but will restore things to the way they were after | 18:20 |
| *** degorenko is now known as _degorenko|afk | 18:20 | |
| clarkb | maybe someone else is willing to look into ES issues as well? | 18:21 |
| jesusaur | clarkb: so still no connectivity even over private addresses? | 18:21 |
| anteaya | pleia2: big focus on mercury at the moment, very good energy for publishing and speaking | 18:21 |
| anteaya | pleia2: and books too! | 18:21 |
| clarkb | jesusaur: there was some, enough to tcp handshake and move a small amount of data but after that it died and timed out against hte 600 second timeout | 18:21 |
| clarkb | siva_krish: you will have to serialize them | 18:21 |
| clarkb | siva_krish: so A then B then C | 18:21 |
| *** cody-somerville has joined #openstack-infra | 18:22 | |
| fungi | assuming they're all in the same git repository anyway | 18:22 |
| tonyb | Can we exclude grenade jobs on xenial? http://logs.openstack.org/71/347871/1/gate/gate-grenade-dsvm-ubuntu-xenial/f47b9c4/logs/grenade.sh.txt.gz#_2016-07-28_16_59_29_099 | 18:22 |
| tonyb | siva_krish: That's a better question #openstack-dev | 18:22 |
| clarkb | tonyb: gah I thought we had that tested | 18:22 |
| clarkb | tonyb: yes we can do that, also why would grenade be so forceful about that? | 18:23 |
| clarkb | maybe our tests were flipping the force bit | 18:23 |
| clarkb | use the FORCE luke | 18:23 |
| tonyb | :) | 18:23 |
| siva_krish | tonyb sure will move it to openstack-dev . Thanks clarkb! | 18:23 |
| fungi | tonyb: good question, i'm trying to remember how we handled that when we did the precise to trusty transition (since the job needs devstack to start on the previous branch to test changes for the target branch) | 18:23 |
| *** claudiub|2 has joined #openstack-infra | 18:23 | |
| fungi | i want to say we ran grenade tests for the release lagged on the prior distro version or something | 18:24 |
| clarkb | fungi: are you sure? | 18:24 |
| clarkb | anyways thats the easy fix rather than a complete revert so I will push that in a moment | 18:24 |
| *** ayoung has quit IRC | 18:24 | |
| *** ddieterly has joined #openstack-infra | 18:25 | |
| tonyb | clarkb: Thanks. | 18:25 |
| *** kzaitsev_mb has joined #openstack-infra | 18:25 | |
| tonyb | siva_krish: I'll meet you there ;P | 18:25 |
| openstackgerrit | Merged openstack-infra/system-config: Clean up puppet-apply testing on precise https://review.openstack.org/348434 | 18:28 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Keep running grenade on trusty for now https://review.openstack.org/348505 | 18:28 |
| clarkb | fungi: tonyb ^ | 18:28 |
| *** baoli has quit IRC | 18:29 | |
| tonyb | clarkb: Thanks | 18:30 |
| krotscheck | fungi: No, I just glossed over it | 18:32 |
| *** electrofelix has quit IRC | 18:33 | |
| *** cdent has quit IRC | 18:33 | |
| clarkb | infra-root can has review on https://review.openstack.org/#/c/348505/1 ? we might want to consider pushing it through quickly too to avoid too much disruption | 18:35 |
| *** alee has quit IRC | 18:35 | |
| anteaya | you just need one more project-config core review | 18:35 |
| clarkb | and a gerrit admin to force merge if we want to do that | 18:36 |
| *** csomerville has joined #openstack-infra | 18:36 | |
| anteaya | yeah I wouldn't be opposed to a force merge, two nova patches in the gate failing on it | 18:37 |
| clarkb | tonyb: btw a bunch of the acceptable distro versions in the grenade list are no longer supported by their upstreams | 18:38 |
| clarkb | tonyb: seems like we should probably clean that up a bit | 18:38 |
| *** azvyagintsev_h has quit IRC | 18:38 | |
| *** tonytan4ever has quit IRC | 18:38 | |
| clarkb | (also its weird to me that we could assert vivid is ok but not xenial) | 18:38 |
| tonyb | clarkb: Yeah. It's all about who verified them and when | 18:38 |
| *** cody-somerville has quit IRC | 18:39 | |
| tonyb | clarkb: I *needed* vivid so I did that work but xenial wasn't interesting for $reasons | 18:39 |
| clarkb | also its a warning that becomes an error :/ | 18:39 |
| clarkb | tonyb: but vivid is dead now :P | 18:39 |
| clarkb | don't vivid | 18:39 |
| clarkb | so is wily and f22 I think | 18:40 |
| clarkb | and I probably don't want to know what a kvmibm1 is | 18:40 |
| tonyb | clarkb: Sure. but is wasn't when I needed it and there isn't a big push to clear the list | 18:40 |
| tonyb | clarkb: You *really* don't wnat to know what that is but it's still a thing | 18:41 |
| *** roxanagh_ has joined #openstack-infra | 18:42 | |
| *** david-lyle has quit IRC | 18:42 | |
| pabelanger | clarkb: +3 | 18:42 |
| clarkb | pabelanger: ty | 18:42 |
| pabelanger | clarkb: not know how to force merge, but can enqueue it into gate | 18:43 |
| pabelanger | don't* | 18:43 |
| *** ociuhandu has joined #openstack-infra | 18:44 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for nova/neutron dns mitaka bug 1607412 https://review.openstack.org/348445 | 18:44 |
| clarkb | hrm I think it has slightly wrong logic though | 18:44 |
| openstack | bug 1607412 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode fails server build on newton side with "Unrecognized attribute(s) 'dns_name'"" [Undecided,New] https://launchpad.net/bugs/1607412 | 18:44 |
| clarkb | wee complicated list of regexes | 18:45 |
| *** rdx_matt has quit IRC | 18:46 | |
| *** roxanagh_ has quit IRC | 18:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Keep running grenade on trusty for now https://review.openstack.org/348505 | 18:47 |
| clarkb | pabelanger: anteaya tonyb ^ I had to move the regex to be after the default gate-grenade-dsvm regex other wise that one would override. I also excluded liberty like the default one does | 18:47 |
| clarkb | I noticed that in zuul status there was a change running bot trusty and xenial even though it was master | 18:48 |
| clarkb | which helped me realize ^ was a thing | 18:48 |
| anteaya | clarkb: thanks for looking, +2 on new changeset | 18:49 |
| clarkb | fungi: so this is where the current plan falls over hardest I think. Its explicit but we end up with a massive set of rules that we have to accomodate when making these cahnges | 18:49 |
| fungi | pabelanger: you add your gerrit account temporarily to the project bootstrappers group, then review the change setting code-review +2 verify +2 workflow +1 and then there will be a submit button after you have saved the review | 18:49 |
| sdague | clarkb: http://logs.openstack.org/93/347593/2/gate/gate-grenade-dsvm-ubuntu-xenial/4269c21/logs/grenade.sh.txt.gz ... | 18:49 |
| sdague | oh, you just discovered that | 18:49 |
| clarkb | its almost 2k lines worth of job rules that we are at the mercy of (both to preserve old behavior and to accomodate new stuff) | 18:49 |
| clarkb | sdague: ya tonyb pointed it out, fix is up | 18:50 |
| pabelanger | fungi: ack | 18:50 |
| sdague | clarkb: great, cool | 18:50 |
| clarkb | would their be opposition to me patching grenade to make that warning just a warning :) | 18:50 |
| clarkb | s/their/there/ | 18:50 |
| fungi | clarkb: i don't know that the branch selection logic for grenade jobs was any easier last time around. we still had to make an exception for it so we ran it on precise the first cycle while other jobs had switched to trusty | 18:50 |
| sdague | clarkb: you actually need to patch devstack stable/mitaka | 18:50 |
| sdague | and I was theoretically opposed to that | 18:50 |
| sdague | however, I think if it makes this easier, I'm fine with it now | 18:51 |
| clarkb | fungi: the tricky part is not the first patchset you are right that isn't any simpler. its what the second patchset had to do | 18:51 |
| pabelanger | clarkb: +3 | 18:51 |
| clarkb | fungi: if we don't accomodate the entire ruleset of 2k lines of rules we trip over existing stuff that may be there | 18:51 |
| sdague | it's actually proposed somewhere, I can probably go revive it | 18:51 |
| *** ddieterly is now known as ddieterly[away] | 18:53 | |
| clarkb | fungi: the old method had the benefit of setting the function once for all jobs and not needing to think through all of the rules belwo it | 18:53 |
| clarkb | you did have to sort through the logic in the function(s) though | 18:53 |
| clarkb | but those were far shorter than what our job rules are currently | 18:53 |
| *** baoli has joined #openstack-infra | 18:54 | |
| clarkb | pabelanger: to force merge a thing you add yourself to the project-creators group in gerrit, review the change so it has the necessary votes (+2 V, +2 CR, +1 W) and hit submit | 18:54 |
| clarkb | fungi: thoughts on doing that here for this change | 18:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement a new job for undercloud testing only https://review.openstack.org/346147 | 18:55 |
| pabelanger | clarkb: okay, I've enqueue it into gate already. Will force merging now be okay? | 18:56 |
| clarkb | pabelanger: oh if its in teh gate already thats good enough | 18:56 |
| clarkb | pabelanger: that will double check I didn't write a abd yaml file | 18:56 |
| pabelanger | clarkb: okay | 18:56 |
| dhellmann | fungi : it looks like zuul-cloner is not installed on the node where we're running the tag-releases job. Is that something I can control through a project-config patch somehow? maybe bindep? | 18:57 |
| clarkb | pabelanger: also if you do add yourself to project creators for whatever reason try and remember to remove yourself after you are done doing $task. Otherwise you get all the buttons in all the projects and you might accidentally approve thinsg you shouldn't | 18:57 |
| pabelanger | clarkb: okay, I'll remember that for the next one | 18:58 |
| *** kzaitsev_mb has quit IRC | 18:59 | |
| *** ddieterly[away] is now known as ddieterly | 18:59 | |
| *** EricGonczer_ has quit IRC | 18:59 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/system-config: Fix emaint sync call https://review.openstack.org/348514 | 19:00 |
| *** ddieterly is now known as ddieterly[away] | 19:02 | |
| clarkb | silly idea time | 19:02 |
| *** akshai has quit IRC | 19:02 | |
| *** fifieldt has quit IRC | 19:02 | |
| anteaya | oh its now is it? | 19:03 |
| clarkb | we could go to a single base image that we use everywhere that includes caches for things like git repos but also chroot/container images for ubuntu/debian/centos/fedora/whatever | 19:03 |
| clarkb | then jobs could just opt into whatever they watn. Of course this will make image building a hundred times more complicated and balloon the image size so not a good idea | 19:03 |
| AJaeger | clarkb: only if we assume that the Linux kernel is not part of the equation | 19:03 |
| AJaeger | clarkb: some tools, like ovs, might need a specific kernel... | 19:04 |
| fungi | dhellmann: oh, that's an interesting problem. we typically install zuul-cloner into a virtualenv via a diskimage-builder element. signing01.ci.o.o is managed by puppet instead since it's a long-lived machine so we'd need some solution to puppet zuul-cloner onto non-single-use nodes | 19:04 |
| pabelanger | clarkb: sounds like too much work, clearly docker is the solution | 19:04 |
| fungi | dhellmann: definitely can't use bindep there. jobs won't have root permissions so will be unable to install system packages or write outside the job workspace for that matter | 19:04 |
| AJaeger | fungi: How about the proposal node? Right now the "venv" environment does not use constraints - and if the proposal node has no zuul-cloner, we cannot switch that... | 19:05 |
| clarkb | pabelanger: AJaeger ya its got a lot of drawbacks. the kernel one is a good one. I know sytemd is becoming more tightly coupled to kernel versions too and we might run into trouble tehre too | 19:05 |
| fungi | AJaeger: right, same scenario | 19:06 |
| prometheanfire | nodepool/elements/cache-bindep/install.d/50-cache-pkgs ONLY caches right? | 19:06 |
| prometheanfire | no install/compile? | 19:06 |
| AJaeger | fungi, that's sad ;( | 19:06 |
| prometheanfire | I think it's currently installing for gentoo, which is slow lol | 19:06 |
| * AJaeger had hoped to be able to use constraints everywhere. | 19:06 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/system-config: Fix emaint sync call https://review.openstack.org/348514 | 19:06 |
| openstackgerrit | Merged openstack-infra/project-config: Keep running grenade on trusty for now https://review.openstack.org/348505 | 19:07 |
| fungi | prometheanfire: the goal of the cache-bindep element is to pre-download packages needed by any jobs that might run on nodes booted from that image, so that the package installer can later install them directly from disk during job runtime | 19:07 |
| pabelanger | clarkb: joking aside, it would be a fun experiment to try | 19:07 |
| prometheanfire | fungi: so in gentoo's case, no compile/install | 19:07 |
| *** jsavak has quit IRC | 19:07 | |
| prometheanfire | gotint | 19:07 |
| prometheanfire | gotit | 19:07 |
| *** acoles is now known as acoles_ | 19:08 | |
| fungi | prometheanfire: righth, assuming gentoo's package manager still compiles binaries on demand at install time, then i guess expect to cache by downloading the package source and then the package compilation takes place at the start of each job | 19:08 |
| *** jsavak has joined #openstack-infra | 19:08 | |
| mtreinish | fungi: so I just did some more reading on mqtt, and we probably will want to use tls for our internal traffic before too long. The auth creds are on the wire in plain text | 19:09 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add Gentoo support to nodepool https://review.openstack.org/310865 | 19:09 |
| prometheanfire | fungi: ya, that'll be rough | 19:09 |
| prometheanfire | qemu, et all | 19:09 |
| fungi | mtreinish: we _could_ consider running germqtt on review.openstack.org | 19:09 |
| prometheanfire | I added --fetchonly to the cache | 19:09 |
| fungi | mtreinish: then it's all over localhost | 19:09 |
| prometheanfire | fungi: what we should probably do is run a binhost for some of this | 19:10 |
| mtreinish | err, actually that would be worse. It's the mqtt auth creds which are plaintext | 19:10 |
| prometheanfire | because bindep-fallback calls this many packages to be installed | 19:10 |
| prometheanfire | >>> Emerging (1 of 225) dev-libs/libyaml-0.1.6::gentoo | 19:10 |
| AJaeger | clarkb: so, for unit tests we should be able to use such an approach, for devstack I doubt it | 19:10 |
| bkero | Hehe openstack-infra gentoo binhost | 19:10 |
| prometheanfire | takes a couple of hours | 19:10 |
| fungi | prometheanfire: right, assuming gentoo is like the bsds and has an intermediate "compiled but not yet installed" state for packages, that would be one option | 19:10 |
| *** gouthamr has quit IRC | 19:10 | |
| prometheanfire | it does | 19:10 |
| mtreinish | fungi: germqtt is localhost to mosquitto right now. (I was thinking more for future other services talking to firehose) | 19:10 |
| bkero | https://wiki.gentoo.org/wiki/Binary_package_guide | 19:11 |
| prometheanfire | called a binpkg, it's what I use on my servers at home | 19:11 |
| mtreinish | fungi: if we moved it to review.o.o then we'd definitely be sending the service user password in plaintext | 19:11 |
| fungi | prometheanfire: oh, right so like the option in bsds for precompiling a package tarball | 19:11 |
| prometheanfire | bkero: what we ill probably have to do is take an image, run bindep on it, installing all the things while generating binpkgs | 19:12 |
| AJaeger | fungi, do you see a solution on adding zuul-cloner to proposal and release nodes? | 19:12 |
| prometheanfire | once we have those generated we'd take them off and mirror them | 19:12 |
| prometheanfire | we'd just have to do that every so often (about monthly) to keep up with churn | 19:12 |
| prometheanfire | but for now it's just a time inconvienience | 19:12 |
| fungi | mtreinish: oh, i had you backwards. right the germqtt to gerrit credentials are over ssh, so already encrypted | 19:13 |
| fungi | mtreinish: so if germqtt runs on the firehose server the publisher creds for the mqtt service go over the loopback | 19:13 |
| *** jcoufal_ has quit IRC | 19:13 | |
| bkero | prometheanfire: What about adding some of the deps to the nodepool image? | 19:13 |
| fungi | AJaeger: sorry, dance card for the day is oversubscribed already. i'm sure there is one but i don't have time to design it today | 19:13 |
| *** rbuzatu has joined #openstack-infra | 19:14 | |
| mtreinish | fungi: yep, so it's not super critical right now. But we'll want to get the tls stuff sorted before we start enabling other services to talk mqtt | 19:14 |
| AJaeger | fungi: ok, let's discuss another time. I know now that I cannot enable it for now;) | 19:14 |
| fungi | AJaeger: chances are we could reencode the same install mechanism from dib into a script called from a puppet exec and just not run that on single-use node image builds since dib is already handling it there | 19:15 |
| fungi | AJaeger: the bigger challenge is taht we also don't have any solution designed to build and update a local git repo cache on those nodes | 19:15 |
| prometheanfire | bkero: what do you mean? | 19:16 |
| fungi | AJaeger: though i guess we could just accept that and let zuul-cloner use its fallback of cloning over the network | 19:16 |
| fungi | which is basically equivalent to what those jobs do now | 19:16 |
| prometheanfire | bkero: nodepool generates from diskimagebuilder, we could compile it each time if we wanted I guess | 19:16 |
| AJaeger | yeah, that's an option. Let's discuss another time and if anybody wants to look into this, it would be appreciated. I can't do it myself ;( | 19:16 |
| *** hashar has joined #openstack-infra | 19:17 | |
| bkero | prometheanfire: Yeah, that's what I mean. Have an element that installs a lot of the dep packages for quicker tests. | 19:17 |
| *** Sukhdev has quit IRC | 19:17 | |
| fungi | clarkb: happen to remember if there's any way to set up networking on instances in rackspace without using glean or their custom nova-agent? like does cloud-init know how to get network configuration from their environment at all? | 19:18 |
| *** jcoufal has joined #openstack-infra | 19:18 | |
| clarkb | fungi: I think if you havea really new cloud-init or possibly one patched wit the rax patch if it hasn't merged yet (it was supposed to merge after nova accepted the code iirc) | 19:18 |
| *** elo has quit IRC | 19:18 | |
| prometheanfire | bkero: that's basically what's already happening, right now the element installs as normal, though it's designed to be --fetchonly | 19:18 |
| *** eric_lopez has joined #openstack-infra | 19:18 | |
| *** rdx_matt has joined #openstack-infra | 19:18 | |
| bkero | That's half the battle | 19:18 |
| *** kzaitsev_mb has joined #openstack-infra | 19:19 | |
| *** fifieldt has joined #openstack-infra | 19:19 | |
| sdague | clarkb: so how much suck is the grenade / trusty thing for master? | 19:19 |
| clarkb | sdague: the change has merged to fix it, next zuul config reload should pick it up | 19:19 |
| sdague | ok | 19:19 |
| prometheanfire | bkero: yep, it's more of a question if we want to make an exception to the --fetchonly policy for gentoo since install time takes so long | 19:20 |
| *** hashar has quit IRC | 19:20 | |
| *** hashar_ has joined #openstack-infra | 19:20 | |
| prometheanfire | bkero: these are the chagnes I have to make right now https://review.openstack.org/#/c/310865/ | 19:20 |
| prometheanfire | this would be nice to be merged, not the biggest blocker (as it seems to pass when it fails...) https://review.openstack.org/#/c/348514/ | 19:22 |
| bkero | prometheanfire: I'll read the patch after lunch, it's kind of big. :) | 19:23 |
| prometheanfire | bkero: yarp | 19:23 |
| bkero | prometheanfire: do you think it's in a state to merge, or more work needs to be done? | 19:23 |
| prometheanfire | bkero: I think it's ready to merge, I removed the WIP on it | 19:23 |
| prometheanfire | the next work is puppet modules methinks | 19:23 |
| *** abregman has joined #openstack-infra | 19:23 | |
| clarkb | looking at these ES tcpdumps it looks like things are going great after a timeout (new connection created), but then it just stops | 19:24 |
| clarkb | like one side is waiting for more data | 19:24 |
| jesusaur | clarkb: is the logstash output buffer too large? | 19:24 |
| clarkb | jesusaur: its set to flush at 1024 events. How do I check or know if the resulting buffer is too big? | 19:25 |
| jesusaur | also, is log_processor working or is that waiting on logs? | 19:25 |
| jesusaur | I'm not sure how to check how full that buffer is :( | 19:25 |
| clarkb | the log processor workers are blocking the send to logstash | 19:25 |
| clarkb | basically their backlog queues fill and they stop doing stuff waiting for logstash to process stuff ahead of it | 19:25 |
| jesusaur | but l_p workers should output number of lines which should be reasonably similar to logstash events | 19:26 |
| dhellmann | fungi : so, ideas? we've previously said we didn't want to use pip to install stuff on this node | 19:27 |
| clarkb | I suppose I could try tuning down the flush threshhold so that it tries to get events out quicker/more often | 19:27 |
| *** fifieldt has quit IRC | 19:27 | |
| dhellmann | fungi : what if I replace zuul-cloner with a little shell script that does the parts I need? | 19:27 |
| openstackgerrit | Merged openstack-infra/release-tools: Update the UPPER_CONSTRAINTS_FILE url to include the new branch https://review.openstack.org/348027 | 19:27 |
| fungi | dhellmann: yeah, i was just discussing options above with AJaeger since jobs running on the proposal.slave node have similar needs (to be able to support upper constraints) | 19:27 |
| dhellmann | fungi : oh, I missed that part of the scrollback | 19:28 |
| dhellmann | I do something like that on my dev box, so I think I have the commands we'd need | 19:28 |
| *** rbrndt has joined #openstack-infra | 19:28 | |
| fungi | dhellmann: having the script do that in the short term would be fine, and we can simplify/standardize it once zuul-cloner is accessible on that machine | 19:29 |
| sdague | clarkb: when is the next zuul config reload? | 19:29 |
| dhellmann | for the release scripts I could just put those in the shell function already wrapping zuul-cloner, but if AJaeger has a need it could be a separate script too | 19:29 |
| dhellmann | fungi, AJaeger : ack, I'll work on that then | 19:29 |
| pabelanger | sdague: should be any moment | 19:29 |
| sdague | ok | 19:29 |
| sdague | just notice there are a lot of patches that are going to need recheck | 19:29 |
| *** kzaitsev_mb has quit IRC | 19:29 | |
| AJaeger | dhellmann: if you read backscroll, my problem is: venv environment should get constraints setup, so I want to setup the git trees properly. We can use a custom macro for this... | 19:30 |
| *** matrohon has joined #openstack-infra | 19:30 | |
| AJaeger | ... but if there's a script called zuul-cloner it would be simpler ;) | 19:30 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout/puppet: switch ovb to multinode for some modules https://review.openstack.org/347964 | 19:30 |
| *** tonytan4ever has joined #openstack-infra | 19:30 | |
| *** ddieterly[away] is now known as ddieterly | 19:31 | |
| dhellmann | AJaeger : your needs may be more complex than mine. I just need a working copy of the repo so I can tag it. It sounds like you need that repo to have a specific configuration? | 19:31 |
| fungi | dhellmann: yeah, chances are the job could just replace zuul-cloner with git clone for now (there's no local git cache on those machines anyway so that's what it's amounting to in post/release pipelines) | 19:32 |
| AJaeger | dhellmann: I need the specified change checkout out - and I need constraints repo with the same branch if it exists | 19:32 |
| pabelanger | sdague: clarkb: ansible-playbook was OOM on puppetmaster, next loop in 13mins | 19:32 |
| sdague | ok | 19:32 |
| *** Kelso has quit IRC | 19:32 | |
| sdague | how are the mirrors doing? as I'm stilly baby sitting that patch to try to get things in for the release | 19:33 |
| fungi | pabelanger: since it's just a layout change, we could hand-patch it on zuul.o.o and tell it to reload its config? | 19:33 |
| fungi | no idea if 13 minutes is enough to make a difference though | 19:33 |
| *** EricGonczer_ has joined #openstack-infra | 19:34 | |
| pabelanger | fungi: let me check patch | 19:34 |
| dhellmann | fungi : oh, in that case, yeah, I may just do that to keep things simple. AJaeger, that would let you code up your version to work however you need it to. what do you think? | 19:34 |
| clarkb | django-mptt-urls just went by on bandersnatch | 19:34 |
| clarkb | so its still going | 19:34 |
| AJaeger | dhellmann: give it a try and let's see how I can built on top of it. | 19:35 |
| pabelanger | fungi: we should patch run_all.sh to accept ansible-playbook args, then we could just use --limit zuul.openstack.org | 19:36 |
| fungi | dhellmann: AJaeger: though i would like to get zuul-cloner and some mechanism to maintain a periodically updated git cache onto those nodes. i just don't know an eta for it getting done | 19:36 |
| AJaeger | fungi: yeah, that would be better. | 19:36 |
| AJaeger | BUT fungi, looking at two or so log files, I didn't see much difference between gerrit-git-prep and zuul-git-pre-upper-constraints in time ;( | 19:37 |
| *** fifieldt has joined #openstack-infra | 19:38 | |
| *** maishsk_ has joined #openstack-infra | 19:38 | |
| AJaeger | That would be a worthwhile exercise, whether it brings us timewise significant improvements. Or whether it's more reliability... | 19:38 |
| *** maishsk has quit IRC | 19:39 | |
| sdague | clarkb: oof, that's taking a long time. :( | 19:39 |
| *** maishsk_ is now known as maishsk | 19:39 | |
| fungi | AJaeger: i expect it's more a reliability improvement than a performance improvement | 19:41 |
| *** claudiub|2 has quit IRC | 19:41 | |
| clarkb | I am tempted to try a full ES restart since its not really doing anything anyways | 19:41 |
| fungi | AJaeger: though also it reduces the load on our git servers for each job we move to zuul-cloner with a local cache | 19:41 |
| clarkb | and we know that previous 1.7 versions have had deadlocking problems | 19:41 |
| clarkb | anyone not ok with me doing that to the ES cluster? | 19:42 |
| *** rdx_matt has quit IRC | 19:43 | |
| AJaeger | fungi, venv is used from different jobs - so those jobs that run on ubuntu-xenial could use zuul-cloner while those that use a long-lived node could use the other option - that alone would be an improvement on reliability/git load. | 19:45 |
| *** Kelso has joined #openstack-infra | 19:45 | |
| AJaeger | let's see what dhellmann comes up with ;) | 19:45 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 19:45 |
| dhellmann | fungi : I agree, and I'd be happy to switch back to using it when that happens. | 19:46 |
| openstackgerrit | Merged openstack-infra/project-config: Add an experimental grenade job for networking-midonet https://review.openstack.org/342535 | 19:46 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 19:46 |
| dhellmann | AJaeger : my fix was going to be to replace the call to zuul-cloner in the shell function I'm already using, but if you want me to start a shell script I can do that, too | 19:46 |
| AJaeger | dhellmann: let me review your change and if it's beneficial, we can change to a shell script. I fear our needs our too different and I need to go with a different solution... | 19:48 |
| dhellmann | AJaeger : yeah, that's my impression, too | 19:48 |
| *** jkilpatr has quit IRC | 19:48 | |
| *** jcoufal has quit IRC | 19:49 | |
| AJaeger | but I might get some inspiration from your change ;) | 19:49 |
| *** rbuzatu has quit IRC | 19:49 | |
| *** woodster_ has quit IRC | 19:49 | |
| AJaeger | dhellmann: so, please tell me what you propose | 19:49 |
| *** eric_lopez has quit IRC | 19:49 | |
| *** jcoufal has joined #openstack-infra | 19:50 | |
| *** ccamacho has joined #openstack-infra | 19:51 | |
| *** HeOS has quit IRC | 19:54 | |
| *** jsavak has quit IRC | 19:54 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use oooq to install the undercloud https://review.openstack.org/348530 | 19:54 |
| dansmith | has gerrit been restarted multiple times today? | 19:54 |
| *** alee has joined #openstack-infra | 19:55 | |
| fungi | dansmith: ps says it was last restarted yesterday | 19:56 |
| *** Teresita-Warrior has quit IRC | 19:56 | |
| *** cloudtrainme has quit IRC | 19:56 | |
| dansmith | fungi: hmm, okay, because multiple times today I've been logged in, went to make a comment, hit save and get kicked out to the login screen | 19:56 |
| dansmith | in a single browser session | 19:56 |
| *** cmspence has joined #openstack-infra | 19:57 | |
| dansmith | and I say unpleasant words when that happens | 19:57 |
| clarkb | wow thats scary, I restarted the ES cluster and now the doc counts are climbing | 19:57 |
| cinerama | greghaynes, ianw: we're meeting again today right? | 19:57 |
| clarkb | dansmith: it does that if you aren't logged in with that tab. I run into it fairly frequently too | 19:57 |
| fungi | dansmith: yeah that's definitely weird. i wonder if we somehow reintroduced the 1-minute timeout that was supposed to be 1-month? | 19:57 |
| greghaynes | cinerama: Yep | 19:57 |
| clarkb | dansmith: it doesn't like running in a couple hundred tabs I ahve found | 19:57 |
| *** asettle has joined #openstack-infra | 19:58 | |
| *** asettle has quit IRC | 19:58 | |
| fungi | i assumed that's what dansmith meant by "a single browser session" (no other tabs) | 19:58 |
| dansmith | fungi: clarkb: okay, but this is definitely newish behavior in the last six months or so.. I never used to have a problem except across a weekend | 19:58 |
| clarkb | fungi: dansmith from what I have seen it has to do with reusing old tabs that aren't lgged in which then log out your newer tabs | 19:58 |
| dansmith | fungi: no I mean ~6 tabs | 19:58 |
| *** jsavak has joined #openstack-infra | 19:58 | |
| dansmith | clarkb: well, I know what behavior you're talking about, but it used to be that if I started fresh on monday with 1 tab, logged in, and spawned them all from there, I could make it all week no problem | 19:58 |
| clarkb | say you have tab 1 from last week, it gets logged out, you open new tab and log in, then if you go back to odl tab and use it the new tab picks up that you should be logged out or something | 19:58 |
| fungi | dansmith: oh, yep. log into one of them, then go force refresh all the others without clicking on anything | 19:59 |
| jesusaur | clarkb: so now its looking like the hold-up was on the elasticsearch side? | 19:59 |
| dansmith | now it seems to happen to me multiple times a day, and at least every morning | 19:59 |
| clarkb | jesusaur: ya | 19:59 |
| clarkb | jesusaur: the cluster itself wasn't logging anything or otherwise showing signs of distress | 19:59 |
| *** gildub has joined #openstack-infra | 19:59 | |
| clarkb | but we are doing about 20k events per second right now I think | 19:59 |
| dansmith | clarkb: yeah, I know what you're talking about, but I'm careful not to trip that by killing all my tabs every time I have to re-login | 19:59 |
| *** rossella_s has quit IRC | 19:59 | |
| fungi | dansmith: basically if you get logged out and log back into one tab, then go to click things on another tab that was still on the old now invalid session, it invalidates your new session from the other tab when you have to relog in | 20:00 |
| *** matt-borland has quit IRC | 20:00 | |
| dansmith | what's annoying is that it will let me navigate around in a tab showing that I'm logged in until I need to do something as me, like make a comment and then BOOM | 20:00 |
| clarkb | dansmith: also there is a way to not lose your comments. Instead of clicking login click the go anonymous button, copy pasta all your comments out, then log in then copy pasta them back | 20:00 |
| *** rossella_s has joined #openstack-infra | 20:00 | |
| clarkb | (seriously derpy work around but has helped me in the past) | 20:00 |
| *** matt-borland has joined #openstack-infra | 20:00 | |
| dansmith | fungi: yeah, I know, which is why I always kill all tabs whenever I have to relogin | 20:00 |
| *** Teresita-Warrior has joined #openstack-infra | 20:00 | |
| dansmith | clarkb: yeah, escape at the shaded screen gets me out to do that, but sooooo annoying | 20:01 |
| pabelanger | clarkb: sdague: zuul has been reloaded | 20:01 |
| fungi | dansmith: but yes, if you're starting to have your sessions expire more frequently then maybe a config change has slipped in on the last restart | 20:01 |
| clarkb | pabelanger: thanks can you keep an eye out that we queue the correct jobs now too? | 20:01 |
| dansmith | fungi: surely seems like it to me | 20:02 |
| *** Apoorva has quit IRC | 20:02 | |
| pabelanger | clarkb: ya, it looks like nova jobs in gate are green now | 20:02 |
| pabelanger | and merging | 20:02 |
| clarkb | pabelanger: great tahnks | 20:02 |
| clarkb | I *think* I have corrected the ES issues by doing a full cluster restart | 20:03 |
| mriedem | when will gerrit let me sort by column? | 20:03 |
| clarkb | which is sadness and goodness all at the asme time | 20:03 |
| mriedem | in my dash? | 20:03 |
| clarkb | mriedem: I think you can get it to order by mod time both backwards and forwards using the api at least | 20:03 |
| pabelanger | fungi: looks like we might have the old zuul.openstack.org still online in DFW. We you working on cleaning that up at some point in time? | 20:03 |
| clarkb | by flipping the a,z to z,a or something | 20:04 |
| fungi | pabelanger: i wasn't but can add it to my to do list | 20:04 |
| fungi | i don't suppose anybody has yet had time today to verify whether a recent change to zuul-cloner stopped actually updating remote branches? | 20:04 |
| *** cloudtrainme has joined #openstack-infra | 20:04 | |
| pabelanger | fungi: thanks | 20:04 |
| prometheanfire | so, looks like I clean up the gathered distfiles (packages that are fetched, should I add an optional variable in order to keep them around? | 20:04 |
| *** baoli has quit IRC | 20:04 | |
| pabelanger | fungi: I haven't tested it | 20:04 |
| jesusaur | clarkb: on the topic of log_processor, I noticed some odd behavior a little while back: the workers send WorkComplete when the first thread adds the url to the queue. But we should probably find a way to not send that until after we fetch and push, ya? | 20:06 |
| *** Teresita-Warrior has quit IRC | 20:06 | |
| clarkb | jesusaur: ya that was just easy mode since we aren't really consuming those events on the other end | 20:06 |
| jesusaur | oh, we aren't? | 20:06 |
| clarkb | nope | 20:06 |
| clarkb | we could if we wanted to do retries and such | 20:07 |
| clarkb | but we deal with ~5 billion events in the backend at any given time and if we lose a few thousand its probably not the end of the world | 20:07 |
| jesusaur | I haven't looked into the gearman protocol much, but thought that it already had logic around clients dieing while working on a job | 20:07 |
| clarkb | ya looks like a normal day for us is 500 million events | 20:07 |
| clarkb | jesusaur: though I think we we really awnted to worry about the data like that we would be better off kafkaing | 20:08 |
| clarkb | otehrwise we would have to build a lot of that reliability on top of gearman | 20:09 |
| fungi | dansmith: no dice. looks like cache.web_sessions.maxAge=7days still | 20:09 |
| jesusaur | clarkb: yeah, its more of a 'nice to have', I just thought we already had a bit more reliability | 20:11 |
| *** roxanagh_ has joined #openstack-infra | 20:12 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Optionally remove portage files https://review.openstack.org/348536 | 20:13 |
| *** ddieterly is now known as ddieterly[away] | 20:13 | |
| clarkb | hrm new gerrit api based things apparently dropped teh a,z z,a paging mechanism | 20:13 |
| clarkb | thats annoying | 20:13 |
| prometheanfire | bkero: https://review.openstack.org/348536 will definitely be needed | 20:14 |
| *** Kelso has quit IRC | 20:14 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: Add Gentoo support to nodepool https://review.openstack.org/310865 | 20:15 |
| *** inc0 has joined #openstack-infra | 20:17 | |
| anteaya | fungi: so this patch is merged yet in the top left where I normally see "Merged" on this patch I see "Not Current": https://review.openstack.org/#/c/348505/1 | 20:17 |
| *** maishsk has quit IRC | 20:17 | |
| *** e0ne has quit IRC | 20:18 | |
| fungi | anteaya: are you looking at the most recent patchset in the patchsets drop-down? | 20:18 |
| *** roxanagh_ has quit IRC | 20:18 | |
| anteaya | ah | 20:18 |
| anteaya | I was not | 20:18 |
| anteaya | my fault, thank you | 20:18 |
| fungi | that's basically what "not current" means | 20:18 |
| fungi | means you're not looking at the "current" (most recently added) patchset | 20:18 |
| anteaya | ah I see | 20:19 |
| anteaya | I guess I need "you fool, wrong patchset" | 20:19 |
| *** Apoorva has joined #openstack-infra | 20:19 | |
| clarkb | corvus: zuul's current status shows changes which are not active as the tip of series in check pipeline | 20:20 |
| clarkb | corvus: any idea why that is happening? | 20:20 |
| clarkb | I wonder if that is a side effect of the queues being non zero and we are in a halfway state | 20:21 |
| *** gyee has joined #openstack-infra | 20:21 | |
| *** matrohon has quit IRC | 20:22 | |
| cmspence | hi, can I get another core to take a look: https://review.openstack.org/#/c/345453/ We've had this review out for a couple days and looking to get our testing turned on. clarkb, thanks for the +2 earlier today | 20:23 |
| inc0 | hey guys, http://logs.openstack.org/70/348270/1/check/gate-kolla-dsvm-deploy-ubuntu-source/242642c/console.html <- I'm trying to figure out this error | 20:24 |
| inc0 | is there any chance whatsoever to ssh to gate machine? | 20:24 |
| *** greghaynes has quit IRC | 20:25 | |
| *** maishsk has joined #openstack-infra | 20:26 | |
| *** sabari has quit IRC | 20:26 | |
| *** HeOS has joined #openstack-infra | 20:26 | |
| clarkb | inc0: have you tried reproducing it locally? | 20:27 |
| inc0 | yeah, no luck tho, where can I get image gates are running? | 20:28 |
| *** kgiusti has quit IRC | 20:28 | |
| clarkb | inc0: however http://logs.openstack.org/70/348270/1/check/gate-kolla-dsvm-deploy-ubuntu-source/242642c/logs/kolla/nova/nova-compute.log shows the issue pretty clearly I think | 20:29 |
| clarkb | the libvirt module doesn't ahve that variable | 20:29 |
| inc0 | so not really, as it has it in centos | 20:29 |
| clarkb | inc0: which have different versions of libvirt | 20:29 |
| *** hemanthm is now known as hemanthm|afk | 20:29 | |
| inc0 | but not python-libvirt | 20:30 |
| clarkb | no but its just exposing what the eunderling libvirt has | 20:30 |
| inc0 | yeah, I know, we install just vanilla libvirt-bin from ubuntu repos | 20:30 |
| inc0 | let me redeploy with fresh images and I'll double-check what versions is it | 20:31 |
| clarkb | http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=05e1b06ab7b30a3d505b9a5799f00eb3ae8acf44 looks like it was added to libvirt just before ubuntu 14.04 released | 20:31 |
| clarkb | so wouldn't surprise me if that is missing | 20:31 |
| pabelanger | EmilienM: looks like tripleo-ci is still doing noop in check, even though there is check jobs now. | 20:31 |
| inc0 | also it used to work for over a year;) | 20:31 |
| tonyb | Have there been issues mirroring pypi packages into (say OVH)? | 20:31 |
| *** coolsvap_ has quit IRC | 20:31 | |
| sdague | tonyb: the mirror is broken everywhere | 20:31 |
| pabelanger | tonyb: yes, we are rebuilding the mirrors | 20:31 |
| clarkb | tonyb: yes, bandersnatch decided it needed to start over so I have a complete mirror sync going niow | 20:31 |
| inc0 | it started to break pretty recently | 20:31 |
| tonyb | okay | 20:32 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Implement basic undercloud sanity checks https://review.openstack.org/346551 | 20:32 |
| clarkb | inc0: maybe you were using cloud archive then stopped? | 20:32 |
| tonyb | I'll hold off on the recheck then ;P | 20:32 |
| clarkb | inc0: if you want the images we use the easiest way to get them is to build them locally. openstack-infra/project-config/tools/build-image.sh | 20:33 |
| clarkb | fcgiapp is about where we are in the sync | 20:33 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add facter output to host_info.txt https://review.openstack.org/347460 | 20:33 |
| inc0 | thanks | 20:33 |
| *** Swami has quit IRC | 20:34 | |
| *** sabari has joined #openstack-infra | 20:34 | |
| clarkb | inc0: my understand of how python-libvirt works is a new version of that can run against many older versions of libvirt but it won't expose things that that libvirt doesn't know about | 20:34 |
| inc0 | yeah, that's true | 20:35 |
| inc0 | however it used to work, that's what I'm saying | 20:35 |
| inc0 | it broke maybe a month ago | 20:35 |
| clarkb | it doesn't look like you capture package info? | 20:36 |
| clarkb | I guess you would need todo that per service/container | 20:36 |
| inc0 | well I know which container is breaking | 20:36 |
| clarkb | http://logs.openstack.org/70/348270/1/check/gate-kolla-dsvm-deploy-ubuntu-source/242642c/logs/system_logs/dpkg.log is for the host machine? | 20:36 |
| *** andymaier has joined #openstack-infra | 20:36 | |
| inc0 | question tho, do you have libvirt in this image? | 20:36 |
| clarkb | no we don't preinstall libvirt in the images | 20:37 |
| *** Swami has joined #openstack-infra | 20:37 | |
| clarkb | looks like 1.2.2 in that dpkg logfile | 20:37 |
| *** greghaynes has joined #openstack-infra | 20:37 | |
| inc0 | yeah...on host | 20:37 |
| inc0 | you won't get this information for docker | 20:37 |
| inc0 | and that actually might be an issue | 20:37 |
| *** ccamacho has quit IRC | 20:39 | |
| *** ccamacho|afk has quit IRC | 20:39 | |
| inc0 | I'll try to purge libvirt on host and see if it helps | 20:39 |
| clarkb | I'm not sure why that would affect the libvirt on your container unless your python-libvirt is talking to the libvirt socket in the host | 20:40 |
| *** ddieterly[away] has quit IRC | 20:40 | |
| inc0 | actually it uses TCP | 20:40 |
| inc0 | and we use net=host | 20:40 |
| inc0 | so there might be problem in that | 20:41 |
| clarkb | ya that seems possible, there may be cross talk that you don't expect due to net=host | 20:41 |
| inc0 | I wonder how our libvirt even starts then | 20:41 |
| tonyb | inc0: Do you have the logs for a workign run? | 20:41 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: use git directly instead of through zuul-cloner https://review.openstack.org/348549 | 20:42 |
| dhellmann | AJaeger, fungi : ^^ | 20:42 |
| inc0 | tonyb, it'll be problem as this issue shows up for some time | 20:42 |
| inc0 | so I think it started when you guys changed images or sth, can't really remember what was that | 20:43 |
| inc0 | but I'm not 100% sure | 20:43 |
| clarkb | inc0: can you clarify on that we haven't changed images for that build in a long time | 20:43 |
| odyssey4me | inc0 the issue is that you're now using libvirt from the UCA repo, but the python wheel you're using was not compiled with the right libraries there | 20:44 |
| clarkb | inc0: thats ubuntu-trusty which has been around for months. It does get daily updates in the various clouds but the fundamentals don't change | 20:44 |
| fungi | dhellmann: i haven't started digging deeply yet, but is that clone_repo() function only used in the jobs that run on signing, and not for other things like release announcements and such? | 20:44 |
| clarkb | odyssey4me: oh! is that the problem? | 20:44 |
| *** inc0_ has joined #openstack-infra | 20:44 | |
| tonyb | clarkb: I think he means the devstack-trusty -> ubuntu-trusty transition | 20:44 |
| odyssey4me | inc0 so the issue is that you've changed to use the UCA repo recently - it has nothing to do with changes in infra | 20:44 |
| clarkb | tonyb: gotcha | 20:44 |
| inc0_ | clarkb, I was about to respond when my net died | 20:44 |
| odyssey4me | clarkb yeah, we encountered that when we switched to using UCA some time ago | 20:45 |
| clarkb | odyssey4me: inc0_ if thats the problem you should be able to install that lib specially or don't use UCA or something | 20:45 |
| pabelanger | inc0_: clarkb: Is it possible bindep-fallback.txt is installing it? | 20:45 |
| clarkb | pabelanger: no I think odyssey4me's theory is spot on | 20:45 |
| fungi | odyssey4me: clarkb: inc0_: so that brings up an interesting challenge with selective use of optional external package repositories. we prebuild a wheel cache against packages present in the standard repo packageset | 20:45 |
| pabelanger | clarkb: ah, just seen that | 20:45 |
| clarkb | we build python-libvirt wheel using older libvirt, then when you UCA you need to make a new wheel to build against the newer libvirt to get the new shiny flags | 20:45 |
| odyssey4me | clarkb inc0 for our source-based builds, we specifically ensure that the python-libvirt module does *not* use the infra-built wheel | 20:45 |
| inc0_ | so UCA is there for some time | 20:45 |
| inc0_ | and it did work before using UCA | 20:45 |
| clarkb | inc0_: yup it should work before using UCA because the wheel would be built properly in that case | 20:46 |
| inc0_ | odyssey4me, yes, but don't forget our libvirt is inside container | 20:46 |
| inc0_ | so that might cause problems | 20:46 |
| odyssey4me | inc0_ this would need to be solved in your build process | 20:46 |
| fungi | probably you need a step to force-rebuild that python-libvirt wheel | 20:46 |
| pabelanger | and a month ago is when our UCA AFS mirror came online | 20:46 |
| fungi | so that it builds against teh correct headers/abi | 20:46 |
| clarkb | fungi: I think we build our wheels for the official distro release packages and if people opt into these other things its on them to make new wheels | 20:47 |
| tonyb | This how I force the rebuild: http://git.openstack.org/cgit/openstack/devstack-plugin-additional-pkg-repos/tree/devstack/lib/libvirt#n34 | 20:47 |
| clarkb | or isntall from source or something | 20:47 |
| tonyb | perhaps Kolla needs to do somethign similar | 20:47 |
| inc0_ | so let me put it this way, it seems we have 2 libvirts running | 20:47 |
| fungi | clarkb: yep, that's basically what i'm suggesting | 20:47 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Break out bootstrapping of subnodes setup https://review.openstack.org/347008 | 20:47 |
| inc0_ | one inside container, another one infra-build | 20:47 |
| pabelanger | clarkb: agree with that statement | 20:47 |
| clarkb | inc0_: no I think the two libvirts was noise | 20:47 |
| *** inc0 has quit IRC | 20:47 | |
| inc0_ | so we don't use python-libvirt that is on host | 20:48 |
| odyssey4me | inc0_ when you install the python packages, simply include '--no-binary libvirt-python' as a pip install option | 20:48 |
| clarkb | inc0_: I would look into fixing the problem odyssey4me points out maybe using tonyb's thing | 20:48 |
| clarkb | inc0_: correct, but you are installing our wheel from our mirror | 20:48 |
| inc0_ | odyssey4me, I'll try that, thanks | 20:48 |
| odyssey4me | inc0_ doing that will force it to install from source | 20:48 |
| inc0_ | ahh | 20:48 |
| clarkb | inc0_: which was previously built against the standard libvirt package on ubuntu | 20:48 |
| *** Kelso has joined #openstack-infra | 20:48 | |
| inc0_ | I get it now | 20:48 |
| odyssey4me | it forces it to build from source with the C libs that are on the host you're installing it onto | 20:49 |
| inc0_ | let me try this | 20:49 |
| *** woodster_ has joined #openstack-infra | 20:49 | |
| odyssey4me | so if UCA is there, it'll activate the extra options... if UCA is not there, it won't | 20:49 |
| *** Kelso has quit IRC | 20:49 | |
| inc0_ | thanks guys for input, I'll play around this | 20:49 |
| odyssey4me | sure thing :) | 20:49 |
| clarkb | inc0_: you should be able to reproduce locally if you use our mirror config | 20:50 |
| *** matrohon has joined #openstack-infra | 20:53 | |
| odyssey4me | pabelanger got a minute to chat about https://review.openstack.org/329892 ? | 20:53 |
| odyssey4me | well, 10 mins or so I guess | 20:54 |
| *** Kelso has joined #openstack-infra | 20:54 | |
| pabelanger | odyssey4me: can we push it until tomorrow morning? | 20:55 |
| odyssey4me | pabelanger sure thing, no worries | 20:55 |
| odyssey4me | ping me when you're ready | 20:55 |
| pabelanger | odyssey4me: will do | 20:56 |
| fungi | if linux wheels included a standardized way to say "built against ubuntu 14.0.4 plus mitaka uca" we might be able to do smarter things there | 20:56 |
| *** aeng has joined #openstack-infra | 20:56 | |
| fungi | but even the distro identification we're using now isn't really standardized | 20:56 |
| *** jamielennox|away is now known as jamielennox | 20:57 | |
| *** trown is now known as trown|outtypewww | 20:57 | |
| dhellmann | fungi : that copy of that function is only used by the scripts in project-config, yes. | 21:00 |
| dhellmann | fungi : there's a separate set of some of these utilities in the release-tools repo, and that's what sends the announcement email, irrc | 21:00 |
| *** Teresita-Warrior has joined #openstack-infra | 21:00 | |
| fungi | dhellmann: so i guess by extension i was asking whether those scripts in project-config are being used to run jobs in other places than the signing node, and sounds like the answer is "no, they're not" | 21:01 |
| *** maishsk has quit IRC | 21:01 | |
| fungi | in which case i'm cool with the change | 21:02 |
| dhellmann | fungi : we run some of those by hand still, but I hope any day now to not have to do that :-) -- but yeah, point taken, and those are only meant to run on the signing node | 21:02 |
| fungi | dhellmann: perfect. hopefully soon we'll be able to use zuul-cloner there too and then we can hopefully be able to drop some duplication as well | 21:03 |
| odyssey4me | fungi yeah, I suppose honestly the way the python works could be altered to be more flexible... but meh... anyone doing it is likely expected to understand the link between the binaries and the code and adapt accordingly | 21:03 |
| *** rhallisey has quit IRC | 21:03 | |
| *** elo has joined #openstack-infra | 21:04 | |
| fungi | odyssey4me: i meant more insofar as providing a "plain ubuntu 14.04" wheel mirror and then an overlaid "ubuntu 14.04 with mitaka uca" wheel mirror | 21:04 |
| fungi | but i think the number of jobs relying on the latter would be a small enough percentage we just let them take the (hopefully imperceptible) performance hit | 21:05 |
| *** maishsk has joined #openstack-infra | 21:05 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Use ovs-interface-nondefault instead of ovs-native job https://review.openstack.org/338944 | 21:06 |
| *** mtanino has quit IRC | 21:08 | |
| odyssey4me | fungi yeah, if the package had a marker of some sort it would be easier to make that work | 21:08 |
| odyssey4me | similar to architecture | 21:08 |
| odyssey4me | but hey, the workaround is a small price to pay | 21:08 |
| fungi | yep | 21:08 |
| fungi | easy solution. go with it. move on to more interesting problems ;) | 21:08 |
| *** yamamoto has joined #openstack-infra | 21:09 | |
| odyssey4me | most certainly | 21:09 |
| clarkb | really numpy and lxml are the vast compile time anyways | 21:09 |
| *** thiagop has quit IRC | 21:09 | |
| clarkb | neither have uca issues | 21:09 |
| *** hurgleburgler1 is now known as hurgleburgler | 21:09 | |
| odyssey4me | clarkb yeah, this is something specific in the way that the libvirt python package exposes the features available in the underlying/supporting binary package | 21:10 |
| *** jkilpatr has joined #openstack-infra | 21:10 | |
| *** jsavak has quit IRC | 21:11 | |
| *** jsavak has joined #openstack-infra | 21:12 | |
| dhellmann | fungi : ++ to moving back to zuul-cloner and dropping duplication | 21:13 |
| *** matrohon has quit IRC | 21:13 | |
| pabelanger | AJaeger: quick question, is tox -elinters valid or should it be tox -epep8? | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| *** siva_krish has quit IRC | 21:14 | |
| *** markusry has quit IRC | 21:15 | |
| fungi | tox -e linters would need to come with a corresponding [testenv:linters] in the tox.ini of any projects where that was run | 21:15 |
| fungi | same as some have a [testenv:pep8] today | 21:16 |
| *** markusry has joined #openstack-infra | 21:16 | |
| *** markusry has quit IRC | 21:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-linters non-voting job https://review.openstack.org/348565 | 21:18 |
| pabelanger | slagle: trown|outtypewww: weshay: you'll be interested in that^ | 21:19 |
| odyssey4me | pabelanger yeah, there was an idea at some point to try to move everyone to the linters target... but that does require all repositories to have the target | 21:19 |
| weshay | thanks | 21:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-linters non-voting job https://review.openstack.org/348565 | 21:20 |
| odyssey4me | I would imagine that if there are projects with multiple lint targets, liaising with them to pipeline them into a single target would be a worthy cause to improve the nodepool usage efficiency | 21:20 |
| odyssey4me | some people appear to feel quite strongly about each job doing something unique though | 21:20 |
| *** yamamoto has quit IRC | 21:21 | |
| weshay | pabelanger, do you have a pointer to where this lives, ansible-role-jobs | 21:23 |
| *** andymaier has quit IRC | 21:26 | |
| *** rbuzatu has joined #openstack-infra | 21:26 | |
| *** maishsk has quit IRC | 21:28 | |
| pabelanger | weshay: openstack-infra/project-config/jenkins/jobs/ansible-role-jobs.yaml | 21:29 |
| *** aysyd has quit IRC | 21:29 | |
| fungi | odyssey4me: right, that's been the main tension between switching to linters or not (though also there's the "this is running flake8 with pyflakes and hacking it's not just pep8" thing to contend with too) | 21:29 |
| weshay | pabelanger, thanks | 21:29 |
| *** jkilpatr has quit IRC | 21:29 | |
| *** rbuzatu has quit IRC | 21:31 | |
| *** PollyZ has quit IRC | 21:32 | |
| *** ddieterly has joined #openstack-infra | 21:33 | |
| *** ddieterly is now known as ddieterly[away] | 21:33 | |
| *** thorst has quit IRC | 21:33 | |
| *** ddieterly[away] is now known as ddieterly | 21:33 | |
| *** xyang1 has quit IRC | 21:33 | |
| *** thorst has joined #openstack-infra | 21:34 | |
| *** baoli has joined #openstack-infra | 21:34 | |
| *** piet has quit IRC | 21:37 | |
| *** thorst has quit IRC | 21:38 | |
| *** burgerk_ has joined #openstack-infra | 21:39 | |
| *** burgerk has quit IRC | 21:42 | |
| *** nwkarste_ has joined #openstack-infra | 21:43 | |
| *** burgerk_ has quit IRC | 21:43 | |
| *** nwkarst__ has joined #openstack-infra | 21:45 | |
| *** edmondsw has quit IRC | 21:46 | |
| *** nwkarsten has quit IRC | 21:47 | |
| *** nwkarste_ has quit IRC | 21:47 | |
| *** Kelso has quit IRC | 21:48 | |
| clarkb | odyssey4me: fungi ya I think the thing that sets static analysis apart from that is failrues should never be complicated to debug | 21:49 |
| clarkb | so its fine to load up a bunch of things and go to town. Unlike say integration testing which as you add a bunch of variables can get annoying to debug when it breaks | 21:50 |
| *** nwkarst__ has quit IRC | 21:50 | |
| clarkb | bandersnatch just did iching | 21:50 |
| clarkb | so many packages | 21:50 |
| *** Daisy has joined #openstack-infra | 21:51 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:51 | |
| anteaya | what was bandersnatch's iching trigram? | 21:53 |
| *** gouthamr has joined #openstack-infra | 21:54 | |
| *** Daisy has quit IRC | 21:55 | |
| *** matt-borland has quit IRC | 21:56 | |
| *** esberglu has quit IRC | 21:57 | |
| clarkb | probably some form of a dependencytree | 21:58 |
| clarkb | elasticsearch still looks healthy and happy | 21:58 |
| anteaya | ah yes, trees are good | 21:59 |
| anteaya | yay healthy elasticsearch | 21:59 |
| *** ddieterly has quit IRC | 22:01 | |
| *** mctaylor has quit IRC | 22:01 | |
| *** Apoorva_ has joined #openstack-infra | 22:01 | |
| *** mdrabe has quit IRC | 22:02 | |
| *** PollyZ has joined #openstack-infra | 22:04 | |
| *** Apoorva has quit IRC | 22:04 | |
| *** mmandel has quit IRC | 22:09 | |
| *** ddieterly has joined #openstack-infra | 22:10 | |
| *** asettle has joined #openstack-infra | 22:10 | |
| *** tonytan4ever has quit IRC | 22:10 | |
| *** ddieterly has quit IRC | 22:11 | |
| *** kzaitsev_mb has quit IRC | 22:12 | |
| *** hashar_ has quit IRC | 22:13 | |
| *** cmspence has quit IRC | 22:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add diskimage-builder nodepool test https://review.openstack.org/345816 | 22:14 |
| ianw | clarkb: ^ that just had a minor reshuffle in response to AJaeger's nit ... if you could take another look that would be great | 22:15 |
| clarkb | ianw: will queue it up, any luck figuring out what was crashing those f23 VMs? | 22:16 |
| *** gouthamr_ has joined #openstack-infra | 22:16 | |
| *** asettle has quit IRC | 22:17 | |
| *** gouthamr has quit IRC | 22:19 | |
| *** abregman has quit IRC | 22:20 | |
| ianw | clarkb: f24 :) no ... not yet. i've only tried on rax so far, need to get one happening on another provider. not sure how to force that, other than to turn rax off | 22:22 |
| *** yamamoto has joined #openstack-infra | 22:22 | |
| ianw | clarkb: but for nothing to come out on kernel net console ... to just disappear & shut down ... odd | 22:23 |
| clarkb | ianw: probably easier method would be to just boot one manually then run the reproduce.sh script on it from a previous job | 22:23 |
| clarkb | rather than waiting for nodepool to boot and crash one | 22:23 |
| clarkb | heh AJaeger likes to be able to read his jenkins jobs :P | 22:25 |
| ianw | clarkb: true. my local testing basically of that didn't crash. i'm just suspicious about rax ... they always manage to be a bit different :) | 22:25 |
| clarkb | ya would be neat to see it run on kvm just to see if it stays running | 22:26 |
| *** Kelso has joined #openstack-infra | 22:26 | |
| clarkb | (osic is probably quickest if you have to pick a cloud to use) | 22:26 |
| ianw | ok, i'll start with testing there in a bit | 22:27 |
| *** yamamoto has quit IRC | 22:27 | |
| *** Kelso has left #openstack-infra | 22:27 | |
| *** nwkarsten has joined #openstack-infra | 22:29 | |
| *** timl0101 has quit IRC | 22:29 | |
| clarkb | lets-encrypt-icecast2 just went by, this is taking far longer than I expected | 22:29 |
| *** krtaylor has quit IRC | 22:31 | |
| bkero | lol | 22:31 |
| *** nwkarsten has quit IRC | 22:33 | |
| dansmith | clarkb: fungi: okay, I have had one tab open since we last spoke and it just bombed on me again | 22:35 |
| dansmith | I dunno what to do to debug it further, but just want to complain | 22:35 |
| *** EricGonczer_ has quit IRC | 22:37 | |
| clarkb | your cookies aren't getting cleared are they? | 22:37 |
| clarkb | thats the only other thing that comes to mind as a possibility | 22:37 |
| dansmith | I dunno how that would happen, and isn't happening for any other sites | 22:38 |
| dansmith | I have another gerrit instance open in another tab, but that has never been a problem before, nor should it be of course | 22:38 |
| *** jsavak has quit IRC | 22:39 | |
| *** sarob has joined #openstack-infra | 22:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add some v3 notes https://review.openstack.org/348579 | 22:42 |
| *** Teresita-Warrior has quit IRC | 22:45 | |
| clarkb | dansmith: something like an adblocker maybe | 22:45 |
| dansmith | clarkb: hmm, okay I guess I should try disabling that on r.o.o and see | 22:46 |
| *** sarob has quit IRC | 22:46 | |
| *** sarob has joined #openstack-infra | 22:46 | |
| *** yamahata has quit IRC | 22:47 | |
| clarkb | dansmith: looking at some requests on my end GerritAccount: "stuff" seems to be where that is stored under the request cookies | 22:47 |
| clarkb | dansmith: you could maybe check if that is being supplied next time you run into it | 22:47 |
| jhesketh | Morning | 22:48 |
| *** rbrndt has quit IRC | 22:49 | |
| clarkb | #status log performed full restart of elasticsearch cluster to get it indexing logs again. | 22:50 |
| openstackstatus | clarkb: finished logging | 22:50 |
| clarkb | jhesketh: morning | 22:50 |
| craige | o/ | 22:51 |
| *** camunoz has quit IRC | 22:52 | |
| clarkb | dansmith: ya if it happens again I would check whether or not you have a GerritAccount cookie in browser for review.openstack.org as thats the only other thing I can figure is causing the problem | 22:54 |
| dansmith | clarkb: check at what point, as soon as the shaded warning screen comes up? | 22:54 |
| clarkb | yup | 22:55 |
| dansmith | okay | 22:55 |
| clarkb | in firefox you go to preferences, privacy, then remove individual cookies which will give you a whole list by site | 22:55 |
| clarkb | or you can pull up the developer tools and refresh and see if the request sends a GerritAccount cookie | 22:56 |
| *** PollyZ has quit IRC | 22:56 | |
| *** dimtruck is now known as zz_dimtruck | 22:58 | |
| *** claudiub|2 has joined #openstack-infra | 22:59 | |
| *** yamahata has joined #openstack-infra | 23:00 | |
| *** roxanagh_ has joined #openstack-infra | 23:01 | |
| *** ams__ has quit IRC | 23:04 | |
| *** ddieterly has joined #openstack-infra | 23:04 | |
| *** pradk has quit IRC | 23:04 | |
| *** roxanagh_ has quit IRC | 23:06 | |
| *** tpsilva has quit IRC | 23:08 | |
| *** tonytan4ever has joined #openstack-infra | 23:11 | |
| *** sdague has quit IRC | 23:11 | |
| *** ddieterly is now known as ddieterly[away] | 23:13 | |
| *** tonytan4ever has quit IRC | 23:16 | |
| *** aderyugin has quit IRC | 23:17 | |
| *** andymaier has joined #openstack-infra | 23:18 | |
| *** kgiusti has joined #openstack-infra | 23:18 | |
| *** Apoorva_ has quit IRC | 23:18 | |
| *** Apoorva has joined #openstack-infra | 23:19 | |
| *** kgiusti has quit IRC | 23:19 | |
| *** kgiusti1 has joined #openstack-infra | 23:19 | |
| *** tonytan4ever has joined #openstack-infra | 23:23 | |
| *** yamamoto has joined #openstack-infra | 23:24 | |
| *** tonytan_brb has joined #openstack-infra | 23:24 | |
| *** dchen has joined #openstack-infra | 23:24 | |
| *** kgiusti1 has quit IRC | 23:26 | |
| *** kgiusti has joined #openstack-infra | 23:28 | |
| *** tonytan4ever has quit IRC | 23:28 | |
| *** ddieterly[away] has quit IRC | 23:28 | |
| *** yamamoto has quit IRC | 23:29 | |
| *** kgiusti has quit IRC | 23:30 | |
| ianw | pabelanger / clarkb : going to put some notes in https://etherpad.openstack.org/p/f24-bringup-debug | 23:31 |
| *** psilvad has quit IRC | 23:33 | |
| *** sdague has joined #openstack-infra | 23:38 | |
| *** xarses has quit IRC | 23:39 | |
| *** sdague has quit IRC | 23:43 | |
| *** hurgleburgler has quit IRC | 23:48 | |
| *** mctaylor has joined #openstack-infra | 23:51 | |
| *** tonytan_brb has quit IRC | 23:52 | |
| ianw | clarkb: i think we require ntp-perl pkg on fedora for ntp-wait | 23:52 |
| ianw | well i don't think, i know :) | 23:52 |
| ianw | and interestingly, it doesn't seem to be working | 23:53 |
| *** mctaylor has quit IRC | 23:57 | |
| clarkb | ianw: is it hanging or just failing to update the clock? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!